devopsy.kinsta.cloud

Category: DevOps and Open Technologies

Editorial Calendar for February 2016

  • Aqua Security Acquires tfsec to Advance DevSecOps

    Aqua Security Acquires tfsec to Advance DevSecOps

    Aqua Security today announced it has acquired tfsec, an open source project that provides a static analysis scanner for infrastructure-as-code (IaC) that is designed to be integrated within a DevOps workflow.

    Amer Jerbi, Aqua Security CTO, said tfsec extends the security platform the company already provides in a way that advances adoption of DevSecOps best practices. In addition, to prevent developers from misconfiguring cloud infrastructure using tools such as Terraform, the scanner surfaces examples of code that provide developers with insights into how to properly configure infrastructure.

    Jerbi noted that tfsec is already integrated with Aqua Trivy, an open source tool that scans container images for vulnerabilities. Integrations with the rest of the Aqua Security portfolio will follow, added Jerbi.

    Tfsec co-founders Liam Galvin and Owen Rumney will join Aqua Security as cloud engineers. The challenge Aqua Security is trying to address is the need to enable organizations to proactively prevent cybersecurity issues from arising in production environments rather than only identifying vulnerabilities that need to be fixed, said Jerbi.

    Aqua Security tfsec

    The decision to acquire tfsec comes in the wake of a series of high-profile breaches when many organizations are scrambling to make sure software supply chains are secure. Cloud platforms are rife with misconfigurations mainly because developers often lack the expertise needed to ensure infrastructure is secure after they have provisioned it. Organizations are embracing DevSecOps best practices in the hopes of educating developers to assume more responsibility for application security. However, developers are not going to acquire that expertise overnight. The tfsec scanner provides a way to enforce security policies today while at the same time affording the time to teach developers how to avoid misconfigurations.

    It’s unclear how many cybersecurity events arise because of cloud misconfiguration issues. The cloud service providers have adopted a shared responsibility model that makes the organization employing the cloud responsible for both the applications deployed on those clouds and for how that infrastructure is provisioned. Many developers that provision cloud infrastructure themselves within the context of a DevOps workflow often assume the cloud service provider is doing more than it is to prevent misconfigurations. Cybercriminals, meanwhile, have become very adept at scanning for those misconfigurations.

    There are, of course, a lot of layers and nuances when it comes to DevSecOps. Arguably, Jerbi said, the most critical thing for most organizations is to simply get started. Tools that can be quickly and easily integrated within an existing DevOps workflow with minimal disruption are critical, Jerbi added.

    In the meantime, DevOps teams should expect cybersecurity teams to become more involved in their workflows, especially as government mandates aimed at securing software supply chains become stricter. Eventually, as organizations gain confidence in the ability of their DevOps teams to secure their application environments, those cybersecurity teams will refocus their efforts on hunting for threats. In the meantime, DevOps teams would be well-advised to make those cybersecurity professionals feel as welcome as possible, given the application security mandate most of them have been given.

  • Best Practices for AWS Tagging With Yor

    Best Practices for AWS Tagging With Yor

    On the surface, AWS resource tags are simple, informational key:value metadata that you can assign to most AWS resources. They also have many hidden benefits, as outlined in this AWS Tagging Best Practices guide for improving cloud operations. There are six use cases author Brian Yost calls out in particular:

    1. AWS console organization and resource groups
    2. Operations support
    3. Cost allocation
    4. Access control
    5. Automation
    6. Security risk management

    In summary, tags can simplify operational processes, improve security posture, speed up mean time to resolution (MTTR) and even save money if implemented consistently and persistently. But, unfortunately, in large organizations, that’s a tedious and often faulty process.

    To help teams overcome that challenge and make the most of cloud resource tags, Bridgecrew (recently acquired by Palo Alto Networks) recently launched a new open source tool, Yor. Yor is designed to automatically add consistent, helpful tags to infrastructure-as-code (IaC) templates starting with Terraform, CloudFormation and Serverless Framework. If run locally or added to a continuous integration (CI) pipeline, Yor will add tags like git organization and repository, unique trace ID and commit modifier details to IaC resources that carry through to cloud resources. For example, with Yor, you can tag your aws_instance code block in Terraform, and those tags will show up in your running EC2 instances.

    This post will explore how to leverage Yor to achieve the six tagging use cases outlined in the AWS Tagging Best Practices guide.

    AWS Console Organization and Resource Groups

    Using tags to organize and filter resources is the simplest and most common use case for tags. If you add tags to resources, such as EC2 instances or database resources, you can filter based on those tags. 

    If your AWS account has thousands or tens of thousands of resources in a region, filtering is necessary to manage those resources. For example, if you have hundreds of web servers, filter by the git_repo tag Yor added to quickly filter down to the web server(s) for your application.

    Yor

    In that same vein, you can set up resource groups to organize multiple resources across AWS services based on tags. For example, if you want to hunt down the VPC, EC2, internet gateway or database created by your Terraform repository, use Yor’s git_repo tag to group together multiple resources into a single resource group.

     

    Yor 

    Operations Support

    Yor excels at simplifying and speeding up operations support. In the words of Brian Yost, “For example, Level 1 support teams could use tags to direct workflow and perform business service mapping as part of the triage process when a monitoring system triggers an alarm.” Yor automatically tags resources with the git organization and modifiers of the cloud resource. This makes it easier to identify the right team and user to patch issues. 

    If you work in a GitOps organization, this is even more critical. Following GitOps best practices means all changes should be performed in the IaC template, not directly on the cloud resource. So if your security team identifies an open port 22 in runtime using policy-as-code or CSPM tools, such as Bridgecrew, Yor’s tags will make finding the owner of the IaC template easier and faster, lowering MTTR.

    Yor takes this a step further. By adding the repository, file and trace tags, Yor helps teams identify not only the code owner but also the exact file and resource block that created that resource misconfiguration.

    Yor

    Cost Allocation

    Large companies often have different profit and loss (P&L) statements for each of their business units. At the end of a reporting period, such as a fiscal quarter, finance teams allocate the expenditures made across the company to the various cost centers.

    For traditional IT teams, this usually involves very broad, general estimates. For example, if two organizations share a server rack, you could allocate costs based on virtual CPU time or disk space. Once that scales into dozens of organizations and ephemeral workloads, that allocation is much more difficult. Cloud costs can be just as difficult to allocate. If you have thousands of resources across types (virtual machines, serverless, managed Kubernetes workloads, managed databases, streams, CDNs, etc.) with differing life cycles and life spans, tying those resource costs back to a cost center is difficult.

    Cloud tagging simplifies that process. Assuming each team has a git organization, Yor will automatically tag resources with the git_org or custom tags can group git_modifiers into cost center tags, and AWS Cost Explorer will let you filter by those tags to easily allocate costs. You can also export those costs by tag(s) to a CSV file for import into other tools for analysis.

    Yor applies those tags reliably and consistently so that finance teams don’t need to dig through different organization name versions, such as combining dev_team_one and devTeam1, or manually categorizing untagged resources.

    Yor

     Access Control

    AWS identity and access management (IAM) policies support the ability to limit control of resources using tags. This simplifies dividing up resources by teams using complex IAM rules.

    For example, if you want developers to access only the EC2 instances they provisioned, you can use the git_modifiers tag added by Yor. Yor will automatically tag resources with the git users who edited the IaC code. Then, Yor will add an AWS IAM policy to limit access based on that tag.  You’ll also want to add a policy to deny actions such as ec2:CreateTags and ec2:DeleteTags so users can’t just add themselves to other resources.

    This is where Checkov, an open source policy-as-code tool, can be leveraged for additional control. Yor can enrich the context for Checkov’s checks, enabling finer-grained policy control. For example, Yor will populate the user identification of whoever modified an IaC template, and Checkov can make sure that only approved team members modified the code. The following is an example of a custom check to determine that only security personnel modified CloudTrail settings.

    If the git_modifier tag added by Yor is anything other than “Nimrod,” “Rotem,” or “Naor,” Checkov will show a failed policy check for that custom check for that resource.

    Automation

    By leveraging tags in combination with other tools, such as Lambda functions and backup plans, you can apply automation to specific resources, such as targeting resources for archive, update or deletion.

    The AWS Tagging Best Practices guide uses as an example a situation where development resources are turned off during non-business hours. AWS recently released a tool called Instance Scheduler for just such a use case. It is a CloudFormation stack with a Lambda function and DynamoDB instance that automatically starts and stops EC2 and RDS instances based on a pre-defined schedule. 

    Instance Scheduler takes a tag with a key:value pair; for example, “Schedule” (or a custom key) as the key and the schedule as the value. The schedules can be either one of the preconfigured schedules in the DynamoDB table, like office-hours, or a custom one.

    Using simple custom tags in Yor, you can tag every resource in a repository with a schedule by using environment variables such as adding the following to the Yor GitHub Action YAML file:

    This results in tags being added to every resource in that IaC directory. With the seattle-office-hours tag, Instance Scheduler will turn off EC2 instances at 5:00 pm Pacific Time and on again at 9:00 am Pacific Time Monday through Friday. This saves money by stopping instances automatically during downtime.

    Yor

    Another popular automation use case is for backups and archival plans. AWS can automatically apply different backup plans based on the tags applied to resources, such as storage and database resources. For example, if certain organizations, such as those that process financial data, need things archived forever, but others only need one year of backup, applying backup plans can move the short-term storage to cheaper options, such as Glacier, and eventually delete them on a schedule.

    Security Risk Management

    The final use case is risk management. The risk level of cloud resources differs based on whether they are in test or production environments and the sensitivity of the data those resources process and hold. This is another area where you can use simple tags like git_repo to filter down to repositories for applications that deal with sensitive data. You can also leverage complex custom tags. For example, EC2 instances attached to databases with “customer” in the name will get a data_level:sensitive tag.

    IT service management tools can apply higher priority to patching and updating resources with tags denoting sensitive data. This is another area where you can leverage Checkov’s graph-based checks in combination with Yor tags to assign higher severity levels to policy violations for resources with sensitive data tags or production environment tags. Assigning priority helps to focus developer attention and mitigate risk faster.

    These six tagging use cases are just a few examples of how you can leverage tags to be more efficient and improve your cloud security posture in AWS. The power of Yor’s tracing capabilities enables development and operations teams to work together without requiring complex triage to find the right person and IaC templates. With Yor’s extensibility and custom tagging capabilities, we can’t wait to see what other use cases the community comes up with!

  • Dynatrace Applies AI to Surface App Vulnerabilities

    Dynatrace Applies AI to Surface App Vulnerabilities

    Dynatrace has enhanced the security module to its observability platform that leverages its Davis artificial intelligence (AI) engine to automatically identify the software libraries and open source packages that represent the greatest security risk.

    Ajay Gandhi, vice president of product marketing for Dynatrace, said the Davis Security Advisor, being added to the Dynatrace Application Security Module, makes it easier for IT teams to understand which vulnerabilities need to be remediated first.

    Davis Security Advisor aggregates vulnerability data in real-time and prioritizes remediation based on the number of vulnerabilities, severity, impact to sensitive data, public exposure, how widely known the vulnerability is, and whether the code impacted is facing the internet. That approach enables Davis Security Advisor to monitor all software libraries used in preproduction and production environments in a way that sharply reduces the number of false positives that would otherwise be generated, noted Gandhi.

    Dynatrace

    As application environments become more complex—in part due to increased adoption of microservices that have lots of dependencies—it’s become impossible for IT teams to implement DevSecOps best practices in workflows without some form of AI augmentation, added Gandhi.

    In fact, a recent Dynatrace survey found 89% of chief information security officers (CISOs) noted cloud-native architectures and container runtime environments, used primarily to construct microservices, have made it more difficult to detect and manage software vulnerabilities.

    In theory, application developers are taking more responsibility for remediating security vulnerabilities within their applications. The challenge is they often inadvertently use an older version of a library that has known vulnerabilities. Hopefully, that issue gets addressed before an application gets deployed. There are, however, plenty of cases where, for one reason or another, a library with known vulnerabilities gets deployed anyway.

    Making matters more challenging, many vulnerabilities are discovered after an application has been deployed. Developers don’t always have access to a software bill of materials that would make it easier to determine what libraries have been employed in what application or, for that matter, which libraries are dependent upon one another.

    Gandhi said the Davis Security Advisor resolves that issue via continuous topology mapping of dependencies to identify those potential security issues. Once identified, the updates that need to be made can then be slipstreamed into a DevOps workflow.

    Analyzing code for vulnerabilities is, of course, not a new idea. Training an AI model to perform that task on behalf of DevOps teams does promise to greatly accelerate the process at a time when organizations are under more pressure than ever to secure their software supply chains in the wake of a series of high-profile breaches.

    It’s just not clear yet who inside organizations might drive adoption for AI platforms capable of identifying vulnerabilities in a way that helps prioritize remediation efforts. In some cases, cybersecurity teams have a vested interest in reducing the number of potential security incidents.

    The one thing that is apparent is shipping insecure code faster only leads to more negative outcomes. There may not be such a thing as a ‘great’ application that happens to be insecure. Instead, the assumption will be that any applications with known vulnerabilities, no matter how great, are too flawed to be deployed given all the security risks most businesses currently face.

  • Hasura Extends Scope of GraphQL API Platform

    Hasura Extends Scope of GraphQL API Platform

    During its online an HasuraCon ’21 conference, Hasura today announced that it has made it possible for developers to employ GraphQL application programming interface (APIs) to launch queries against data residing in multiple databases.

    At the same time, Hasura announced a preview of schema sharing capabilities along with support for workflows based on GitOps processes that it will make generally available later this year. The Hasura Cloud platform can now deploy local migrations from staging to production with a simple git push in addition to accessing previews of applications on every pull request to the git repository.

    The schema sharing capability added to the Hasura GraphQL Engine is intended to bridge the gap between beginner and power users by allowing them to share and install samples of permissions, relationships and metadata. In addition to improving collaboration, the goal is to make it easier to onboard software engineers as projects evolve and become more complex.

    Originally developed by Facebook, GraphQL has emerged as a popular alternative to REST APIs because it enables developers to retrieve data at a more granular level, which results in improved performance.

    Hasura is a provider of an instance of a GraphQL engine the company has developed around a project that is now hosted by the GraphQL Foundation, an arm of the Linux Foundation. The GraphQL Foundation counts among its members Airbnb, Amazon Web Services (AWS), Apollo, Coursera, Elementl, Facebook, GitHub, IBM, Intuit Hasura, Paypal, Prisma, Shopify and Twitter.

    GraphQL provides a complete description of the data made available via an API and is organized in terms of types and fields rather than traditional endpoints. In addition to clarifying what data is available via that API, developers can employ types to avoid writing manual parsing code.

    Since last summer, Hasura has been making available a pilot of a GraphQL implementation that is capable of supporting remote joins across disparate databases. That capability is critical because data sets that are related to a specific business process may wind up being stored in multiple databases.

    Hasura

    Tanmai Gopal, Hasura CEO, said the open source Hasura GraphQL Engine has already been downloaded more than 250 million times this year. Downloads in 2020 reached 100 million. The challenge is that while GraphQL offers a superior alternative to REST APIs, most of the API tooling that exists within organizations is still designed around REST APIs, noted Gopal.

    To accommodate that reality, Hasura has made a tool available that will convert a query designed using GraphQL into a set of calls that are compatible with REST APIs. Developers still gain the benefits of being able to launch more granular queries, noted Gopal.

    REST APIs, in general, are becoming more problematic as organizations embrace microservices-based applications. It often takes minutes to refresh a simple dashboard because of the amount of data that REST APIs are trying to pull from various data sources. As organizations embrace digital business transformation, they are also encountering application latency issues that can be traced back to the amount of data being called from multiple microservices.

    Hasura says in addition to providing a more efficient method to access data, Hasura is also working on adding caching capabilities to its core engine to further improve performance.

    It’s not clear at what rate GraphQL might replace REST APIs, but DevOps teams should assume organizations will wind up employing a mix of both for years to come. The challenge is going to be determining when to make the transition as organizations become more dependent on APIs.

  • Grafana Labs Acquires k6 to Add Open Source Load Testing Tool

    Grafana Labs Acquires k6 to Add Open Source Load Testing Tool

    During its online GrafanaCONline 2021 conference today, Grafana Labs revealed it has acquired k6, a provider of an open source load testing tool.

    Raj Dutt, Grafana Labs CEO, said load testing will complement the observability platform Grafana Labs currently provides based on open source Prometheus software. Most of the output of any load testing processes are typically evaluated using a platform that analyzes metrics, he noted.

    The k6 platform has been gaining traction among developers that require access to a tool that makes it simpler to dynamically scale load testing up and down as required versus having to guess how much infrastructure might be required to test an application workload, said Dutt. That capability not only makes it simpler to integrate testing within a DevOps workflow, Dutt added, but also serves to reduce testing costs.

    In general, Dutt noted testing is shifting further left as developers assume more responsibility for quality assurance. The challenge is finding a way to enable developers to automatically incorporate testing within a continuous integration (CI) process that allows them to easily observe how synthetic tests are executing, added Dutt.

    There’s no doubt developers are running more tests before promoting workloads into a production environment as part of an effort to reduce the number of issues that might be encountered when an application is deployed in a production environment. In many cases, the support calls that arise now go directly to developers who now have a vested interest in reducing the volume of those calls.

    The issue that developers are contending with is as they embrace microservices, the dependencies that exist across multiple applications makes it more challenging to successfully deploy those applications. Grafana Labs has been making a case for a cloud platform that makes it simpler to centrally view data collected from instances of Prometheus distributed across multiple application development environments. Prometheus, now being advanced under the auspices of the Cloud Native Computing Foundation (CNCF) that oversees the development of Kubernetes, is now being deployed more widely by IT teams to observe legacy monolithic applications alongside emerging microservices-based applications.

    It’s not clear the degree to which organizations are also employing dedicated application testers. In many cases, those testers will surface issues that developers overlooked (or simply chose to ignore) in the face of tight application delivery deadlines.

    In the longer term, however, it’s also apparent artificial intelligence (AI) will be playing a larger role in application testing. Someone will still need to review those tests, but many more rote tasks will increasingly be automated.

    One way or another, the total cost of testing is declining, thanks in part to increased reliance on open source testing tools that, in many cases, developers are simply downloading themselves. Those tools are reducing much of the licensing friction and cost restrictions that developers might otherwise encounter.

    In fact, the easier it becomes to test applications, the better overall application quality will become in the months and years ahead. At at time when organizations have never been more dependent on software, that potential improvement to the overall state of application quality can’t arrive soon enough.

  • Using Rust to Build Distributed Applications

    Using Rust to Build Distributed Applications

    Rust is an open source programming language that was initially developed at Mozilla Research and became increasingly popular for building distributed applications. It has been referred to in the Stack Overflow Survey as “the most-loved programming language” by software developers for five consecutive years. Moreover, companies like Amazon and Microsoft have recently revealed that they are using Rust to develop the software components that are responsible for ensuring application safety.

    Rust allows the implementation of safe and predictable code by leveraging memory safety and speed. It is the ideal choice to create code without incurring undefined or unpredictable behavior thanks to its features such as consistent runtime behavior and a static type system.

    Although Rust can be used for any system programming application, it is mainly implemented to build applications like simulation and game engines, browser components and file and operating systems. Rust syntax is similar to C++, but it addresses issues that C and C++ normally struggle with such as developing code without security vulnerabilities, general memory errors and problems in writing concurrent programs.

    The Advantages of Rust in Distributed Programming

    Rust tackles the performance and reliability problems of other low-level programming languages by using a careful compiler which examines all the variables used and referenced. As the code is compiled, Rust uses monomorphization by creating exclusive and high-performance code for each type of channel. It prevents programmers from paying performance penalties due to its flexibility. Based on the principle that you don’t pay for what you don’t use, the code written in Rust is enhanced and abstractions are solved at compile time making the program safe and without runtime overhead.

    Rust doesn’t have a garbage collector, which allows programmers to build abstractions with predictable performance. In Rust, there is no garbage to collect because the moment something would have become garbage, it gets freed immediately at compile time, making the program much faster. Moreover, Rust uses a borrow checker to check where data is used across the program, and by following a set of rules it establishes where data needs to be dropped or initialized. It provides the same advantages of a garbage collector with the efficiency and predictability of manual management.

    Another crucial characteristic of Rust is its asynchronous I/O. In Rust, asynchronous programming is widely implemented and well-integrated. It enables developers to effectively scale on a certain number of cores and exploit the operating system resources. By running several units of code on each application, this feature enhances performance and responsiveness of the operating system.

    Rust prevents some of the software engineering problems that could critically damage the application development process. These issues concern concurrency and memory safety which can cause very often denial of service, code vulnerabilities or compromised secrets. Examples of these frictions that are common in other languages such as C or C++ include uninitialized variables, undefined behavior, invalid iterators and unexpected variable modification.

    Another great advantage of using Rust is the Rust developer community, an extremely friendly and welcoming community. There are multiple events, chat rooms and forums to participate in that makes it even more pleasant to build distributed applications with Rust.

    Thanks to its ability to fix concurrency and memory issues, to compile and run safe and predictable code and provide a fast implementation, Rust is quickly increasing its popularity and establishing itself as the ideal choice for building distributed applications.

  • Lightbend Launches Serverless Managed DevOps Service

    Lightbend Launches Serverless Managed DevOps Service

    Lightbend today unfurled a cloud service based on a serverless framework that provides developers with a managed DevOps platform to build applications that dynamically scale resources up and down as required.

    Brad Murdoch, executive vice president for strategy at Lightbend, said Akka Serverless provides access to its Java development platform environment that runs atop Kubernetes via a declarative application programming interface (API) based on the gRPC protocol. Support for additional open APIs, such as GraphQL, is planned, added Murdoch.

    At the same time, Lightbend revealed today that Jonas Bonér has assumed the role of CEO in place of Mark Brewer. Previously CTO and founder of the Akka Project, Bonér is a co-author of a reactive manifesto that defines the core principles required to build applications using a reactive programming model.

    Akka Serverless, available via an open beta program that provides access to limited amounts of storage, is scheduled to be generally available in the fourth quarter on the Google Cloud Platform (GCP), with support for additional cloud services planned.

    The service is made possible because Lightbend extended an existing open source Cloudstate serverless framework to create a platform-as-a-service (PaaS) environment to support large amounts of distributed stateful data that developers employ to build applications, noted Murdoch. That capability overcomes the performance limitations of existing function-as-a-service approaches, based on other serverless computing frameworks, that don’t support stateful application development, added Murdoch.

    In fact, one of the reasons that adoption of serverless frameworks has been limited thus far is the inability to be employed within the context of a stateful application that requires a lot of access to data, said Murdoch.

    Lightbend addressed that issue by enabling event sourced entities persist each change in state as an event that Akka Serverless writes to a journal. That journal can be used to replay events to reconstruct state at a particular time, debug or provide an audit. At runtime, a key distinguishes each entity instance from all others.

    Akka Serverless is the latest in a series of managed DevOps services that automatically scales as developers request new resources. There is no need for developers to configure and manage a database on which to build their application. Many organizations are migrating to these services because they would rather focus their time and effort on building applications versus managing a DevOps platform, said Murdoch.

    It’s not clear to what degree organizations will embrace managed DevOps services, which are starting to appear on various cloud platforms. However, the implication is that by choosing these types of services, organizations may not need to hire, for example, as many site reliability engineers (SREs). The challenge, however, going forward is that most of the managed DevOps platforms are currently tied to a specific cloud service at a time when most organizations are routinely using more than one cloud.

    Of course, developers will need to continue to manage continuous integration (CI) processes, but, in time, the continuous delivery side of the DevOps equation may be incorporated within a managed service. Regardless of the path forward, as DevOps processes continue to become more automated, the pace at which applications can be deployed is about to significantly accelerate.

  • Why API Quality Is Top Priority for Developers

    Why API Quality Is Top Priority for Developers

    It is no secret that web APIs have become increasingly important to the operation of modern businesses. According to RapidAPI’s Developer Survey and Insights report, 61% of developers used more APIs in 2020 than 2019, with 71% planning to use even more in 2021. We are no longer tied to a small number of monolithic service providers. Businesses tend to gravitate toward specialized managed services, such as billing and authentication, which are backed by APIs. Business models for new products and services are constructed based on APIs. Trust has become a necessity for APIs. APIs that perform consistently and with high quality earn trust, and those that do not are abandoned for a competitor.

    Why Does Quality Matter?

    If a market exists for an API, it’s a given that there will be competition. To remain competitive, APIs must build and maintain trust with their consumers; sales and marketing can only go so far. The highest quality APIs will inevitably capture the largest slice of market share. Quality isn’t a destination, either—it is a collection of goals that require continuous investment.

    What Does Quality Mean for an API?

    Just like the code that drives an API, the quality of an API itself is quantifiable. The following facets describe some broad categories used to measure API quality. The weight of each facet will vary depending on how business-critical the API will be when in production.

    Resilient. An API is resilient when it continues to operate in the face of adverse conditions. Can an API continue to operate (even at reduced capacity) if the database is down? How does it handle network traffic spikes?

    Robust. A robust API is liberal in what it accepts, but conservative in what it sends. Will it crash when presented with a poorly formatted request, or respond with instructions on how the caller can fix the problem themselves? Can the code be restructured without breaking the API contract?

    Secure. API security is vital in most deployments. Are efforts made to check for OWASP Top 10 Web App security risks? Are open source vulnerabilities known and addressed in a timely manner? Are secure endpoints gated with authentication?

    Discoverable. APIs that are well documented and well structured are much easier to consume and integrate with other services.

    Consistent. A consistent API is one that changes without negatively impacting existing consumers. Changes are backward compatible or versioned as new features are introduced. API consumers should know far in advance if a breaking change will be coming if it cannot be avoided.

    How Do I Get There?

    Processes, tools and, most importantly, culture, are the key to high-quality APIs. Let’s explore how we can approach each API quality facet.

    Resilient. Processes such as chaos engineering, load testing and manual quality assurance can uncover situations where an API is failing to handle unexpected situations. Deploying your API to a cloud provider with a compelling SLA instead of your own hardware and network shifts the burden of infrastructure resiliency to a service, freeing your time to build features for your customers.

    Robust. A comprehensive suite of automated tests isn’t always sufficient to provide a robust API. Edge cases, unexpected code branches and other unplanned behavior may be triggered by requests that were not considered when writing the test suite. Traditional automated tests should be complimented by fuzz testing to help uncover hidden execution paths. Canary and blue-green deployments can further expose your API to the real world in a limited manner and prepare for when even more unexpected requests will hit.

    Secure. API security is a moving target. It is expected that most APIs are built on layers of open source libraries and frameworks. Software composition analysis is a necessity to stay on top of zero-day vulnerabilities by identifying vulnerable dependencies as soon as they are discovered. OWASP guidance is a must-have—directing API developers to implement attack mitigation strategies such as CORS and CSRF protection. Application logic must be well tested for authorization and authentication.

    Discoverable. For REST APIs, the OpenAPI initiative provides a consistent language for describing the API. Many frameworks, such as Spring Boot, can even generate OpenAPI documentation directly from your code. gRPC-based APIs offer similar benefits by providing both instructions for accessing the API and building the clients to communicate with it. Adding support for GraphQL allows developers to connect multiple APIs together with consistent tooling. Whichever direction you choose, it is vital to keep API documentation up to date and accessible.

    Consistent. To be consistent, the API development process should make incompatible changes obvious. Code reviews can catch many breaking changes right at the source. API contract testing and integration testing can be used to discover API changes as part of the build. When working in a microservices environment, staging environments may alert engineers if an API change will cause problems before production.

    Quality Builds Trust

    According to the RapidAPI survey, 58% of executives said participating in the API economy was a top priority for their organization. The figure was even higher in certain industries such as telecommunications, where the number is 89%. Whether you are building APIs for internal use or for public consumption, your goal should be to earn the trust of API consumers. Investing in the culture, tools and processes to build quality APIs will pave the way to building and maintaining that trust.

  • Grafana Labs Advances Open Source Visualization and Observability

    Grafana Labs Advances Open Source Visualization and Observability

    During the online GrafanaCONline conference today, Grafana Labs announced that version 8.0 of its open source visualization software widely employed by DevOps teams is now generally available. In addition, version 1.0 of Grafana Tempo, an open source back end platform for collecting distributed traces that are needed to drive observability across an application environment, is also generally available.

    At the same time, Grafana Labs revealed that Grafana Tempo is now available as a service on Grafana Cloud.

    Version 8.0 of Grafana adds support for additional types of visualizations such as streaming panels, state timeline, status history, bar chart and histograms in addition to a revamped unified alerting system that enables DevOps teams to view and edit alerts generated by Grafana and open source Prometheus monitoring platform via a common user interface. The notification system adopted to achieve that goal is based on a Prometheus alert manager that provides capabilities such as grouping, deduping and silencing of notifications alongside advanced message templates.

    The latest release of Grafana also improves startup and graph rendering performance, thanks to a major reduction in initial download size and graph rendering time.

    The enterprise edition of Grafana has also been updated to provide support for fine-grained access controls, improves caching of back end data sources and improved reporting capabilities such as live dashboard links that can be embedded in email.

    Overall, Grafana Labs claims there are now more than 750,000 active installations of Grafana that organizations are employing to create visualizations that correlate data collected from multiple back end sources.

    In addition to contributing to Grafana and Grafana Tempo, the company is running a cloud service that makes multiple instances of Prometheus running on different clusters accessible from a central console.

    Torkel Ödegaard, creator of the Grafana project and chief Grafana officer for Grafana Labs, said Grafana Tempo promises to further advance observability thanks mainly to its ability to employ object storage to capture complete distributed traces. Rather than only sampling traces, it’s now feasible to trace every request. Grafana Tempo is also compatible with any open source tracing protocol, such as Jaeger, Zipkin and OpenTelemetry, and can be integrated with metrics, logs and dashboards via a managed Grafana Cloud service.

    It’s not quite clear how quickly organizations are embracing distributed traces to improve application observability or, for that matter, to what degree traces might eliminate the need for certain types of metrics or log analytics. The one thing that is certain is DevOps teams require more context than ever to proactively manage complex IT environments. Distributed traces make it simpler to track interactions between application services. The challenge is they can require IT teams to store a lot more data.

    In time, of course, distributed tracing will become more commonly employed as developers more routinely instrument applications based on microservices that tend to have lots of dependencies that are difficult to discover in the absence of some type of distributed tracing capability. Of course, distributed tracing has been around, in some form or another, in application performance management (APM) platforms for some time. The difference now is that, as open source software, it’s becoming more affordable to implement.

  • The Future of Low-Code is Open

    The Future of Low-Code is Open

    The low-code market is seeing meteoric rise across the world, as companies try to keep up with digitization demands and shrinking IT budgets. Even as we witness increasing low-code adoption among professional as well as citizen developers, an intriguing question comes to mind – What lies ahead for low-code, and could it ever become a mainstream approach for modern development teams?

    The answer may well be an open source, low-code platform that offers high productivity, while supporting seamless integration with the overall fabric of modern software development practices within an enterprise.

    It’s feasible to assume that low-code will evolve to become open low-code, resulting in greater innovation and agility.

    To further understand what this means, let’s dive deeper. What are open systems?

    According to Wikipedia, open systems are computer systems that provide some combination of interoperability, portability and open software standards.

    Over the years, the software industry has seen great benefits from designing, implementing and consuming open systems. TCP/IP protocol standards, UNIX systems, Web browsers, REST APIs – all of these are shining examples of open standards that went on to become highly successful and widely adopted. By remaining open, they enabled higher interoperability, streamlined development and fostered rapid innovation.

    Low-code is now at a critical stage in its adoption curve. For the last few years, we have seen citizen developers successfully execute shadow IT with low-code and churn out applications at a breathtaking pace. Today, low-code platforms are hardened for enterprise use, are programmed to understand the scalability and security needs of a complex application and have integration capabilities mature enough to seamlessly fit in with existing tools and technologies. As a result, we are now seeing greater adoption of low-code within the professional development community, covering a diverse set of use cases from simple dashboards to complex applications. The natural next evolution of low-code is that it becomes mainstream within enterprise IT, and is used to build mission-critical applications and systems. So, how does this next phase play out for application development?

    The Case For an Open Low-Code Platform

    The approaches and techniques of modern software development teams has changed dramatically to meet the demands of modern, software-enabled business. Developer velocity and organizational agility have become key benchmarks for high performing software development teams, as those metrics have a direct correlation with software excellence and superior business performance. According to a research report by McKinsey, teams that belong to the top quartile with regards to developer velocity score have 60% higher total shareholder returns and 20% higher operating margins. Such teams can experiment faster and release code to production without delays.

    As application teams continue to embrace low-code for mainstream development, it is vital that low-code platforms support developers so they are encouraged to employ modern application development and delivery best practices. Low-code should introduce minimal disruptions to a developer’s working practices and workflow. Also, it is important that low-code can integrate seamlessly with the fabric of the overall enterprise application architecture. This can happen if a low code-platform is open standards-based and flexible so that the rest of the enterprise application infrastructure can coexist with it.

    What Makes a Low-Code Platform Open?

    Developer-Centric Features

    Developers like control, flexibility and a unified experience. They find comfort in sticking to their preferred languages, tools and application development and delivery workflows. A low-code platform that offers productivity with equal flexibility, with a focus on building robust enterprise architecture, is bound to be the future of application development. Platforms should focus on providing a unified developer experience across concept, design, integration, development and deployment stages of the app life cycle, employing a technology stack that is modern, best-of-breed and cloud-native. It’s equally important to provide a way for developers to easily bring any external innovations into the low-code platform.

    Visualization, Customization and Ownership

    Many low-code platforms do not generate 100% of an application’s code. Also, most of the code generated by proprietary platforms is also proprietary, and quite often remains hidden and not easily accessible or extensible. A platform that generates open standards-based, real code is a great asset, especially for professional developers building complex applications that require extensive customization and deep integration with enterprise tooling (security, testing, debugging, integration etc.). The code generated should be based on developer-friendly, best-of-breed application stacks and standard design patterns. This way, application teams will have complete familiarity with and understanding of the code. Enabling full export of the code allows teams to own the code created by the platform.

    Flexible Application Architecture

    The application architecture should be loosely coupled, supporting microservices and micro front ends that can be independently built, deployed and scaled. This way, architecture can support cloud-native application development easily. Also, all other aspects of the application life cycle should allow for plug-and-play capability. This includes, but is not limited to, plugging in custom UI components (widgets, templates), custom security providers, custom back end code, logging frameworks, event driven systems, etc. A plug-and-play model ensures that development teams can integrate custom providers that are fine-tuned for the enterprise.

    Modern Development Best Practices

    Modern application development practices have evolved to allow teams to experiment faster and release code to production at a never-seen-before pace. Optimizations in performance and scalability have resulted in applications that can support millions of end users. As developers warm up to low-code, the platforms should align with and implement all modern development practices while building applications. The idea is to minimize friction in a developer’s journey towards low-code, so that they continue to leverage the same design principles, application tooling and enterprise integrations as they do in the complex programming world.

    Infrastructure-as-Code (IaC)

    Developers need a way to continuously deploy software so there is always a version of the application ready for production. Low-code platforms should support an IaC option, so code generated is always deployable seamlessly on the developer’s infrastructure of choice. Platform should integrate to build, test and release systems (version control systems, CI/CD, artifact repos, container image repos, Kubernetes clusters and public or private cloud instances, for example). This way, artifacts built by low-code are continuously integrated and deployed to the enterprise’s operational systems.

    Low-code is at an inflection point within enterprises, as it becomes the platform of choice for digital transformation and application modernization. This is the opportunity for low-code platforms to become a key ingredient of an enterprise application architecture. An open low-code approach will allow application development teams to benefit from the underlying best practices prevalent within the organization.

    Low-code is not merely a productivity tool; it has the potential to be a technological and cultural catalyst that drives enterprise innovation and business agility.