devopsy.kinsta.cloud

Category: Continuous Delivery

Continuous delivery is a software engineering discipline that allows organizations to consistently develop and release quality software at a rapid pace. It is achieved by automating the software development lifecycle, enabling tasks such as development, testing and release to occur concurrently and continuously.

5 Unusual Ways to Improve Code Quality
Code quality refers to how well-written and maintainable software code is. It encompasses factors such as readability, efficiency, scalability, reliability, maintainability and security. High code quality ensures that software is easy to understand, modify and extend, has fewer bugs and is less likely to break down. It also reduces development time and costs and improves the overall user experience. One common approach to determine code quality is to use automated tools that analyze the code based on predefined quality standards and generate a report with a numerical score.

Why is Code Quality Important?

Generating high-quality code ultimately enhances the end user’s experience. Focusing on creating quality code can also help consumers understand the safety and reliability of the software or program they’re using.

Low-quality code can affect user experience and satisfaction, making it critical to write high-quality code that functions properly. When developers write low-quality code, it can create vulnerabilities that expose the software to cybersecurity threats, resulting in bugs and functionality issues and impacting the software’s reliability.

Here are some additional reasons high-quality code is important:
- Enhances code readability: Readability is important because it leads to higher rates of understanding among other developers. A code that’s easy to read is also easy to improve.
- Improves the level of program sustainability: Program sustainability measures how long a code can survive with minimal changes. High-quality code can last for years, making it more sustainable.
- Increases transferability: Transferring code from one platform to another is a complicated process. It’s much easier for developers to transfer high-quality code, as they only need to make minimal changes.
- Lowers costs related to program or software: When software lasts for a long time with few errors, it can save a company money. Instead of paying for additional time to rewrite low-quality code, the company can rely on the original, high-quality source code.
5 Unusual Ways to Improve Code Quality

Here are a few ways you may not have thought of which can help your development team improve code quality and end-user satisfaction.

Bug Bounty

A bug bounty program is a type of crowdsourced security testing in which an organization offers rewards to independent security researchers or ethical hackers for identifying and reporting vulnerabilities in their software or systems. By offering rewards for vulnerabilities, organizations can incentivize security researchers to find and report security issues before they can be exploited by attackers.

Bug bounty programs can improve code quality in several ways:
- Incentivize security testing: Bug bounty programs incentivize security researchers to test an organization’s software or systems for vulnerabilities. This can increase the amount and quality of security testing that is done, leading to a more secure product.
- Encourage responsible disclosure: Bug bounty programs encourage security researchers to report vulnerabilities directly to the organization rather than publishing them publicly or selling them on the black market. This allows organizations to fix vulnerabilities before they are exploited by attackers.
- Provide feedback on code quality: Security researchers who participate in bug bounty programs can provide feedback on code quality which can help organizations to improve their development practices and write more secure code in the future.
Auto-Generated Documentation

Auto-generated documentation is a type of documentation that is generated from code comments and metadata as part of an automated development pipeline. It is typically generated using tools such as Javadoc, Doxygen, or Sphinx, which parse code comments and metadata to generate HTML, PDF, or other formats of documentation.

Auto-generated documentation can improve code quality in several ways:
- Improved code readability: Auto-generated documentation can provide clear and concise descriptions of code functionality, making it easier for developers to understand the code and its purpose.
- Better code organization: Auto-generated documentation can help developers organize their code by providing a clear hierarchy of classes, methods, and functions, making it easier to navigate and maintain the codebase.
- Consistency: Auto-generated documentation can ensure that documentation is consistent across the codebase, reducing the risk of errors and misunderstandings.
- Quality assurance: Auto-generated documentation can be used as a quality assurance tool, helping developers to identify errors and omissions in their code comments and metadata.
Coding Conventions

Coding conventions are a set of guidelines and standards that outline how code should be written and formatted to promote consistency and readability. They typically cover aspects such as naming conventions, code structure, formatting, and documentation.

By following coding conventions, developers can improve code quality and make their code easier to read and understand, which makes it easier to maintain and modify. They also help ensure that code is consistent across a project or organization, which can make collaboration and integration of different code segments easier.

Typical elements that coding conventions dictate include:
- Naming conventions for variables, functions, and classes.
- Indentation and formatting of code blocks.
- Use of whitespace and comments to improve readability.
- Guidelines for handling exceptions, errors, and input/output.
Quality Gates

Quality gates are a set of checkpoints or criteria that software must meet before it can progress to the next stage of development. They are typically used in software development processes to ensure that the software is of a certain quality and is ready for release.

Quality gates can be automated or manual, and they are typically implemented at key points in the development process, such as after unit testing, integration testing, or acceptance testing. They can include metrics such as code coverage, code quality, functionality, performance and security.

By enforcing quality gates, development teams can catch and address issues earlier in the development process, which can reduce rework, save time and ensure that the final software product meets the required quality standards.

Smart Code Snippets

Smart code snippets, also known as intelligent code snippets or live templates, are pre-written code fragments that can be inserted into a code file with placeholders for variables or user-defined inputs. They are designed to help developers write code more efficiently and accurately by reducing the amount of repetitive or boilerplate code they need to write and by providing contextual awareness and intelligence.

Regular code snippets are static pieces of code that can be inserted into a file as needed. They typically contain pre-written code blocks or templates that can be customized to fit the specific use case.

In contrast, smart code snippets are dynamic and can adapt to the context of the code being written. For example, a smart code snippet might include placeholders for variables that need to be defined or automatically fill in common code constructs based on the context of the code.

Conclusion

The five lesser-known ways to improve code quality discussed in this article offer a starting point for teams looking to take a fresh perspective on their development process. Improving code quality is critical for the success of any software development project. While there are many established methods and best practices for improving code quality, there are also some less conventional approaches that can be surprisingly effective. By adopting a creative and open-minded approach to software development, teams can identify new ways to improve code quality and build more reliable, efficient and secure software.

Image Source
March 20, 2023
Four Technologies Transforming Data and Driving Change

Data is getting a facelift in 2023 courtesy of event-driven architecture (EDA). These four technologies are set to transform DevOps and IT operations over the next year–whether it be how companies make use of data or the customer experience they offer.

1. Applied Observability: A Powerful Source of Decision-Making

Applied observability has been recognized by Gartner as a key 2023 strategic tech trend at its most recent Orlando Symposium. It ensures that the right data is delivered at the right time for rapid action based on confirmed stakeholder actions rather than intentions. “Observable” data include key digitized artifacts that appear when any stakeholder takes any kind of action. Applied observability feeds these observable artifacts back in a highly orchestrated and integrated approach to accelerate organizational decision-making and allows the business owner to track how long an action took to fully process at every point in the workflow.

But unearthing these insights requires an event-driven approach to software architecture. Distributed tracing, for example, is a method of tracking application requests as they flow from front-end devices to back-end services and databases. By definition, and in contrast to traditional tracing, distributed tracing can be visualized to show a searchable, graphical picture of when, where and how a single event flowed through an enterprise, regardless of the number of hops the workflow took to fully process.

Consider the potential traceability and end-to-end observability benefits across a payment ecosystem underpinned by EDA. Embedding distributed tracing into an event mesh emits trace events in OpenTelemetry format so banks can collect, visualize and analyze them in any compatible tool. This empowers them to not only confirm that a given message was published but easily understand exactly when and by whom, where it went, down to individual hops, who received it and when … or why not.

2. Hyper and Intelligent Automation: Taking Automation to the Next Level

Gartner and IDC identified both hyper and intelligent automation in their yearly technology trends for 2023. Hyperautomation moves automation up a level, adding more intelligence to automation and using a broader set of tools so that previously un-automatable tasks can be automated. Hyperautomation initiatives come in many different shapes and sizes and are being seen across a wide range of industries, from banking and insurance to manufacturing and health care.

But hyperautomation requires an organization to be underpinned by an EDA and–more specifically, an event mesh–to maximize success. An event mesh is an interconnected network of event brokers that allows data to be pushed in real-time to parts of the organization where the data is needed. It does this dynamically, meaning that new event types can be added at any time and interest in events can be registered to allow a seamless interchange of data for the applications that are interested in using it.

Take the aviation industry as an example. An event mesh streams information such as flight routes, delays, cancellations and mileage accruals between applications, connected devices and people anywhere in the world, instantly. With an event mesh, information about events can be continuously streamed to multiple systems; filtered so each system only receives the data it needs. This ultimately enhances the customer experience as passengers, pilots and crews are notified in real-time when something relevant occurs on each and every flight.

3. The Acceleration of the Event-Driven API Economy

The API economy has exploded. Research from MIT Initiative on the Digital Economy showed that over a four-year period, businesses using APIs saw 12.7% more growth in market capitalization compared to those that did not adopt APIs.

But the API economy is changing as event-driven architecture and asynchronous event-driven APIs become increasingly important to companies striving to operate in real-time. Using both synchronous and asynchronous methods can result in an application environment in which system resources are most effectively used.

AsyncAPI and RESTful APIs will be the next evolution of the API economy. API management platforms will need to adapt quickly to this new reality. As a prime example, Forrester recently spotlighted a global biotech company that is blazing a trail in the API-driven area. As the case study demonstrates, the organization understood that EDA is a complement to RESTful APIs–another tool in their digital business toolbox, filling in the gaps of a REST-only approach.

As a result, they planned a unified event + REST approach from the outset. Both APIs and events are managed as enabling digital products, with the governance and life cycle of both unified as one process. A platform team builds a unified API + event platform for app dev teams rather than building each as a separate siloed platform. The results include faster speed to market and new business opportunities. The company also embraces FAIR data principles: Data that is findable, accessible, interoperable and reusable. A digital marketplace consisting of both events and APIs makes the data findable and machine-readable via Open API Spec (OAS) and AsyncAPI metadata. Data can then be accessed on demand via APIs and events push data in real-time.

4. The Metaverse Continues at Pace

Driven by technologies including digital twins, augmented reality and virtual reality, McKinsey estimates the metaverse will be worth $5 trillion by 2030. Data in real-time will be the vital common denominator to link the digital and physical divide and optimize the Metaverse. McKinsey lists real-time data as a vital element to facilitate adoption of the metaverse in several key industries including retail, banking/financial services and transportation.

The metaverse cannot be static. It needs to be in real-time, in motion. It needs an EDA and a real-time event mesh to support it.

For example, if an avatar visits a retailer in the metaverse, the retailer will need to keep track of countless user actions such as how long someone spent looking at a particular product, which products got the most interest, etc. The metaverse will produce a significant amount of data that companies will be interested in analyzing in real-time. This data can be used to make decisions from real-time adjustments to inventory to incentivizing commerce through real-time offers/discounts/coupons and even dynamically pricing products.

Designing a Data-Driven Future With new Tech

Together, these four predictions have the potential to revolutionize company operations, communications and decision-making–but to take advantage of their full potential and reap success, these four trends will need EDA by their side.

March 17, 2023
Neural Hashing: The Future of AI-Powered Search

In recent years, vector embeddings have emerged as a promising technology to power information retrieval projects. Vectors use mathematics to represent words and encapsulate meaning. Combined with deep learning algorithms, you can build powerful AI search applications. However, recent advances have shown that another kind of technology – neural hashes – can outperform vectors while avoiding the tradeoffs that people experience with vector search.

Neural hashing’s superiority over vectors will soon make it accessible, affordable and the primary method for delivering AI-powered search for e-commerce, enterprise search and other areas where search data is constantly growing and changing. Even as early as 2023, neural hashing will be the primary method for delivering AI-powered search.

First Came Keyword Search

Most of us are familiar with the basics of keyword search technology, which most search engines still use. When someone submits a query, the keyword search engine will perform a simple lookup and display results by relevance. Keyword algorithms employ techniques such as natural language processing (NLP), lemmatization, typo tolerance, synonym libraries and more to perform the lookups.

Keyword search engines work well for the most part but can fall short when it comes to long-tail queries, i.e., those longer and more specific queries which represent half or more of search queries. Despite its powerful keyword technology, you can never write enough rules or stuff your site with enough keywords to cover every edge case. Being able to improve long-tail query results, however, can be a huge benefit to an organization, which is why more companies are beginning to look for new technologies to complement their keyword search engines.

Then Came Semantics

Semantic, or AI-powered search, can understand concepts. Semantic search goes beyond matching to understanding. For example, if you searched “something for a headache” on a website, a keyword search engine would look for items tagged with those specific words, turning up results only if they can match some or all of the text. But a semantic search engine would understand the concepts being described and offer solutions—such as pain medications.

Semantic search has traditionally been employed using vector embeddings. You can use vectors for all kinds of objects, too—images, videos, audio, etc. Vector-powered search engines can be incredibly smart, but processing and maintenance make them too expensive to run in production for most enterprise use cases. This is true, especially for certain industries where data is dynamic. In retail and e-commerce, for example, that inability to pivot and scale quickly is a real detriment, as brands need to update data that is unpredictable (e.g., user-generated content) and constantly in flux (many online sellers update their search databases hourly).

The solution is neural hashing, which refers to the process of compressing vectors with neural networks (deep learning algorithms which mimic the human brain) into hashes or binary vectors. These hashes retain 96% or more of the vector information while using a fraction of the storage space. A semantic search engine built using neural hashing is nearly or just as accurate as vector search but about 500 times faster.

Better Results With Hybrid Search

When it comes to AI-powered search, neural hashing will eliminate the tradeoffs between speed, accuracy and costs. It can deliver results in single-digit milliseconds with no additional overhead. As good as it is, it’s even better when combined with a keyword search. We can use hybrid search—both technologies—simultaneously; most queries are a combination of keywords and concepts.

For example, what if rather than looking for a type of product, a shopper is searching for help with getting rid of a pesky symptom? Or, has a question-style query? As search data indicates, people are often myopically focused on a problem that they desperately want to solve, such as:

● Leaky faucet
● Stress headache
● Stained carpet

They only know they have an urgent problem. Of course, they’re looking for a solution, but they typically have no idea what particular product they need in order to get relief. They’re still using some keywords, however, like “faucet,” “stained” or “carpet.”

Each query can be scored by both the hashing and keyword engines to deliver a superior result for every query. Hybrid search can be combined with traditional search engine features like natural language processing (NLP), autocomplete, learning-to-rank algorithms, filters and more to optimize the end-user experience.

When it comes to search, speedy, accurate results are the name of the game. In order to make searches as intuitive, fast, and scalable as possible while still remaining cost-effective, search and discovery platforms should explore what hybrid search powered by neural hashing has to offer.

It has been shown that neural hashing can make highly intelligent AI fast and cost-effective at scale and without the need to bring on highly specialized engineers for every project. We appear to be at another turning point in the world of artificial intelligence, and neural hashing is going to be the driving force behind many future innovations.

March 17, 2023
What NetOps Teams Should Know Before Starting Automation Journeys
Automation has captured the human imagination since the earliest days of recorded history. A fanciful example of this comes from Homer’s classic “The Iliad” in the form of a bronze robot called Talos, which was designed to automatically monitor and defend the island of Crete from marauding intruders. The appeal of automation for IT organizations, including network operations (NetOps) teams, may not be so life-and-death but it’s undeniable, nonetheless. More and more NetOps teams are turning to automation to achieve higher network uptime (by reducing manual errors), increase efficiency and reduce costs and help deliver a better and more consistent operational experience for all users.

Due to the allure of these promised benefits, the temptation for NetOps teams to integrate automation as soon as possible may be strong. However, it is important for these teams to remind themselves that automation is a tool, not the destination itself. Indeed, simply overcoming the technical challenges of deploying automation may not result in the successful outcome teams want, without identifying and defining the intended outcome of what their automation journeys should be.

This article will explore some practical steps and best practices that NetOps teams should use throughout their own automation journeys.

Getting the Journey Started on the Right Step

A well-known maxim in business is that you should visualize success and start with the end in mind when embarking on a major project or initiative. This is a good reminder for NetOps teams as they begin their automation journey. That is, these teams should be able to answer the question, “What is our desired end state when it comes to automation?” before they even evaluate what tools or technologies they may want to consider.

Teams who have this mindset will have a clearer understanding of their overall mission as well as an enhanced view of the various projects, processes and products they can improve through automation. It is critical for NetOps teams to take the time to put in the work of defining the desired IT experience at the end of the automation journey, even if the task itself may be difficult. For example, this experience may help define how NetOps engineers will interact with an automation tool through its code, its dedicated user interface, or even its generic service portal. As the saying goes, “no pain, no gain.” The teams who invest in the time and effort to plan ahead will be far more likely to maximize the benefits of automation.

What to Automate?

In addition to visualizing what success looks like, it is important for NetOps teams to identify what tasks or processes they should be automating. This is important because there is a vast universe of NetOps tasks that can be automated. Taking the time to identify tasks to be automated will generate huge benefits for the NetOps team as it will help them select the correct automation strategy and toolset that will deliver them the results in an efficient way. And while specific projects may be unique to each NetOps team, in general, there are two broad categories of tasks that are ideal for automation:

1. Highly repetitive tasks. These are manual or semi-automated tasks that are not necessarily hard but can be cumbersome for NetOps teams. Examples of repetitive tasks that can be automated include assigning an interface into a specific VLAN or perhaps updating network security policies or rules.

2. Highly complex tasks. These can be more strategic and difficult tasks that can be prone to human error due to their scope, scale, and complexity. Automating these tasks may require additional planning and steps to ensure success. Examples of complex tasks that are ideal for automation include service redirection from one device to another or BGP peering with third parties.

Think Big, Start Small With NetOps

Like all people, NetOps professionals enjoy the results of a job well done. So, while the vision of their automation journey may be big, it’s important to start with a small, short-term project that can be completed quickly. There are a couple of benefits to this approach:
1. Quick automation wins will give NetOps teams confidence for future projects.
2. Projects like this can generate data and feedback that NetOps teams can convert into learnings and insights for the next project.
This approach can also be applied to bigger, more complex automation projects. Instead of taking on the entire scale of the project at once, NetOps teams can break it down into smaller components. For example, infrastructure-as-code (IaC) is a broad category of automation entailing the managing and provisioning of systems through machine-readable definition files. There are many considerations to large-scale IaC projects such as identifying the use case, e.g., Day 0 or Day 2 operations. The advantages of this approach are the same as with the quick-win scenario: There is a better likeliness of success and more immediate feedback and data to guide the NetOps teams through this entire process.

Finally, as talented as most NetOps teams are, they are not likely to have all of the automation expertise in-house at any given time. So, it is important that they address skills gaps (automation, programming) through strategic vendor partnerships, internal/external recruitment and additional training for their teams. Again, instead of recruiting or investing en masse, NetOps teams are well-advised to take the time to plan, execute, learn and modify as they go along.

Summary

The human fascination with automation is stronger than ever. The industry buzz around one particular form of automation—especially artificial intelligence and machine learning—is indicative of this. NetOps teams have much to gain from investing in the tools and skills required to do automation right.

Automation of data center network pods, self-contained zones that represent a section of the overall data center, is a good example of a strategic project requiring forethought and planning to execute well. A logical first step in this process might be to anticipate how to manage future changes when automation is in place. Options for this may include code manipulation, Git (open source code management) or CI/CD pipelines, all elements of infrastructure-as-code automation. Or it may involve improving user experience through a specific UI interface or through a self-service portal. The next step then might be to decompose this large configuration into smaller bits starting with the registration of the new switches into the controller. Once this is done, we can now go into port configurations and then eventually move toward larger and more sophisticated elements such as the automation of VLAN assignments and L3 peering.

For long-term and large-scale automation projects like this, it is especially important to follow the recommended key steps outlined in this article, which are:

1. Start with the end in mind. It’s critical that the teams take the time and effort to define what success looks like, especially in the form of the IT experience they want to achieve.
2. Know what you want to automate. There is an endless supply of tasks and processes that can or should be automated. Prioritizing automation projects that NetOps wants to take on is a major milestone in any automation journey.
3. Think big, start small, learn as you go: Quick automation project wins are key to building confidence and “muscle memory” for future projects. For bigger projects, the best approach is to break them down into more manageable projects that can provide vital data and insights during the journey.
March 16, 2023
DevOps Adoption in Salesforce Environments is Advancing

A global survey of DevOps professionals building apps in Salesforce environments found 44% of respondents reported their organization has a continuous integration/continuous delivery (CI/CD) platform in place, with another 39% planning to adopt one.

The survey, conducted by Gearset, a provider of a DevOps platform for building custom applications in Salesforce environments, polled 1,254 DevOps professionals that are building and deploying custom applications on top of the Salesforce software-as-a-service (SaaS) platform. The survey also found nearly half (49%) implemented change monitoring, with another 33% planning to follow suit. More than half (52%) implemented source-drive workflows, with another 33% similarly planning to follow suit. A total of 41% of respondents noted that, over the last 12 months, they have more custom code to manage.

Overall, the survey found 54% of respondents reported increased productivity, followed by improved quality and faster releases at 41% each. Nearly three-quarters (72%) reported they are able to deploy changes from one environment to another within four hours. Well over a third (39%) of this group are able to complete a deployment in less than an hour. Just under a quarter (23%) are also able to release code multiple times a week.

However, only 18% can release a new feature to production in less than a day. Nearly half can complete the release within one day to a week. A total of 43% claimed that less than 10% of releases include a bug or error, with 46% reporting they can roll back a release in a day or less, if needed. Just over a third can restore a process in a day or less.

The biggest obstacle to DevOps adoption is lack of experience (41%), followed by a third (29%) of respondents that reported difficulties with collaboration and another 29% that experienced limited buy-in from the rest of the application development team. Only 30% received DevOps training once a month, with 50% citing additional CI/CD and automation training as their biggest need.

Despite these issues, a full 98% are generating a return on their DevOps investments, with 59% of teams reporting returns of over $10,000 per month and 40% reporting returns in excess of $50,000 per month. The top two priorities for the next 12 months are to consolidate systems and processes and improve flexibility, tied at 47% each, followed closely by 43% that noted that digital business transformation is a top priority.

David Runciman, technical content manager for Gearset, said that while Salesforce applications are based on unique object architecture, it’s apparent that organizations are extending those applications using, for example, low-code tools more quickly. The issue that creates is, as organizations move to deploy these applications, they are encountering bottlenecks that conspire to slow the pace of innovation, he noted.

The maturity level of DevOps teams building these applications will naturally vary. However, as the number of applications being built—especially by professional developers—increases, it’s only a matter of time before more organizations embrace DevOps best practices to bring order to the chaos, noted Runciman. While there may be some citizen developers building applications in Salesforce environments, the bulk of enterprise-class applications are built by professional developers that typically appreciate DevOps best practices, he added.

The issue remaining is just how quickly those DevOps teams will be able to implement DevOps best practices as demand for additional applications grows.

March 16, 2023
How Bazel and GitHub Can Fix the Dependency Availability Problem

Recently, GitHub upgraded the internal version of Git they use to produce repository archives. You’ve probably used these archives before if you’ve downloaded a .zip or .tar.gz file from a repository at a particular version. GitHub produces those archives on demand using Git archive and caches them for a short time.

Upgrading Git regularly is a generally good idea, but this change regrettably broke a huge number of Bazel projects. Bazel is a widely used, free, open source tool for building and testing software, so this breaking change is a big deal. What happened? Most Bazel projects fetch at least some of their dependencies using rules in their WORKSPACE files like this:

http_archive(

name = “com_github_bazelbuild_buildtools”, sha256 =

“05eff86c1d444dde18d55ac890f766bce5e4db56c180ee86b5aacd6704a5feb9”, strip_prefix = “buildtools-6.0.0”,

urls =

[“https://github.com/bazelbuild/buildtools/archive/refs/tags/6.0.0.ta

r.gz”],

)

See that /archive/refs/tags/ part of the path? That’s the endpoint I’m talking about.

This is bare-bones dependency management: Bazel attempts to download an archive from the first URL in the list; it tries the next URL if the first is not available and so on. Bazel then checks the file’s SHA-256 sum against the known value and, if it’s correct, extracts the archive and proceeds with the build.

The Git upgrade caused a change in archives’ SHA-256 sums. I think there was a small change in zip compression, but it doesn’t really matter—any variation in file ordering, alignment or compression causes the archives’ SHA-256 sums to change even though the extracted contents are the same.

This is at least the third time Bazel builds have broken that I can remember. This has also been discussed extensively before. I’m writing this in the hope that we can make our systems more resilient and avoid these kinds of problems in the future.

Since GitHub made the change that triggered this, they naturally get the immediate blame from the community, though I think it’s mostly undeserved. Upgrading dependencies (especially Git and especially if you’re GitHub) is a reasonable thing to do. To my knowledge, GitHub has not documented a guarantee that files returned by the archive endpoints have stable SHA-256 sums. It’s a mistake for users to rely on a guarantee that was never made. It’s tempting, of course, because it’s easy, but it’s a mistake nonetheless.

This is a classic example of Hyrum’s Law:

“With a sufficient number of users of an API, it does not matter what you promise in the contract: All observable behaviors of your system will be depended on by somebody.”

Since these updates have broken Bazel (and presumably others) a few times now, I’d really like to see GitHub clarify in documentation whether users should or should not depend on stable archive SHA-256 sums. A GitHub engineer commented that this is not stable, but product managers and support engineers have commented at other times that is stable. I don’t really think discussion comments count since they’re not discoverable. Only official documentation is authoritative.

I haven’t actually found any documentation for these release archive URLs, so I’m not sure where this clarification should go. It’s not part of the REST API. Linking to releases is pretty close.

If archive SHA-256 sums are guaranteed to be stable (now or in the future), I think documenting and testing that would let us all sleep easier at night.

If archive SHA-256 sums are not guaranteed to be stable, it wouldn’t be a terrible idea to inject a little chaos to prevent people from depending on them. For example, in Go, the iteration order of elements in a map is undefined. To prevent developers from depending on iteration order (and tests from breaking when the hashing algorithm is tweaked), the Go runtime adds a random factor into the hashing algorithm, so the iteration order is different every time a program runs. Something similar could be done here with archive file order or alignment. I wouldn’t suggest gratuitously breaking this API, but if it needs to change anyway for some reason in the future, it would be a good idea to add something like this.

What Could the Bazel Community do Better?

Bazel developers should not rely on stable archive SHA-256 sums unless that stability is guaranteed and documented by GitHub. More importantly, developers should not rely on dependency artifacts being available on GitHub at all: A library author could delete their project at any time.

I’ll point to Go modules as a model of a great dependency management system designed to solve this exact problem. The Go team operates proxy.golang.org, a mirror for all publicly available Go modules. Internally, the proxy stores actual files for each module and does not need to regenerate them. The proxy protocol is open and easy to implement as an HTTP file server, so you can run your own proxy service for better availability. I’d love to see something like this happen for Bazel, especially if it’s operated by Google. It is not technically difficult to build a service like this, but there are a lot of thorny issues around handling abuse and legally distributing software with unrecognized licenses, and Google has already figured out those issues for Go.

Until such a service exists, developers can protect themselves by copying their dependencies to their own mirror. A GCS or S3 bucket works fine.

Library authors can and should protect their users by providing static release artifacts (not dynamically generated archives), and mirroring those. For example, check out the http_archive boilerplate for rules_go:

http_archive(

name = “io_bazel_rules_go”,

sha256 =

“dd926a88a564a9246713a9c00b35315f54cbd46b31a26d5d8fb264c07045f05d”, urls = [

“https://mirror.bazel.build/github.com/bazelbuild/rules_go/releases/d ownload/v0.38.1/rules_go-v0.38.1.zip”,

“https://github.com/bazelbuild/rules_go/releases/download/v0.38.1/rul es_go-v0.38.1.zip”,

],

)

The file rules_go-v0.38.1.zip is created by the rule authors and attached to the release; it’s not dynamically generated.

It’s also copied to mirror.bazel.build, which is a thin frontend on a GCS bucket shared by many rule authors in the Bazelbuild organization.

One other tip: If you’re feeling adventurous enough to use an experimental, undocumented feature (to make your build more stable! Really!), you can configure Bazel’s downloader to rewrite those GitHub URLs to point to your own mirror.

Aside: SHA-256 of archives or contents?

It’s unfortunate that a change to the Git archive that does not affect extracted contents of an archive can still change its SHA-256 sum. Bazel absolutely does the right thing by checking the sum of the downloaded file before extracting its contents.

This is the (delightfully named) Cryptographic Doom Principle. If Bazel only authenticated the contents of an archive, it might be possible for an attacker to exploit a vulnerability in Bazel’s zip parser before the archive is authenticated. Since Bazel authenticates the archive before extracting it, the pre-authentication attack surface is very small.

Closing Thoughts

When you’re designing software, think carefully about how it’s going to be used. If there’s a right way and a wrong way to do something, make sure the right way is easier and more obvious. Better yet, make the right way the only way.

I think this is a case where Bazel’s dependency management is too limited: To use http_archive safely, you need to set up an HTTP mirror with copies of your dependencies. That’s too much work for users, especially new users who aren’t aware of the hazards. A more complete dependency management system should include an artifact registry or a read-through caching system with at least one public implementation. I was hoping Bazel modules and the Bazel central registry would provide that, but the central registry only includes module metadata: Module content is separate, specified in URLs that still frequently refer to the unstable GitHub endpoint.

March 16, 2023
Addressing Software Supply Chain Security

It’s essential for organizations to learn more about the software supply chains they rely on and the steps needed to secure them. In just the past few years, we have seen a major uptick in malicious actors exploiting vulnerabilities in software supply chains to facilitate attacks on organizations. However, it’s important to remember that these attacks are not a new phenomenon. In fact, they’ve been around for decades. Therefore, the need to secure the software we rely on is not new, though the industry turning its attention to this problem is.

Software supply chain attacks are also far from simple. Given the fact that software supply chains are highly complex and made up of a number of constantly changing components, the avenues that malicious actors can take to pull off an attack are plentiful. For example, software artifacts are partially made up of components downloaded from open source software repositories. These platforms have become a favored attack avenue, given how frequently they are used by developers and the trust extended to them by development organizations. According to GitHub, anywhere from 85-97% of enterprise codebases come from open source repositories. Two of these repositories, npm and PyPI, have seen a nearly 300% spike in the number of attacks on their platforms in the past four years, impacting thousands of software supply chains globally.

One example of the kinds of software supply chain attacks that are now common is IconBurst, a major attack from last year where more than two dozen typosquatted packages were discovered on npm. As with other recent attacks, the IconBurst attackers used typosquatting – uploading malicious packages to repositories with names that closely resemble popular and legitimate packages on the same platform. Developers, in a rush to complete their work, are often tricked into downloading these typosquatted packages. In the case of the IconBurst campaign, for example, one of the malicious packages discovered was downloaded over 17,000 times!

Given the significant increase in the number of software supply chain attacks, organizations must make more of an effort to understand how these attacks work and then take steps to secure their software supply chains from compromise.

Say Goodbye to “Shift Left”

To begin with, development organizations need to reevaluate the mantra to “shift left.”

For years, industry leaders promoted the notion of “shifting left” as a call to marry application security with application development: Embedding secure development tools and practices into the continuous integration/continuous delivery (CI/CD) process. But times are changing and the industry is beginning to understand that while shifting left is necessary, it is not sufficient to secure software supply chains.

“Don’t shift left; shift everywhere!” said Tanya Janca, founder and CEO of WeHackPurple. Janca was speaking as a member of a recent Techstrong panel, “Software Supply Chain: It’s All About The Code” at the Predict 2023 virtual event. Janca pointed out the industry’s past fixation on “shift left” and the gaps in security that this caused since threats to the software supply chain not only happen during development but also throughout the entire software release process.

In fact, the notion of shifting left may do development organizations and developers a disservice: Leaving the work of securing applications entirely to them. “It’s unfair to expect developers to write perfect code,” said Jeff Williams, co-founder and CTO of Contrast Security, another member of the panel.

Software engineers are already under pressure to develop software quickly to meet development and project milestones. That makes it difficult for them to implement perfect security measures, like vetting every single line of third-party code they use while also watching out for social engineering attacks.

A New Approach to Secure the Software Supply Chain

For organizations looking to secure the software supply chains they rely on, they must choose the right tools. But much of the secure development and application security testing (AST) technology out there won’t cut it. For instance, tools such as dynamic or static application security testing (DAST/SAST) are critical components of the secure development life cycle (SDLC). However, they don’t provide holistic coverage of all of the supply chain threats out there, including software tampering and the risk posed by compromised open source and third-party libraries. On the other hand, tools such as software composition analysis (SCA) can help vet open source components but often fail to detect malicious modules or provide total security coverage of software supply chains. Knowing that your application depends on a malicious open source package after the fact is of little consolation when you release code at the speed of DevOps.

This is why modern tools that go beyond these capabilities should be considered. “It’s about giving developers the tools they need to be successful,” said GitLab’s CPO David DeSanto at the event. Experts in the space agree that change is needed but that developers and development teams aren’t set up for success as it stands.

According to Mat Matthews, an experienced practitioner in the DevOps space: “While developers are responsible for ensuring clean code, it takes a cross-functional team with a specific focus on security to reduce the attack surface against supply chain attacks.”

To accomplish that, organizations need to look to their security operations centers (SOCs) to take on a greater role in helping to secure software supply chains. Close collaboration between development and SOC teams is the surest way to provide end-to-end security for an organization’s software supply chains. And, supplied with the right tooling, SOCs and development teams working together will be better positioned to scan software packages; achieve greater visibility into development processes; detect deviations from a set baseline; and pursue possible supply chain compromises before they can cause real harm to software publishers and their customers.

As threats and attacks on software supply chains evolve, so must protections. A modern software supply chain security program should aim to protect the entire software development process, from the developer IDEs and their plugins to the open source components, CI/CD workflows and the release pipelines, while also being equipped to spot and pursue compromises at any point in the development life cycle. Only then will development teams be able to prevent harmful attacks like SUNBURST, IconBurst and countless others that have happened since.

March 15, 2023
ReversingLabs Adds Ability to Detect Secrets in Application Binaries

ReversingLabs today announced it added an ability to detect secrets exposed in application binaries to its Software Supply Chain Security (SSCS) platform.

Tomislav Peričin, chief software architect for ReversingLabs, said this addition will make it easier for DevSecOps teams to identify secrets that are inadvertently left in applications as plain text or that can be discovered because of weak cryptography, scripts that have been included in directories that have secrets configuration files, packaging automation mistakes, compromised developer accounts or the activities of malicious insiders.

The SSCS platform can identify more than 250 secret types out of the box, including private keys, version control, certs, tokens and more. Once they are identified, the platform also surfaces their precise location, which services are affected and if those secrets are exposed or leaked elsewhere. The platform also identifies third-party, open source, testing keys and other commonly shared secrets in addition to which secrets have already been exposed. Armed with those insights, it then becomes possible to better prioritize remediation efforts on actionable issues to reduce overall manual triage fatigue, noted Peričin.

ReversingLabs also provides guidance for sensitive information policies, including documentation of public exposures and breakdowns of secrets used for web service access credentials, web service access tokens, web service API keys and webhook service access keys.

Most software composition analysis (SCA) tools used to secure software supply chains are limited to identifying issues in source code. As applications become more modular, however, the odds authentication credentials such as login credentials, application programming interface (API) tokens and encryption keys might be left exposed because of a developer mistake substantially increase. The overall goal is to make it possible for DevSecOps teams to discover these issues in binary code before an application is deployed, noted Peričin. That “just in time” approach to secrets management includes canary token management and custom detection policies to better ensure application security.

In general, secrets within applications are becoming more challenging to secure as cybercriminals increasingly scan repositories and continuous integration/continuous delivery (CI/CD) platforms to find them. The ultimate goal is to inject malware into a software component that will be used in multiple downstream applications. Once activated, that malware will then create a backdoor through which cybercriminals can exfiltrate data or launch a full-blown ransomware attack. It was the latter type of attack that led to the U.S. government creating a formal National Cybersecurity Strategy to, in part, lock down the software supply chains employed by federal agencies to build applications.

A recent survey published by ReversingLabs, however, found that while 87% of respondents agreed that software tampering has become a more frequently used cybersecurity attack, only 37% said they have any means to detect it. Only 7% said they could detect software tampering at each phase of the software development life cycle.

It’s not clear to what degree best DevSecOps practices are currently being implemented within application development teams, but as more attacks are launched against software supply chains, it’s now only a matter of time before most DevOps teams are routinely required to scan code for exposed secrets and known vulnerabilities long before code is allowed anywhere near a production environment.

March 14, 2023
NETSCOUT Taps F5 to Optimize Custom App Performance

Through its alliance with F5, NETSCOUT Systems has extended the reach of its nGeniusOne enterprise performance management platform to support custom applications.

The nGeniusONE platform supports more than 1,000 voice, video and business data applications using NETSCOUT InfiniStreamNG appliances in combination with nGeniusONE to provide visibility into any infrastructure environment, including data centers, private and public cloud and co-location facilities. The integration with F5 extends those capabilities to add support for custom applications via integration with F5 BIG-IP server pool definitions that can now be directly imported into nGeniusOne.

Eileen Haggerty, area vice president for enterprise product and solutions marketing for NETSCOUT, said that approach enables nGeniusOne to access clear, unencrypted traffic from the BIG-IP Local Traffic Manager.

As more cloud-native applications built using microservices are deployed across distributed computing environments, network latency is becoming a bigger DevOps challenge. A wide range of networking issues can adversely impact application performance, but without any visibility into network traffic, it can be challenging to determine the root cause of an issue. NETSCOUT has deployed a global network of appliances to surface those issues.

A recent report published by Cloud Security Alliance (CSA) found the average enterprise uses 464 custom applications and expects to deploy 37 new applications in the next 12 months. At that rate, the number of organizations likely to encounter networking issues is only going to increase, noted Haggerty.

It’s not clear to what degree network operations (NetOps) teams are becoming integrated with DevOps teams that build and deploy custom applications. The need for both teams to collaborate more effectively is becoming more apparent as more application code is deployed at the network edge. In some cases, developers are now programmatically invoking network resources directly, while other organizations provide developers access to a portal to self-service their requirements within the context of a set of pre-defined guardrails. Regardless of approach, the need for greater visibility into the underlying networks that custom applications depend on has become critical.

Exactly how much that need will drive organizations to embrace one networking vendor over another remains to be seen. DevOps teams are starting to exercise more influence as organizations realize how much they depend on custom software to drive digital business processes. DevOps teams need to be able to understand whether an issue is the result of a networking problem or an inherent problem with the code written or the way an application has been deployed.

Of course, DevOps and NetOps teams don’t share a common nomenclature so the cultural divide between these teams can be wide. The simpler it becomes for NetOps teams to share visibility into networking services the easier it becomes to bridge that divide. Developers should be able to see for themselves how changes to application code might potentially improve network performance. In the absence of any proof to the contrary, most DevOps teams will simply assume there is a fundamental flaw in the networking services being provided.

March 13, 2023
Improving the DevOps Process for Mobile App Developers
Development teams—perhaps especially mobile development teams—have heavily invested in systems to automate their processes and accelerate the delivery of mobile apps. From build, test and release to tracking and monitoring, the mobile DevOps team depends on systems like Fastlane, Bitrise, Jenkins, Azure Pipelines and GitLab—and really, that list just scratches the surface.

The mobile app market moves so fast that automation is the only way mobile DevOps teams can keep up with their competitors and rapidly changing customer expectations. They have to iterate and release rapidly—the more frequently a publisher puts out new features, the more highly customers rate their app. In the GitLab 2022 Global DevSecOps Survey, 70% of respondents said their teams release code every day or every few days.

Unfortunately, there’s one major component of mobile app development that sits outside of these automated processes in most mobile DevOps teams: Security. For the most part, developers still implement security manually, and the process for ensuring apps are secure mostly comes down to code scanning and penetration tests. Again, from the GitLab survey, 53% of developers are running static application security testing (SAST), but unfortunately, the data from those scans is often not feeding back into developer workflows. Less than three out of 10 teams (29%) pull scan results into a report for developers.

Additionally, there’s still a large disconnect between the security and development teams. Nearly half (47%) of security pros said that developers miss more than three-quarters of the bugs in the code, leaving them for the security teams to find, and more than half (56%) said it was hard to get developers to prioritize fixing code vulnerabilities. In fact, prioritizing vulnerability remediation was security professionals’ biggest challenge. Additionally, it’s interesting that when it comes to shift left security, the emphasis appears to be on early code scanning, not actually building security into apps earlier in the process.

Data-Driven Decisions and Security Automation

First, DevOps teams need to integrate data about the security of their mobile apps early in the process so they can make informed decisions about what protections to incorporate into the next build. Scanning information and results from penetration tests clearly need to feed back to the development team as quickly as possible—there’s no point in doing these tests if the information remains unavailable and not acted on.

But mobile devices have the capability to collect and send much richer data about the security threats apps are actually facing in the field. By collecting this data, DevOps teams can make data-driven decisions about which threats are the highest priority to combat.

However, as noted earlier, it does little good to collect data if it’s never used. And the slow pace of manual security implementation does not allow DevOps teams to implement protections quickly or nimbly enough to keep pace with the rapidly changing threat environment. DevOps teams need to automate the build, testing, release, tracking and monitoring of security to the same degree as every other aspect of an app’s development. Specifically, they need:
- A system that can store, provide version control and audit security in every release
- An automated system that can build the desired protections into the app within the organization’s existing CI/CD processes
- Automated verification that the protections slated for implementation are actually included in the release
- A feedback system from data collected in the field, including data about the effectiveness of protections already implemented. This proves security measures are working and reinforces the value of the DevSecOps process.
With this combination of data and automation, mobile app developers can transition from DevOps attempting to shift left security towards a DevSecOps process that is completely data-driven. Instead of reacting to the latest threat that’s making headlines in the news or working off gut feelings about the direction the threat environment is moving, DevOps teams can look at trending threat data from their very own apps, slicing it according to growth, geography, device, OS version and a multitude of other filters. As a result, the organization can pinpoint exactly which threats are emerging as the next big thing so they can defend against them early.

And with the assistance of automation, the DevOps team can keep up with the trending data, building security protections into the app within days or even hours of making a decision on what to include.

To improve the DevOps process, security has to shift left, but moving code scanning and penetration testing earlier in the process won’t do much good to harden protection without real-time data from the field and automation for security implementation. What DevOps needs is a transition to data-driven DevSecOps.
March 13, 2023