devopsy.kinsta.cloud

Tag: code

A Deep Dive Into Flaky Tests

In software, tests are imperative for quality assurance as they ensure that your application is working as expected, irrespective of modifications in the source code to comply with changing requirements. However, you may have come across a specific category of flaky tests; these tests are unstable and unpredictable because they may pass or even be in identical conditions, i.e., have the same configuration without code, data or environmental changes. This article discusses flaky tests, why they occur and some best practices to avoid them.

An Overview of Flaky Tests

A flaky test is defined as a test that can pass or fail regardless of any changes to code, data or environment. This inconsistency in test results can stifle development by breeding disbelief, and delay production rollouts due to the need for unnecessary debugging. This phenomenon puzzles developers and undermines their trust in testing; hence, a systematic approach is needed to deal with this problem.

Flaky tests are inconsistent since a test can fail on one occasion and then succeed on another while using a static test environment and codebase. There are many reasons behind this inconsistency including timing issues such as race conditions and reliance on external systems which might behave unpredictably, among others.

Impact on Software Development

Flaky tests can greatly affect the workflow in SDLC leading to a waste of time and resources when teams chase after errors that are nondeterministic, unreliable and hard to find inherently. The following are the key areas where flaky tests can impact in a typical SDLC process.
• Delayed software delivery
• Reduced trust in test results
• Increased maintenance costs
• Decreased test coverage
• Increased debugging effort
• Impediments to CI/CD

Common Causes of Flaky Tests

To determine a proper strategy for minimizing the impact, we must first understand why tests become flaky.

• Concurrency Issues: When tests are executed concurrently, i.e., multiple tests contend for shared resources at the same time, deadlocks, race conditions may occur and these in turn might lead to unstable outcomes.
• External Dependencies: When your tests need interaction with external systems such as APIs or databases, this introduces flakiness because of network latency as well as different response times.
• Non-Deterministic Logic: When your tests rely on random elements such as dates, times or user input, i.e., anything that cannot be predicted with certainty, they can have different outcomes.
• Unstable Environment: When the environment where the tests are conducted lacks proper isolation, there might be differences in application performance, configurations and availability rates, which might eventually cause a test to work once but fail another time under identical conditions.

How to Detect Flaky Tests

There are several strategies you can adopt to detect flaky tests, such as using different methods and tools to find inconsistency and unreliability patterns within the outputs generated by them.

Analyzing Historical Test Results
Examining historical test results is a reliable approach to uncovering unreliable tests. You can do this to identify irregularities or patterns within your data sets while comprehending the common trends or occurrences that may be useful when estimating the likelihood of a test failure.

Using CI Tools With Built-In Flaky Test Detection Capabilities
You can also leverage CI tools that have built-in flaky test detection capabilities to identify tests that have inconsistent behavior. By using such tools, you can detect flaky tests at an early stage in the SDLC process thereby saving you time and effort in identifying flaky tests manually.

Execute Tests Multiple Times
You can execute your tests multiple times using testing frameworks or as part of a CI/CD pipeline to determine variability in the test outcomes. If nothing was changed in the code or the environment, yet the test results indicated anomalies and inconsistencies, then that is a good indicator of flakiness.

Monitoring and Documenting
Another strategy is to keep an eye on test execution history to determine any anomalies in the test results over some time. By recording metrics such as, failure rates, you can understand how consistent, stable and trustworthy the tests are.

How to Fix Flaky Tests

Here are the key strategies you could adopt to fix flaky tests.

Isolate the Test
Firstly, you should identify and quarantine the flaky test. Then, replicate it to determine the root cause, i.e., why a test becomes unstable. Remember to reset or clean the state before and after every test.

Refrain From Using Random Data
Since random data can cause unpredictable behavior, you should ensure your test doesn’t depend on random or unexpected data. It would help if you used fixed or predefined values for user input and deterministic algorithms or methods to generate or process data. If you need random data, ensure you log the data appropriately to enable a rerun of the tests.

Make Your Tests Robust
Your tests should be equipped with retries, timeouts or waits for handling network or performance issues. They should also be adept at being executed under various environments and conditions. Create assertions meant to verify an expected result or behavior by checking ranges, patterns and other pertinent parts correctly designed for that purpose.

Eliminate External Dependencies
You can use mocks, stubs or fakes to reproduce external system behavior without using real dependencies. You must ensure that your tests don’t rely on external factors or affect other tests.

How to Maintain a Test Suite Free of Flaky Tests
To build a test suite devoid of flaky tests, you need to be proactive and follow some recommended practices.

Remove Duplicate Tests
While it is essential to review tests, it is equally important to get rid of repetitive tests, as they slow the testing process. Regularly maintaining the tests ensures accuracy and reliability, thus making the test results more precise and actionable.

Continuous Monitoring
Establish a monitoring system that continuously monitors test execution and output to help identify discrepancies quickly and eradicate them from the test process. You should also provide your team with the necessary tools (monitoring tools, reporting systems and necessary hardware) to run the tests.

Prioritize the Flaky Tests
Prioritize them based on the level of business risk, the amount of effort needed to fix them and test timing. There is a lot more value in focusing on tests that validate critical business workflows. Tests seldom used in a business workflow should be low priority and ignored. Additionally, specific flaky tests need much time and effort to fix, which should be removed during test maintenance.

Analyzing Test Metrics
You should maintain records of common issues to help your peers learn from past problems and solutions. To get insights, identify trends and patterns, and make the right decisions. It would help if you collected relevant metrics such as test execution times, failure rates and flakiness patterns. It would also help if you imparted training on writing reliable and robust tests, what flaky tests are and how to deal with them.

Conclusion

While eradicating flakiness is challenging, you can always trace its origin and handle it pronto. Managing flaky tests is an ongoing iterative approach where you identify, understand and fix intermittent or unpredictable tests. You can mitigate the challenges of flaky tests by adopting the strategies and best practices outlined in this article.

July 3, 2024
Sapient.ai Automation Platform Leverages AI to Create Unit Tests

Sapient.ai today launched a testing automation platform that makes use of artificial intelligence (AI) and automatically creates unit tests as the applications are developed.

Company CEO Rishi Singh said Sapient Codeless makes use of large language models (LLMs) that have been specifically trained to create unit tests and then automatically run them as needed. It analyzes code and then automatically generates declarative tests based on the inputs and outputs of the methods used to build the application.

Additionally, Sapient Codeless automatically adapts to changes in the codebase, ensuring tests remain up to date in a way that eliminates the need for ongoing test maintenance.

The overall goal is to make it simpler to keep pace with the amount of code that needs to be tested as more organizations make use of AI assistants to write code, noted Singh.

That approach provides the added benefit of significantly reducing the amount of testing code that DevOps teams would otherwise need to manage alongside all the other code they already manage, he added. In some organizations, there is as much as five times more boilerplate testing code that DevOps teams need to manage than there is actual code that ultimately finds its way into an application, said Singh.

As the pace at which applications are developed continues to accelerate, it’s clear existing manual approaches to creating unit tests will not scale. It’s expected many organizations thanks to the rise of AI will build and deploy more applications in the next two years than they might have in the past decade.

However, if most of those applications are not tested more thoroughly before they are deployed most DevOps teams will find themselves trying to debug and troubleshoot more applications than ever long after they have already been deployed in a production environment, noted Singh.

AI-Infused Automation Platforms Make Things Easier

It’s not clear to what degree organizations may need to run more tests versus focusing more on running the right test at the right time. Much time is wasted running tests in a sequence that doesn’t surface a failure until late in the process. In theory, automation platforms infused with AI should make it easier to identify code that is likely to fail a specific test sooner. In effect, DevOps teams should be able to more easily run the right test at the right time.

In the meantime, the debate over who should run what type of test at any given time continues unabated. Many developers are happy to run tests that surface issues as they write code but are not nearly as engaged when tests are run, long after they have moved on to another project. In many cases, they would rather see a DevOps team manage a significant portion of the testing process on their behalf.

Regardless of approach, the one thing that is clear is the overall quality of the applications being deployed needs to improve. The issue now is finding a way to achieve that goal without increasing the overall level of toil already being experienced.

June 25, 2024
Harness Survey Surfaces Raft of DevOps Challenges

A survey of 500 software engineering leaders (38%) and practitioners finds nearly half reporting they can’t release code to production without risking failures, with 39% noting their code fails to push to production at least half the time.

Conducted by Wakefield Research on behalf of Harness, a provider of a continuous integration/continuous delivery (CI/CD) platform, the survey also finds that when code does need to be rolled back more than two-thirds resort to manual processes.

Overall, 60% of respondents also noted they are still releasing code on a monthly or quarterly basis. A full 59% of developers said application security requirements limit their ability to release code frequently.

Nick Durkin, Field CTO for Harness, said one of the primary reasons that security remains a bottleneck is mainly because too much responsibility for it has been shifted left toward developers, versus relying on DevSecOps platforms infused with artificial intelligence (AI) to automatically identify and resolve issues in a way that empowers everyone involved in software development.

In general, AI should be used to eliminate toil such as creating and running tests as part of a larger effort to eliminate the need to babysit tools and platforms, added Durkin.

For example, 42% of the practitioners surveyed said deploying code isn’t fast or efficient, with 44% noting that testing code end-to-end isn’t efficient. More than two-thirds (67%) said they need to wait a week to completely test code and, even then, 32% said there isn’t high unit test coverage.

More than half (52%) of the developers surveyed attributed burnout as a primary reason their peers left their jobs, with just under a quarter (23%) working overtime for at least 10 days a month. A full 97% said they regularly switch contexts between on average 14 different tools.

Well over half (54%) reported it takes longer than a week to learn a new tool. A full 59% said it takes a week to build internal tooling. More than a quarter (28%) said it takes a day to build an artifact.

Nearly two-thirds (62%) of developers have also experienced scope creep as requirements expand, which makes them less confident to execute.

Report Finds Onboarding New Hires Takes Too Much Time

New hires don’t seem to provide any immediate relief either, because onboarding them takes an average of 100 days, according to 71% of respondents.

Additionally, 40% of developers said their organization doesn’t enforce good security and governance policies. A total of 41% don’t have automated security and governance policies, while 42% said they don’t have robust identity and access management policies.

A full 62% of the executives surveyed said they would prefer to address these issues by adopting an integrated DevOps platform but the rate at which that transition is occurring is unclear. Many organizations have invested heavily in legacy DevOps platforms that they are often reluctant to rip and replace. In many cases, organizations are transitioning to a modern DevOps platform for new application development projects while continuing to employ legacy platforms to update existing applications.

Very few would argue that software development workflows are optimal. The challenge is determining how best to move forward from where most organizations are today, in a way that minimizes as much disruption as possible at a time when organizations have never been more dependent on software.

May 17, 2024
Skillsoft Taps ChatGPT to Improve Developer Training

Skillsoft today revealed it plans to add generative artificial intelligence (AI) capabilities to its Codecademy online training platform to provide developers with hands-on experience using ChatGPT to write code.

Kunal Ahuja, general manager for Codecademy consumer business at Skillsoft, said the goal is to give developers a more curated path toward teaching them how to define prompts that will result in better code being generated. These enhancements will be made available to Codecademy Plus and Pro users as well as enterprise customers in the coming months, he noted.

There’s no doubt that developers are relying more on generative AI to write code, but not all the output generated by platforms that are based on a general-purpose large language model (LLM) is going to be consistent. Codecademy provides a more immersive experience that enables developers to refine their prompt engineering expertise, said Ahuja.

AI-enabled capabilities include code explanations and project hints to improve debugging skills, along with other real-time feedback surfaced via a virtual coding assistant. There are also courses such as “Prompt Engineering for Software Engineers” available.

Finally, Skillsoft is also providing access to a mock interview simulator that developers can access to help determine if they are ready for a job interview.

As it becomes more apparent that prompt engineering is a skill application development teams will need to master to build and deploy software, there’s a clear need to teach nearly everyone in IT how to craft prompts efficiently. Ultimately, the number of prompts used to generate code will have an impact on the total cost of building software.

In the longer term, Skillsoft plans to build its own LLM using data it collects via its platform to provide training using an LLM that has been trained using code that has been curated, said Ahuja. One of the issues with a general-purpose LLM is that it is trained using examples of code pulled from across the web that varies widely in quality. That can result in outputs of code that have known vulnerabilities or that simply run inefficiently. Of course, it may be just as likely that a developer augmented by generative AI is, depending on their level of expertise, actually delivering higher-quality code.

Despite those concerns, however, it’s still important to provide developers with the training they need today to remain competitive, said Ahuja.

It’s not clear how much code these days is being generated by machines versus humans, but as the volume increases, the number and size of the software builds that need to be managed is only going to increase. Hopefully, as much AI will be applied to the DevOps workflows used to manage those software builds as there is being applied to writing code, but in the short term, at least, DevOps teams might soon find themselves overwhelmed.

The one thing that is certain is DevOps teams have a vested interest in making sure the code being written by developers is of the highest quality possible. After all, whenever there is an issue with that code, it usually falls to the DevOps team to fix it.

Image source: https://vecteezy.com/programming-background-with-person-working-with-codes-computer-e1679047506780.jpg

February 20, 2024
Improve Efficiency With Smaller Code Reviews
In software development, code review plays a significant role in helping detect issues in your application’s source code early and promote collaboration among your team members. Breaking down code reviews into more manageable portions can yield great benefits. Review cycles are shorter and more convenient to comprehend when they are smaller. In other words, smaller reviews are easier to comprehend and provide feedback cycles. As a result, we can reduce development, deployment and debugging times while reducing the scope of change in each iteration.

This article presents a discussion on code reviews, why they are important, their benefits and why smaller code review cycles are more effective than larger ones.

What is Code Review?

Code review is defined as a software development process which involves reviewing code written by other developers. As part of the development process, it aims to improve code quality, ensure compliance with standards and identify and fix problems early on. Code reviews play a role in software development as they help catch errors before they turn into significant issues.

Peer reviews allow developers to detect errors or defects in each other’s code. By reviewing each other’s code, developers are able to identify bugs and flaws in the code. It helps deliver code that is secure and sustainable, adheres to standards and detects and resolves problems early in the development life cycle.

This can either be a formal process, where code is systematically evaluated according to a checklist or set of guidelines, or an informal process, where code is reviewed ad hoc. Code review can be performed by the same developers working on the code or by other developers with expertise in the relevant domain or technology.

By engaging in the exchange of feedback and actively participating in code reviews, we can ensure that the code is well-organized, easy to comprehend, adapt and extend over time. To be effective, it’s essential that everyone on the team is on the same page when it comes to their expectations and goals regarding these reviews.

Here are the key benefits of code review at a glance:
- Identify bugs and issues
- Improve code quality, consistency and maintainability
- Enforce coding standards
- Foster team collaboration
- Promote knowledge sharing
Best Practices of Code Review

Here are some of the best practices you should follow to improve your code review process:
- Review small units of code
- Set clear objectives and guidelines
- Involve multiple reviewers
- Provide constructive and respectful feedback
- Use code review tools
- Prioritize high-risk areas
- Balance attention to detail with time efficiency
- Encourage collaboration and learning
- Set realistic deadlines
- Follow up and close the loop
Automating the Code Review Process

Automated code reviews can help you enhance collaboration amongst your team and reduce the time spent in each cycle. Your developers can improve code quality and maximize productivity by automating the development workflow.

Here are the key benefits of automating the code review process:
- Enhances code quality
- Improves productivity
- Enhances code quality
- Improves collaboration
- Fosters learning opportunities
The Problem With Large Code Reviews

The traditional approach to code review often involves reviewing large chunks of code in a single session. Although this approach is initially efficient, it can lead to multiple problems. You will need more effort and spend more time reviewing large chunks of code. If code reviews are extensive and take longer to complete, it can often result in decreased attention to detail. Due to this, the likelihood of missing crucial errors and reviewer fatigue will increase, resulting in delays in feedback and completing the code review process.

The Benefits of Smaller Code Reviews

When the code review cycle is small, it becomes easier to track deployment changes easily. Smaller code reviews continuously improve development, deployment and debugging times and reduce the radius of a change set. As a result, it promotes faster deployment and helps identify issues easily and early in the software development life cycle.

Once a code review cycle is complete, the code goes through many steps before it finally reaches the production phase. Monitoring the application’s performance and detecting bugs (if any) is much easier when you make only minimal changes to your code. When the changes are small, it is easier to track the application’s behavior and detect bugs.

Here are some benefits of adopting smaller code reviews:

Focused Review

Reviewing smaller chunks of code enables you to analyze the code in a more focused and in-depth manner. This helps reviewers focus on specific functionality or components, leading to more accurate feedback.

Reduced Cognitive Load

Limiting the amount of code to review reduces cognitive overload for both developers and reviewers. Smaller code review cycles can be more efficient because they break down the review process into more manageable chunks. As a result, you’ll be able to evaluate code quality in a much more effective and efficient manner and identify potential issues or improvements more easily.

Faster Feedback Loop

Smaller code reviews can significantly speed up the feedback loop in the development process. Hence, feedback cycles will be faster, i.e., you will be able to receive feedback faster, resulting in quicker iterations and faster software delivery. The smaller the review, the easier it is to digest and the faster the feedback cycle.

Enhanced Collaboration

Smaller code reviews allow for more focused discussions and collaboration between reviewers and developers. Most importantly, since you can handle concerns effectively and efficiently, you can engage in more productive discussions. The efficiency and effectiveness of code reviews improves when the expectations of all stakeholders are aligned.

Measuring Success

To measure the success of smaller code reviews, consider the metrics and approaches given below:

Defect Detection Rate

This is a measure of the number of defects or issues identified during a code review cycle relative to the number of defects found later in testing or in the production environment. Note that a higher defect detection rate indicates you’re able to detect defects effectively.

Review Efficiency

Calculate the time it takes to review a chunk of code to understand the pace and efficiency of the code review process.

Feedback Quality

Assess the quality of the feedback provided during reviews, such as checking if the feedback is constructive, clear and actionable. Note that if the feedback is of high quality, it would typically yield better code improvements and learning opportunities for your team.

Cycle Time

The duration of the code review process is an indicator of efficiency. A shorter cycle time suggests an efficient review process, but quality should not be compromised.

Automated Checks Integration

Evaluate how effectively automated tools are integrated into the code review process to detect common issues and reduce the manual effort required in reviews.

Knowledge Sharing

Assess how well the code review process facilitates knowledge sharing and cross-team learning. This can be observed through the diversity of reviewers and the nature of the discussions in the review process.

Bug Rates Post-Release

Monitor the incidence of bugs or issues in the code post-release. A decrease in this rate can indicate that the code review process is effectively catching and addressing potential problems before release.

Conclusion

Code review helps produce code that is highly performant, scalable, secure and compliant with the recommended standards. Code reviews can make the codebase easily understood, flexible and scalable. A collaborative approach to code review can yield several benefits, such as promoting a culture of continuous development and learning, maintaining an organized codebase with clean code and clear structure, helping detect and resolve issues quickly, etc.
February 13, 2024
Swimm Adds Generative AI Chat Tool for Documentation

Swimm today announced it has added a chat tool that enables developers to use natural language to surface insights into code.

Company CEO Oren Toledano said the /ask Swimm tool makes it simpler to launch queries that enable developers to better understand how code was constructed.

The /ask Swimm tool aggregates documentation along with other related data to surface factors that are not evident in the code itself, such as documentation of business decisions, product design considerations and decisions concerning why specific architectural choices were made. It automatically captures and updates code-related knowledge to provide a continuous feedback loop as code, documentation, files and repositories are created and updated to provide deeper insights into code that goes beyond what might have been actually documented.

The overall goal is to provide a mechanism that enables developers to understand how and why code was constructed within the context of an integrated development environment (IDE), said Toledano.

Swimm has previously made available a platform that uses generative AI platform to create a static analysis of documentation. That capability should make it simpler for organizations to track documentation at a time when generative AI platforms such as ChatGPT are exponentially increasing the amount of code being written. In theory, those platforms should be able to also create documentation for that code, but there will still be a need for tools to track and analyze it.

In the longer term, Swimm plans to continue to extend its usage of generative AI to provide deeper levels of insights across entire applications and software ecosystems to address that challenge, noted Toledano.

In the short term, however, it’s clear that AI is making developers more productive than ever. In fact, the volume of code being generated might soon overwhelm existing DevOps pipelines and workflows. Most DevOps teams will need to revamp those workflows as the pace at which more applications than ever are being developed and deployed faster than ever.

It’s still early days as far as generative AI adoption is concerned, but it’s clear many developers are already using it to write code. How much of that code makes it into a production environment is difficult to determine, but a lot of that code is going to be of varying quality. A general-purpose AI platform such as ChatGPT was trained using code collected from all across the web. The code generated by these platforms is only as good as the examples used to train it, all of which were created by human developers who may have made mistakes, such as including code that has known vulnerabilities.

Regardless of whether code was generated by a machine or a human, the probability that documentation will need to be analyzed as part of a code review process is always high. The challenge and the opportunity is to determine the best way to apply AI to a longstanding challenge that is now being exacerbated by the existence of AI tools that, with each passing day, are only becoming more accessible to developers.

January 17, 2024
Sonar Adds Secrets Detection to Code Analysis Portfolio

Sonar has added a secrets detection capability to its portfolio of tools for analyzing code and DevOps workflows.

Designed to be applied to both finding secrets in code repositories and as applications are developed using an integrated development environment (IDE), this capability is being added to SonarLint, SonarQube and the managed SonarCloud service to identify, for example, passwords, application programming interface (API) keys, encryption keys, tokens, database credentials and other private information.

Sonar co-CEO Olivier Gaudin said the goal is to eliminate secrets that are accidentally or maliciously stored in source code to improve software supply chain security.

While it’s important to discover secrets in codebases, Sonar is making a concerted effort to make it simpler for developers to discover secrets before they find their way into a codebase that is one step away from a production environment, noted Gaudin. The SonarLint static analysis tool the company provides can now discover secrets in microseconds using the syntactic and semantic analysis engines the company has now extended, he added. No developer is going to wait five to 10 minutes for a code scan to be completed, said Gaudin.

A Learn as You Code tool also explains how each secret found in a code segment poses a security risk. The overall goal is not only to remove secrets but, ultimately, reduce the number of instances where secrets are discovered in code, said Gaudin.

Most organizations today are at varying stages of embracing DevSecOps best practices to improve software supply chain security. It’s not clear how readily secrets management is being embraced as part of those modernization efforts, but as cybercriminals routinely scan for application secrets stored in clear text that can be used to compromise application security, the need to encrypt secrets becomes a higher priority.

There is, of course, no shortage of tools for detecting secrets. Sonar is now simply making a case for managing secret detection as part of a larger effort to reduce vulnerabilities in code that cybercriminals can exploit. That approach should help reduce the total cost of securing a software supply chain at a time when forthcoming regulations are going to make organizations that deploy software more accountable for application security.

In the meantime, thanks to the rise of generative artificial intelligence (AI), the volume of code that needs to be scanned will likely exponentially increase. The issue with general-purpose AI platforms such as ChatGPT is they were trained using code of varying quality that often contains vulnerabilities that are now sometimes incorporated into the code they generate. On the plus side, however, many developers have little to no application security expertise, so it’s just as feasible that ChatGPT will create more secure code than a human developer. Regardless of how the code was created, however, no one will know for sure how secure it is without scanning it first.

Hopefully, as AI progresses, the quality of the code being generated will only continue to improve. Until then, DevOps teams should continue to assume all code is suspect until proven otherwise.

December 18, 2023
Sourcegraph Launches Cody Generative AI Tool to Write Code

Sourcegraph today made generally available an open source Cody tool that leverages generative artificial intelligence (AI) tools to write and fix code.

Version 1.0 of Cody invokes StarCoder, an open source large language model (LLM) for creating code made available by Hugging Face and a chat tool that makes use of dense-sparse vector retrieval system for code and documentation to invoke GPT-4 Turbo, Claude 2, GPT-3.5 Turbo, Claude Instant and Mixtral-8x7B LLMs.

Designed to be deployed as an extension of VS Code, Cody is available in a free edition as well as a professional edition, dubbed Cody Pro, that Sourcegraph supports. Cody Pro will be free until February 2024, after which it will cost $9 per month. There is also a version for the JetBrains editor available in beta, along with an experimental Neovim offering.

Cody is also integrated with the code search tools provided by Sourcegraph to provide development teams with additional context that, depending on the use case, delivers a completion acceptance rate of 30% or higher for code.

In addition to enabling developers to write code faster, Coy also simultaneously invokes the graph technology Sourcegraph created to provide additional context to a prompt that makes it simpler to surface relevant code by parsing code and producing a graph that captures the semantic understanding of code. Cody, for example, can search the codebase for files and documents that can be used to suggest a rough plan of attack for implementing a new feature or resolving issues that surfaced in a bug report.

Cody can also create, save and share custom commands to make it simpler to reuse the prompts created to write code.
Future versions of Cody will be more tightly integrated with the Sourcegraph universal code graph to provide additional context. In addition, Cody will soon leverage deeper graph context to make autocomplete suggestions based on symbols defined elsewhere.

Sourcegraph CEO Quinn Slack said the combination of graph technologies and LLMs reduces latency while helping to ensure the code generated by the LLM doesn’t produce hallucinations such as type errors and imaginary function names.

In addition, Sourcegraph is committed to applying AI and graph technologies beyond the codebase itself to provide deeper insights into the environment code will eventually be deployed in, added Slack.

It’s not clear how much developers are relying on generative AI tools such as Cody and GitHub Copilot to write code faster, but it’s clear the pace at which software can be written is accelerating. That increased pace of application development will eventually require DevOps to revisit workflows that were not designed to manage codebases that, in the months ahead, will significantly increase in size.

Each developer will most likely have a personal preference for one generative AI tool versus another, depending on the quality of the code generated. The issue is that no matter how good any of those tools are, however, the code generated will still need to be reviewed by a human before it can be deployed in any production environment.

December 14, 2023
Hitting the Accelerator for Change: How Open Source Drives Software-Defined Cars

Every new car on the road today contains hundreds of millions of lines of code, and that’s a good thing because the technological functions they enable add greatly to the car’s ultimate value. For over 100 years, automotive consumers coveted cars because of their paintwork, their leather interiors and the speeds they can travel. But now, their real and perceived value is influenced by technology features. Technologies such as electric vehicles, self-driving and other software-defined services are what drive the value of the car—and it’s no longer too nerdy to describe cars as ‘software-defined.’

Navigating the Road to the Future

For car manufacturers and their suppliers, the challenges of the software-defined car are already very much in focus. Original equipment manufacturers (OEMs) have been hiring new workers and re-training existing ones to navigate these challenges, such as cybersecurity and connected technologies. Ten years ago, OEMs such as Renault and Volkswagen had few in-house software engineers. Fast forward to today, and both companies have numerous resources focusing on software.

Software will soon become ever more central to the vehicles we drive, and that will soon drive us, as vehicles become more connected, autonomous, shared and electric (CASE). In the next few years, autonomous vehicles will soon be handling tens of terabytes of data per day, gathered by sensors such as LIDAR and cameras. Customers are increasingly expecting smartphone-esque features, which involves keeping pace with the very latest apps and technologies.

Combined with features such as passenger entertainment systems and remote diagnostics, the requirement for software and processing power in vehicles is going to grow at an ever-increasing pace. Open-source software can provide automakers with the power to keep up.

You Must Shift Gears

Inevitably, these processing power requirements will require a seismic shift in the way OEMs work. Currently, OEMs are struggling to maintain existing systems‒while at the same time attempting to develop new platforms. It’s a hangover from years in which vehicles were not connected or had extremely basic connected services. This has led to a situation where it’s difficult to upgrade the software at pace due to the requirements and constraints of the hardware.

There are lessons for car-makers in the smartphone world. Think about how a software update can be applied across multiple different smartphones. An iOS update works just as well across a new device or a device from four years ago. One iPhone will see multiple updates, adding new features and boosting security.

For car manufacturers, this won’t be an easy shift: The industry needs more defined standards to allow an easier interface between hardware and software. The industry also needs to reconsider how it develops hardware and how this relates to software. These challenges will require a new way of thinking for car manufacturers as they transition to a world where time cycles are critical. I believe that cross-industry collaboration is needed to accelerate the development of autonomous vehicles.

Keep an Eye on Moving Parts

Complexity is building at an increasingly rapid pace in the car industry, and this shows no sign of slowing down. In a report on the industry, McKinsey noted that the average complexity of individual software projects in the automotive sector has grown by 300% over the last decade. By 2025, the number of lines of code in the average luxury car will have doubled from today’s figure of 100 million lines of code, according to a prediction by Volkswagen‒which is already more than a military aircraft like the F-35. In full-self-driving cars, the figure could potentially reach a billion lines of code, analysts believe.

It’s not just the software that is becoming more complex, the hardware is too. Modern vehicles already have up to 150 Electronic Control Units (ECUs), often scattered through the vehicle, close to sensors. Increasingly, manufacturers are moving towards a model where these ECUs are combined and consolidated, connected by an ethernet network. As the amount of data vehicles deal with increases, it’s unlikely that all of this will be directly uploaded to cloud servers; instead, much of it will be processed inside the vehicle before ‘edited highlights’ are uploaded to the cloud for processing.

This has important implications for vehicle maintenance and also poses cost issues. Car manufacturers face challenges when it comes to managing the ever-growing complexity of their systems while trying to keep annual recurring costs down.

Watch the Hacker Threat

The need for cybersecure vehicles is also growing, particularly as we see self-driving vehicles enter the fray. In 2015, two security experts showed off a hack where they were able to ‘switch off’ a Jeep driving at 70mph down an American highway. It marked the dawn of the ‘hackable’ car—although the seeds of this were sown decades earlier when the first ‘connected’ components were unveiled in the 80s and were not built to be resilient to hackers. A 2021 report on global automotive security was damning, saying, “There have been 110 Common Vulnerabilities and Exposures (CVEs) that year related to the automotive industry, 33 in 2020 alone compared to 24 in 2019.”

Ultimately, the automotive industry faces increasing regulation in the face of these threats. As governments begin to regulate software within the automotive industry, those designing the products and software will become more accountable.

Open Source and Safety … Always

For car manufacturers, there are also important issues related to functional safety—unsurprising when you are dealing with a device that weighs a ton or more. This directly impacts software development, with additional costs for any system that has an impact on driver or passenger safety.

Car manufacturers need to learn from the aviation industry, where systems related to safety are rigidly separated from those that are not, reducing costs due to compliance.

It’s often difficult to reuse existing software in cars today, as it can require porting to a different configuration. Costs can rise, not in a linear fashion, but exponentially. How much money will it cost to write, implement and test 100 million lines of code ‒ or a billion? This is the challenge manufacturers are facing. When it comes to managing these costs, open source will become ever more important and not only will allow the reuse of existing software, but will enable other manufacturers to contribute to said software.

Software-Defined Cars

The future is rushing towards us at speed—and in it, cars are increasingly software-defined. In this future, open source offers OEMs a highly attractive solution to the considerable hurdles ahead, from safety issues to cybersecurity. Open source will unleash the power of the ambitions of the car industry—not just to compete with the tireless innovation of other industries, such as mobile, but to exceed it. For consumers, cars are already the most complex devices they own: and they could be the most innovative. The flexibility and scalability of open source will allow car-makers to free their imagination and create the innovative, software-driven car of the future.

December 8, 2023
Improvement Cycles and AI

It’s that time of year when pundits start to crank out their predictions for the next twelve months! I’ve been asked to contribute to a private group (and occasionally, Techstrong Group asks me to contribute to their annual list), so I won’t thrill you with my predictions just yet; but I will discuss the cycles and trends that previous predictions have shown.

Back when I was testing physical servers, we thought of throughput bottlenecks in terms of where the problem was. The circle would go disk->memory-> CPU, and in the case of things like NAS, insert network in there somewhere, too. When one bottleneck was resolved, another would crop up. And users will always seem to be pushed to the next bottleneck.

We see similar cycles in DevOps. First, Agile made development faster, and that caused us issues throughout the rest of the process. Then, DevOps helped with the build and a little with the testing and deployment steps, but in the early days it was more standardization than speeding up. And since then, we have been in that cycle. We’re seeing GitOps speed up and automate deployment and upgrades—based upon the automation that came before—we’re seeing test and security tools feeding directly into the IDE to speed resolution, we’re seeing container automation cause “cloud-first development” (I hate that name; Kubernetes has cloud-like interfaces, but is not precisely cloud) to be fully automated, and thus faster.

And we’re looking at that cycle continuing. No doubt you will hear a lot of predictions that start with “AI …” For the most part, AI is a tool and/or technique. Listen for what it is improving because it is doing very little that’s actually new. (Delete list of things that sounded alarmingly like those predictions I said I wouldn’t make.)

All in all, we are still living in a very cool technological time. We can have a tool generate code snippets of the dull work for us so we can focus on the important bits. Some shy away from this use of “AI,” but honestly, we already use libraries to do the grunt work—I haven’t written to a screen buffer in forever—because we don’t have to. Same story—use the tools that make you productive. Some of the more obscure bits of Linux command lines are far better dealt with by asking an AI to give me examples than searching through tons of docs across several platforms. Never was much of a fan of man output for complex commands, this makes it easier. The nice bit is that bash and UNIX utilities don’t change much over time, so answers can be a bit dated and perfectly accurate. That’s not true for much of IT.

The environment has consistently gotten more complex, while the building blocks have gotten easier to use. This allows things to keep moving and changing while it offers us better ways to serve the organization’s needs. This does change at some point—infinite change is indeed a thing, but any technology does have its limits—but I honestly can’t see the point at which we are content with our IT tools and stop looking for more. Oh, most organizations go through those cycles—organizationally, once an infrastructure and process works correctly, it is likely to hang around until better is found—but the industry will keep changing.

And you’ll keep learning and changing. That’s one prediction I will make for next year. Teams and IT staff will grow and learn new things, because at its heart, that is what we do. We learn new things and keep the lights on. In good years, we offer sweet new tools and functionality to the company or our customers. But either way, we keep things online. Year after year. You all rock, so that’s an easy prediction.

November 1, 2023