devopsy.kinsta.cloud

Category: Continuous Testing

Continuous testing is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate.

Site Reliability Engineering (SRE) Comes of Age in 2022

The site reliability engineer (SRE) role is still gathering steam across organizations. In January 2022, LinkedIn listed SRE as the 21st job with the highest global demand throughout the past five years. That’s pretty high for such a specific tech role. And, looking to the future, it appears the SRE practice will only continue to gain adoption as a method to support high availability, reliability and improved digital customer experiences. The SRE approach is also essential to meet service-level agreements (SLAs) and internal service-level objectives (SLOs).

I recently met with Kurt Anderson, SRE architect, and Emily Arnott, content writer from Blameless to get their take on the top trends and predictions for the SRE practice in 2022. According to the Blameless team, 2022 will likely see increased adoption of the SRE role throughout companies and across internal departments.

Just as manufacturing sets high standards for its quality assurance, digital applications now require the same high reliability to meet inflated user expectations. To address these new realities, SREs have their work cut out for them—they must confront how to unify disparate DevOps tools and must drill farther down to unearth specific user experience qualms—all without adding to alert fatigue.

Increasing SRE Adoption

As mentioned above, the SRE role has seen tremendous growth in recent years and doesn’t show signs of slowing. SRE as a practice was first popularized by Google in 2015. Today, they define it simply as “what happens when you ask a software engineer to solve an operational problem.” This equates to maintaining high levels of reliability for digital services by creating errors budgets, designing SLAs/SLOs and optimizing process automation over time.

To Arnott, we’re still seeing SRE adoption take off at an exponential rate as more and more companies embrace Google’s SRE doctrine. “During the pandemic, more and more companies were forced to adopt a digital strategy. Reliability has to be fundamental to that strategy.” Arnott foresees SRE becoming a fruitful role at startups, which may initially have been intimidated by the prospect.

“The practice of SRE is a scale issue,” Anderson explained. It can be difficult for time-pressed companies to enact an SRE program when their attention is already so divided. However, as new requirements force warp-speed innovation, we’re seeing greater DevOps adoption. And the SRE role, consequentially, is becoming more of an imperative.

More Unification of Multi-Tool Condition

Most DevOps professionals would agree—we live in a multi-tool world. For example, multi-cluster management concerns are bringing a plethora of new tools to the fold. And computing environments are just as diverse, often combining multiple clouds and composed of hybrid estates that house various clusters of microservices. Even within the same organizations, different teams may opt for differing DevOps tooling. Anderson believes that SREs can bring “more unification to a fractured DevOps adoption across a single company.”

As companies adopt more software, there’s inevitably some degree of tool sprawl, Anderson said. For example, there may be competing methods to monitor the use of digital services. The platform team may be using Datadog, an app team may be using New Relic and another team may be using Prometheus. These tools don’t speak the same language and generate different logs, creating silos.

For Arnott, the future involves accepting the innate complexity of the microservices era and using an agnostic layer that can receive data from anywhere to identify meaningful incidents.

Observability Gets a Bigger UX Emphasis

The SRE role is definitely a trend that has been growing and will continue to grow in importance, Anderson said. And as more organizations turn digital, it’s becoming necessary; with this shift comes enormous pressure to exceed user experience expectations. “Competition and people’s expectations are so high right now,” said Arnott. “If you are unreliable, it’s unacceptable these days.”

Whereas DevOps focuses more on getting applications into production, SRE takes a more UX-focused approach. Take Netflix, for example, which has invested heavily in a core SRE team that ensures robust and reliable user-facing services. “A key underpinning to keeping our customers happy and streaming is a strong focus on reliability,” according to the Netflix Technology Blog.

To cater to this new paradigm, teams will require a deeper understanding of observability based on particular cohorts of users, said Arnott. SREs require precise user experience contexts, such as an uptick of lags or increased error responses. But they also must discover the contributing factors behind these bugs. “SLOs and error budgets are like taking your temperature—they tell you if something is wrong. They don’t tell you why you have a fever.”

SRE Takes Cues From Manufacturing

As reliability becomes more and more central to a company’s brand presence, it’s only natural that something that important would be elevated into the executive suite, Anderson explained. As a reference example, he pointed to the fact that manufacturing industries often appoint a chief reliability officer. In traditional manufacturing, product lines undergo a statistical process control with rigorous testing to ensure quality. Analogous positions exist across industries, such as chief quality officers in financial services.

The takeaway is that as reliability becomes more of defining competitive metric, tech companies may require a similarly high-level C-suite position to align on that idea and to direct teams to implement SRE practices. Of course, the necessity for such a title will entirely depend on organizational size and makeup.

Greater Reliance on SLOs to Avoid Broken SLAs

A service-level agreement (SLA) is typically a formal contract built around the reliability of a digital service with an external partner, which, if breached, may result in financial penalties or other forms of restitution. This agreement could exist between a software provider and an end user but is more strictly enforced in a B2B context.

On the other hand, a service-level objective (SLO) is more of an internal target. SLOs are often set at a lower threshold than SLAs. For example, perhaps an SLA ensures a user login fails no more than 10 out of one million attempts, but the SLO is set at two out of one million.

Setting internal SLOs helps keep engineers abreast of modest customer experience issues before they become a major incident that results in fines. “Some companies treat SLOs as just another alert; they should be representative of user experience,” said Anderson. It makes sense that as SRE initiatives begin to mature, companies will slowly start to adopt even more competitive goals to deliver superior user experiences and avoid unsatisfied partners.

Final Thought: Don’t Play The Blame Game

A big part of the SRE doctrine is the concept of blameless postmortems. Because when you’re in the middle of an incident, the worst thing to do is point fingers. “Blaming triggers a defense mechanism and when you’re in a defensive mode, you can’t respond well,” said Anderson.

Instead, it’s better to keep an open mind when performing these retrospectives—accept that what occurred happened for a reason, and seek to discover that reason to make positive improvements. “What can we change at a systemic level to become more resilient?” inquires Arnott. “What is the origin of that mistake? What information was lacking? What communication should have happened to prevent this?”

Typically, accidents don’t occur intentionally or due to incompetence but rather due to holes in the process. By not playing the blame game, companies are better positioned to fix issues as they progressively release new features. Then, hopefully, with the right SRE approach in place, teams can advance their digital services to meet the high stakes of today’s modern user experience expectations.

January 25, 2022
DevOps for Salesforce: CI/CD Challenges

DevOps encompasses a wide variety of technical and social practices for improving the development, delivery and operation of IT systems. Salesforce has become more than a CRM system; it evolved into a full-featured ecosystem and a low-code platform that made development far simpler and more approachable than traditional code-based platforms. Cross-functional teams build on Salesforce to create custom apps and integrations with other enterprises systems without requiring the rigorous approaches that are common with traditional technologies like Java.

Now, DevOps for Salesforce has emerged to enable teams to build applications faster. And as teams have built more and more customizations onto the Salesforce platform, the level of complexity has exploded. DevOps has emerged as a critical capability for teams to bring this complexity under control.

The Primary Challenge of Deployments

The primary challenge that most Salesforce teams face is deployments. Deployments on Salesforce are not just a matter of moving files, they require sophisticated management of thousands of XML files and careful management of dependencies. A number of commercial companies have appeared to help manage the Salesforce development life cycle, and the Salesforce development experience (DX) initiative has unlocked vastly more possibilities for teams to automate existing processes and adopt new workflows.

The need for more structured and rigorous approaches to Salesforce development has increased the need for teams to learn from traditional software development and DevOps teams and understand how traditional techniques like modularization and continuous integration can be adapted to the Salesforce platform. Technology and knowledge transfer from the broader DevOps community to the Salesforce community remains an active area of exploration.

Working in Parallel

It was the adoption of continuous integration by the extreme programming (XP) community in the 1990s that drove intense interest in the need for automated testing, especially unit testing. This allows teams to create exact copies of each other’s work and develop in parallel. Working in parallel allows for far faster innovation since work doesn’t have to be precisely sequenced.

But there’s a cost to working in parallel; these workstreams eventually need to be integrated into a cohesive whole. Software projects following a “waterfall” methodology may delay integration until near the end of the project. But these parallel work streams diverge naturally over time, meaning that integration can take months and require enormous amounts of rework.

Continuous Delivery: A Distinct Practice

Continuous integration (CI) avoids this issue by merging work early and often (at least daily). “If it hurts, do it more frequently,” is a mantra in the CI world, where the challenge is to resolve conflicts in small batches rather than letting them accumulate.

CI creates a constant risk of failure for software, since teams working on one part of the system may be making changes to shared components that can cause other teams’ code to fail. Automated testing provides a layer of protection by allowing functionality to be checked and rechecked thousands of times with minimal human intervention. Creating automated tests requires more effort than executing a manual test, but the cost of re-running each test becomes negligible. Thus, automated testing is a perfect solution for high-velocity teams who need fast and ongoing ways to ensure they are not regressing as they build new capabilities.

Continuous delivery (CD) is a distinct practice that’s often mentioned in the same breath as CI. Continuous delivery was popularized by the 2010 book of the same name by Jez Humble and Dave Farley. The practice of CD extends the idea of CI by aiming to deploy code to production in small batches on a regular basis; ideally at least daily. Whereas CI addresses the risks of developing in parallel, CD addresses the risks of delaying feedback from end users.

Software is built to solve specific problems. But each attempt at a solution by developers is a hypothesis that needs to be tested and confirmed. Only the people who actually experience that problem can tell developers whether these solutions are sufficient. So developers must get fast feedback from end users to have confidence in what they’ve built. Teams that neglect this practice are operating on assumptions and optimism; the longer they delay delivery the more the risks of those assumptions compound. Poor adoption and user experience is to be expected when teams don’t practice continuous delivery.

Increasing Pressure

CI creates added pressure to test quickly, CD increases that pressure significantly. Delivering small batches of changes reduces the risk of each change and makes them faster and easier to roll back. But by definition, we are changing production systems that may be used by thousands or millions of people each day. A failure caused by CI may waste a few hours of a developer’s time and delay delivery of a change. But a failure caused by CD can prevent thousands of people from doing their jobs, corrupt production data, expose security vulnerabilities and worse.

When developers integrate their work in CI, the goal is to run a fast set of tests that confirms nothing major has broken. But work being delivered to production needs to go through a battery of comprehensive tests to ensure it meets all functional and structural requirements. In a waterfall-style project, pre-release testing may involve scores of testers working for months. In continuous delivery, teams must be able to execute tests at every level of the system and have robust confidence that no failures have occurred.

Comprehensive testing requires testing the user interface, testing processes that span multiple systems, loading and confirming data using APIs, security testing and more. Continuous delivery has driven the need for automated testing to new levels and led to entirely new approaches like canary deployments. Canary deployments are a form of shift-right testing where a feature is initially delivered to a small subset of end users to check for issues before rolling it out to everyone.

The Essence of Lean

It’s often said that the main driver of success in DevOps is culture. The acronym CALMS is used to summarize the five aspects of DevOps: Culture, automation, lean, measurement and sharing.

Culture refers primarily to creating a culture of continuous learning and experimentation. The practices described above are not formulaic and require a growth mindset and a culture that prioritizes psychological safety and information flow.

Automation is the aspect of DevOps most commonly emphasized (including in what’s been written here). Automation enables dramatic efficiencies and is the central miracle of technology, but automation is not a panacea without addressing these other aspects of DevOps.

Lean refers to the manufacturing practices pioneered in the 1930s at Toyota and further developed and built upon in the late 1980s and 1990s. The essence of lean is to do more with less by understanding and optimizing the value stream by which work is actually performed.

Measurement refers to the importance of taking a data-driven approach to optimizations. Measurement provides teams with a way to gauge whether process improvements are effective or not.

Finally, sharing refers to the importance of sharing information, practices and resources between individuals, teams and organizations.

January 24, 2022
Embracing Security by Design in Development

The number of data breaches publicly reported in 2021 far exceeded the total for 2020, making 2021 a record year. With cyberattacks on the rise and bad actors and ransomware gangs demanding millions in ransom, cybersecurity has taken on new urgency.

In addition to more sophisticated hacking techniques, the rapid pace of tech adoption during the pandemic is another contributing factor fueling cybersecurity risks. A recent survey by Forrester shows 74% of companies attribute recent cyberattacks to vulnerabilities in technology put in place during the pandemic. As companies raced to enable remote work, develop new applications and roll out new consumer-facing digital solutions, cybercriminals saw new opportunities to identify vulnerabilities and to attack.

Cybersecurity threats can affect every business regardless of size and every individual, regardless of who they are or where they live. Even the most protected and advanced organizations are at risk: One of the world’s biggest cybersecurity companies recently fell victim to one of the largest breaches in history. It’s time for all companies to rethink their approach to boosting security in the software development process.

As a nearshore development partner for businesses around the world, we have embraced a ‘security by design’ mindset to keep our clients’ builds secure. To put security first and ensure it remains at the forefront of every engagement, we adopted a parallel build systems process to improve the integrity of the code we deliver. This also reduced the overall threat of potential attacks. How did we do this? There are three key components to developing and implementing a security-by-design mindset.

Embedded Security Expertise

It is critical that security experts become official members of every software development team. These specialists are there every step of the way from design to delivery, finding and fixing security flaws along the development journey. Instead of being brought in at the end of the process, they are deciding on new solutions and implementing protections in real-time alongside experts in design, user experience and coding. Each plays a role and works as an integrated member of the team providing advice and suggestions during Agile cycles and sprints.

Predictive Analytics

A recent survey showed that 96% of third-party applications deployed in cloud infrastructure contain known vulnerabilities. That rising risk can make it possible for bad actors to infiltrate thousands of organizations. Threat hunting tools can spot, anticipate and predict vulnerabilities in software development. Tools can automatically scan code as it is written to vet and diagnose vulnerabilities. Our team scans code while in production and reports risks and vulnerabilities and then immediately consulting with the rest of the team to work on the next secure version.

Continuous Learning

Security is changing quickly, and it’s never been more critical to ensure developers have the skills necessary to defend and protect corporate infrastructure, applications and digital solutions. With the threat landscape constantly evolving, new skills and continuous learning are required to stay ahead of the latest threats and fraud techniques. Part of our training is focused on deploying white hats, or ethical hackers, to breach our development environments and the applications we deploy. These experts then set about identifying weaknesses and helping to improve security. They also take our training to the next level and push our experts to continually learn and educate themselves on the latest threats and solutions. By upgrading their skills, security practitioners can stay ahead of cybercriminals.

Every day, enterprises face thousands of attacks on their systems, and malware continues to grow in frequency, volume and complexity. These attacks are not just coming from criminals looking to make a buck; there are growing fears of nation-state attacks escalating. These are often seen as the more dangerous threats to businesses. The best defense against cyberthreats is a strong offense anchored by a security by design mindset.

January 24, 2022
CircleCI Adds Free Unlimited Access Tier to CI/CD Service

CircleCI has added a free unlimited access tier to its continuous integration/continuous delivery (CI/CD) platform.

Jim Rose, CircleCI CEO, said in contrast to other ‘freemium’ offerings, CircleCI’s free tier provides developers and DevOps teams with frictionless access to all the premium capabilities of the platform. Those capabilities include up to 6,000 build minutes per month, unlimited users, the ability to fine-tune pipelines using caching, concurrency, test splitting, security tools and health metrics.

CircleCI claims more than 200,000 teams now employ its CI/CD platform. The freemium tier is intended to make it easier for a DevOps team to take advantage of a CI/CD platform as they begin an application development project, noted Rose. As that project evolves and matures, many of those teams will eventually require commercial support and other services provided by CircleCI, he added.

The free tier of the CircleCI cloud service comes as many organizations attempt to accelerate the rate at which they can build and deploy applications. Digital business transformation initiatives, in particular, have become a much higher priority in the wake of the COVID-19 pandemic. A free tier of a CI/CD service is likely to encourage more experimentation simply because it reduces the level of overhead associated with setting up an application development environment.

As the number of applications being developed continues to steadily increase, competition among CI/CD platform providers is becoming more intense as the number of organizations adopting DevOps best practices to accelerate application development increases. At the same time, more DevOps teams than ever are relying on cloud services to host CI/CD platforms in the wake of the COVID-19 pandemic.

It’s still not clear whether DevOps teams will opt to rely on a proprietary platform versus an open source platform. While an open source platform prevents organizations from vendor lock-in, the ease of use of a cloud service may lessen those concerns as the cost of switching from one cloud service to another declines.

One way or another, usage of CI/CD platforms will increase sharply in 2022. It’s not uncommon for different DevOps teams within the same organization to employ different CI/CD platforms despite efforts to encourage standardization. In the name of productivity, DevOps teams tend to exercise a fair amount of autonomy within many organizations.

The biggest challenge, of course, will not necessarily be building applications but deploying them. Despite the availability of CI/CD platforms, the processes that most organizations rely on to deploy an application are still largely manual. In the age of the cloud, however, it should become easier to automate application delivery thanks to the APIs cloud service providers expose. In fact, 2022 may be the year that DevOps teams finally master CD. Providers of CI/CD platforms, of course, have a vested interest in making sure the full promise of an integrated CI/CD platform finally becomes an everyday DevOps reality.

January 21, 2022
Securing Software with Intelligent Pipelines

One of the biggest cybersecurity risks involves vulnerabilities in the application layer. After all, the best firewall is useless if the web application itself is vulnerable. Many companies have worked to mitigate these risks by investing in their AppSec programs. According to a recent whitepaper written by ESG (commissioned by Synopsys), 71% of companies surveyed now use AppSec tools for more than half of their software projects. Notably, over two-thirds of companies already use 11 or more automated application security testing (AST) tools, such as SAST, DAST, IAST, fuzz testing and container scanning solutions.

This is due, at least in part, to the fact that tool manufacturers have now made their products “DevOps-ready” and support suitable integrations with CI/CD pipelines. This makes it tempting to simply let AppSec scanners run in the pipelines, but that can introduce other problems.

Problems with AppSec in CI/CD Pipelines

Too many results: Developers can be inundated with findings, yet only a small percentage are likely to pose such a high risk that they need to be fixed immediately. But the prioritization guidelines are often formulated in separate documents and are ambiguous.

Development pipelines are slowed down: Build pipelines often run at frequent intervals; every second to every minute. Scans with AppSec tools may take several minutes or even hours.

Manual AppSec activities are left out: Not all AppSec activities can be automated, such as architecture risk analyses, threat models and penetration tests. Nevertheless, these are an essential part of the AppSec strategy.

Intelligent pipelines (i.e., intelligent, purpose-optimized automation and orchestration of the various AppSec tools and activities) are ideal for overcoming this challenge. Combined with the consolidation of scan results, a new category of solutions has emerged here, which Gartner dubbed application security orchestration and correlation, or ASOC for short, in 2019.

How Pipelines Become Intelligent

The “intelligence” lies in deciding which tools need to run at what time and what to do based on the results. So instead of scanning the entire codebase with AppSec tools at every commit, it dynamically decides which scanner needs to run and to what extent. This decision can take into account various parameters such as the scope of the actual code change, the risk profile of the application or the development stage of the software.

The risk profile of the application should also be considered. Web applications that are accessible from the internet and process sensitive data pose a greater security risk than an internal tool for generating documentation. Such risk profiles usually emerge from prior architectural risk analyses and threat models.

Furthermore, the scope of AppSec testing should be appropriate to the development stage of the application. Individual commits of a feature branch should be checked mainly by static code analysis for passwords and API tokens contained in the code and compliance with coding guidelines, such as SEI CERT, to support rapid development. Later on, during the merge request into the main branch, more extensive scans should be added, including deeper data flow analyses, which then detect cross-site scripting or SQL injection attacks. A longer runtime can be accepted here since such merge requests usually have to be approved according to the dual control principle.

Code Security Policies

The core of intelligent pipelines lies in individual guidelines or policies. These define when specific AppSec activities are executed. Additionally, these policies describe how to proceed with the combined results (e.g., whether the code may be integrated into the master branch or the web application may go live).

The policies are described in a configuration file; this is policy–as-code. Just as with other as-code methods, this enables or improves uniqueness, reproducibility and automation. Simple policies can be created according to the “If this, then that” principle. For example, a software composition analysis (SCA) scan can be triggered when either the project file changes (package.json, go.mod, pom.xml, etc.) or new files or directories are added, but not when only existing source code files are changed.

Conclusion

Whether with or without orchestration tools, anyone who wants to make their software more secure must inevitably consider which AppSec activities make sense at what time. In other words: The right scan at the right time. Furthermore, it should be determined how to proceed with the results before ordering or even automating a scan. To do this, you need to understand what the greatest risks are and how to protect the software from those risks.

Intelligent pipelines help to implement and, above all, automate an AppSec strategy. A corresponding strategy is a prerequisite. But even (or especially) without a clear strategy, it makes sense to look at the architecture of an intelligent pipeline, as this both encourages collaboration between development, AppSec and DevOps teams and raises the right questions that lead to a successful AppSec strategy.

January 20, 2022
Harness CD Platform Now Available via Source-Available License

Harness today announced a free community edition of its continuous delivery (CD) platform is now available under a source-available license rather than a more traditional open source license.

Scott Sanchez, chief marketing officer for Harness, said the goal is to make CD as accessible to developers and DevOps teams as their current CI platform. Harness already makes an open source edition of its continuous integration (CI) platform available.

The Harness CD platform is available under a Polyform Shield license that provides free access to the platform, including the right to use, modify and distribute it but not launch a competitive offering. A source-available license is an alternative to an open source license that allows an organization to potentially create a fork of the Harness CD that could, for example, be delivered as a cloud service. That specific issue has led to scenarios where vendors that create an open source project find themselves competing for revenue against a cloud service provider that is not necessarily as committed to contributing and maintaining the primary open source project. At this juncture, it’s not clear how many other vendors that make software available under an open source license might take a similar tack.

While most organizations that embraced DevOps have been able to successfully implement a set of CI best practices, the number of organizations that have implemented CD is not nearly as high. One reason for that is automating deployments across platforms has never been especially easy. Each platform tends to be customized to the point where it is difficult to automate application deployments. In effect, each platform is its own unique type of “snowflake.”

However, as the number of digital business transformation initiatives continues to steadily increase, the need to regularly update application experiences is going to require more organizations to implement a set of CD best practices. A source-available license should help increase the number of DevOps teams that can access and experiment with a CD platform, noted Sanchez. The challenge, of course, is that many of those applications are now based on microservices with components widely distributed across an IT environment.

Historically, CI and CD have been synonymous when viewed within the context of a CI/CD platform. It’s not clear to what degree DevOps teams will look to better leverage the CI/CD platforms they already have in place rather than deploying a dedicated CD platform for IT operations teams and letting developers choose whatever CI platform they prefer. In general, a recent survey conducted by the Continuous Delivery Foundation found that 80% of respondents that work for companies with two or more employees are involved in DevOps to some degree.

Regardless of approach, the coming year is shaping up to be one where many more organizations take DevOps initiatives to the next level. Arguably, as more organizations realize how dependent they are on software to differentiate themselves from rivals, the ability to deliver new application experiences has become critical. At the core of those efforts is a set of DevOps best practices that will ultimately determine just how competitive those organizations will be.

January 19, 2022
Testlio Adds Fused Testing to Automate Testing Platform

Testlio announced today it is adding fused testing capability to its application testing platform; Testlio is used by IT organizations to manage application testing teams consisting of internal employees and external contractors vetted by Testlio.

Fused testing describes the combination of automated and manual testing IT organizations use to improve overall productivity. Fused testing strikes a balance between tests that can be automated and those that still need to be performed by a human.

To achieve that goal, Testlio has partnered with Applitools, BrowserStack, Headspin, Mabl and Sauce Labs. Failure signals and session artifacts from these tools can be pushed to the Testlio platform to streamline the triage of test results. Testlio is making six device slots and 100 hours per month of automated testing processing time available at no additional cost.

Finally, Testlio has set up an automated testing center of excellence (CoE) as part of an expansion of its professional consulting services it provides to help organizations implement testing best practices.

Testlio CEO Steve Semelsberger said those integrations will make it simpler to augment human testers and increase the rate at which applications can be tested as many organizations are accelerating the rate at which those applications are being built and deployed.

As organizations seek to implement continuous testing within the context of a DevOps workflow it’s apparent there will need to be more reliance on automation to keep pace with application development, he added.

Through its platform, Testlio provides organizations with access to more than 10,000 freelance application testers on demand. That network of testing professionals can create, run, optimize and diagnose automated tests using Appium and Selenium frameworks written in programming languages such as Python, Java and Node.js. Only 3% of the freelance application testers that apply to join the Testlio network have been accepted, noted Semelsberger.

The degree to which organizations prefer to rely on internal employees versus contractors to test applications will naturally vary. However, it’s difficult to predict what volume of tests will need to be performed—and when they should be performed—to preserve application development timetables. Contractors provide IT organizations with a degree of testing flexibility that is becoming an absolute requirement given the current challenges of hiring and retaining professional application testers, said Semelsberger.

Application testing is, of course, difficult to apply uniformly no matter how many testers are available. In addition to providing access to experienced freelance testers, Testlio also provides a platform for managing the overall testing process. The challenge is then aligning those testing tasks with the various application development projects the organization has launched.

Regardless of the approach to testing, the need for it has never been more critical. As organizations embrace a wide range of digital business transformation initiatives, a much higher percentage of the applications are now externally facing. The tolerance for suboptimal application experiences among customers and partners is much lower than that of internal employees. As such, first impressions, like it or not, are going to determine the success and failure of most of those application initiatives.

January 19, 2022
How Test Automation Can Help Drive DevOps Success

Testing can be one of the more time-consuming aspects of the DevOps process. It can be exhaustive, frustrating, requires a lot of hands-on labor hours and people seldom enjoy it. But testing is vital, and no developer should simply skip it.

Recently, a strong trend has emerged wherein DevOps experts encouraged developers to automate their testing procedures. Test automation can bring your DevOps projects to the next level, save time and maximize pipeline capacity for continuous testing.

Test Automation: Definition and Benefits

Test automation minimizes the amount of human labor required in testing. In test automation, tools and frameworks write testing “scripts” that work to ensure the high quality of your application.

DevOps can use test automation in a variety of different ways–it can even take advantage of test cases and detect bugs in your code. This automatic testing procedure speeds up the process, getting your product to the market much more rapidly. The automatic quality checks can also practically eliminate the risk of human error in the process of continuous testing.

Test automation can also ease the burden of the repetitive nature of some DevOps testing, simplifying and speeding up the process significantly. Finally, automation promotes a high degree of reliability throughout your testing process. Test automation can also optimize your process.

Test Automation in the Context of DevOps

DevOps is all about continuous development—the product is created, deployed, tested, and sent out to be consumed by the end users.

Since it came on the scene, quality assurance (QA) has gained importance. The traditional approach to QA (wait until the product deployed to go through regression and functional testing) was much improved with the advent of DevOps. DevOps integrates the QA team into the whole of the development process, merging all players into a unified team. This integration requires a great deal of coordination across the different stakeholders.

This also allows for a high degree of agility and speed in the process, speed which the DevOps methodology then magnifies through a high amount of automation in continuous testing (which can then be configured to run automatically).

Deciding on the Best Tools for You

To build up an effective test automation process, you must find the right software workflow tools for you and your DevOps team. These tools should support and manage your test assets from a central location, passing your product through the testing process much more quickly and maximizing efficiency. That said, as helpful as having an automated testing process can be to enhance the efficiency and effectiveness of your software, using tools that don’t fit with your team and goals will present potentially significant problems for your organization. There is a staggering array of test automation tools available on the market, and they all have their strengths and drawbacks. No single tool will satisfy every need related to automated testing, so your goal needs to be to find the ones that will work the best with your unique needs and desires.

What to Keep in Mind When Choosing a New Tool

Cost (for both licensing and support): Budget is an eternal tyrant, and your team needs to go into this procurement work with a solid understanding of your financial limits. If you have stringent limits there, there are some types of open source software like Selenium that could be useful. These tend to require a substantial degree of technical skill, so they may not be as helpful for you—if that’s a weakness of your team.

Type: There are several different types of DevOps testing tools. Language is one of the more important factors when deciding on one–make sure the tool operates in either the same language as your application or at least one in which your team can efficiently operate. The intended platform for the tool is also notable, as is the intended audience (i.e. whether the product aims more towards developers or testers).

DevOps integration support: It’s always important to double-check that your tool supports both DevOps tool integration and continuous integration. If you don’t effectively coordinate your tools, it will undoubtedly slow down your process.

Training: Depending on the level of expertise within your DevOps team, you’ll need to ensure that there is an appropriate degree of training on how to use the team—for the people who will be using it.

Quality test reports: Test reports with both depth and breadth are crucial to success in continuous testing. These help notice defects and their root causes, then allow you to test the effectiveness of your analysis.

Excellent customer support: You will want to use a tool that will serve your needs well. That means a tool that understands your specific situation and can help you work through complications.

The Necessity of Automation Skills

All of this aside, you cannot just rely on the strength of your test automation tools to get you through the testing process. If you decide to implement test automation without having people on your team with the skills to manage it, you may end up in a challenging position riddled with inefficiency.

Here’s what you need from a test automation software engineer: They must have a broad base of knowledge that encompasses test automation tools. Beyond that, they need to understand how to dive deep into software testing functionality and the technical complexities that undergird the project. They should also know how to write scripts alongside development, collaborating with both operations and development teams. That collaboration between test automation, development and operations will have a deeply important role in writing test scripts and maximizing the coverage of your tests.

A test automation software engineer’s skills offer invaluable stability. Indeed, tools are often built on a foundation of skills. If you have a resourceful and capable software engineer on hand, you’ll be much more prepared for the fluctuations of the market than if you’re relying on a single tool. If you decide to focus on only one tool, your team risks limiting itself and potentially falling behind the times and needs of your customers.

The Dangers of Automation

Automation is one of the foundations of DevOps’ success. But even so, you must strike a delicate balance. If you automate too much, you may end up hurting your product.

When you start connecting too many different tools to your project, you begin to risk unwanted outcomes that practically cancel out the investment you made in DevOps in the first place. If you try to automatically test things that automation can’t effectively measure (general user experience, for example), you’ll end up with worthless data. Your various automation tools can start to create more work than they’re resolving if you need to spend a great deal of effort in managing them.

That said, if you under-automate, you’ll be losing out on workflow efficiencies that your competitors are using, making your team less competitive. Thus, it’s necessary to try and balance the two. The first step to striking this balance is having a thorough understanding of your requirements vis-a-vis testing, development and deployment. If you have that under your belt, you can find the automation tools that work best for your goals. Once you have the right combination of automation tools, you’re golden.

Also, test automation doesn’t negate the need for manual testing—and it isn’t designed to. Of course, certain kinds of manual testing can be eliminated by automation (especially the repetitive kinds).

But you still need manual testers behind the wheel examining your product more thoughtfully and at a deeper level than an automated program. That deep-level variety of manual testing can often discover issues that automatic testing failed to see. It can also provide a more thorough level of insight into the user experience that automatic testing will seldom be able to provide.

Benefits of Test Automation

We’ve covered a lot of deep ideas in this piece. But at the end of the day, test automation is beneficial for a few simple reasons: it reduces human error, simplifies your tests, allows you to deliver products much more quickly and leads to a much more reliable quality assurance structure.

That said, test automation won’t be a panacea for all your team’s ills. It will work, but only if you’ve chosen the right tools and you have people on your team with the skills to use them effectively. Once again, don’t rely on continuous automated testing as the be-all and end-all of your quality assurance practice. Several QA questions show that a human presence is much better at resolving certain types of problems and over-automating can cause significant inefficiencies.

Test automation can be a wildly useful tool if you use it well. These tips will help you take your organization to the next level of efficiency and excellence.

January 19, 2022
Innovation With Software Architectural Excellence

Software architecture plays an irreplaceable role in enabling businesses to deliver value efficiently. This is evident in how digital natives have been challenging incumbents and forcing them to rapidly digitize and transform how they operate and innovate. Accordingly, achieving architectural excellence is a continuous process of efficient implementation, advanced planning, and executive leadership engagement. Furthermore, it’s essential to recognize that software architectural strategies and planning shouldn’t be limited to IT. Enabling collaboration between IT and business leaders is necessary as software architecture plays a key role in delivering value, making processes robust, and tackling the next wave of changes in value creation and delivery.

The Goal of a Good Software Architecture

A good software architecture facilitates the ongoing business processes function robustly and ensures business capabilities are competitive against threatening challenges. Beyond this, the software architecture should also become the fundamental aspect of driving the next waves of business changes—whether they are processes, models, or innovative products and features. These changes result from competitive forces increasing digitization, transformation, and ecosystem-based consolidation, making change the overarching constant in various business aspects. Accordingly, conventionally viewing a software architecture at a limited software or product level doesn’t work as well as it used to. The practice of viewing software architecture at the business and ecosystem-level gains even more emphasis as digitization and innovation make businesses and ecosystems digitally intertwined in how they interact.

The following fundamental aspects describe how to innovate with architectural excellence:

Security and Privacy

Customers are increasingly becoming wary of how their data is handled and protected—and are even reluctant to do business if an organization doesn’t have clear policies on how it protects and handles customer data. On the other hand, regulators are constantly improving their compliance requirements to protect customers. This landscape is changing at different rates across multiple countries, posing challenges and equally creating opportunities for fast adapters. All of this implies that businesses need to ethically manage customer data along with protecting against the constantly evolving threat landscape. To satisfy this need, it is necessary to revise the existing software architecture and involve senior executives in making critical decisions in this respect. Moreover, architectural planning cycles should come down to a few months—instead of years. This is key to building and implementing innovative solutions rapidly to tackle changing regulatory landscape and customer expectations.

Satisfy User Needs

Since 2008, the smartphone has been changing customers’ experiences and expectations. Digital disruptors are continually challenging incumbents and threatening their survival. In this paradigm shift, businesses should understand their customers more deeply, quickly adapt their business models and introduce new products to fulfill customer needs. It requires architectural agility that offers the flexibility to innovate business models in shorter cycles and improve them faster with customer feedback. This also means keeping a long-term view towards software architecture planning and strategy development that enables seamless innovation and eliminates process and performance bottlenecks. In this view, software architecture improvements and decisions should align with business objectives and strengthen business capabilities, not just the number of implementations or technical solutions the IT has delivered.

Empower Developers

Today, more and more businesses need developers to help them digitize and transform. Accordingly, empowering developers is essential, and software architecture is the fundamental way to help them drive innovation and build critical business capabilities. Developer empowerment is usually measured in terms of developer velocity — it defines how enabled developers are to be agile and how well their talent is utilized. To improve developer velocity, provide them with a well-defined software architecture to focus on quality and consistent development. Also, the software architecture must have scope for customization so that developers can utilize it to its full potential.

While the above describes the overarching approach, the following sections detail the fundamental aspects of achieving innovation with architectural excellence.

Scalability

The software architecture should support scaling applications seamlessly on-demand to accommodate spikes in user growth and scale back to save operational costs. Accordingly, this approach requires deploying your application in the cloud while ensuring not just the underlying infrastructure but also the multiple components in your entire system scale without friction or intervention. High performance should accompany this to ensure customer satisfaction and deliver a superior experience. This means that your system should perform ideally at any scale at any time.

Portability

Can the system itself or components of it be deployed in any cloud platform—public, private or hybrid cloud? How much does it cost to move in terms of time, effort, and cost? In a world of hyperscale cloud providers and cutting-edge cloud services, a company’s IT ecosystem shouldn’t be locked into one platform or vendor but should utilize the best available services and technologies currently—that are also future-ready to improve performance and optimize overall costs.

Compliance

Businesses should develop a compliance framework that aligns people, processes and strategies with regulatory requirements to ensure compliance and avoid non-compliance risks. Also, the framework should be work-in-progress to accommodate upcoming changes from regulators across the board.

Maintainability

Similar to the source code of software components, software architecture should be easily maintainable—meaning that the software architecture shouldn’t introduce bottlenecks when modifying and improving it.

Maintainability issues could take many forms: is the software architecture outdated for the current technical and business landscape? Is it using legacy versions that are incompatible when other components and dependencies are updated? Ensuring the software architecture’s maintainability helps facilitate innovation and move at a competitive pace. If at all unsure where to start improving or redefining a software architecture, start with maintainability.

Cost

IT is often under constant scrutiny to justify its expenditure, and it is complicated to measure IT’s meaningful contribution to the business outcomes. As a result, IT constantly faces the risk of budget cuts and challenges in optimizing costs. Typically, the software architecture is let to grow organically, leading to duplication of system and efforts and inconsistent data across systems. To overcome this, control the software architecture on an ongoing basis, which, in turn, allows for more predictable cost-control.

In this fast-changing landscape of digital, businesses should maintain flexible architectures that facilitate innovation and continually improve the software architecture using agile methodologies to gain a competitive advantage in the business capabilities. This practice is much needed, especially when digital disruptors are increasingly threatening to take it all—leaving laggards behind.

January 17, 2022
How AI Can Improve Software Development

Change is constant and inevitable. While some changes are welcome, others are just a necessary evil.

Continuous calls for digitalization combined with a pandemic-prompted emphasis on transforming business operations, force organizations across industries to modernize their legacy systems. The emergence of cloud computing has shrunk organizations’ on-premise IT footprints. However, with 96 of the world’s largest 100 banks, 90% of the world’s largest insurance companies, 23 of the 25 largest retailers in the U.S. and 71% of the Fortune 500 still using mainframe systems, any modernization efforts—resulting in increased risk of downtime or defects—will frequently lead to disastrous consequences.

So, while making changes can be expensive, uncertain, and lead to customer loss and/or embarrassment in the marketplace, modern business demands that systems change to continue to deliver value. The problem with mainframe modernization, however, is that today’s code search tools, linters and program analysis tools are deficient when it comes to mitigating the risks associated with maintaining and improving systems.

Modernization: A Knowledge Problem

With 10,000 mainframes in active use globally, the average organization annually spends anywhere from 60% to 80% of its IT budget just on maintenance. But, as experienced and proficient programmers retire or move on, many companies are realizing that the specialized domain knowledge of the industry and institution that those developers used to create and update these complex critical systems is walking out the door with them.

As the developers who originally wrote the code have long since moved on, and documentation leaves a lot to be desired, developers new to a system are unable to acquire comprehensive knowledge of the system’s functionality, at a detailed or fast enough level, to become adequately productive and proficient. That lack of speed comes with significant risk and cost to any organization.

Recognizing the full and true intent of the functionality written in code is not simple. Developers today spend roughly 75% of their time searching through source code, to identify the code representing the functionality that needs to be changed. Complicating matters further, understanding the code is not enough. To effectively and efficiently maintain and support any system, software developers must know precisely what an application actually does—and how altering code in one part of the system impacts it as a whole. But since the code representing a behavior that needs to be changed can be littered throughout the system, developers might think they’re making a simple tweak, when really they may be breaking the entire system (and not even knowing it). The risk of change resulting in unintended consequences is real.

All of that being said, how can developers improve their efficiency and effectiveness?

The Shortcomings of Today’s Tools

With code repositories growing unchecked over time to extraordinary sizes, those responsible for maintaining and preserving a system’s functionality say it’s becoming increasingly challenging to find bugs in code without machine assistance. And with debugging being an arduous, time-intensive endeavor, developers are increasingly turning to code search tools, linters, and static and dynamic analysis tools to analyze millions of lines of code, flag errors and suggest fixes.

Whether the tool excels at bug localization, code visualization or “error” detection, entirely too many tools are insufficient when it comes to actually identify the specific lines of code that require attention. Program analysis tools only illustrate code in ways that developers still have to interpret (potentially incorrectly) and draw their own conclusions. While code search tools might speed the rate at which developers can build a mental model of the code, such tools are notorious for false positives, and developers still have to undergo the mentally challenging endeavor of piecing that mental model together to identify the code, and then safely make changes. Worse yet, code completion tools—tools that less-experienced developers disproportionately rely on—can actually propose imprecise changes (one prominent tool offers incorrect code 71% of the time), and that introduces risk.

The cherry on top of this problematic sundae is that today’s tools don’t have any way to effectively validate the scope and accuracy of a proposed change. The influence of a suggested change can be tough to identify, so testing for comprehensive validation is frequently next to impossible.

Ideally, there has to be a better way to mitigate risk while making changes to legacy systems. Enter artificial intelligence. By employing a unique approach to AI, the code repository truly becomes a knowledge repository, enabling many capabilities unrealized before now.

A Developer’s AI-Enabled Accessible Knowledge Repository

By leveraging AI to automate the identification of the specific lines of code that require attention, developers can simply ask this AI-driven knowledge repository where behaviors are coming from—and quickly identify the code associated with that behavior. This puts AI squarely in the position of intelligence augmentation, which is key to leveraging its capabilities.

This novel approach of AI reinterprets what the computation represents and converts it into concepts, therefore “thinking” about the code in the same way humans do. The result is that software developers no longer have to unearth the intent of previous developers encoded in the software to find potential bugs. Even better, developers are able to overcome the inadequacies of automated testing by using AI to validate that they haven’t broken the system before they compile or check in the code. The AI will forward simulate the change and determine whether it’s isolated to the behavior under change. The result is the bounds of the change are confined to the behavior under change so that no unintended consequences arise.

While AI is not yet advanced enough to safely make changes on its own, it can serve as a knowledge repository with the added capability of directing developers precisely where to make any needed changes. With AI, programmers—even on their very first day on the job — can make any necessary fixes with confidence that they won’t break the entire system.

Unlike its code search, code navigation, code completion and program analysis tool counterparts, AI fundamentally empowers developers to safely, efficiently and effectively make any necessary changes. With AI — that is fortified by a knowledge repository that understands source code the same way a human does — developers can truly do more with less.

January 13, 2022