devopsy.kinsta.cloud

Category: DevOps Practice

How to Create Bug-Free Blockchain Apps
While all developers strive for bug-free code, it’s particularly crucial in a blockchain deployment where sensitive data or other confidential info is being exchanged, such as in health care or finance. However, some businesses have learned that lesson the hard way. Cryptocurrency exchange Binance recently revealed a devastating security breach that resulted in a loss of more than $40 million. While this is a large sum, a security breach could be even more costly for businesses that lose private messages, confidential contracts and more.

Rebuilding (and regaining customer trust) is not where organizations should be spending their time. A better way forward is for company leadership to mandate a blockchain test automation strategy from the very beginning.

While most software testers are familiar with testing web applications, they may not be sure how to approach blockchain testing. The good news is, some of the same focus areas of web testing apply to blockchain testing, too (functional, performance and security). However, blockchain involves a greater number of tests, effort and focus areas, such as infrastructure orchestration, staging and simulating scenarios in a distributed environment, which requires an even greater level of expertise.

Because of these additions, fixing a bug in a blockchain app requires several extra steps and, as a result, it is more time-consuming and costly than fixing bugs in other apps. For example, to reproduce bugs or verify code, developers should perform these steps multiple times:
1. Form a new endpoint.
2. Deploy new code to the new endpoint.
3. Migrate the current data to the new back end.
4. Suspend the old back end.
5. Update all front ends to the current version.
After that, verifying the bug-fix performed by testers or continuous integration (CI) tools will require these steps again. This complexity makes it difficult to release bug-free blockchain applications, yet doing so puts organizations and their customers at risk of financial harm. Due to the high-risk nature of deploying blockchain applications, it’s critical that testing is included in the managerial vision and strategy.

To release blockchain apps with bug-free code, businesses should follow these guidelines:

Integrate Test Automation Into Your Blockchain Testing Strategy

Test automation can relieve testers from performing thousands of orchestrating tests manually, which is both difficult and inefficient. A good system architecture should be scalable, maintainable and testable, so teams can test components independently along with mocked components. If a test requires a different environment or infrastructure, infrastructure as code (IaC, such as Puppet, Chef, Ansible and AWS is invaluable in setting up appropriate test environments. Another crucial part of the strategy is to provide the infrastructure with declarative definitions and allow developers to implement codes that support test automation.

Run Unit and Integration Tests to Cover Both Back End and Front End Code

Blockchain applications contain at least a back end (a set of smart contracts or chain codes running on a blockchain) and a front end (where users can interact with data stored in the blockchain). Performing unit tests will cover both back end and front end code, and, because of its critical role to blockchain apps, its primary focus should be on the chain codes. Integration testing, however, should cover the various types of oracles and front end applications. Performing additional tests will eliminate any holes that can threaten the back end by verifying the integration between the UI and chain codes.

Perform Tests in a Local Infrastructure

Though public blockchains (such as Bitcoin or Ethereum) have several test-nets, it’s optimal to perform the tests in a local infrastructure. Local infrastructures are lightweight, faster and more stable for unit and integration tests. For example, in a Ganache local infrastructure, it’s possible to run more than 1,000 unit tests on five Ethereum smart contracts with approximately 3,000 deployments in an hour or less.

Continuously Update Security System Test Suites

Blockchain is a multi-party infrastructure and in some cases, it is a public network. As a result, these applications will be visible to many stakeholders and may even be open-sourced to the world. This level of transparency means that hackers are able to study the programs and plan attacks based on a particular system. Continuously updating security system test suites will ensure applications are protected against all common security vulnerabilities.

Minimize Continuous Testing Pipelines for Each Phase of the Development Process

Layering integration and unit tests in a local infrastructure with continuous updates to the security system test suites creates a continuous testing pipeline that is essential to high-quality blockchain code. Continuously verifying code changes in multiple environments, such as local blockchain and test-net, ensures high-quality code is delivered to the production blockchain. These tests, however, can last hours or days for each code change, so it’s important for teams to minimize and create lean pipelines that focus on test stages, test types and test environments, and develop multiple pipelines for code at each phase of the development process.

Build Customized Tests Suites to Eliminate Prospective Cheating Holes

Test teams also need to customize test suites in the pipelines to cover both functional and security testing. Covering security cases is important, but unit test modules should also exercise smart-contract-unit potential security threats, such as overflow and re-entrancy, to eliminate as many prospective cheating holes as possible.

Due to the nature of decentralization and orchestration of underlying infrastructure, developing bug-free code for blockchain applications presents many more challenges than other applications. By taking best practices from traditional software testing, especially test automation, and customizing them for blockchain development, businesses can achieve bug-free blockchain coding and protect their most vulnerable applications.

— Thong Nguyen
May 31, 2019
Are Your Development Processes Truly Agile?
Many organizations are making a move to agile-powered environments, but agile development is more than a name. How do you determine if your organization is truly agile? The evaluation starts with the Agile Manifesto. If you’re following the principles laid out in the manifesto, that’s a good sign that you’re on an agile journey.

For those not familiar with the manifesto, however, let’s review its points quickly. The Agile Manifesto’s core is simple—the primary points of the manifesto include the following:

Individuals and interactions take precedence over processes and tools. This means the organization and its leaders place more value on people than they do on processes. People respond to business needs and they drive the development process within the organization.

Working software takes precedence over comprehensive documentation. This is not about not providing any documentation at all, but you should streamline documentation. Any documentation that does exist should be necessary only for current and future development.

Customer collaboration over contract negotiation. Engaging customers throughout the development process makes it easier for your people to meet the needs of your customers. Doing so means all business case features are clear and useful to customers.

Responding to change over following a plan. This means change is welcome throughout the project, each project and allows for priorities to shift over time versus following a rigid or scripted development plan.

The Agile Manifesto 12 Steps

Essentially, the core values of the Agile Manifesto can be fully detailed in 12 steps, or principles. These principles support the key points of the manifesto and provide additional detail as to how the practices can be followed in an agile environment. Again, for those who currently follow the Agile Manifesto, the following may be redundant. For those new to the concept, these steps provide clarification and direction to drive agile processes overall.
- Self-organizing teams encourage great architectures, requirements and designs.
- Regular reflections on how to become more effective.
- Frequent delivery of working software.
- Simplicity.
- Support and trust for those involved in the process.
- Enable face-to-face interactions.
- Working software is the primary measure of progress.
- Agile processes to support a consistent development pace.
- Attention to technical detail and design enhances agility.
- Collaboration between the business stakeholders and developers throughout the project.
- Customer satisfaction through early and continuous software delivery.
- Accommodate changing requirements throughout the development process.
Based on these principles, there are three points—the final three listed here—that we often find the most compelling. A colleague of mine, Tobias Spöcker, scrum master and QA engineer, says these points naturally float to the top for an organization such as TOPdesk.

The reasons for this are fairly obvious. We create products that are sold to thousands of customers around the world. To pinpoint any development problems, we look to the scrum process described by the scrum guide, with an emphasis on the last parts of the process. For example, the following illustration is edifying for explaining this process:

Notice in the “product release” phase illustrated above that the team listed here focuses on finished work for the development sprint. This portion of the model should be able to present a product incrementally to the stakeholders, including the customer. Ideally, these increments are released through a continuous delivery model. Here are two principles that further illustrate the point: Collaboration between the business stakeholders and developers throughout the project and customer satisfaction through early and continuous delivery of the software.

There’s an additional consideration when providing value: Develop what is right; don’t simply develop something right.

Adding Value

At TOPdesk, providing the proper value to customers is a major goal for us. To achieve our organizational goals, we consider the impacts on our customers so when we start on a new product development project, we always work with some of our customers so they are involved in the development process, which makes the process and the product better for everyone. Those of our customers that have participated in our development processes usually tell us that they find the experience valuable and that being a part of the process also helps them when they use the solution. These same customers also express satisfaction in our efforts—as partners in the development process—and with us as an organization.

Direct Feedback is the Key to Better Development

The immediate and direct feedback we receive also helps to achieve another of our key operational principles: accommodating changing requirements throughout the development process. Through the partnership with our customers during the development phase, we’re able to receive regular, constructive feedback—even criticism—that leads to changes and evolutions in the product map and development stream. This feedback facilitates refinement of the most important processes and features, and moves them to the top of the priority list, leading to a project that’s undertaken with a focus on the most valuable areas so that development capacity (on the part of our organization) is spent the most wisely.

In Sum

The agile mindset and the scrum framework are powerful tools if they are used in the right way. When considering following any development principles, don’t go in blindly. It is a good idea to see which parts of the toolbox help your organization be the most agile, efficient and customer-focused, and utilize the processes and principles that work best for your organization.

Consider direct feedback from customers, which can be implemented immediately into the product scope. Provide simple-to-use customer communications vehicles, including metrics built into the software, surveys and feedback portals. Live demos of the product, when ready, can be extremely effective in helping you collect copious feedback efficiently. However, if your feedback loop is too big—meaning there are too many cooks in your kitchen—the whole process will quickly balloon beyond profitable, resulting in a less agile development process.

The projects we develop using these principles are the ones that deliver the most value to our customers the most quickly. Including feedback from customers makes the development team more confident in what they are creating, and our customers tell us that they feel their needs and input have been heard, which they greatly appreciate.

— Nancy Van Elsacker Louisnord
May 31, 2019
How AI is Transforming Testing Today

Currently, there is much debate about how artificial intelligence (AI) will transform our lives. One of the technology’s key benefits lies in its power to transform traditional approaches to app development and software testing. Gartner identified AI-driven development as one of its top three strategic technology trends for 2019.

Digital transformation is dependent on better-quality software, and software testing matters more than ever. However, the growing complexity of our software-driven world means that new, smarter and quicker forms of testing are critical in the race to build better software faster. But in a hyper-connected and complex environment, this is no simple task, and this is where AI-driven testing can help.

AI Transforming Testing

Like any new technology, there are exaggerated claims. However, when it comes to testing, AI is already delivering real improvements and making intelligent automation a reality. AI has moved far beyond an academic concept and is driving business benefits as opposed to hints to those organizations that have adopted it. Below are four of the key ways that it’s already transforming testing.

Auto-Generation of Test Scripts

There are undoubtedly complex test cases for which AI can’t auto-generate all the code. However, AI can easily auto-generate the code that clicks the buttons on the screen, fills out a form, logs into the app and other similar functions. Organizations that have deployed AI-driven testing are able to auto-generate 80% of code reliably, delivering substantial efficiency improvements. Along with auto-generation, AI achieves auto-maintenance: Tests are rebuilt when something in the digital product changes, ensuring continuous quality is maintained and further reducing the resource burden on human testers.

Test Optimization

If you’ve ever used Amazon, then you know that recommendation engine technologies are not that futuristic. AI can be applied to testing in the same way to help companies determine which test is most likely to find a defect based on the risk information gathered. Rather than taking a haphazard approach to testing, AI enables you to home in on the areas at risk—not only delivering efficiency gains but also ensuring a quality experience that delights.

Release Impact

Neural networks, combined with test history and data from current test runs, can predict how an upcoming release will impact users. For example, will customer satisfaction go up or down? Armed with this information, companies can make any adjustments necessary to ensure their users continue to be delighted by their digital experience.

Customer Impact

This is a key benefit of deploying AI-driven testing as you can determine the impact on the business from release updates. Neural networks, machine learning and other AI techniques can be used to understand how technical factors impact business performance. For example, are load times negatively impacting conversion rates? By understanding how releases will affect the business, you can make changes that will quickly have a positive impact.

What Does the Future Hold for AI-Powered Testing?

In addition to the core benefits that AI-driven testing is already delivering, organizations can expect to see additional benefits within the next 12 months that will further accelerate their ability to fully automate the entire testing life cycle.

Failure Determination

Current testing compares the result of a test with a pre-defined known good result. That methodology simply will not work with dynamic systems, such as an internet of things (IoT) device that interacts with multiple other devices which are constantly changing. AI is a key part of the solution: A well-trained AI system can determine whether a real user would consider the actual result correct or not.

Persona Testing

One size doesn’t fit all, and the ability to understand how your app reacts under different personas will allow organizations to understand what parts of the app are particularly good or bad for different personas. Armed with this information, you can optimize your app for your target audiences, ensuring an experience that delights.

As every organization races toward digital transformation, delivering a high-quality digital experience is essential for survival. To create compelling digital experiences, companies need tools that can navigate applications intelligently, learn where quality issues are most likely to occur and identify the critical data correlations that will help developers resolve issues quickly. This is where AI is already delivering significant advantages to organizations that have adopted it.

It’s clear that AI-driven testing is not a passing fad. Rather than taking a wait-and-see approach, it’s time to jump in and experience how AI can help transform app development and software. With AI-driven testing making a real and measurable impact on the business, you can’t afford to be late to this revolution.

— Antony Edwards

May 30, 2019
Textbook DevOps vs. Real-World DevOps

DevOps is one of those things—like Marxism or baking macarons—that is easy to understand in theory, but much messier to implement in practice. Real-world DevOps rarely aligns with textbook DevOps or the DevOps practices that organizations should follow in theory.

That is true for a number of reasons. Let’s walk through some of the biggest below.

Your Company Is Not Organized in the Way DevOps Theory Assumes

According to DevOps theory, every company has an IT department and a development department. Doing DevOps boils down to making those departments work well together.

The reality, of course, is that not all companies organize their staff or technical operations in this way. Some don’t silo developers and IT engineers by default. Others (especially smaller companies) don’t have developers or IT staff on hand at all; they outsource that work.

These companies can still benefit from DevOps principles such as collaboration and automation. But they should avoid fixating on DevOps teachings regarding the organization of technical work.

Real DevOps Skills are Hard to Find (And Expensive to Obtain)

“DevOps engineer” is now a trending job title. But few technical training programs have caught up with the demand for DevOps.

Most universities still silo majors and courses in software development away from those that focus on IT. Other types of technical training programs function similarly.

What this means is that it is more difficult to hire a DevOps engineer, or other technical staff with a broad set of development and IT skills, than it may first appear. DevOps engineers exist, but they are usually experienced staff who learned their real-world DevOps skills on the job. They’re more expensive to hire and difficult to poach from their current gigs.

Full Automation Is a Pipe Dream

The DevOps mantra tells us that we should automate everything, from software testing and builds to deployment, production-environment monitoring and incident response.

Automation is great in theory. But the reality is that you can never automate everything. Human engineers will still need to respond to problems occasionally, or to unanticipated issues that arise during testing, deployment and production management.

An important point to emphasize here is that even if you successfully automate virtually all of your processes, the mere fact that human engineers must be available to respond when something does go wrong undercuts the idea that you can rely on automation alone. Humans still have to watch the automated processes, even if they rarely need to intervene. The simple act of watching and being available to respond when necessary is a significant burden.

Agility Conflicts With Simplicity and Reproducibility

In some respects, DevOps is a contradiction. On the one hand, it prioritizes agility and scalability. On the other, it emphasizes simplicity and reproducibility.

What this means in practice is that DevOps teams adopt microservices and containers in the interest of maximizing agility. But microservices and containers are much more complex to manage than monoliths and virtual machines.

I suspect the reality is that most DevOps organizations end up erring toward more complex technologies and architectures. They ultimately find greater value in them than they do in keeping things simple.

Still, the idea that you can be agile and be simple at the same time doesn’t apply well in the real world. (And sure, automation, infrastructure as code and the like can help in simplifying the management of complex agile technologies, but they only go so far.)

Most Organizations Still Aren’t Doing DevOps

If you spend your time surrounded by other people who have adopted DevOps and implemented it successfully, it’s easy to assume that the rest of the world is like you.

The reality, however, is that we’re now a decade into the age of DevOps, and yet a majority of organizations have made only halting steps toward implementing basic DevOps processes. If DevOps were half as easy to implement as it appears to be when you read the latest book about how to do DevOps, many more companies would be doing CI/CD by now.

Conclusion

DevOps delivers real benefits, and it’s important to continue preaching the DevOps gospel to organizations that can benefit from DevOps philosophy and practices.

However, it’s important to balance the conversation about DevOps with the reality: Real-world DevOps is much messier than it often seems in theory. Not many organizations have the configuration or resources to do textbook DevOps.

— Chris Tozzi

May 30, 2019
Citizen Developers Drive Productivity and Innovation

It was once written that “software is eating the world,” “APIs are eating the world” and “APIs are eating software.” In 2019, a more accurate statement might be, “APIs (or application programming interfaces) fuel the software that’s still eating the world.”

In today’s world, there is an ever-constant demand for software, enterprise applications and mobile applications, and most company IT departments can’t keep up. Enter the citizen developer. Citizen developers are helping organizations close the demand gap and innovate more quickly. But who are they and what role do they fill within an organization? And how do they contribute to overall technology innovation within a company?

Citizen Developers and the Need for Speed

The ongoing need to develop software quickly and efficiently has given rise to a new type of developer. A citizen developer is a non-professional developer who builds new business applications using existing tools, development and runtime environments sanctioned by corporate IT. They also can access various online tutorials to learn and master their application-building skills.

Today’s software development environment, powered by APIs, is ripe for the citizen developer. APIs, mobile application development kits and the cloud have created a no-code/low-code, building block-like approach to building software. Less code and a more simplistic software development path allow citizen developers to create much-needed apps quickly. This building-block approach cuts development time, translating to faster time-to-value.

Speed is critical not only in the creation of new apps, but also in the end user experience of these apps. With the building-block model, citizen developers use best-of-breed technologies for each block—making the resulting UX optimal (and fast) for an internal user or customer end user. Internal users are able to use fast, effective search within employee directories, company intranets and project management solutions, for example.

For end users, this need for speed addresses their ever-growing expectations. When someone searches for hotel deals in Phoenix, for example, they expect accurate, personalized and fast results. Even a one- or two-second delay can translate to lost opportunity and revenue.

The Workplace

In the digital workplace, citizen developers serve a key purpose. For starters, they know firsthand about the challenges employees face at work; they also know which applications or tools are critical to getting the job done. These non-IT developers (perhaps in a business or marketing role) help create apps for timesheet entry, productivity tracking, project workflow reporting and more. In one study, 76% of businesses stated that at least some of their apps were developed outside of their IT department—and 20% even stated that most of their applications were created outside of IT.

Citizen developers often use low-code tools or visual tools with some degree of coding required. Low-code allows developers to design and develop software applications with minimal hand-coding, while low-code platforms enable users to build applications with modern user interfaces, integrations and data quickly without writing thousands of lines of complex code. Also, citizen developers may work alongside software developers and database experts to develop apps with custom coding to meet specific enterprise or company needs.

Citizen Developers and IT: Working Together

When citizen developers work alongside IT, two things happen. First, by creating simple apps on their own, they enable IT to focus on developing more complex applications. Second, when building more complicated or customized applications, IT can use citizen developers to derive valuable insight. This data helps with testing or tweaking business app prototypes. With low-code, citizen developers with some technical skill start with a prototype and then can iterate with the IT team during the developer phase and later integration of systems, custom coding and security processes. Also, developers, who are often asked to understand more of the business side of the organization, gain this insight working with workplace citizen developers. And the results are promising: 82% of citizen developers said they witnessed a reduction in inefficient tasks while 71% reported an increase in team productivity.

In the past, IT spearheaded application development projects from beginning to end. Now, with citizen developers becoming an integral part of the workplace, IT’s role changes a bit. While no longer in full control, IT does give recommendations to the business department and selects development components and building blocks.

Just the Beginning

When it comes to search software, citizen developers have the potential to create personalized and customized experiences. While some types of front-end search are simple to create, developing a personalized and customized search experience is more complicated. This is where the current work lies and in doing so, we can build a new community of search-focused citizen developers.

This is only the beginning, as there will always be a demand for developers (both traditionally trained and citizen developers). In the coming years, we will see more citizen developers as there literally aren’t enough engineering schools to train students to meet business demand. They are helping to democratize application and software development across the board—and we can expect to see businesses and consumers continue to benefit.

— Julien Lemoine

May 29, 2019
5 DevOps Terms That We All Use, Yet No One Can Define

I’m all for DevOps. Sometimes, however, I have a hard time talking about DevOps. The reason is because many of the key terms related to DevOps lack clear and precise definitions.

Let me explain …

DevOps is such a big deal that it has bred its very own lexicon. Search Google for “DevOps terms,” and you’ll find a slew of “dictionaries” and “glossaries” dedicated to defining the various terms that go hand in hand with DevOps.

Yet despite these resources, some deep ambiguities remain regarding the actual meaning of various terms that you hear frequently in DevOps-centric conversations. Let me walk through some of the biggest.

Continuous

DevOps is all about continuous this and continuous that. Continuous integration. Continuous delivery. Continuous deployment.

To me, the problem with the word “continuous” in this context is that it lacks an exact meaning. No one is integrating, delivering or deploying software in a literally continuous fashion. That would mean having zero delays between different releases, which is not the case. Even organizations that do CI/CD really well manage deliveries and deployments of only once per day or so.

Maybe instead of “continuous,” we should just say something like “fast.” Because that’s what “continuous” really means in most cases.

CI/CD

Speaking of CI/CD, that’s another term that is ripe with ambiguity. It’s clear enough what “CI” stands for (continuous integration). But does the “CD” stand for “continuous delivery,” “continuous deployment” or both? Or does it not matter?

Moreover, does a your application delivery process qualify as a CI/CD pipeline simply by including a continuous integration server (to do the CI) and some kind of automated deployment tool (to cover the CD side of things)? Or does a CI/CD pipeline also necessarily include build and test automation? I think most people would say the latter, but the term itself does not imply that.

Microservices

Microservices are all the rage in the era of DevOps. Depending on how you define a microservice, however, they have been all the rage for decades, long before anyone was talking about DevOps.

The idea of distributing applications into multiple services is not new. That’s what SOA was all about in the 2000s. Going back further, there were microkernels, a concept popular in the 1980s. In microkernel architectures, the operating system kernel runs as a set of distinct services, rather than as a monolith.

The question, then, is what makes a microservice, or a microservices architecture, different from other types of distributed, service-based architectures. Maybe it’s the size of the service, but there’s no official definition of what distinguishes a microservice from a non-microservice.

Agile

It is often said that DevOps builds upon the principles of agile software development. Those principles were defined, at a high level, in the early 2000s and spelled out in the “Manifesto for Agile Software Development.”

It’s clear enough what agile means, but I’ve long struggled to piece together the exact relationship between agile and DevOps. Is DevOps “agile applied beyond the software team,” as one DevOps company puts it? Or are agile and DevOps distinct practices that just happen to have similar goals?

It’s hard to say, because there is little consensus about what exactly “agile” means in the context of DevOps.

DevOps

Last but not least is “DevOps” itself. Different people have defined DevOps in very different ways. AWS says it is “the combination of cultural philosophies, practices, and tools.” Other definitions focus on philosophy, and leave tools and specific practices mostly out of the picture.

Blurring the definition of DevOps further is the fact that virtually every tech company in the world calls itself a DevOps company today, with little concern for how, specifically, its products or services meet that nomenclature.

What does DevOps really mean? I don’t think we’ll ever have a singular answer to that question.

Conclusion

In fact, I don’t think there will ever be clear definitions for any of the terms I’ve listed above. I’m not suggesting there should be.

But I do think it would be helpful to the DevOps community as a whole to recognize the ambiguity in many of these terms. We use these words as if everyone knows what they mean, but the reality is that that is often not the case. If you want to use terms like “continuous” or “agile,” do so, but remember to clarify what those terms mean to you, because you can’t assume that they mean the same thing to others.

— Chris Tozzi

May 28, 2019
Are Self-Service Machine Learning Models the Future of AI Integration?

DevOps teams seeking to step up their mojo in developing cutting-edge artificial intelligence (AI) features are facing a big skills bottleneck when it comes to data analytics and machine learning modeling. As a result, the market is seeing an influx of self-service machine learning models and machine learning-as-a-service offerings designed to help development teams more easily integrate AI capabilities into their software.

This is coming in direct response to an explosion in demand for AI capabilities in the enterprise. According to Gartner analysts, AI adoption in the enterprise tripled in the past year. A report last fall from MIT Sloan Management Review and Boston Consulting Group found that 91% of enterprises believe that AI will deliver new business growth to them by 2023.

The trouble is that folding AI functions and predictive analytics into software requires a whole new level of expertise in data science and machine learning from cross-functional DevOps teams. They need added skills to choose the right algorithmic approaches, acquire and manage data, train the models and integrate them into the code base and underlying infrastructure so that everything works properly under the hood.

“We have heard from customers everywhere that they want to adopt machine learning but struggle to actually get models into production,” said Eric Boyd, vice president of cognitive and AI for Microsoft.

Ali Farhadi, Xnor.ai CEO

DevOps leaders today already face an uphill battle to keep their developer ranks staffed with well-trained software engineers. Piling on additional requirements for very specialized machine learning and data science expertise further strains those recruitment efforts. According to the most recent Harvey Nash/KPMG CIO Survey, the top one and two technical realms that suffer the biggest skills shortages today are in data science and AI, with 46% and 38% of CIOs respectively reporting recruiting pain in those areas.

“There’s a huge imbalance between the demand in the market and the supply of the very best AI experience to actually train these models,” said Ali Farhadi, CEO of Xnor.ai, a Seattle firm that just last week released a self-service platform for developers. Called AI2Go, the platform provides pre-trained deep learning models to quickly integrate AI features such as facial recognition and object classification directly into their software.

“We want to enable anyone who can code to benefit from AI,” Farhadi said.

The Xnor release comes just a few weeks on the heels of a big launch from Microsoft of several self-service machine learning model tools aimed firmly at MLOps teams—those DevOps teams implementing machine learning capabilities. The new services from Microsoft roll out capabilities to manage the code, data and environments used by MLOps teams throughout the machine learning deployment life cycle.

“Azure Machine Learning service’s MLOps capabilities provide customers with asset management and orchestration services, enabling effective ML lifecycle management,” explained Jordan Edwards, senior program manager for Azure MLOps, in a blog post on the topic. “With this announcement, Azure is reaffirming its commitment to help customers safely bring their machine learning models to production and solve their business’s key problems faster and more accurately than ever before.”

Tools such as these are not always going to obviate the need for data scientists and machine learning/AI specialists on DevOps teams. As Farhadi noted, even a simple service like AI2Go requires development teams to choose the right models for the right situations. Nevertheless, this burgeoning market could take the pressure off teams by minimizing the scale of specialized recruiting, while reducing bottlenecks and dreaded re-work.

“Machine learning projects usually involve several people—data engineers, data scientists, DevOps and others. Conventionally, at each stage of the machine learning life cycle, most, if not all, of these people end up doing an excess of custom coding, redundant work and manual trial and error,” said Craig Stewart, senior vice president of product management and product marketing for SnapLogic, which last fall introduced a self-service machine learning add-on for its integration platform. He noted that one of the primary goals is speed, to help cross-functional teams “build, train, validate and deploy high-performing models faster.”

— Ericka Chickowski

May 28, 2019
Get Testing Bottlenecks Out of Your Pipelines

I first learned the term “deployment pipeline” when I read “Continuous Delivery: Reliable Software Releases through Build, Test and Deployment Automation,” by Jez Humble and David Farley back in 2009. End-to-end automation of the build, deploy, test and release processes kick in each time a team member commits a change to the source code repository.

Chapter 5 of the book notes that the forms of testing common to all projects include manual test stages which “… assert that the system is usable and fulfills its requirements, detect any defects not caught by automated tests, and verify that it provides value to its users. These stages might typically include exploratory testing environments, integration environments, and UAT (user acceptance testing).”

Humble and Farley include these manual stages in the deployment pipeline, also known as a continuous integration (CI) pipeline, build pipeline or delivery pipeline. However, from what I observe, I think a lot of times people forget about these important manual stages.

Though we can do more and more with test automation, thanks to improving technology, manual testing activities are here to stay in most business domains. My own previous team realized this as we started to move toward continuous delivery (CD) a couple of years back, with twice-weekly deploys to production. Smaller changes and more frequent deploys means lower risk of regression failures. Yet, we still struggled with how to fit in important manual testing activities.

Whenever my cross-functional delivery team has a problem, I look for a way to make it visible, to help us talk about it and brainstorm ideas to solve it. My “aha” moment around manual stages came when I learned techniques to visualize our pipeline from Abby Bangser and Ashley Hunsberger. My team knew we had these manual testing activities and that we wouldn’t feel confident about releasing changes without them, but we weren’t really thinking about them as stages in the pipeline.

Visualizing Your Pipeline – Including the Manual Stages

Figure 1: A Continuous Delivery Pipeline (my diagrams are inspired by Katrina Clokie’s pipeline diagrams in “A Practical Guide to Testing in DevOps”)

Getting your team together to visualize your pipeline is a great method to find ways to improve it. Gather around a table with cards or sticky notes, or draw on a whiteboard—what are all the steps in your pipeline, including the manual ones? In Figure 1, the manual stages in a team’s CD pipeline are shown in yellow. All the automated and manual things need to happen before the business decides to push the button and deploy to production. It’s easy to see that these manual stages can become a bottleneck. Manual exploratory testing is likely to take longer than the automated test suites. If user acceptance testing needs to be done by certain business stakeholders and they aren’t available at the right time, that holds up the release. If the team is looking to move to continuous deployment, where every change to the product goes out to production as soon as all stages complete successfully, these manual stages will be in the way.

When you get your whole team including developers, database experts, operations specialists and product owners involved in solving these problems, you may discover innovative ways to approach them. In my team’s case, we decided everyone on the team needed to help with exploratory testing so we could do it in a timely manner. Testers helped the developers learn exploratory testing skills with hands-on workshops and by pairing with them. Then, each developer pair working on a story was tasked with doing the exploratory testing at the story level before declaring the story finished. We wrote exploratory testing charters at the epic or feature level as stories in our backlog. When enough stories were complete, anyone on the team, including testers and product owners, could pick up those charters.

If we couldn’t complete all the necessary exploratory testing before the next production deploy, we used feature flags to hide the changes from customers until we could complete the testing. We could go ahead and deploy the changes and wait to release them once we felt confident they would not cause any customer pain.

Asynchronous Testing Stages

Using feature flags or toggles is a good way to allow the team to approach some manual testing activities asynchronously, so they don’t hold up the deployment pipeline. If some required manual accessibility, testing takes an extra day to complete or the UAT has to be postponed for a week, the deploy still happens. A change could be toggled on for a small percentage of users and observed in production to see if any unexpected behavior occurs. Whatever activities are needed to safely release a particular change don’t have to hold up other changes that are ready for customer use.

Figure 2: Asynchronous stages in a deployment pipeline

Figure 2 is an example of a continuous deployment pipeline, but this asynchronous approach to the manual stages works for continuous delivery as well. Each time a team member commits a change, if all the automated stages in the pipeline pass, the release candidate build will be deployed to production. The manual testing activities are not inline with the other steps and don’t hold back the deployment.

If we’re ready to deploy a feature but need to finish those manual testing stages before we release it, developers can put in a feature flag set to “off” to prevent it from being exposed to customers. When the asynchronous manual stages are all complete and successful, the feature flag can be flipped on in production. This is the actual release, customers can now use the feature. The feature flag should be removed as soon as possible, as too many feature flags at once has serious risks.

Using feature flags or toggles to permit deploying changes to production before all testing stages are finished is one way to avoid bottlenecks and succeed with continuous delivery or deployment. Continuous delivery and deployment success depend on doing frequent small changes, not big-bang releases. A lean approach to feature flags, keeping the inventory low, is advisable.

Visualize It With Your Team

We’ve looked at one way to keep manual pipeline stages from holding up deploys, but there may be other solutions for your team. And there are lots of other key success factors for CD, such as keeping changes small and frequent. There’s lots to discuss and brainstorm about as your team, or a representational cross-section of it, gets together and experiments with a visual model of your pipeline. Talk about what you need to learn from each pipeline stage, who needs to know it and whether the pipeline is providing actionable feedback.

As a tester, I’ve sometimes questioned aspects of the pipeline that nobody else had thought about. Having team members with development, operations, database and other skills present means more potential for creative solutions. Our pipeline provides a major feedback loop and we want that feedback as fast as we can get it. Writing each stage on a card and arranging them on a table or wall often triggers ideas for parallelizing more stages or realizing a particular stage isn’t necessary or can be done locally by developers.

Automate all the things where it’s feasible. Use the time you save to optimize the way you accomplish manual testing activities and possibly automate more of them. Master techniques such as feature flagging so that you can take the time needed to feel confident about every new feature you release.

References

“Continuous Delivery: Reliable Software Releases through Build, Test and Deployment Automation,” Jez Humble and David Farley, https://continuousdelivery.com

“A Practical Guide to Testing in DevOps,” Katrina Clokie, https://leanpub.com/testingindevops

“Feature Toggles, AKA Feature Flags,” Pete Hodsgon, https://martinfowler.com/articles/feature-toggles.html

— Lisa Crispin

May 24, 2019
Successful DevOps Initiatives Start With Company Culture
What initially drove the need for embarking on the DevOps journey in your team or at your company in the first place? Was it:
- Market shift?
- New leadership or an acquisition?
- New product or service idea?
- Cost savings or consolidation?
- All of the above?
Whatever drove the need, my guess is there were pre-sales demos, reorganizations and training on new tools that were promised to solve all your DevOps problems along the way. Now, I must break your heart just a little: Buying new tools or apps will not, on their own, ensure positive DevOps outcomes.

A tool-first approach is ineffective. In fact, it may actually generate backlash from the individuals or groups who are invested in protecting the status quo and result in a wasted implementation effort. Imposing new methodologies or processes across teams through new tools can cause people to dig in, generating responses ranging from passive-aggressive to overtly destructive—both contributing to failure.

It Starts With Culture

Your success on the DevOps journey starts with culture. Culture transcends tools, methodologies and processes, making use of all but not relying on any. Culture was, and always will be, present, and much like continuous development—routinely referenced in DevOps—it is continuously influenced by everyone and everything in its environment.

The successful implementation of DevOps must begin with a conscious effort to reassess and evolve a company’s culture, supported by the tools. For example, continuous deployment as an initiative will only find traction when members of the team all believe, philosophically, it is best for the organization. It requires a certain kind of individual.

A culture of rapidly evolving and adapting teams focused on continuously discovering, developing and delivering demands a commitment to evolving toward a more contemporary integral DevOps toolset, not the other way around.

Culture Alignment as a Competitive Advantage

Cultural alignment is not a buzzword bingo box that gets checked off. One’s sense of alignment with a clearly articulated mission and set of values, coupled with a way of working in a team and supported by the right tools, generates velocity and efficiency. This alignment becomes a competitive advantage.

But, doing so demands all participants to be more conscious of how what they do affects everyone and everything else, in real time.

Culture Change is Difficult but Achievable

It can be very daunting or even scary if your company culture has a long history, producing images of the Borg or Hive, depending on your age bracket. Changing a culture is difficult. Here are some thoughts on where you might start.

Learning agility: Focus on learning to dance with VUCA (Volatility, Uncertainty, Complexity and Ambiguity) instead of being consumed by it.

Diversity: Commit to working with people with wide-ranging intellect, backgrounds, titles, CVs, life experiences, ways of learning, gender and energy, instead of working with your mirror.

Trust: Build a work environment based on psychological safety—a sense of confidence that the team will not embarrass, reject or punish someone for speaking up. Knowing takes time and deliberate effort.

Leadership: Adapt and promote a leadership style such as the 3 C’s of change leadership–communicate, collaborate and commit.

Coaching Culture: Consider championing a coaching subculture by applying a methodology such as LACE to work through difficult conversations:
- Listen to understand.
- Ask powerful questions.
- Challenge and support with feedback.
- Establish next steps and accountability.
Worth It

We started with a little heartbreak. But I have had enough conversations about DevOps success and failure to believe that you already knew it was not just about tools. We have all felt the weight of culture when it is not in alignment. Employing some or all of these recommendations will set you down the path of creating your own language that is a requirement for building a culture. Now stop looking at the next great messaging protocol and start working on building an agile, aligned culture.

— Joshua Platt
May 24, 2019
The Essential Toolbox for DevOps Analytics

With DevOps teams under constant pressure to improve software delivery cycles while maintaining—and improving—quality, many have turned to continuous testing (CT), implementing it as part of their overall development strategy. CT offers a number of benefits to DevOps teams, but it also introduces new challenges, specifically infinite amounts of data that require analysis. As more teams integrate CT into their strategy, it is clear that analyzing, understanding and filtering the results data quickly is critical to preventing bottlenecks in the DevOps process. The problem is, teams simply don’t have the time needed to analyze data.

Current research indicates that teams spend anywhere between 50 and 72 hours per regression cycle analyzing test results. This includes filtering out noise and assessing failures that may impact their software releases. However, the amount of time DevOps teams have to review and qualify test results has shrunk dramatically in recent years—from days to hours or even minutes, in some cases—leaving teams scrambling.

With pressure to accelerate velocity and deliver consistent, high-quality experiences, DevOps team require new analytics solutions to help them manage large test-results datasets in a CI/DevOps environment.

Let’s explore five essential tools that enable DevOps teams to quickly and efficiently analyze data, triage issues and act upon failures with the best possible insights.

Executive Dashboards

Dashboards have evolved immensely over the years, allowing dev managers and QA managers to easily examine the pipeline, see CI trends related to time, build health and more. When looking for the best solution, make sure it includes quality heat maps and CI dashboards, which make it easy to spot an anomaly in the CI pipeline and allow managers, DevOps teams and practitioners to quickly drill down into the single test report and the issue.

Single Test Report Visibility With Advanced Reporting Artifacts

With a single test report, practitioners have access to the entire flow of test steps, video, logs, screenshots and Jira–even the ability to drill down into the source code. This makes it quick and easy to locate an issue. For teams, these means more time fixing and less time searching.

Cross-Platform Reports

These analytics reports show the UI/UX simultaneously across multiple screen sizes, resolutions and platforms. Since the boundaries between the digital platforms need to be seamless, whether on desktop browsers or mobile devices, having the ability to see the UI/UX on multiple form factors (and layouts) at the same time is essential to assess overall quality.

DevOps Noise Reduction Tools Powered by AI and Analytics

Today, finding which error classifications are true application bugs is critical to efficiency. Noise reduction tools that are powered by AI allow teams to filter out problems caused by various issues that are not software defects—for example, device connectivity issues and wrong scripting. Having a tool with the ability to locate each failed test execution, the root cause and the classification category is a huge productivity boost for both the test engineer and the developer.

DevOps and Actionable Insights

Analyzing data is one thing, knowing what to do it with it is another. Today, there are a number of analytics solutions on the market that provide teams with both capabilities. Once an issue is detected, classified and reported, having deep insight brings practitioners halfway toward the resolution of the issue.

To succeed in software delivery, teams not only need better automation as part of their processes, test flows and CI/CD workflows, but also an analytics platform to manage all of their test results data. Since DevOps involves a collection of team members from all parts of the software delivery lifecycle (SDLC) process, the central platform needs to meet the needs of all team members. As you work to build your next test analysis toolbox, consider the five features above to efficiently evaluate the data, act upon it and deliver iterations and features with confidence.

— Eran Kinsbruner

May 23, 2019