devopsy.kinsta.cloud

Category: Doin’ DevOps

  • Continuous Testing Practices – Part 3

    Continuous Testing Practices – Part 3

    In my prior blog, Continuous Testing – The Quest for Quality at Speed, I described five tenets and some of the practices for continuous testing to help with understanding what continuous testing is. In my consulting work, I find it necessary to use 15 categories of practices to assess an organizations’ continuous testing capabilities. Given the large number of practices, I am publishing the complete list in three parts.

    This article is part three, which covers the following categories of continuous testing practices.

    • Security tests
    • Pre-flight testing
    • Integration testing
    • Pre-production testing
    • Test in production

    You can read part one here and part two here.

    Security Tests

    • Credentials and secrets needed to operate any test are vaulted.
    • Customer sensitive data in application databases used for testing is obfuscated from misuse by people involved in testing.
    • New security tests that are necessary to test a software change are created together with the code and integrated into the trunk branch at the same time the code is. The new tests are then used to test the code after integration.
    • Security tests for each DevOps pipeline stage are automated and may be selected automatically according to predefined criteria.
    • Test results that indicate possible security concerns are tagged for security analysis.
    • Dynamic or interactive application security tests exercise the application to check for security vulnerabilities. Results of these tests are delivered to developers through tools and feedback loops native to their organization.
    • Attack patterns, abuse cases and tests are built according to application module profiles.
    • Unit, functional and integration tests around—and especially outside—boundary conditions are tested. Tests include error handling, exception handling logic and negative tests.

    Pre-Flight Testing

    • Products are architected to support modular independent packaging, testing and releases. In other words, the product itself is partitioned into modules with minimal dependencies between modules. In this way the modules can be built, tested and released without requiring the entire product to be built, tested and released all at once.
    • Prior to committing software changes into the integration or trunk branch, pre-flight tests are conducted on the software change integrated with the tip of the integration branch within a private instance, to ensure the changes will not break the integration branch.
    • Requirements for testing are included in agile backlog planning procedures.
    • Pre-flight tests include unit testing, static code analysis scanning, functional, regression and performance tests. During this stage, feature flags or toggles may be installed in the software and each flag setting is verified to make sure the flags work. New automated tests that validate the changed software are created and tested, for use in later stages of the pipeline and subsequent regression tests.
    • Pre-flight tests, conducted prior to integration, include unit testing, static code analysis scanning, functional, regression, performance and security tests. During this stage, feature flags or toggles may be installed in the software and each flag setting is verified to make sure the flags work. New automated tests that validate the changed software are created and tested for use in later stages of the pipeline and subsequent regression tests.
    • Prior to committing software changes into the integration or trunk branch, pre-flight tests are conducted on the software change integrated with the tip of the integration branch within a private instance to ensure the changes will not break the integration branch.
    • Test-driven development (TDD) is used to ensure tests are created ahead of the code to be tested. TDD is most often applied for unit level tests but may also be applied for functional testing. A key advantage of TDD is that it ensures working tests are available for use in all pipeline stages. Without using TDD, there is a tendency for test creation to fall behind the code. Another advantage of TDD is that tests are created without the bias of knowing how the code was written, because it hasn’t been written yet.
    • From a testing point of view, microservices introduce the requirement to validate the contract of each service with any other services that use it. The independence of the microservice or any interservice dependencies need to be well tested. Performance and reliability considerations of operating services over a network need to be verified. If the application changes, affected microservices, or any dependent group(s) of microservices also need to be regression tested.
    • Containers offer the opportunity to package test resources in special test containers so they can be conveniently and immutably invoked on-demand for testing changes with elastic scaling benefits.
    • A strategy ensures that changes to a database, or changes to an application that uses a database, perform as expected, during each stage in the continuous delivery pipeline.

    Integration Testing

    • Software changes are tested incrementally by a cross-functional team over all stages of the continuous delivery pipeline.
    • During the code commit stage, assessments are conducted on the committed code to validate that the pre-flight test results satisfy acceptance criteria for integration into the integration branch.
    • During the build stage, a select group of tests are run on each build to ensure the integrated build satisfies build integration acceptance testing criterion.
    • Containers offer the opportunity to package test resources in special test containers so they can be conveniently and immutably invoked on-demand for testing changes with elastic scaling benefits.
    • API contract testing is used to test microservices interfaces.
    • Test code and data for each application are stored and maintained in test containers so they can be managed using the same container creation, orchestration and monitoring capabilities as the application.

    Pre-Production Testing

    • After the integration and build stage, functional, performance, regression, customer acceptance and security tests are conducted to ensure the images resulting from the build or set of builds satisfy functional, performance, security and delivery assessment criteria.
    • Testing during deployment include dark launch strategies such as A/B testing. feature flag rollouts, green/blue deployments and canary testing.
    • Release assessment criteria are defined and use test results to decide whether a release meets release acceptance criterion.

    Test In Production

    • Testing during deployment uses strategies that minimize the impact of failures such as A/B testing. feature flag rollouts, green/blue deployments and canary testing.
    • Automated tests verify each deployment.
    • Chaos engineering is used for testing the resiliency of applications, deployment infrastructure and pipelines.
    • Tests for disaster recovery and restoration of services, infrastructure and pipelines are conducted periodically.
    • Fire drills on applications, pipelines and infrastructures are used to test procedures for detection and recovery of applications, pipelines and infrastructures.

    What This Means

    Continuous testing (CT) practices are critical to success with DevOps. Many organizations underestimate both the importance and the practices needed to engineer Continuous Testing. This article is part of a three-part series that provides a comprehensive list of practices that can be used to understand and assess an organization’s Continuous Testing capabilities. To learn more about my blueprint for continuous testing, refer to my book Engineering DevOps.

  • Continuous Testing Practices – Part 2

    Continuous Testing Practices – Part 2

    In my prior blog, Continuous Testing – The Quest for Quality at Speed, I described five tenets and some of the practices for continuous testing to help explain what continuous testing is. In my consulting work, I find it necessary to use 15 categories of practices to assess an organizations’ continuous testing capabilities. Given the large number of practices, I am publishing the complete list in three parts.

    This article is part two, which covers the following categories of continuous testing practices:

    • Test automation
    • Test tools
    • Test infrastructure management
    • Test scripts
    • Test results

    You can read part one here.

    Test Automation

    • Test automation requirements are defined.
    • New automated tests are created for each new feature in compliance with the test strategies for the product or service.
    • Automated test cases are reviewed and tested to ensure they satisfy the purpose of the test.
    • Automated test cases are compliant with test case standards for the product or service.
    • Automated test cases are designed to run independent of each other.

    Test Tools

    • All testing tools in a pipeline are integrated into a common framework which enables orchestration of test resources and automation of test tasks.
    • A test framework supports automation of everything needed for testing including test cases, test tools, applications, test infrastructures, and test data.
    • A test framework can be invoked, controlled and released through an application programming interface (API) so that it can be integrated into a DevOps continuous delivery toolchain.
    • Features provided by test frameworks include capabilities to orchestrate the test infrastructure, select tests, control the execution of tests, monitor tests in progress and report test results, logs and verdict data.
    • A selection criterion is defined for test tools. The criterion includes requirements for coverage, automation features, support, compatibility with the automation framework, compatibility with test progress reporting systems.
    • Test tool software versions are maintained in a source version management system.
    • A common test automation framework is defined for the entire organization to ensure all test tools can integrate and be orchestrated with common test automation practices.
    • Test automation tools are available for testing all features of an application that can be automated.
    • Manual testing tools are available for testing features that must be tested manually. These tools can report results to a common test management reporting system.
    • Test authoring tools support input error checking and features which help encourage scripts development according to best practices. Examples of such features include the ability to create and use a repository of known good test utilities for control and analysis of the system under test.
    • DevOps-ready test tools are employed that easily integrate into a test framework and toolchain and elastically scale on demand vertically or horizontally, to match the varying capacity and workload demands of tests of software changes going through the continuous delivery pipeline.
    • DevOps-ready tools can be orchestrated, scaled, invoked, controlled and monitored from an API.

    Test Infrastructure Management

    • Standards for describing resources and topologies for test configurations are documented.
    • Techniques used to accelerate the time for executing tests are used such as vertical and horizontal scaling of test campaigns.
    • Test resources and topology files are maintained in a software version management system.
    • Orchestration and automation tests are conducted to verify the software that supports orchestration and automation layers satisfy their assessment criterion
    • The test infrastructure used for each stage in the pipeline is a replica of production configuration variations, as much as is practical.
    • Infrastructures for testing are elastic, with the ability to stand up and release (i.e. orchestrate) infrastructure configurations as needed for use with specific tests. Typical techniques that are employed to orchestrate test environments include dynamic infrastructure configuration tools, infrastructure-as-code (IaC) strategies, cloud services, testing-as-a-service (TaaS) providers and use of tools to stand up and release applications and test resources packaged in immutable containers.
    • Database migration, replication and restore tools also need to be tested to ensure they can function quickly enough to keep up with the speed of the DevOps continuous deployment pipeline cadence.
    • Health checks on the test environment are used to reduce the chance of false negative test verdicts that could be caused by test environment failures.

    Test Scripts

    • Test scripts are maintained in a software version management system.
    • Guidelines for test scripts and test suites practices are documented.
    • Test utilities, tests and test suites follow guidelines for documentation, test results reporting, reuse and maintenance practices.
    • Test script peer reviews are conducted to encourage collaboration and to ensure test script best practices are used.
    • Each test script has a unique identification.
    • The test type of the script is identified (for example, sanity test, build test, smoke test, security, acceptance test, conformance test, functional test, unit test, patch test, bug fix test, integration test, regression test, performance test, system test, other).
    • Evidence, including an execution log, demonstrates that each verdict of tests work.
    • Tests determine at least one verdict, which may be ‘Pass,’ ‘Fail’ or ‘Inconclusive.’
    • Verdict and assertions priorities are recorded or reported.
    • Tests are data-driven. Data inputs control execution of the test.
    • Each test script is is tagged with a codified risk level.
    • Failure verdicts reported by the script include a tag for risk priority.

    Test Results

    • Techniques are used to ensure test results analysis keeps up with accelerated tests. For example, test design techniques, analysis tools built into test frameworks, test results dashboards and/or test results analyzers that run in parallel with the tests.
    • Quality assurance goals include specific thresholds for test coverage prior to release such as run-to-plan (RTP) and pass-to-plan (PTP) minimum thresholds. RTP is a measure of how many tests of the total planned tests have completed. PTP is a measure of how many of the completed tests pass. For example, 95% RTP and 90% PTP.
    • Quality goals include a measurement and threshold for reliability. For example, S-curves for pass rates can indicate reliability when the pass rate stabilizes for a period. Example: 95% pass rate for three consecutive test cycles.
    • Time to recover a failed service is routinely tested and used to calculate mean-time-to-recover (MTTR).
    • Monitoring and management tests are conducted to verify monitoring and management capabilities of the pipeline satisfy their assessment criterion.
    • Technical and business metrics are designed to monitor the success of DevOps testing for technical continuous improvement and to justify business cases needed to implement those improvements.
    • Input and output metrics and acceptance criterion exist for each stage in the value stream.

    What This Means

    Continuous testing (CT) practices are critical to success with DevOps. Many organizations underestimate both the importance and the practices needed to engineer continuous testing. This article is the second in a three-part series that provides a comprehensive list of practices that can be used to understand and assess an organization’s continuous testing capabilities. To learn more about my blueprint for continuous testing, refer to my book Engineering DevOps.

  • Continuous Testing Practices (Part 1)

    Continuous Testing Practices (Part 1)

    In my prior blog, Continuous Testing – The Quest for Quality at Speed, I described five tenets and some of the practices for continuous testing (CT) to help those understand what continuous testing is. In my consulting work, I find it necessary to use 15 categories of practices to assess an organizations’ continuous testing capabilities. Given the large number of practices, I am publishing the complete list in three parts.

    This article is part one, which covers the following categories of continuous testing practices:

    • Continuous testing tenets
    • Leadership
    • Culture
    • Test strategy and plans
    • Test management

    Continuous Testing Tenets

    • The continuous testing principle “shift left” is practiced, in which tests are conducted as early in the pipeline as possible.
    • The continuous testing principle “fail early” is practiced, in which tests are arranged so that the most likely problems are found in the early stages of the pipeline.
    • The continuous testing principle “fail often” is practiced, in which tests are run frequently and with many different conditions such as different variations of operating systems, production topologies, browsers and production versions of services.
    • The continuous testing principle “test fast” is practiced, in which tests are run in quick cycles to get to a verdict as quickly as possible.
    • The continuous testing principle “be relevant” is practiced in which tests are focused on the most important tests and results for each purpose and stakeholder.

    Leadership

    • Leaders establish a definition and vision for software quality.
    • Leaders establish a definition for test coverage.
    • Leaders clarify how everyone’s roles fit with the CT strategy.
    • Leaders establish incentives for CT performance.
    • Leaders proactively sponsor quality assurance testing as a strategic element of their budgets and operations rather than viewing testing as a cost to avoid. This includes budgeting time and money for DevOps test training, test resources, test frameworks, test tools, test management and establishing policies for assessments.
    • Operational and capital expense budgets are planned together with all stakeholders to ensure all stakeholders agree there is enough opex and capex budget for the organization to accomplish quality goals for the budget period being planned.
    • Cost accounting practices ensure the capex and opex expenditures are tracked and made visible to stakeholders.
    • Schedules for CT improvement projects are identified and tracked.
    • Skills development programs provide individuals and teams opportunities to learn CT practices.
    • The performance of continuous quality assurance programs are monitored and made visible to stakeholders.
    • Leaders participate in creating and maintaining guidelines for codifying and tagging business risk priorities.

    Culture

    • Quality is everyone’s concern. Responsibility for attaining high quality software products is shared by everyone on a cross-functional team.
    • Cross-functional team members collaborate for testing. This reduces confrontation and friction between people who find failures and those who fix the code.
    • Dev teams embrace the creation of good tests and analysis of test results.
    • Ops team members contribute to test planning, test creation, test execution and test results analysis.
    • Policies for test creation and test coverage are agreed upon by the cross-functional team including Dev, QA, security and Ops team members.
    • A test architect is a role within the organization. The role has expertise in CT practices and helps cross-functional teams understand and apply them.

    Test Strategy and Plans

    • Test strategies and plans are created for each product or service and reviewed by cross-functional teams including development, QA, Ops and security teams.
    • Test strategies and plans identify the type of tests, and the extent of testing activities, for each stage in the value stream.
    • Test requirements and tasks are included in agile planning procedures for each theme, epic and user story.
    • Test strategies and plans define a test coverage standard, test case standards, and input/output criterion for each pipeline gate.
    • Test plans identify the test cases and all the supporting requirements to implement and execute the test cases as required satisfying the test strategy and test management goals.
    • Test strategies and plans identify the approach for system and acceptance testing.
    • Test strategies and plans identify any tests that are to be conducted in production.
    • Test strategies and plans identify any tests that are to be conducted for security testing.
    • Test strategies and plans identify a target for test automation and manual testing for each type of testing. For example, 100% of all tests in the integration stage and 85% of test in the pre-production stage is a typical goal.
    • The test strategy requires that changes to a database or changes to an application that uses a database have tests defined for the database and the application use of the database.

    Test Management

    • All aspects of test management, such as test planning, test guidelines, test code version management and version management of test results are handled by tools that are well-integrated into the continuous delivery pipeline and any test framework being used for test automation.
    • Schedules for all test tasks are identified and tracked.
    • New features have a scheduled task to ensure test goals are met for each new feature.
    • Each test case has associated attributes that can be used to find tests according to the attributes. For example, an attribute could be type of test and the feature purpose of the test.
    • Test environments and resources needed to satisfy requirements to run each test are identified and available programmatically for scheduling purposes.

    What This Means

    Continuous testing practices are critical to success with DevOps. Many organizations underestimate both the importance and the practices needed to engineer continuous testing. This article is the first in a three-part series that provides a comprehensive list of practices that can be used to understand and assess an organization’s continuous testing capabilities. To learn more about my blueprint for continuous testing, refer to my book Engineering DevOps.

  • Learning ‘The Third Way’ of DevOps – Continuous Improvement

    Learning ‘The Third Way’ of DevOps – Continuous Improvement

    I refer to the third way of DevOps–continuous experimentation and learning, as simply continuous improvement. In some sense, there is no way to learn the third way of DevOps because it is, by definition, a moving target. It does make sense to discuss what is needed to create a continuous learning organization for DevOps continuous improvement that builds on mastery of the first way and second way of DevOps.

    As indicated in my blog Learning ‘The Second Way’ of DevOps – Continuous Feedback and my prior blog Learning ‘The First Way’ of DevOps – Continuous Flow, it is clear many organizations are struggling to realize well-engineered DevOps, even the first way of DevOps, continuous flow.

    The first way of DevOps, continuous flow, is the foundation for DevOps. Once mastered, an organization will naturally want to master the second way of DevOps to achieve the higher level of benefits that continuous feedback offers, when engineered well. Once the second way of DevOps is achieved, then an organization will naturally want to master the third way of DevOps, to sustain and grow their DevOps capabilities.

    DevOps is an evolving body of knowledge. There is no one-stop shop that covers all roles and the complete range of skills required for DevOps. A DevOps-competent workforce requires an ongoing commitment to DevOps skills development and mastery. Enterprises must approach DevOps training strategically to leverage training resources to their best advantage.

    Learning Continuous Improvement

    There are no training resources dedicated solely to the third way of DevOps—continuous improvement. Many of the same resources as the first way of DevOps and the second way of DevOps do have some relevant information, though. Analysis of DevOps results, third-party research sources, industry sources such as publications, articles, conferences and tool vendors roadmaps for DevOps and related frameworks including site reliability engineering (SRE) and AIOps are good sources of information that stimulate self-directed teams to seek the training information that is most relevant to the organization’s DevOps improvements.

    Why Is Learning Continuous Improvement Important to Engineering DevOps?

    The third way of DevOps—continuous improvement—requires an organizational commitment to continuous learning and mastery at the most advanced level. Typical skills needed include implementation and operation of advanced practices for microservices design and operations, advanced continuous testing, multi-pipelines, value stream management, SRE, observability, DevSecOps and AIOps. Retention and development of highly skilled and hard to find experienced DevOps professionals and SREs makes it important to develop skills in-house.

    How Is Learning Continuous Improvement Accomplished?

    The following are recommended engineering practices for learning continuous improvement:

    • A continuous learning culture encourages individuals to continuously gain knowledge and exchange experiences and ideas for DevOps and SRE.
    • Mentoring is an explicit program.
    • Retrospectives are structured to thoroughly analyze significant, unexpected events and determine solutions that will prevent the events in the future.
    • An active online library of DevOps and SRE training resources is maintained that tracks the latest and most innovative approaches to DevOps and SRE.
    • DevOps and SRE teams evolve their skills and advance their careers by engaging with industry thought leaders to continuously update training content.
    • Partner with trusted education and training partners ensure the highest-quality learning content and tools are made available to DevOps and SRE teams.
    • A portion of DevOps and SRE teams time is allocated for training and education.
    • Cross-team sharing of recommended engineering practices is proactively pursued using the concepts of Yokoten—proactive sharing of practices between teams.

    What Is Needed to Implement Learning Continuous Improvement?

    The following are recommended engineering practices for implementing a learning program for continuous improvement.

    • Assign roles and responsibilities to experts for defining a training program for DevOps continuous feedback (e.g., DevOps sensei, training PM, evangelist).
    • Conduct a continuous feedback DevOps skills gap assessment.
    • Define a training strategy that best address the gaps and fits the needs and priorities of the organization.
    • Define a training budget that includes expenses and time allocations per role and skill.
    • Determine which skills and roles can be covered by inexpensive self-study resources.
    • Determine which skills and roles warrant more costly commercial training.
    • Identify training goals for each role.
    • Ensure recruiting job profiles match skill requirements for DevOps skills that are most relevant to the organization’s strategic goals.
    • Create a continuous learning culture that encourages individuals to continuously gain knowledge and exchange experiences and ideas for DevOps and SRE.
    • Build an active online library of DevOps and SRE training resources that tracks the latest and most innovative approaches to DevOps and SRE.
    • Engage with industry thought leaders to continuously update con tent for DevOps and SRE teams to evolve their skills and advance their careers.
    • Partner with trusted education and training partners to ensure the highest-quality learning content and tools are made available to DevOps and SRE teams.
    • Put in place means to make training resources visible and available.
    • Set training goals for teams and individuals.
    • Put in place incentives and recognition programs for training.
    • Track training progress.

    Some of the best sources of DevOps training resources are listed below:
    1. The DevOps Institute
    2. Partners of The DevOps Institute, such as Learning Tree International and Xellentro.
    3. Udemy.com
    4. Recorded talks and publications of IT Revolution
    5. Books such as my own Engineering DevOps, by Marc Hornbeek.

    What This Means

    The third way of DevOps—continuous improvement—requires an organizational commitment to continuous learning and mastery at the most advanced level. Typical skills needed include implementation and operation of advanced practices for microservices design and operations, advanced continuous testing, multi-pipelines, value stream management, SRE, observability, DevSecOps and AIOps. Analysis of DevOps results, third-party research sources, industry sources such as publications, articles, conferences, and tool vendors roadmaps for DevOps and related frameworks including site reliability engineering (SRE) and AIOps are good sources of information that stimulate self-directed teams to seek the training information that is most relevant to the organization’s DevOps improvements.

    Organizations must approach DevOps training strategically and continuously to leverage training resources to best advantage. To learn more about my blueprint for learning all things DevOps, refer to my book Engineering DevOps.

  • Learning ‘The Second Way’ of DevOps – Continuous Feedback

    Learning ‘The Second Way’ of DevOps – Continuous Feedback

    As indicated in my prior blog Learning the ‘First Way’ of DevOps–Continuous Flow, many organizations are struggling to realize well-engineered DevOps, even the ‘first way’ of DevOps – continuous flow.

    The first way of DevOps, continuous flow, is the foundation for DevOps. Once mastered, an organization will naturally want to master the second way of DevOps to achieve greater benefits that continuous feedback offers when engineered well. DevOps is an evolving body of knowledge. There is no one-stop shop that covers all roles and the complete range of skills required for DevOps. A DevOps-competent workforce requires an ongoing commitment to DevOps skills development and mastery. Enterprises must approach DevOps training strategically to leverage training resources to their best advantage.

    Learning Continuous Feedback

    There are no training resources dedicated solely to the second way of DevOps—continuous feedback. Many of the same resources used to learn the first way of DevOps have some relevant information, though. DevOps monitoring, release automation and security tools publications, articles and tool vendors are a good source. A DevOps expert who is experienced with mature DevOps can select the training information that is relevant to the specific continuous feedback tools and feedback metrics used by the enterprise.

    Why This is Important to Engineering DevOps

    Continuous feedback requires a mature understanding of DevOps concepts and practices. Specific skills are needed for implementation and operation of advanced monitoring tools and practices for application performance, database performance, advanced continuous testing, advanced release management, infrastructure performance and continuous security.

    How Is Learning Continuous Feedback Accomplished?

    The following are recommended engineering practices for learning Continuous Feedback.

    Roles and responsibilities are assigned to experts for defining a training program for DevOps continuous feedback (e.g., DevOps sensei, training PM, evangelist). Continuous feedback DevOps skills gap assessments are conducted periodically. Dev and Ops training strategies are designed to address gaps and priorities of the organization for DevOps continuous feedback. A training budget that includes allocations of expenses and time allocations per role exists to support continuous feedback. Continuous Feedback goals and SLAs are defined for each role.

    Recruiting job profiles match skill requirements for DevOps skills relevant for continuous feedback. Training resources are available and visible to those that need continuous feedback training. Training goals set for teams and individuals include goals for continuous feedback. Incentives and recognition programs support Continuous Feedback training. Continuous feedback training progress is tracked and visible.

    What Is Needed to Implement Learning?

    The following are recommended engineering practices for implementing a learning program for continuous feedback.

    Assign roles and responsibilities to experts for defining a training program for DevOps continuous feedback (e.g., e.g., DevOps sensei, training PM, evangelist). Conduct a continuous feedback DevOps skills gap assessment. Define a training strategy that best addresses the gaps and fits the needs and priorities of the organization. Define a training budget that includes expenses and time allocations per role and skill.

    Determine which skills and roles can be covered by inexpensive self-study resources. Determine which skills and roles warrant more costly commercial training Identify training goals for each role. Ensure recruiting job profiles match skill requirements for DevOps skills that are most relevant to the organization’s strategic goals.

    Put in place the means to make training resources visible and available. Set training goals for teams and individuals. Put in place incentives and recognition programs for training. Track training progress.

    Some of the best sources of DevOps training resources are listed below:

    1. The DevOps Institute
    2. Partners of The DevOps Institute, such as Learning Tree International and Xellentro
    3. Udemy.com
    4. Recorded talks and publications of IT Revolution.
    5. Books such as my own “Engineering DevOps,” by Marc Hornbeek.

    What This Means

    To accomplish the benefits of the second way of DevOps—continuous feedback requires a commitment to continuous learning and mastering skills beyond what is needed for the first way of DevOps–continuous flow. Specific skills are needed for implementation and operation of advanced monitoring tools and practices for application performance, database performance, advanced continuous testing, advanced release management, infrastructure performance and continuous security. Organizations need a trained workforce to succeed with DevOps. Organizations must approach DevOps training strategically and continuously to leverage training resources to their best advantage. To learn more about my blueprint for learning all things DevOps, refer to my book Engineering DevOps.

  • Make a Plan for Test Automation

    Make a Plan for Test Automation

    The amount of testing that we could be doing is massive. Most of us don’t look at testing across the spectrum and all-inclusively, but let’s do that for a second.

    We have functional testing at the code level, which is reasonably well automated already, if your shop is using such automation. Then we have integration testing, which is a superset collection of API testing, code integration testing, usability and regression testing. Integration testing is automated, and sometimes that’s done well, but this massive space is still growing. Next up would be security testing, which covers much of the same code that other testing is interested in from a strictly security focused perspective. Then, there is actual usability testing, which is more important for some areas than others—embedded needs far less usability testing than web apps, for example—and has different levels of automation at this point in time.

    Really, when you think about it, the complexity of this environment is massive. We not only need to test an array of different functionalities from an array of different perspectives, but we have to do it on an array of platforms. Testing on Windows is different than testing on Linux; testing in the cloud can use different tools than testing in-house; clients can be on any given platform, depending on the application and target market … we really do have a beast, here.

    And testers have always been behind, or have not covered things well. This is not a ding on testers, this is the result of the complexity at hand and the fact that, in many instances, one line of code can go wrong in a bunch of different ways.

    Enter automation, which is growing rapidly in the world of continuous integration (CI). Automated testing tools range from broad to very focused, and all help facilitate testing. The DevOps increase in development means more testing, and testing was already behind. This has been discussed pretty thoroughly in the past, but it finally feels like we’re getting to a point where we have viable solutions to the problem. If testers become test developers that review results of automated tests, and through CI those automated tests are increasingly worked into the process of development, we’ll see testers getting better coverage and quality seeing a continued increase.

    But that does mean you should keep your eyes on the market as it continues to grow, and pick tools that offer the most functionality that is most useful to your test team. And get them up and running. More testing means more work setting up tests. Even no-code test solutions for UI require someone to record the actions that are necessary to complete the test, and that takes someone’s time. But the return is long-term testing that doesn’t require a person until a red flag is raised by automation.

    I’m all for it. Both testing and security have the most to gain from automation, because they’ve been the most behind historically, and corner-cutting has cost organizations dearly. So, let’s get with it; check out some new test tools (there are plenty listware articles giving you a place to start), and let’s make great software better!

  • How to Build a People-First Engineering Team

    How to Build a People-First Engineering Team

    When we think about successful technology teams, there is a tendency to primarily focus on the technology being created rather than the people who are creating it. While both approaches can lead to successful products, a technology-first approach is often the default when software engineers become managers without necessarily having the right tools to manage effectively. Becoming an engineering manager requires an entirely new skill set that involves different communication styles, time management, organization, tools and software, scoping work, learning how to effectively delegate and identifying the right people to complete the team.

    As a CTO, and having gone through this transition, I have noticed that the more a manager is able to be in sync with the people on their team, the less burnout they experience, the higher the employee retention rates and the better work their employees will do.

    While most people won’t be entirely people- or technology-focused, here is a quick overview of the differences:

    • A team-oriented manager knows the personal and professional struggles their people are experiencing, and there’s a sense that the team they run is all working together on a mission toward the same goal. When you talk to this type of manager, they will naturally gravitate towards giving status updates on the wellbeing of individuals.
    • Managers who tend to see the team through a more technical lens will instead identify parts of the project or technological or strategic roadblocks that are impacting the team.

    The same issues will be dealt with very differently by these different types of managers. It can be beneficial to record yourself when you are talking about your team to identify where your focus is, and evaluate if that feels right for you.

    Building Effective 1:1 Communication with Team Members

    An individual contributor is normally accustomed to arriving at weekly 1:1s and working with their manager to solve any problems they may have. When they become a manager, it becomes their role to coach team members to solve their own problems, in addition to building trust. This role change takes preparation, and, if you are unsure how to do it well, you may find it helpful to draft out an agenda that you can follow if needed. You can also gain knowledge about this process by talking to more experienced managers.

    What do you talk about in your 1:1 meetings? Even if you are more a technology-oriented person, 1:1 meetings with your team should be less about the status of projects and more about how the person you are speaking to is doing and how you, as a manager, can help. Having started as a software engineer myself, and now as a CTO, I have found the below questions to be helpful as I navigate how best to help my team thrive in their roles.

    1. “How are you doing?” If you are speaking with a team manager, another great question is, “How is your team doing?” You want to genuinely understand the status of the individual outside of the technical aspects of the job.
    2. “Do you have any feedback for me?”
      I always ask for feedback. For a while, they may not have any; but I often find they start with a small one like “Better options for snacks.” This is an excellent opportunity to build trust by improving the work experience. The next time, they will probably share more until you receive really important feedback, while also improving the relationship. The important thing to remember, here, is to respond to the feedback.
    3. “On a scale of zero to 10, how much fun are you having at work?”
      I really want people to have fun at work. You can ask them to give you a number between zero and ten. When they respond with a number, say, six, a great follow-up question can be to ask what is needed to change the situation so that number becomes a seven. How can you help get them there? I have heard a wide spectrum of answers to this question, ranging from easy to complex; from things like permission to be unresponsive for an hour a day to the desire to learn the skills needed to do a different type of job.
    4. “Where do you see yourself in the next two years and 10 years?”
      Helping your team define both long and short term goals is a really important part of being a manager. Specifically for engineers, it is a great idea to think about if you want to continue moving up in an engineering role, or if making the switch to a managerial role is a better fit.
    5. “How can I help you get to the next level in your career?”
      It is common to get sidetracked when balancing your day-to-day work with your personal life. You can also use 1:1 meetings to help your team members identify what they can do to hit their short- and long-term goals, and how to maintain a better work-life balance.

    Supporting Colleagues

    This last year has been tough for most people, to varying degrees, which has solidified the idea that we need to be there for team members during difficult times; this goes beyond the pandemic. That can mean sending their favorite meal via delivery, sending a care package or simply making it very clear that when someone on your team needs to step out or take time off, that is fine – no questions asked. To be able to be offline and not need to do a long handoff is so important when you are going through a hard time.

    During this move to remote work, as in many leadership teams, my company has gone against the cultural norm of keeping life and work separate. We have acknowledged that it is too difficult to do that right now, and very publicly recognized that everyone is going through something. It has felt like mental bandwidth is filled up with things going on outside of work, so as team members, it is good to make sure work doesn’t add to that pressure.

    In a way, this pandemic period has broken down barriers between work and “life” that has made it almost obligatory to focus on people first. From my perspective, this has been a breakthrough in how we can do business moving forward. By thinking about each individual on your team, everyone can be uplifted and heard, which can be the impetus to more innovation. If you are operating at your max with no slack in your system and little support from managers and teammates, you lose creativity. As innovative teams, this experience has been a reminder to build people-first principles into our cultures to promote a healthy work-life balance that lends itself to innovative solutions and better results.

  • Learning ‘The First Way’ of DevOps – Continuous Flow

    Learning ‘The First Way’ of DevOps – Continuous Flow

    Many organizations are struggling to realize well-engineered DevOps, even ‘the first way’ of DevOps – continuous flow.

    DevOps is an evolving body of knowledge. There is no one-stop shop that covers all roles and the complete range of skills required for DevOps. A DevOps-competent workforce requires an ongoing commitment to DevOps skills development and mastery. Enterprises must approach DevOps training strategically to leverage training resources to their best advantage.

    Learning Continuous Flow

    The first way of DevOps, continuous flow, is the foundation for DevOps.

    Why Is Learning Continuous Flow Important to Engineering DevOps?

    A knowledgeable, skilled workforce is needed to meet strategic objectives. Without a solid understanding of DevOps concepts and terminology, communication between people and teams is inefficient and a source of bottlenecks and misunderstandings. Developing hands-on skills for the tools and workflows is needed to understand and implement DevOps pipelines. There is an industry-wide shortage of skilled DevOps candidates. New skills foster innovation. Knowledge workers reduce production costs and reduce mistakes. Having skilled coworkers makes for a satisfying working environment. Employee retention is better. Competitive advantages are derived from knowledge.

    How Is Learning Continuous Flow Accomplished?

    The following are recommended engineering practices for learning continuous flow.

    People are encouraged to be continuous learners with an ongoing commitment to DevOps skills development. DevOps training is approached strategically to ensure DevOps training is well aligned to business goals. DevOps skills gap assessments are conducted and used to define and refine the training strategy. Leaders and cross-functional teams are encouraged to take DevOps fundamentals training. Leaders define a training strategy that best fits the needs and priorities of the organization to meet critical skills gaps.

    The training strategy considers which skills and roles can be covered by inexpensive self-study resources. Determine which skills and roles warrant more costly training. DevOps training has a distinct budget. Compliance to DevOps training programs are tracked. DevOps leadership training is offered to management and staff that have a role relevant to DevOps. DevOps tools training includes the recommended engineering practices for using each DevOps tool in the Service Catalog. Staff are encouraged and supported to take DevOps certifications that are relevant to their assigned roles.

    A DevOps training program is defined and includes a recommended curriculum for each role including leaders, developers, QA, Ops, and infrastructure and security staff that are operating in a DevOps environment. Recommended DevOps training resources are cataloged, and the catalog is visible for all stakeholders. Merit and recognition programs encourage employees to master DevOps skills.

    All staff is allocated time for DevOps training. DevOps mentoring programs supplement formal training. Job swapping and immersive training is encouraged and planned. People are encouraged to attend external conferences and meetups and share what they learn with other team members.

    What Is Needed to Implement Learning for Continuous Flow?

    The following are recommended engineering practices for implementing a learning program for continuous flow.
    Assign roles and responsibilities for DevOps training leadership (e.g., DevOps Sensei, Training PM, evangelist). Conduct a DevOps skills gap assessment. Define a training strategy that best addresses the gaps and fits the needs and priorities of the organization. Define a training budget that includes expenses and time allocations per role and skill.

    Determine which skills and roles can be covered by inexpensive self-study resources. Determine which skills and roles warrant more costly commercial training. Identify training goals for each role. Ensure job recruiting profiles match skill requirements for DevOps skills that are most relevant to the organization strategic goals. Put in place the means to make training resources visible and available. Set training goals for teams and individuals. Put in place incentives and recognition programs for completion of training.

    Some of the best sources of DevOps training resources are listed below:

    1. The DevOps Institute
    2. Partners of The DevOps Institute, such as Learning Tree International and Xellentro
    3. Udemy
    4. Recorded talks and publications available through IT Revolution
    5. Books such as my own Engineering DevOps, by Marc Hornbeek.

    What this Means

    Organizations need a trained workforce to succeed with DevOps—even ‘The First Way’ of DevOps—continuous flow. Organizations need to approach DevOps training strategically to leverage DevOps training resources to their best advantage. To learn more about my blueprint for learning all things DevOps, refer to my book Engineering DevOps.

  • 7 Step Transformation Blueprint for SecDevOps

    7 Step Transformation Blueprint for SecDevOps

    The struggle for continuous software security is obvious, but the solutions are not.

    As I indicated in my prior blog, SecDevOps is the Solution to Cybersecurity, a security-first mindset, coupled with SecDevOps-specific practices, provides an opportunity to achieve true continuous security. But, in reality, how can an organization accomplish SecDevOps?

    This blog explains how to apply my 7 Step Transformation Blueprint to SecDevOps.

    Leadership and engineering solutions require the persistent, methodical application of skills and practices toward leading and designing solutions that achieve business and team goals, including continuous security. While driven by visionary ideals, engineering requires practical, disciplined, progressively refined implementations using carefully chosen dimensions of people, process and technology solutions. At any point in the engineering life cycle, the goal is to achieve a balanced solution while evolving practices towards maturity.

    The seven step transformation engineering blueprint prescribes seven steps for achieving and continuously refining digital transformation methodically, no matter what the goals or current level of maturity are. The seven steps are visioning, alignment, assessment, solution, realize, operationalize and expansion. Each step considers the people, process and technology aspects of the transformation.

    Step One – Visioning

    Top leaders define a strategic vision for the digital transformation for the organization including a motivating vision statement, measurable goals, team values, and major implementation tactics. Identify senior sponsors that will own the transformation at the strategic level. Include key partner organizations that need to be strategically aligned to the transformation. In a SecDevOps transformation, a vision for a security-first mindset and associated SecDevOps practices are called out as the highest priority for implementation tactics supporting the vision.

    Step Two – Alignment

    Leaders and key team members who are most important to the implementation of the transformation align specific measurable goals and tactics for selected “model” applications. Specific measurable goals around continuous security are set in this step.

    Step Three – Assessment

    For the current state of selected applications, capabilities are discovered and assessed, deep-dive assessments are conducted for specific topics, and a current state value stream map is created relative to the organization’s goals. My earlier blog, DevSecOps Practices Gap Assessment, explains my recommended approach for conducting an assessment for security.

    Step Four – Solution

    An expert team performs analysis of assessment data and formulates a future state value stream roadmap including themes, epics and user stories and obtains alignment with key stakeholders. My earlier blog, 9 Pillars of Continuous Security Best Practices, outlines a comprehensive set of practices to consider when building a roadmap for any continuous security solution.

    Step Five – Realize

    Proof of concept (POC) trials are conducted to validate solution choices. Trials of security tools and integrations of those tools into the SecDevOps platforms also would be conducted during this step. The solution is validated with selected applications and use cases. Training is conducted as the solution is deployed to production. Governance practices for the new solution are activated.

    Step Six – Operationalize

    Deployed improvements are monitored and controlled with metrics. Retrospectives are conducted to create actionable prioritized lessons learned for continuous improvement. Chris Tozzi’s article 6 DevSecOps Metrics for DevOps and Security Teams to Share suggested metrics that can be developed and leveraged, both for this step and to drive ongoing improvements as the use of SecDevOps practices expand.

    Step Seven – Expansion

    Once continuous flow (the first way of DevOps) is realized for a select set of applications, the organization can safely expand the solution(s) to other applications across the organization. Further transformation cycles will lead to realization of continuous feedback (the second way of DevOps) and continuous improvement (the third way of DevOps) and apply it to SecDevOps.

    What This Means

    SecDevOps strategies and solutions are complex. The seven step transformation blueprint described in this blog can help organizations build a strategy and implement SecDevOps as an important part of their digital transformation. To learn more about how to apply the blueprint to continuous security (and other key elements of digital transformation) refer to my book Engineering DevOps.

  • Le Morte d’Arthur— A DevOps Engineering Journey

    Le Morte d’Arthur— A DevOps Engineering Journey

    The stories of King Arthur have fascinated me from the time I was young. Now that I am old DevOps_the_Gray, I enjoy relating the story of King Arthur to my experiences with DevOps. I hope you find this blog interesting and relevant too.

    Imagine a boy raised in Britain by the wizard Merlin who becomes king because he can pull the magic sword Excalibur from a stone. Once king, he unites warring tribes, forms a brotherhood of knights and creates a round table to be sure everyone who sits around it has equal status as they pursue the quest for the Grail.

    DevOps as the Grail of Software Creation and Delivery

    To engineer DevOps, even the first way of DevOps—continuous flow (harmony)—requires faithful employment of leadership (King Arthur), collaborative team culture (the Knights of the Round Table), disciplined engineering processes (laws of Camelot), technology (Excalibur) and automation (the magic of Merlin, Archimedes and Nimue). The ultimate pursuit of the third way of DevOps—continuous improvement (quest for the lost Grail)—requires persistent commitment to engineering discipline and noble practices.

    DevOps also requires a persistent evangelist (Nimue, the Lady of Avalon) and leadership (Morte d’Arthur) to be forever vigilant and ready to lead DevOps’ evolution into the future (King Arthur is forever ready to rise again with Excalibur).

    Both King Arthur and DevOps depend on the virtuous tenets of chivalry/culture, laws/ processes and use of Excalibur/technology.

    Arthurian tales represent a masterful integration of historical events with misty, imagined magical episodes and, like DevOps, embody tenets of chivalry and respectful moral leadership. While the means can be elusive, the results can be magical. The sword Excalibur and the search for the Grail are powerful symbols that remind me to always try to engineer technology for good and noble purposes.

    The Arthurian Analogy Expresses the Importance of DevOps

    “Then there entered into the hall the Holy Grail covered with white samite, but there was none might see it, nor who bare it.” —Le Morte d’Arthur, BOOK XIII CHAPTER VII

    “…said Sir Gawaine, we have been served this day of what meats and drinks we thought on; but one thing beguiled us, we might not see the Holy Grail, it was so preciously covered. Wherefore I will make here avow, that to-morn, without longer abiding, I shall labour in the quest of the Sangreal.” —Le Morte d’Arthur, BOOK XIII CHAPTER VII

    Like the Grail, DevOps is too often regarded as an ethereal thing, earnestly sought after and pursued. To do DevOps well means treasures and miraculous benefits; satisfying, harmonious production; and competitive advantages in abundance.

    Like Arthur’s battles, hard work and skills are required for success with DevOps.

    “King Arthur rode in the battle exhorting his knights to do well, and himself did as nobly with his hands as was possible a man to do; he drew out Excalibur his sword, and awaited ever whereas the Romans were thickest and most grieved his people, and anon he addressed him on that part, and hew and slew down right, and rescued his people; and he slew a great giant named Galapas, which was a man of an huge quantity and height, he shorted him and smote off both his legs by the knees, saying, Now art thou better of a size to deal with than thou were, and after smote off his head. There Sir Gawaine fought nobly and slew three admirals in that battle. And so did all the knights of the Round Table.” —Le Morte d’Arthur, BOOK V CHAPTER XII

    DevOps, At its Best, is Engineering Greatness

    In my experience, well-engineered DevOps always starts by defining a great goal. Greatness is won by careful, inspirational leadership, measured tactics, superior technology, a culture of collaboration, courage and unwavering persistence in applying skills to the task until the goal is attained. To engineer is to do the noble work of engineering. DevOps engineering requires the skills of elite leaders and engineers that have a profound understanding of DevOps practices.

    DevOps’ Enduring Legacy and Future

    “Yet some men say in many parts of England that King Arthur is not dead, but had by the will of our Lord Jesu into another place; and men say that he shall come again, and he shall win the holy cross.” —Le Morte d’Arthur, BOOK XXI CHAPTER V-VII

    DevOps implementations – and DevOps itself – are not immortal.

    As DevOps leaders, experts, and evangelists come and go, the perceived value of DevOps may wax and wane. The fame of innovative new technologies expires ever more quickly with the acceleration of technological systems facilitated by stacking technologies higher and higher.

    Epilogue

    The stories of King Arthur teach us to lead passionately when needed to triumph against DevOps challenges and to continue the pursuit of perfection. I hope you find this Arthurian DevOps analogy as relevant and interesting as I do, and I hope that it adds a little uniqueness and fun to your DevOps understanding. If you would like to see more on this analogy, I refer you to the only DevOps book with the sword Excalibur on the cover. Now, on with the quest of engineering DevOps!

    Marc Hornbeek – DevOps_the_Gray, esq.