devopsy.kinsta.cloud

Tag: Engineering DevOps

  • Revolutionizing the Nine Pillars of SRE With AI-Engineered Tools

    Revolutionizing the Nine Pillars of SRE With AI-Engineered Tools

    In my blog Rapid Strategic SRE Assessments Accelerate IT Transformations published last year, I classified site reliability engineering (SRE) into nine pillars of SRE practices—a comprehensive framework that covers the full scope of SRE practices. This nine pillars SRE framework is consistent with the SRE blueprint published by the DevOps Institute. Each of these pillars represents a vital area of SRE implementation and, together, they ensure a holistic approach to aligning development and operations for effective collaboration, uneventful continuous deliveries and reliable, highly scalable, safe operations.

    The current generation of AI-engineered tools is now being applied to each of the SRE pillars and has the potential to significantly increase efficiency, reliability and scalability for all SRE practices.

    Just as SRE practices complement DevOps practices, these nine pillars of SRE complement the nine pillars of DevOps explained in my recent blog Revolutionizing the Nine Pillars of DevOps With AI-Engineered Tools.

    AI Applications for the Nine Pillars of SRE

    The following briefly explains how the current generation of AI-engineered large language model (LLM) tools are being applied or could be practically applied to improve each of the nine pillars of SRE.

    Culture: AI can help identify patterns and trends in system behavior that may not be immediately evident. This can assist in shifting the wisdom of production further to the left. Tools like Datadog’s Watchdog can be used to automatically detect and surface anomalies in your system behavior, for example.

    Toil Reduction and Automation: AI can help reduce toil by automating more complex tasks that were previously challenging to automate. Machine learning models can predict system behavior, enabling proactive responses. AI operations platforms like Moogsoft or BigPanda can automate incident management, from detection to remediation.

    SLAs/SLOs/SLIs and Error Budgets: AI can assist in refining SLAs, SLOs and SLIs by providing accurate predictions of system performance and identifying potential bottlenecks. Tools like Nobl9 can use machine learning to better predict and define SLOs based on historical performance data.

    Measurements and Observability: AI can help analyze vast amounts of data from monitoring and observability systems, identifying patterns and correlations that may be difficult for humans to detect. Tools like Dynatrace and New Relic use AI to analyze monitoring data, detect anomalies and correlate events.

    Anti-Fragility: AI can contribute to the development of more resilient systems by analyzing system behavior under stress and identifying vulnerabilities. The data gathered from chaos experiments using tools like Gremlin can be used in conjunction with machine learning models to predict system behavior.

    Work-Sharing and Incremental Technical Debt: AI can help analyze code and infrastructure changes to assess their impact on technical debt. Tools like DeepCode and SonarQube use AI to provide guidance on how to manage this debt incrementally.

    Deployments: AI can be used to optimize deployment strategies by predicting the impact of new releases. A tool like Harness can use machine learning to optimize deployment strategies and minimize risks.

    Performance Management of Apps and Infrastructure: AI can analyze application and infrastructure performance data to identify optimization opportunities and predict capacity requirements. Tools like Turbonomic leverage AI to optimize resource allocation in real-time.

    Incident Management: AI can assist with incident management by automating the detection and triage of incidents, helping to quickly identify the root cause. Tools like PagerDuty use AI to automate detection, triage and suggest potential remediation steps, leading to faster resolution times.

    Pitfalls and Challenges

    Applying AI to SRE is a complex process with certain challenges. Here are some potential pitfalls along with ways to address them:

    Lack of Quality Data: AI and machine learning models are only as good as the data they are trained on. Inadequate or poor quality data can lead to inaccurate predictions and insights.
    • Prioritize data hygiene and governance. Collect comprehensive and diverse data from your systems; ensure that it is well-structured and free of errors and store it in a way that’s easily accessible for training AI models.

    Over-reliance on Automation: While AI can greatly enhance automation, relying on it too heavily without human oversight can lead to missed signals or overcorrections in response to false positives.
    • Maintain a balance between automation and human oversight. Use AI to support decision-making, not replace it entirely. It’s important to have experienced SREs review AI outputs regularly to ensure they make sense and are beneficial.

    Underestimating the Need for AI Expertise: Implementing AI is not just about buying and deploying a tool. It requires a deep understanding of AI and machine learning principles, as well as the ability to interpret the results correctly.
    • Invest in training your team on AI and machine learning principles or consider hiring AI specialists. Having in-house AI expertise can help ensure that you’re using AI tools effectively and interpreting their outputs correctly.

    Ignoring the Importance of Integration: AI tools need to work seamlessly with your existing infrastructure and processes to be effective. A lack of integration can lead to data silos and inefficiencies.
    • When selecting AI tools, consider how easily they can be integrated into your existing infrastructure and workflows. Use APIs and other integration methods to ensure that your AI tools can access the data they need and that their insights are easily accessible to your SRE team.

    Neglecting Ethics and Privacy: AI tools often need access to sensitive data to operate effectively, which can raise ethical and privacy concerns.
    • Implement robust data privacy and security measures. Ensure that you’re following all relevant regulations and industry best practices for data privacy. Use anonymization and other data protection techniques to protect sensitive data.

    Summary

    Artificial intelligence is revolutionizing SRE practices by automating complex tasks, analyzing vast amounts of data and making proactive predictions. AI can reduce toil, enhance system understanding, and streamline incident management. It allows SREs to shift further left by integrating predictive insights into the development process, thereby enhancing the culture of proactive issue prevention. AI also aids in defining more precise service level objectives, error budgets, and optimizing deployment strategies based on predictive analytics. In the realm of incident management, AI aids in detecting, triaging, and mitigating incidents faster by identifying patterns and suggesting remediations. Overall, AI is fast becoming an essential tool for SREs, facilitating a shift from reactive problem-solving to proactive management, thereby increasing system reliability, performance, scalability and efficiency.

  • How AI Transforms DevOps Infrastructure

    How AI Transforms DevOps Infrastructure

    In my recent article Revolutionizing the Nine Pillars of DevOps with AI-Engineered Tools, I explained that the elastic infrastructure pillar involved automated provisioning and management of computing resources, often leveraging cloud-based solutions for scalability and resilience. This includes practices like infrastructure-as-code (IaC).

    In this article, I explain how AI can help manage cloud resources for DevOps by predicting future needs based on usage trends and automatically adjusting resources to optimize cost and performance.

    Here are some common DevOps use cases that IaC practices aim to address, along with potential AI tools that can assist in each scenario:

    Private and Public Cloud Agnosticism:
    • Use Case: Needing to deploy and manage infrastructure across multiple cloud providers or environments without vendor lock-in.
    • AI Tool Example: AI-powered cloud management platforms (e.g., Turbonomic, CloudHealth) can analyze resource utilization, performance and cost across various cloud providers to optimize workload placement.

    Reducing Infrastructure Configuration Bottlenecks:
    • Use Case: Avoiding manual and error-prone configuration processes that delay deployment and cause inconsistencies.
    • AI Tool Example: Intelligent configuration management tools (e.g., Puppet, Chef, Ansible) can automate the provisioning, configuration and management of infrastructure resources.

    Infrastructure Cost Optimization:
    • Use Case: Optimizing resource allocation and minimizing infrastructure costs.
    • AI Tool Example: AI-driven cost optimization platforms (e.g., ParkMyCloud, Cloudability) can analyze usage patterns, recommend resource resizing or right-sizing and suggest cost-saving opportunities.

    Infrastructure Sharing:
    • Use Case: Sharing infrastructure resources securely across multiple teams or projects.
    • AI Tool Example: AI-powered resource scheduling and sharing tools (e.g., Kubernetes with cluster autoscaling) can dynamically allocate and release resources based on demand and workload requirements.

    Infrastructure Quality:
    • Use Case: Ensuring consistency, reliability, and adherence to best practices in infrastructure deployments.
    • AI Tool Example: AI-based infrastructure validation tools (e.g., Terrascan, Checkov) can scan IaC templates, provide security and compliance checks and offer recommendations for improvement.

    Infrastructure Security:
    • Use Case: Integrating security controls and policies into the infrastructure deployment process.
    • AI Tool Example: AI-driven security tools (e.g., Aqua Security, Sysdig Secure) can analyze infrastructure configurations, detect vulnerabilities and enforce security policies during deployment.

    Dynamic Orchestration:
    • Use Case: Needing to dynamically manage and scale infrastructure resources based on workload demands.
    • AI Tool Example: AI-powered orchestration platforms (e.g., Kubernetes with horizontal pod autoscaling) can automatically scale resources up or down based on metrics such as CPU utilization or request latency.

    Integration and Control of Infrastructure with CICD Pipelines:
    • Use Case: Seamlessly integrating infrastructure provisioning and deployment with CI/CD pipelines.
    • AI Tool Example: AI-enabled CI/CD platforms (e.g., Jenkins with ML plugins, Harness) can leverage machine learning algorithms to optimize pipeline execution, provide intelligent recommendations and automate release processes.

    Monitoring Infrastructure Performance:
    • Use Case: Monitoring and ensuring the optimal performance of infrastructure resources.
    • AI Tool Example: AI-powered monitoring solutions (e.g., Dynatrace, Datadog) can leverage machine learning to detect anomalies, predict performance issues and provide actionable insights for infrastructure optimization.

    Rapid Restore of Infrastructure:
    • Use Case: Quickly recovering and restoring infrastructure in the event of failures or disasters.
    • AI Tool Example: AI-driven backup and recovery tools (e.g., Veeam, Rubrik) can automate backup processes, intelligently prioritize data restoration and leverage machine learning for predictive data recovery.

    Challenges With the Transformation to AI-Engineered Infrastructure

    An organization may face a number of challenges when transitioning to AI tools for infrastructure, as follows:

    Lack of Expertise and Skill Set:
    • Challenge: Organizations may lack the necessary expertise and skill set to implement and operate AI tools effectively.
    • Solution: Invest in training and upskilling programs for employees to acquire the required knowledge and skills. Partnering with external consultants or hiring experts in AI and infrastructure can also bridge the expertise gap.

    Data Availability and Quality:
    • Challenge: AI tools heavily rely on high-quality data for training and accurate decision-making, but organizations may face challenges in obtaining relevant data and ensuring its quality.
    • Solution: Implement robust data governance practices to ensure data availability, quality and consistency. Invest in data collection, storage and processing frameworks that support AI initiatives. Data cleansing and preprocessing techniques can be employed to improve data quality.

    Integration Complexity:
    • Challenge: Integrating AI tools into existing infrastructure and workflows can be complex, especially when dealing with legacy systems and heterogeneous environments.
    • Solution: Plan for seamless integration by conducting a thorough assessment of existing systems and infrastructure. Leverage APIs, automation frameworks and standardized protocols to enable smooth integration. Adopt modular and scalable architectures that facilitate the incorporation of AI components.

    Ethical and Legal Considerations:
    • Challenge: AI tools introduce ethical and legal considerations, including privacy, security, bias and regulatory compliance, which can pose challenges during infrastructure transformation.
    • Solution: Develop clear guidelines and policies around data privacy, security and ethical considerations. Perform rigorous testing and validation to detect and mitigate biases in AI models. Engage legal and compliance teams to ensure alignment with relevant regulations and standards.

    Change Management and Cultural Shift:
    • Challenge: The adoption of AI tools often requires a cultural shift within the organization, including changes in mindset, processes and collaboration between teams.
    • Solution: Develop a change management strategy to communicate the benefits of AI tools and address any concerns or resistance. Foster a culture of experimentation, learning and collaboration. Encourage cross-functional teams to work together and share knowledge. Provide ongoing support and training to ensure successful adoption and adaptation to the new tools and practices.

    Summary

    Incorporating AI into IaC practices for DevOps brings transformative benefits that drive efficiency, agility and scalability. By leveraging AI tools, organizations can optimize resource allocation, streamline deployments, enhance security and improve performance monitoring. Embracing AI for IaC empowers teams to automate and intelligently manage infrastructure, resulting in faster time-to-market, cost savings and increased productivity. Unlock the full potential of DevOps by embracing AI and revolutionize your infrastructure management with the power of intelligent automation.

    It’s important to note that challenges and solutions are to be expected when transforming to the use of AI, and the specific context and requirements of each organization may vary. Therefore, it’s advisable to assess the organization’s unique needs and constraints to tailor the solutions accordingly.

  • How AI Addresses DevOps Monitoring and Observability Challenges

    How AI Addresses DevOps Monitoring and Observability Challenges

    In my recent article Revolutionizing the Nine Pillars of DevOps with AI-Engineered Tools, I explained that the continuous monitoring pillar involves monitoring observable systems and application performance in real-time so that teams can detect and react to problems quickly, often before they impact users. It’s crucial for maintaining application health and informing future improvement efforts.

    In this article, I explain how AI can analyze logs and metrics to predict potential system failures or performance degradation, allowing for proactive maintenance and issue resolution.

    Continuous Monitoring and Observability Use Cases

    CI Anomaly Detection: AI can analyze historical data to detect anomalies during the continuous integration phase. Any unusual change can be flagged for review before it proceeds to the next phase. Tools like IBM Watson Anomaly Detection can help identify these anomalies by using AI to detect patterns and irregularities.

    Code Quality Assurance: AI can be used to analyze the code in the development stage to ensure its quality, which can help reduce bugs and vulnerabilities. Tools like DeepCode and Codota use AI to identify potential problems and suggest improvements based on a learned database of code and solutions.

    Test Case Optimization: AI can help optimize the selection of test cases in continuous integration (CI). Using historical test data, AI can identify which test cases are most likely to find new defects. Tools like Testim.io can help with this by using AI to prioritize testing based on risk and change impact.

    Predictive Analytics in CD: AI can analyze historical deployment data and predict potential issues during continuous delivery (CD). This can help preemptively solve problems, reducing downtime. Tools like Splunk use AI and machine learning to provide predictive analytics for operations data.

    Automated Rollbacks: In the case of continuous deployment, AI can be used to automatically roll back deployments that are causing issues. Tools like Harness use machine learning to understand typical application behavior, automatically reverting to the last stable state if anomalies are detected.

    Infrastructure Optimization: AI can help optimize the usage of resources in cloud environments. Tools like CAST.AI and Turbonomic use AI to continuously optimize your infrastructure, ensuring better performance while reducing cost.

    Incident Management: AI can help automate the incident management process, from detection to resolution. Tools like BigPanda and Moogsoft AIOps use AI to aggregate, correlate and analyze alerts from various sources, reducing noise and speeding up incident resolution.

    Log Analysis: AI can analyze logs and identify patterns that would be difficult for humans to spot. Tools like Logz.io use AI to provide cognitive insights into log data, providing a deeper understanding of the data.

    Security Threat Detection: AI can analyze patterns and detect security threats more effectively. Tools like Darktrace use machine learning to detect unusual behavior in real-time, thereby detecting potential threats before they cause damage.

    Network Monitoring: AI can predict network outages by analyzing traffic patterns. Tools like Kentik use AI to proactively identify potential network issues before they affect users.

    Challenges When Transforming Continuous Monitoring to use AI

    Here are some challenges organizations might face when transitioning existing CI/CD pipelines to incorporate AI into continuous monitoring and observability, along with possible solutions:

    Data Quality and Availability: The effectiveness of AI-engineered tools largely depends on the quality and quantity of the data they’re given. Insufficient or poor-quality data can result in inaccurate insights or predictions. Implement effective data governance and management practices to ensure data quality and accessibility. Data should be thoroughly cleaned and properly labeled to facilitate the training of AI models.

    Skills Gap: Adopting AI-engineered tools requires new skills that the existing IT team may not possess. There might be a lack of understanding of how to use these tools effectively. Provide comprehensive training to your DevOps team to bridge the skills gap. Also, consider hiring AI specialists or working with an experienced vendor to help implement and manage the AI tools.

    Resistance to Change: As with any significant transformation, resistance to change can be a substantial barrier. Employees may be concerned about job security or the perceived difficulty of adapting to new tools. Communicate clearly and transparently about the benefits of the AI transition, both at an organizational and individual level. Ensure staff that AI is there to assist them, not replace them. Organize workshops and training sessions to ease the transition.

    Integration with Existing Systems: AI tools need to integrate seamlessly with existing DevOps tools and workflows to ensure that they add value without disrupting operations. Choose AI tools that are compatible with your existing infrastructure or consider implementing integration middleware. Conduct a proof-of-concept (PoC) to ensure the new AI tools integrate smoothly.

    Cost of Implementation: Deploying AI tools can require a significant upfront investment, both for the tools themselves and for necessary infrastructure upgrades. Conduct a thorough cost-benefit analysis to understand the return on investment (ROI) that the AI tools can deliver. Consider starting with lower-cost or open source tools or use cloud-based AI services to reduce initial investment in infrastructure.

    Summary

    As the world of DevOps evolves, the integration of AI in monitoring and observability becomes increasingly important. Whether it’s during the phases of continuous integration, continuous delivery or continuous deployment, or for applications, infrastructure and pipelines, AI can provide exceptional benefits. From anomaly detection in CI, code quality assurance and test case optimization to predictive analytics in CD, AI can transform your operations, delivering rapid and more reliable results.

    However, transitioning to an AI-optimized DevOps environment is not without its challenges. Issues such as data quality, the skills gap, resistance to change, system integration and cost implications must be considered and addressed. But fear not, solutions are at hand. With effective data governance, comprehensive training, transparent communication, smart tool selection and thorough cost-benefit analysis, you can navigate these challenges and reap the rewards of AI integration in your DevOps journey. So, are you ready to embrace the future of AI-driven DevOps? There’s no time like the present to start exploring the possibilities.

  • Harnessing AI in Continuous Delivery and Deployment

    Harnessing AI in Continuous Delivery and Deployment

    In my recent article Revolutionizing the Nine Pillars of DevOps with AI-Engineered Tools, I explained that the continuous delivery pillar is an approach where code changes are automatically built, tested and prepared for release to production; this may be followed by continuous deployment where an approved release is automatically deployed to the production environment. The practices of the CD pillar aim to make releases and deployments less disruptive and more frequent, increasing speed and efficiency.

    In this article I explain how AI can help continuous delivery and deployment processes.

    Continuous Delivery Use Cases

    Orchestration of Test Environments: Teams need a reliable way to prepare and manage various test environments. This includes setting up and tearing down environments, managing versioning and configuration and handling data and resources. AI tools that can help include Harness, and IBM’s UrbanCode, which can automatically spin up and down environments for testing purposes and handle configuration as code.

    Execution of Various Tests: This includes system acceptance tests, regression tests, performance tests, user acceptance tests and security tests. The AI tool that could help here is Parasoft’s SOAtest. Its AI can generate test scenarios, ensure full coverage and automatically adapt to changes in the code base.

    Performance Monitoring of Release-Candidate-Artifact: Tools need to assess the performance of release candidates, identify bottlenecks and compare performance against previous versions. Dynatrace is an AI-powered tool that provides automatic and dynamic baselining of application performance, helping teams to recognize when performance is lagging.

    Automation of Policies for Artifact Acceptance: Rules and policies need to be defined and enforced regarding when an artifact is ready for release. Tools like Harness use machine learning to analyze past deployments and test results to predict the success of future deployments and automate the approval process.

    Continuous Deployment Use Cases

    Orchestration of Deployment Environments: Teams need to manage deployment environments, including setup, versioning, configuration and data management. Spinnaker is an open source tool that uses AI to manage multi-cloud deployment strategies.

    Execution of Deployment Strategies: This includes executing strategies such as Blue-Green, Canary and Feature Flag gradual rollouts. A tool like LaunchDarkly can utilize AI to manage feature flags, analyze usage data and roll out features gradually.

    Testing in Production: Testing the new release in the live production environment is crucial. A tool like Sentry can use AI to monitor application use in real-time and identify and alert teams to issues faster.

    Restoring Production to a Prior Version: When a problem is detected with a new release, teams may need to revert to a previous stable version quickly. StackStorm is an event-driven automation platform that uses AI to define workflows, allowing you to quickly roll back to a previous version in case of failure.

    Updating Documentation and Configuration Management Databases: It’s crucial to keep records up-to-date. A tool like Guru uses AI to automatically update and keep track of documentation, ensuring that it’s always accurate and up-to-date.

    Challenges When Transforming CD to use AI

    Here are some challenges organizations might face when transitioning their existing CI/CD pipelines to incorporate continuous delivery, continuous deployment and AI tools, along with possible solutions:

    Lack of understanding of AI capabilities and uses: One of the first challenges that organizations might face is the lack of understanding of what AI can and cannot do. Some team members may have unrealistic expectations or fear that AI will replace their jobs. Regular training and workshops can help teams understand how AI can enhance their roles rather than replace them. Clear communication about the goals and benefits of incorporating AI can also help allay fears.

    Data privacy and security concerns: Using AI tools often involves handling a large amount of data, some of which may be sensitive or private. There are also concerns about the security of AI tools themselves. Establish a robust data governance policy that includes procedures for handling sensitive data. Also, when selecting AI tools, prioritize those that are known for their robust security measures.

    Complexity in integrating AI tools with existing systems: Not all AI tools will seamlessly integrate with existing systems and infrastructure. This could lead to technical debt and instability in the pipeline. Prioritize AI tools that have wide integration capabilities. Engage the tool’s support team or hire consultants if needed to ensure the integration is smooth and stable.

    Resistance to change: The transition to a new way of working can be met with resistance, especially if team members are comfortable with the existing systems and processes. Change management strategies can help ease the transition. This can involve regular communication about the benefits of the change, training and providing support during the transition.

    Cost of AI tools and training: AI tools can be expensive, and there’s also the cost of training team members to use these tools. Start with a cost-benefit analysis to understand if the transition is worth the investment. Look for AI tools that offer a good balance between cost and features. Consider phased implementation to spread the costs over time.

    Dependence on vendor support: Given the complexity of AI, there may be a high dependence on the AI tool vendor for support. This could become a problem if the vendor is unresponsive or ceases to support the tool. Choose vendors who have a good track record of providing strong customer support. Also, consider tools that have a strong community around them, which can provide additional sources of help and support.

    Summary

    Navigating the landscape of continuous integration, continuous delivery and continuous deployment (CI/CD) pipelines can be complex, but it’s a vital process that’s being revolutionized by the introduction of AI tools. From orchestrating test and deployment environments, managing testing and deployment strategies, to automating delivery tasks and quality checks, AI is playing an increasingly critical role in streamlining and enhancing these processes.

    However, as with any significant change, transitioning to AI-enhanced CI/CD pipelines is not without its challenges. This includes understanding AI’s capabilities, data privacy and security concerns, technical integration difficulties, resistance to change, cost considerations and vendor support dependencies. By identifying and addressing these challenges head-on, organizations can effectively leverage AI’s capabilities, bringing increased efficiency, reliability and value to their software delivery pipelines.

    With proper planning, continuous learning and strategic use of AI tools, organizations can not only overcome these challenges but also thrive, gaining competitive advantages in their software delivery processes. The future of CI/CD is undeniably entwined with AI, and embracing this evolution is an exciting journey that holds the promise of transformative benefits.

  • A New DevOps Database Platform is Needed

    A New DevOps Database Platform is Needed

    The diverse number of database platforms, deployment environments and data variants needed to support the plethora of applications and services required for rapid DevOps continuous delivery lead times, coupled with the challenges of integrating database changes for application pipelines, is a growing challenge for many organizations as they strive to accelerate their DevOps transformations.

    When you stand back and look at the bigger picture, an enterprise with many databases and various data requirements for DevOps continuous delivery presents a problem. At best, it makes things chaotic; at worst, it is a gross impediment to overall organization agility—which undermines the very purpose that DevOps is meant to serve.

    In my book, Engineering DevOps, I explained that databases offer several challenges for CI/CD pipelines.

    In the simplest case, when a database is used solely by one application or microservice, the database pipeline can be integrated with the application pipeline to ensure its changes use the same version management system and change controls to ensure the database changes are kept in sync with the application. However, many database platforms are not as responsive to changes as the applications themselves and become a bottleneck for the pipeline.

    If a database is used by several applications, each with its own DevOps pipeline, then changes to the database must be coordinated with all applications that depend on it. Maintaining a separate database pipeline linked to the application pipelines through CI/CD toolchains, as a strategy, is overly complicated and can often introduce additional bottlenecks and synchronization issues.

    Another challenge with database pipelines is the management of data in the database during the CI and CD stages. Data in production needs to be persistent and is ever-changing, even while the underlying database code is changing. During the CI process, sample data from production needs to be used to ensure testing is realistic, and new test data must be introduced to verify new database capabilities are working.

    During the CD stages, rollbacks to prior versions of database code must avoid rolling back to versions that are no longer compatible with production data. This is a key reason why database changes must follow a disciplined approach that fully tests database changes with applications prior to full deployment. Rolling deployments, in which applications and database changes are deployed gradually in separate clusters, can minimize the blast radius in case of problems. The required coordination of separate systems adds complexity and bottlenecks.

    So, why not simply integrate database changes as artifacts within the same DevOps pipeline as the application?

    Gilad David Maayan’s September 2020 article DevOps Database Strategies and Challenges: Automation, Scalability, and More explained some of the specific challenges that arise with integration of traditional databases with DevOps.

    Database tools were not designed for integration with DevOps tools. These tools tend to be database- and environment-specific, which can make it difficult to incorporate into pipelines designed for flexibility.

    Database performance and ease of management often do not perform well in DevOps environments where multiple applications need to use the database in a short time, such as in the accelerated lead times for DevOps continuous delivery. This is especially true with larger-scale databases. If an application change must sync with a database schema, changing the pipeline can become complicated especially if there is a need to sync across multiple application pipelines to meet continuous delivery goals.

    While DevOps processes are proven for continuous delivery of applications, incorporating database changes into the DevOps pipeline is hindered by large data sets, limited database automation and security concerns. As a result, database development and testing continue to rely on outdated processes, with fixed and often shared instances refreshed from backups. A new DevOps Database platform is needed to provide highly dynamic and efficient cloud-native database solutions for modern enterprise digital transformations, and that platform must include DataOps use cases for DevOps, DevSecOps, continuous testing and governance. This new kind of data repository must be designed to integrate seamlessly with DevOps CI/CD pipelines through flexible APIs and cloud-native architectures.

    Given the points above, traditional database solutions and parallel data pipeline strategies are not sufficient to meet the challenges of enterprise DevOps transformations. What is needed is a new kind of DevOps database platform that allows the database, and database changes, to be as agile as any other artifact in the DevOps pipeline.

    Paul Stanton, VP of product management at Windocks, in his June 2021 YouTube video Windock DevOps Data Repo, said, “Imagine terabyte class database environments managed as DevOps artifacts with entire versioned databases available in seconds.” That’s what we need.

    The following are examples of the capabilities required for such a new DevOps database platform.

    • Automated database provisioning
    • Distributed repositories support
    • Simple declarative builds
    • Dynamically scalable
    • Virtual hard drive images for a wide variety of DBs – Linux, SQL, Oracle and others
    • Distributed versioned database environments working on standard file systems and storage systems delivery writeable data at the speed of DevOps – in other words, in seconds.
    • Deploys equally to all operating systems and cloud environments, Docker containers and Kubernetes.
    • Many clones per virtual machine (VM)
    • Security features including access management, data masking and encryption
    • Compatible with and easily integrates with CI/CD platforms
    • Compatible with multi-cloud (AWS, Azure, GCS)
    • Built-in data security and governance into DevOps data.

    I will explain these more completely in a subsequent article.

    What This Means

    Digital transformations are imperative for modern enterprises and organizations if they are to remain competitive. However, the transformations of many enterprises and organizations are being seriously impeded by having too many database platforms and tools that do not integrate well and that bottleneck continuous delivery goals. A new kind of DevOps database platform, in which the database is treated as any other artifact, is needed to resolve technical problems with database integration to DevOps application pipelines. A new DevOps database platform solution is needed to accelerate time-to-value and realize efficient, safe operation for digitally transforming enterprises and organizations.

  • Continuous Testing Practices Gap Assessment

    Continuous Testing Practices Gap Assessment

    Continuous testing is an essential part of DevOps. To get the maximum value from continuous testing, it is important to use recommended best practices, inclusive of people, processes and technologies. A gap assessment is a great way to efficiently evaluate an organization’s practices for DevOps and determine a strategy for improvement. Gap analysis provides valuable input for formulating a strategy and a roadmap to improve continuous testing.

    My SlideShare, Gap Survey, Assessment and Analysis For Continuous Testing, explains how my continuous testing gap assessment tool greatly speeds up the gap survey, gap assessment and gap analysis process for continuous testing.

    It is important to clarify a few definitions for this blog because I have found that different people and organizations use these terms differently. In this blog, and in my consulting work, I use the following definitions.

    • Gap survey is a discovery tool used to collect information about the current state of practices for a topic such as continuous testing.
    • Gap assessment is a process to determine differences (gaps) between current practices and recommended practices for a topic. Gap assessment is also used to refer to the entire collection of activities in this list.
    • Gap assessment workshop is a meeting in which the gap assessment is validated prior to gap analysis.
    • Gap analysis is the process of determining priorities for reducing gaps found during a gap assessment.
    • Gap assessment tool is software that helps perform gap surveys, gap assessments and gap analysis.

    I define and use a seven-step process for gap assessments.

    The steps are as follows:

    1. Pick a topic to analyze. The same processes and tools can be applied equally well to almost any topic, such as continuous testing.
    2. Determine recommended practices for the topic. Something that is worth doing a gap assessment for, such as continuous testing, will normally be fairly complex with tens if not hundreds of practices. Categorize the practices into groups (I call the groups “pillars”) to make the list of practices more understandable and meaningful. To do this requires some research and discovery work. For example, I have a database of practices and pillars for DevOps, continuous testing, DevSecOps and SRE as well as some other topics.
    3. Enter the practices from step two into a gap assessment tool. The tool can be a simple survey tool you create, a commercial survey tool or something custom-made.
    4. Determine who needs to be included in the gap survey.
    5. Collect data using the gap survey.
    6. Perform a gap assessment.
    7. Perform a gap analysis.

    For the remainder of this blog, I will explain in detail the seven steps for continuous testing.

    Step One: Pick a Topic to Analyze

    In this blog, the topic is continuous testing.

    Step Two: Determine Recommended Practices for Continuous Testing

    For an example continuous testing assessment, I used the 15 practices categories described in this three-part series:
    Continuous Testing Practices Part 1
    Continuous Testing Practices Part 2
    Continuous Testing Practices Part 3

    The 15 practice category pillars are:

    • Continuous testing tenets
    • Leadership
    • Culture
    • Test strategy and plans
    • Test management
    • Test automation
    • Test tools
    • Test infrastructure management
    • Test scripts
    • Test results
    • Security tests
    • Pre-flight testing
    • Integration testing
    • Pre-production testing
    • Test in production

    Step Three: Enter the Practices from Step Two Into a Gap Assessment Tool

    A gap assessment tool, pre-loaded with sample continuous testing practices, can be found in a file called “Continuous Test Practices Assessment Tool v2. It can be downloaded for free from one of the resource pages found on my website Engineering DevOps.

    You can edit the practices categories and add/delete practices from each category if you prefer to make changes.

    Step Four: Determine Who to Include in the Gap Survey

    For the gap analysis to be comprehensive, people in roles that are affected by continuous testing practices need to be surveyed, or at least represented, to ensure their perspectives are included. The following are example roles that typically are included.

    • Business leaders – because they influence culture and budgets for continuous testing.
    • Developers – because they need to design in accordance with continuous testing practices.
    • Project owners – because they influence product work priorities.
    • QA testers – because they need to test in accordance with continuous testing practices.
    • Ops – because continuous testing practices affect operations.
    • Security – because continuous testing affects security practices.

    The survey can be conducted for an individual application, a group of applications, or all the applications in the enterprise. However, it is important that the gap assessment and gap analysis be performed on the organizational segment that is being targeted for improvement.

    Step Five: Collect Data Using the Gap Survey

    A gap survey should allow each surveyed person to enter an importance level score, a practice level score and comments for each practice. All this information is essential for the gap assessment and gap analysis. In the gap survey included within my gap assessment tool, survey respondents are asked to score the importance of each practice as one of 0=Not relevant, 1=not Important, 2=Nice to have, 3=Important, 4=Very important or 5=Critical. Practice level choices are 0=Not sure, 1=Rarely, if ever, 2= Sometimes, 3=Most of the time, 4=Always or 5=We are really good at this. Comments that are relevant to qualify or explain the scores entered for each practice should also be entered, especially if there is any doubt or ambiguity.

    Step Six: Perform a Gap Assessment

    The gap assessment process requires all the practices scores collected from the surveys to be collected and assembled to calculate an aggregate set of scores. Gap scores are calculated using a formula that weights each practice level score with the corresponding importance level score. A visual representation helps to identify practices areas and individual practices that have the highest, most important gaps.

    No matter how professionally written the practices and score definitions are, it is not unusual for some people to misunderstand and to enter scores that they would not have otherwise. For this reason, it is important to ensure that the data collected is validated before conducting the gap analysis. The preferred approach is to conduct a gap assessment workshop with key representatives from each role that participated in the survey. During the workshop, the scores for each practice that have a high deviation between survey responses are discussed and, if necessary, adjusted.

    Step Seven: Perform a Gap Analysis

    The gap analysis process involves extracting the high gap practices and tagging and ranking each of them against solution categories determined by a consultant or topic expert. The results of the gap analysis indicate where solution strategies and implementation roadmaps need to be focused to reduce the most important gaps.

    What This Means

    Those who are serious about DevOps will do well to learn from others and be serious about using recommended continuous testing practices. This blog explained a gap assessment approach, which uses my 15 categories of continuous testing practices. My free Continuous Testing Gap Assessment Tool facilitates performing a gap analysis for continuous testing and outlines priorities to improve DevOps practices for an organization. For those that want to dig deeper on this topic, additional definitions and contextual information related to the way I do gap assessments are clarified in my book Engineering DevOps.

  • Learning ‘The Third Way’ of DevOps – Continuous Improvement

    Learning ‘The Third Way’ of DevOps – Continuous Improvement

    I refer to the third way of DevOps–continuous experimentation and learning, as simply continuous improvement. In some sense, there is no way to learn the third way of DevOps because it is, by definition, a moving target. It does make sense to discuss what is needed to create a continuous learning organization for DevOps continuous improvement that builds on mastery of the first way and second way of DevOps.

    As indicated in my blog Learning ‘The Second Way’ of DevOps – Continuous Feedback and my prior blog Learning ‘The First Way’ of DevOps – Continuous Flow, it is clear many organizations are struggling to realize well-engineered DevOps, even the first way of DevOps, continuous flow.

    The first way of DevOps, continuous flow, is the foundation for DevOps. Once mastered, an organization will naturally want to master the second way of DevOps to achieve the higher level of benefits that continuous feedback offers, when engineered well. Once the second way of DevOps is achieved, then an organization will naturally want to master the third way of DevOps, to sustain and grow their DevOps capabilities.

    DevOps is an evolving body of knowledge. There is no one-stop shop that covers all roles and the complete range of skills required for DevOps. A DevOps-competent workforce requires an ongoing commitment to DevOps skills development and mastery. Enterprises must approach DevOps training strategically to leverage training resources to their best advantage.

    Learning Continuous Improvement

    There are no training resources dedicated solely to the third way of DevOps—continuous improvement. Many of the same resources as the first way of DevOps and the second way of DevOps do have some relevant information, though. Analysis of DevOps results, third-party research sources, industry sources such as publications, articles, conferences and tool vendors roadmaps for DevOps and related frameworks including site reliability engineering (SRE) and AIOps are good sources of information that stimulate self-directed teams to seek the training information that is most relevant to the organization’s DevOps improvements.

    Why Is Learning Continuous Improvement Important to Engineering DevOps?

    The third way of DevOps—continuous improvement—requires an organizational commitment to continuous learning and mastery at the most advanced level. Typical skills needed include implementation and operation of advanced practices for microservices design and operations, advanced continuous testing, multi-pipelines, value stream management, SRE, observability, DevSecOps and AIOps. Retention and development of highly skilled and hard to find experienced DevOps professionals and SREs makes it important to develop skills in-house.

    How Is Learning Continuous Improvement Accomplished?

    The following are recommended engineering practices for learning continuous improvement:

    • A continuous learning culture encourages individuals to continuously gain knowledge and exchange experiences and ideas for DevOps and SRE.
    • Mentoring is an explicit program.
    • Retrospectives are structured to thoroughly analyze significant, unexpected events and determine solutions that will prevent the events in the future.
    • An active online library of DevOps and SRE training resources is maintained that tracks the latest and most innovative approaches to DevOps and SRE.
    • DevOps and SRE teams evolve their skills and advance their careers by engaging with industry thought leaders to continuously update training content.
    • Partner with trusted education and training partners ensure the highest-quality learning content and tools are made available to DevOps and SRE teams.
    • A portion of DevOps and SRE teams time is allocated for training and education.
    • Cross-team sharing of recommended engineering practices is proactively pursued using the concepts of Yokoten—proactive sharing of practices between teams.

    What Is Needed to Implement Learning Continuous Improvement?

    The following are recommended engineering practices for implementing a learning program for continuous improvement.

    • Assign roles and responsibilities to experts for defining a training program for DevOps continuous feedback (e.g., DevOps sensei, training PM, evangelist).
    • Conduct a continuous feedback DevOps skills gap assessment.
    • Define a training strategy that best address the gaps and fits the needs and priorities of the organization.
    • Define a training budget that includes expenses and time allocations per role and skill.
    • Determine which skills and roles can be covered by inexpensive self-study resources.
    • Determine which skills and roles warrant more costly commercial training.
    • Identify training goals for each role.
    • Ensure recruiting job profiles match skill requirements for DevOps skills that are most relevant to the organization’s strategic goals.
    • Create a continuous learning culture that encourages individuals to continuously gain knowledge and exchange experiences and ideas for DevOps and SRE.
    • Build an active online library of DevOps and SRE training resources that tracks the latest and most innovative approaches to DevOps and SRE.
    • Engage with industry thought leaders to continuously update con tent for DevOps and SRE teams to evolve their skills and advance their careers.
    • Partner with trusted education and training partners to ensure the highest-quality learning content and tools are made available to DevOps and SRE teams.
    • Put in place means to make training resources visible and available.
    • Set training goals for teams and individuals.
    • Put in place incentives and recognition programs for training.
    • Track training progress.

    Some of the best sources of DevOps training resources are listed below:
    1. The DevOps Institute
    2. Partners of The DevOps Institute, such as Learning Tree International and Xellentro.
    3. Udemy.com
    4. Recorded talks and publications of IT Revolution
    5. Books such as my own Engineering DevOps, by Marc Hornbeek.

    What This Means

    The third way of DevOps—continuous improvement—requires an organizational commitment to continuous learning and mastery at the most advanced level. Typical skills needed include implementation and operation of advanced practices for microservices design and operations, advanced continuous testing, multi-pipelines, value stream management, SRE, observability, DevSecOps and AIOps. Analysis of DevOps results, third-party research sources, industry sources such as publications, articles, conferences, and tool vendors roadmaps for DevOps and related frameworks including site reliability engineering (SRE) and AIOps are good sources of information that stimulate self-directed teams to seek the training information that is most relevant to the organization’s DevOps improvements.

    Organizations must approach DevOps training strategically and continuously to leverage training resources to best advantage. To learn more about my blueprint for learning all things DevOps, refer to my book Engineering DevOps.

  • Learning ‘The Second Way’ of DevOps – Continuous Feedback

    Learning ‘The Second Way’ of DevOps – Continuous Feedback

    As indicated in my prior blog Learning the ‘First Way’ of DevOps–Continuous Flow, many organizations are struggling to realize well-engineered DevOps, even the ‘first way’ of DevOps – continuous flow.

    The first way of DevOps, continuous flow, is the foundation for DevOps. Once mastered, an organization will naturally want to master the second way of DevOps to achieve greater benefits that continuous feedback offers when engineered well. DevOps is an evolving body of knowledge. There is no one-stop shop that covers all roles and the complete range of skills required for DevOps. A DevOps-competent workforce requires an ongoing commitment to DevOps skills development and mastery. Enterprises must approach DevOps training strategically to leverage training resources to their best advantage.

    Learning Continuous Feedback

    There are no training resources dedicated solely to the second way of DevOps—continuous feedback. Many of the same resources used to learn the first way of DevOps have some relevant information, though. DevOps monitoring, release automation and security tools publications, articles and tool vendors are a good source. A DevOps expert who is experienced with mature DevOps can select the training information that is relevant to the specific continuous feedback tools and feedback metrics used by the enterprise.

    Why This is Important to Engineering DevOps

    Continuous feedback requires a mature understanding of DevOps concepts and practices. Specific skills are needed for implementation and operation of advanced monitoring tools and practices for application performance, database performance, advanced continuous testing, advanced release management, infrastructure performance and continuous security.

    How Is Learning Continuous Feedback Accomplished?

    The following are recommended engineering practices for learning Continuous Feedback.

    Roles and responsibilities are assigned to experts for defining a training program for DevOps continuous feedback (e.g., DevOps sensei, training PM, evangelist). Continuous feedback DevOps skills gap assessments are conducted periodically. Dev and Ops training strategies are designed to address gaps and priorities of the organization for DevOps continuous feedback. A training budget that includes allocations of expenses and time allocations per role exists to support continuous feedback. Continuous Feedback goals and SLAs are defined for each role.

    Recruiting job profiles match skill requirements for DevOps skills relevant for continuous feedback. Training resources are available and visible to those that need continuous feedback training. Training goals set for teams and individuals include goals for continuous feedback. Incentives and recognition programs support Continuous Feedback training. Continuous feedback training progress is tracked and visible.

    What Is Needed to Implement Learning?

    The following are recommended engineering practices for implementing a learning program for continuous feedback.

    Assign roles and responsibilities to experts for defining a training program for DevOps continuous feedback (e.g., e.g., DevOps sensei, training PM, evangelist). Conduct a continuous feedback DevOps skills gap assessment. Define a training strategy that best addresses the gaps and fits the needs and priorities of the organization. Define a training budget that includes expenses and time allocations per role and skill.

    Determine which skills and roles can be covered by inexpensive self-study resources. Determine which skills and roles warrant more costly commercial training Identify training goals for each role. Ensure recruiting job profiles match skill requirements for DevOps skills that are most relevant to the organization’s strategic goals.

    Put in place the means to make training resources visible and available. Set training goals for teams and individuals. Put in place incentives and recognition programs for training. Track training progress.

    Some of the best sources of DevOps training resources are listed below:

    1. The DevOps Institute
    2. Partners of The DevOps Institute, such as Learning Tree International and Xellentro
    3. Udemy.com
    4. Recorded talks and publications of IT Revolution.
    5. Books such as my own “Engineering DevOps,” by Marc Hornbeek.

    What This Means

    To accomplish the benefits of the second way of DevOps—continuous feedback requires a commitment to continuous learning and mastering skills beyond what is needed for the first way of DevOps–continuous flow. Specific skills are needed for implementation and operation of advanced monitoring tools and practices for application performance, database performance, advanced continuous testing, advanced release management, infrastructure performance and continuous security. Organizations need a trained workforce to succeed with DevOps. Organizations must approach DevOps training strategically and continuously to leverage training resources to their best advantage. To learn more about my blueprint for learning all things DevOps, refer to my book Engineering DevOps.

  • Learning ‘The First Way’ of DevOps – Continuous Flow

    Learning ‘The First Way’ of DevOps – Continuous Flow

    Many organizations are struggling to realize well-engineered DevOps, even ‘the first way’ of DevOps – continuous flow.

    DevOps is an evolving body of knowledge. There is no one-stop shop that covers all roles and the complete range of skills required for DevOps. A DevOps-competent workforce requires an ongoing commitment to DevOps skills development and mastery. Enterprises must approach DevOps training strategically to leverage training resources to their best advantage.

    Learning Continuous Flow

    The first way of DevOps, continuous flow, is the foundation for DevOps.

    Why Is Learning Continuous Flow Important to Engineering DevOps?

    A knowledgeable, skilled workforce is needed to meet strategic objectives. Without a solid understanding of DevOps concepts and terminology, communication between people and teams is inefficient and a source of bottlenecks and misunderstandings. Developing hands-on skills for the tools and workflows is needed to understand and implement DevOps pipelines. There is an industry-wide shortage of skilled DevOps candidates. New skills foster innovation. Knowledge workers reduce production costs and reduce mistakes. Having skilled coworkers makes for a satisfying working environment. Employee retention is better. Competitive advantages are derived from knowledge.

    How Is Learning Continuous Flow Accomplished?

    The following are recommended engineering practices for learning continuous flow.

    People are encouraged to be continuous learners with an ongoing commitment to DevOps skills development. DevOps training is approached strategically to ensure DevOps training is well aligned to business goals. DevOps skills gap assessments are conducted and used to define and refine the training strategy. Leaders and cross-functional teams are encouraged to take DevOps fundamentals training. Leaders define a training strategy that best fits the needs and priorities of the organization to meet critical skills gaps.

    The training strategy considers which skills and roles can be covered by inexpensive self-study resources. Determine which skills and roles warrant more costly training. DevOps training has a distinct budget. Compliance to DevOps training programs are tracked. DevOps leadership training is offered to management and staff that have a role relevant to DevOps. DevOps tools training includes the recommended engineering practices for using each DevOps tool in the Service Catalog. Staff are encouraged and supported to take DevOps certifications that are relevant to their assigned roles.

    A DevOps training program is defined and includes a recommended curriculum for each role including leaders, developers, QA, Ops, and infrastructure and security staff that are operating in a DevOps environment. Recommended DevOps training resources are cataloged, and the catalog is visible for all stakeholders. Merit and recognition programs encourage employees to master DevOps skills.

    All staff is allocated time for DevOps training. DevOps mentoring programs supplement formal training. Job swapping and immersive training is encouraged and planned. People are encouraged to attend external conferences and meetups and share what they learn with other team members.

    What Is Needed to Implement Learning for Continuous Flow?

    The following are recommended engineering practices for implementing a learning program for continuous flow.
    Assign roles and responsibilities for DevOps training leadership (e.g., DevOps Sensei, Training PM, evangelist). Conduct a DevOps skills gap assessment. Define a training strategy that best addresses the gaps and fits the needs and priorities of the organization. Define a training budget that includes expenses and time allocations per role and skill.

    Determine which skills and roles can be covered by inexpensive self-study resources. Determine which skills and roles warrant more costly commercial training. Identify training goals for each role. Ensure job recruiting profiles match skill requirements for DevOps skills that are most relevant to the organization strategic goals. Put in place the means to make training resources visible and available. Set training goals for teams and individuals. Put in place incentives and recognition programs for completion of training.

    Some of the best sources of DevOps training resources are listed below:

    1. The DevOps Institute
    2. Partners of The DevOps Institute, such as Learning Tree International and Xellentro
    3. Udemy
    4. Recorded talks and publications available through IT Revolution
    5. Books such as my own Engineering DevOps, by Marc Hornbeek.

    What this Means

    Organizations need a trained workforce to succeed with DevOps—even ‘The First Way’ of DevOps—continuous flow. Organizations need to approach DevOps training strategically to leverage DevOps training resources to their best advantage. To learn more about my blueprint for learning all things DevOps, refer to my book Engineering DevOps.

  • Best of 2019: The Time Has Come for an Engineering Approach to DevOps

    Best of 2019: The Time Has Come for an Engineering Approach to DevOps

    As we close out 2019, we at devopsy.kinsta.cloud wanted to highlight the five most popular articles of the year. Following is the third in our weeklong series of the Best of 2019.

    Do you want to know how to do DevOps? Are you doing DevOps but not satisfied with the results you are getting? DevOps is complex. It does not come with a prescription, installation guide, user’s manual or maintenance manual. It doesn’t even have a standard definition. How can anyone expect to get good results with DevOps without clear and definitive guidance? There are plenty of books that describe different aspects of DevOps and customer user stories, but up until now there has not been a book that frames DevOps as an engineering problem with a step-by-step engineering solution and a clear list of recommended engineering practices to guide implementers.

    My book, Engineering DevOps: From Chaos to Continuous Improvement … and Beyond, provides a step-by-step engineering prescription that can be followed by leaders and practitioners to understand, assess, define, implement, operationalize and evolve DevOps for their organization. It provides a unique collection of engineering practices and solutions for DevOps. By confining the scope of the content of the book to the level of engineering practices, the content is applicable to the widest possible range of implementations.

    This book was born out of my desire to help others to do DevOps, combined with a burning personal frustration. The frustration comes from hearing leaders and practitioners say, “We think we are doing DevOps, but we are not getting the business results we had expected.”

    There are many documented references, courses and other sources of information that explain a multitude of different aspects of DevOps, but there is not a comprehensive step-by-step prescriptive guide that details how to engineer a DevOps solution for any organization given any starting point.

    Engineering DevOps takes a strategic approach, applies engineering implementation discipline and focuses operational expertise to define and accomplish specific goals for each leg of your DevOps journey. This book guides the reader through a journey from defining an engineering strategy for DevOps to implementing The Three Ways of DevOps maturity using engineering practices: The First Way (called “Continuous Flow”) to The Second Way (called “Continuous Feedback”) and finally The Third Way (called “Continuous Improvement”). This book is deliberately intended to be a guide that will continue to be relevant over time as your specific DevOps, and DevOps more generally, evolves.

    Structure of this book

    This book is organized as an engineering reference guide presented in five parts as follows:

    Part I: What Is Engineering DevOps and Why Is It Important? is organized into three chapters. Chapter one, “What is Engineering DevOps?,” explains engineering concepts and terms used in this book that have proven to work with many clients I have encountered during my consulting experiences. My depiction of a DevOps Engineering Blueprint is presented for as a useful, practical “big-picture” reference for discussing engineering the major parts of DevOps. Chapter two, “Nine Pillars of DevOps,” describes a classification of DevOps practices that I have found to be useful characterization of DevOps engineering practices readily applied when engineering DevOps implementations. Chapter three, “Why Is Engineering DevOps Important?,” explains the benefits of taking an engineering approach to engineering DevOps.

    Part II: Engineering People, Processes and Technologies for DevOps provides a comprehensive explanation of recommended engineering practices for the higher levels of the DevOps Engineering Blueprint. DevOps transformations are not normally understood or applied using strict engineering principles. Engineering practices are presented in nine chapters. “How Should DevOps Be Engineered?” includes a discussion of DevOps Engineering Maturity Levels for each of the Three Dimensions (People, Process, and Technology), Twenty-Seven DevOps Engineering Critical Success Factors and Lean DevOps Value-Stream Pipeline Engineering.

    I describe recommended engineering practices for the top layers of the DevOps Engineering Blueprint in the next chapters, which include “Value-Stream Management (VSM),” “Application Release Automation (ARA),” “Version Management,” “Continuous Security (a.k.a. DevSecOps),” “Service Catalog,” “Governance,” “Site Reliability Engineering (SRE),” “Disaster Mitigation” and “Recovery.”

    Part III: Engineering Applications, Pipelines, and Infrastructures Engineered for DevOps provides a comprehensive explanation of recommended engineering practices for the lower levels of the DevOps Engineering Blueprint, presented in six chapters as follows: “DevOps Applications Engineering,” “CI/CD Pipelines Engineering,” “Elastic Infrastructure Engineering,” “Continuous Test Engineering,” “Continuous Monitoring Engineering” and “Continuous Delivery and Deployment Engineering.”

    Part IV: DevOps Seven-Step Transformation Engineering Blueprint provides a description and tools for my approach to realize and evolve DevOps. The seven steps are Visioning, Alignment, Assessment, Solution Engineering, Realization, Operationalize and Expansion. By using these tools, DevOps leaders and practitioners can create, implement, operate and expand their DevOps across the organization. The chapter goes further to explain how to evolve DevOps from a successful First Way DevOps (Continuous Flow) toward realizing more advanced Second Way (Continuous Feedback) and Third Way (Continuous Improvement) DevOps implementations. This part includes a discussion of “Beyond Continuous Improvement”—a look at emerging technologies shaping DevOps in the future, and how you can prepare your DevOps and yourself for the future. This part includes a discussion of how to set up an effective DevOps engineering training program that supports continuous learning of DevOps engineering skills needed to maintain and enhance DevOps.

    Part V: Appendices, Continuous Learning, and References includes materials and sources I have found most useful for engineering DevOps.

    To get a copy of my book “Engineering DevOps: From Chaos to Continuous Improvement … and Beyond,” or start reading for free, click here.

    — Marc Hornbeek