devopsy.kinsta.cloud

Tag: best practices

  • Best of 2021 – Nine Pillars of DevOps Best Practices

    Best of 2021 – Nine Pillars of DevOps Best Practices

    As we close out 2021, we at devopsy.kinsta.cloud wanted to highlight the most popular articles of the year. Following is the fourteenth in our series of the Best of 2021.

    In a prior blog I explained my 27-factor DevOps assessment model which includes nine pillars of DevOps and three dimensions of people, process and technology.

    The nine pillars represent categories of practices. In this blog, I include examples of practices for each of the nine pillars. In a future blog I will explain how to use these practices to conduct a gap assessment for your DevOps practices.

    Leadership Practices for DevOps

    • Leaders demonstrate a long-term vision for organizational direction and team direction.
    • Leaders intellectually stimulate the team by encouraging them to ask new questions and question basic assumptions about the work.
    • Leaders provide inspirational communication that inspires pride in being part of the team, says positive things about the team, inspires passion and motivation and encourages people to see that change brings opportunities.
    • Leaders demonstrate support by considering others’ personal feelings before acting, being thoughtful of others’ personal needs and caring about individuals’ interests.
    • Leaders promote personal recognition by commending teams for better-than-average work, acknowledging improvements in the quality of work and personally complimenting individuals’ outstanding work.

    Collaborative Culture Practices for DevOps

    • The culture encourages cross-functional collaboration and shared responsibilities and avoids silos between Dev, Ops and QA.
    • The culture encourages learning from failures and cooperation between departments.
    • Communication flows fluidly across the end-to-end cross-functional team using collaboration tools where appropriate (for example Slack, HipChat, Yammer).
    • The DevOps system is created by an expert team, and reviewed by a coalition of stakeholders including Dev, Ops and QA.
    • Changes to end-to-end DevOps workflows are led by an expert team, and reviewed by a coalition of stakeholders including Dev, Ops and QA.
    • DevOps system changes follow a phased process to ensure the changes do not disturb the current DevOps operation. Examples of implementation phases include: proof of concept (POC) phase in a test environment, limited production and deployment to all live environments.
    • Key performance indicators (KPIs) are set and monitored by the entire team to validate the performance of the end-to-end DevOps pipeline, always. KPIs include the time for a new change to be deployed, the frequency of deliveries and the number of times changes fail to pass the tests for any stage in the DevOps pipeline.

    Design-for-DevOps Practices for DevOps

    • Products are architected to support modular independent packaging, testing and releases. In other words, the product itself is partitioned into modules with minimal dependencies between modules. In this way, the modules can be built, tested and released without requiring the entire product to be built, tested and released all at once.
    • Applications are architected as modular, immutable microservices ready for deployment in cloud infrastructures in accordance with the tenets of 12-factor apps, rather than monolithic, mutable architectures.
    • Software source code changes are pre-checked with static analysis tools, prior to commit to the integration branch. Static analysis tools are used to ensure the modified source code does not introduce critical software faults such as memory leaks, uninitialized variables, and array-boundary problems.
    • Software code changes are pre-checked using peer code reviews prior to commit to the integration/trunk branch.
    • Software code changes are pre-checked with dynamic analysis tests prior to committing to the integration/trunk branch to ensure the software performance has not degraded.
    • Software changes are integrated in a private environment, together with the most recent integration branch version, and tested using functional testing prior to committing the software changes to the integration/trunk branch.
    • Software features are tagged with software switches (i.e., feature tags or toggles) during check-in to enable selective feature-level testing, promotion and reverts.
    • Automated test cases are checked in to the integration branch at the same time code changes are checked in, together with evidence that the tests passed in a pre-flight test environment.
    • Developers commit their code changes regularly, at least once per day.

    Continuous Integration Practices for DevOps

    • A software version management (SVM) system is used to manage all source code changes. (Git, Perforce, Mercurial, etc.)
    • A software version management (SVM) system is used to manage all versions of code images changes used by the build process. (Git, Perforce, Mercurial, etc.)
    • A software version management (SVM) system is used to manage all versions of tools and infrastructure configurations and tests that are used in the build process. (Git, Perforce, Mercurial, etc.)
    • All production software changes are maintained in a single trunk or integration branch of the code.
    • The software version(s) for supporting each customer release are maintained in a separate release branch to support software updated for each release.
    • Every software commit automatically triggers a build process for all components of the module that has code changed by the commit. The system is engineered such that resources are always sufficient to execute a build.
    • Once triggered, the software build process is fully automated and produces build artifacts, provided the build time checks are successful.
    • The automated build process checks include unit tests.
    • Resources for builds are available on-demand and never block a build.
    • CI builds are fast enough to complete incremental builds in less than an hour.
    • The build process and resources for builds scale up and down automatically according to the complexity of the change. If a full build is required, the CI system automatically scales horizontally to ensure the builds are completed as quickly as possible.

    Continuous Testing Practices for DevOps

    • Development changes are pre-flight tested in a clone of the production environment prior to being integrated to the trunk branch. (Note: “production environment” means “variations of customer configurations of a product.”)
    • New unit and functional regression tests that are necessary to test a software change are created together with the code and integrated into the trunk branch at the same time the code is. The new tests are then used to test the code after integration.
    • A test script standard is used to guide test script creation, to ensure the scripts are performing the intended test purpose and are maintainable.
    • Tests are selected automatically according to the specific software changes. CT is orchestrated dynamically, whereby the execution of portions of the CT test suites may be accelerated, or skipped entirely, depending on how complex or risky the software changes are.
    • Test resources are scaled automatically according to the resource requirements of specific tests selected and the available time for testing.
    • Release regression tests are automated. At least 85% of the tests are fully automated and the remaining are auto-assisted if portions must be performed manually.
    • Release performance tests are automated to verify that no unacceptable degradations are released.
    • Blue/green testing methods are used to verify deployments in a staging environment before activating the environment to live. A/B testing methods are used together with feature toggles to try different versions of code with customers in separate live environments. Canary testing methods are used to try new code versions on selected live environments.
    • The entire testing life cycle, which may include pre-flight, integration, regression, performance and release acceptance tests are automatically orchestrated across the DevOps pipeline. The test suites for each phase include a predefined set of tests that may be selected automatically according to predefined criteria.

    Elastic Infrastructure Practices for DevOps

    • The data and executable files needed for building and testing builds are automatically archived frequently and can be reinstated on demand. Archives include all release and integration repositories. If an older version of a build needs to be updated, then the environment for building and testing that version can be retrieved and reinstated on demand and can be accomplished in a short time (for example, minutes to hours.)
    • Build and test processes are flexible enough to automatically handle a wide variety of exceptions gracefully. If the build or test process for a component is unable to complete, then the process for that failed component is reported and automatically scheduled for analysis, but build and test processes for other components continue. The reasons for the component failure are automatically analyzed and rescheduled if the reason for the failure can be corrected by the system; if not, then it is reported and suspended.
    • System configuration management and system inventory is stored and maintained in a configuration management database (CMDB).
    • Infrastructure changes are managed and automated using configuration management tools that assure idempotency.
    • Automated tools are used to support immutable infrastructure deployments.
    • Equal performance for all. The user performance experience of the build and test processes by different teams are consistent for all users, independent of location or other factors. There are SLAs and monitoring tools that ensure the user performance experience is consistent for all users.
    • Fault recovery mechanisms are provided. Build and test system fault monitoring, fault detection, system and data monitoring and recovery mechanisms exist. They are automated and are consistently verified through simulated failure conditions.
    • Infrastructure failure modes are frequently tested.
    • Disaster recovery procedures are automated.

    Continuous Monitoring Practices for DevOps

    • Logging and proactive alert systems make it easy to detect and correct DevOps system failures. Logs and proactive system alerts are in place for most DevOps component failures, and are organized in a manner to quickly identify the highest-priority problems.
    • Snapshot and trend results of each metric from each DevOps pipeline stage (for example, builds, artifacts, tests) are automatically calculated in process and visible to everyone in the Dev, QA and Ops Teams.
    • Key performance indicators (KPIs) for the DevOps infrastructure components are automatically gathered, calculated and made visible to anyone on the team that subscribes to them. Example metrics are availability (uptime) of computing resources for CI, CT and CD processes, time to complete builds, time to complete tests, number of commits that fail and number of changes that need to be reverted due to serious failures.
    • Metrics and thresholds for DevOps infrastructure components are automatically gathered, calculated and made visible to anyone on the team that subscribes to them. Example metrics are availability (uptime) of computing resources for CI, CT and CD processes, time to complete builds, time to complete tests, number of commits that fail and number of changes that need to be reverted due to serious failures.
    • Process analytics are used to monitor and improve the integration, test and release process. Descriptive build and test analytics drive process improvements.
    • Predictive analytics are used to dynamically adjust DevOps pipeline configurations. For analysis of test results, data may indicate a need to concentrate more testing in areas that have a higher failure trend.

    Continuous Security Practices for DevOps

    • Developers are empowered and trained to take personal responsibility for security.
    • Security assurance automation and security monitoring practices are embraced by the organization.
    • All information security platforms that are in use expose full functionality via APIs for automation capability.
    • Proven version control practices and tools are used for all application software, scripts, templates and blueprints that are used in DevOps environments.
    • Immutable infrastructure mindsets are adopted to ensure production systems are locked down.
    • Security controls are automated so as not to impede DevOps agility.
    • Security tools are integrated into the CI/CD pipeline.
    • Source code for key intellectual property on build or test machines are only accessible by trusted users with verified credentials. Build and test scripts do not contain credentials for access to any system that has intellectual property. Intellectual Property is divided such that not all of it exists on the same archive and each archive has different credentials.

    Continuous Delivery Practices for DevOps

    • Delivery and deployment stages are separate. The delivery stage precedes the deployment pipeline.
    • All deliverables that pass the delivery metrics are packaged and prepared for deployment using containers.
    • Deliverable packages include sufficient configuration and test data to validate each deployment. Configuration management tools are used to manage configuration information.
    • Deliverables from the delivery pipeline are automatically pushed to the deployment pipeline, once acceptable delivery measures are achieved.
    • Deployment decisions are determined according to predetermined metrics. The entire deployment process may take hours, but usually less than a day.
    • Deployments to production environments are staged such that failed deployments can be detected early and impact to customers isolated quickly.
    • Deployments are arranged with automated recovery and self-healing capabilities in case a deployment fails.

    What This Means

    DevOps is a powerful tool that enables many benefits for organizations that use it. Achieving performance efficiently with DevOps depends on following best practices. By following the nine pillars of practices enumerated in this blog, organizations can achieve the performance potential that DevOps has to offer.

  • How to Build a Team That Embraces Metrics

    How to Build a Team That Embraces Metrics

    When we talk about metrics in software delivery, a lot of developers think of execution metrics — things like throughput, delivery and number of deploys. But in reality, those metrics don’t motivate anyone — at least not without connecting them to a bigger picture.

    I’ve worked in software for 23 years. I’m a three-time founder and four-time CTO, responsible for leading a 200+ member distributed engineering organization. I’m passionate about metrics because they can tell me a lot about where things are working and where they are not. On the other hand, there’s a risk that operating metrics become the focus, taking attention away from measures of business value.

    Getting your developers to embrace metrics isn’t about hand-selecting specific metrics to track. It’s about disseminating the bigger picture and sharing measures of business impact that allow your team to see progress against those larger, shared goals.

    When developers understand how their day-to-day work directly impacts the company’s goals and vision, they’re able to understand their impact, too, and will want to do better. When they want to do better, they want execution metrics to get them there.

    Clarify Your Goals

    If you want your developers to not only track their metrics but be excited to use them, there are a few things you need to do first:

    • Get clarity on the goals you’re trying to accomplish as a team and as an organization
    • Figure out how you’re executing against those goals
    • Find out what’s getting in the way

    Developers shouldn’t be obsessing over execution metrics. Rather, you need them to understand what the company is trying to achieve. What is the outcome you’re aiming to deliver to customers and for the business overall?

    If your developers have clarity on the company’s mission and vision, they understand how their work directly impacts the customer and the success of the organization, which is highly motivating.

    Individual contributors will start asking for access to more information so they can diagnose their own problems and improve their work.

    Are we spending too much time on technical investment? Does it feel like we have too many incidents, issues and escalations from customers? Are we far off on our estimates? By answering these questions, your team can determine which metrics to track.

    Don’t Make These Mistakes

    Using one metric to rule them all. A lot of engineering organizations try to rally entire departments around the same execution metrics. Velocity, or throughput, for example, is a great tool to expose other issues, but consistent velocity is not a guarantee of delivering business value. In fact, targeting consistent velocity is likely to result in tradeoffs you didn’t anticipate, like taking shortcuts to get things out the door.

    Treating the metric as the goal. Meeting a metric shouldn’t be the goal; rather, metrics should be viewed as a way to ensure your team is meeting their goals. Organizing around business value is what really helps your team understand the purpose of their work. If you’re just rallying around how many times you can deploy per day, then sure, you can be an amazing software-deploying machine — but you might be deploying changes all day that have nothing to do with the actual goals of the business.

    Being a blocker. Developers need autonomy and flexibility to do their jobs well. Give teams a high-level objective and let them determine which metrics they need to push, instead of giving everybody the same metrics and telling them to get better.

    If engineers don’t have the autonomy to make their own decisions, then they’re going to feel like they’re just part of the machine. As engineers, and as humans, problem solving is what motivates us, along with the knowledge that our skills are contributing to a larger, shared goal.

    A Team That Embraces Metrics: Putting it Into Practice

    There are three channels that are essential to communicating business-value metrics to teams. At the company level, executives will introduce their high-level goals at all-hands meetings — where the company is trying to go this year, how you differentiate in the market — the things that really matter to a business.

    One level down, during organizational or department meetings, is where to communicate how teams should think about the goals from a product and engineering perspective, while still thinking about them within that overall business context.

    It’s also important that this is understood down through the management chain. An engineering manager should be able to explain to their team the overall business goals with local-level reinforcement and how it all ties together.

    When you have that knowledge as a team, you know what you’re capable of executing because you understand what you’re trying to achieve for the company and the customer. That’s when developers start asking for more metrics because they want to understand how to get there faster.

    If You’re Not Hitting Your Goals, You Should be Asking for the Data

    The only real failure is not learning from failure. It should be okay to not hit your numbers. Set a bold target and push yourself, then learn something from your misses. Figure out what you didn’t understand, what got in your way and why.

    Estimation is hard. Trying to plan work is hard. But if you know how much you were able to deliver last quarter, you can focus on getting something similar done this quarter instead of assuming this will be the one quarter where there are no interruptions, your initiatives will all be perfectly defined and nobody will get sick.

    If you’re not hitting your business goals and you’re not looking at your historical data, then you’re missing your opportunity to improve.

    Your Team Embraces Metrics. Now, How Do You Get Them Excited?

    The first time I started using continuous deployment (CD), I was truly amazed at how quickly it allowed my team to put new functionality into the hands of our customers. We could instantly find out if they liked it, and if they didn’t, we would build another version tomorrow.

    As an engineer, CD is a great way to feel more connected to your customers because you get to put something you built in front of them the same day you complete it. You also become interested in a different form of metrics: usage and adoption.

    The connection between your work and the ultimate value to the user is much clearer when the feedback cycle is shorter. CD allows you to get that feedback from customers directly and really see the impact you have on them. That gets people excited.

    There’s no Shortcut

    You can try to drive an entire organization by execution metrics, but it simply won’t be fulfilling for anyone. If your team is aligned around what they’re trying to achieve, they will recognize the value of execution metrics and they will come looking for them.

    There’s no shortcut.

    If you start with the reason and you end with the numbers, you know you’re on the right track. You’ll know you’re getting there because you’ll see the numbers moving, your team will be excited by that, and they’ll be motivated to do even better.

  • DevOps Practices Gap Assessment

    DevOps Practices Gap Assessment

    Using DevOps to continuously deliver software has proven value for organizations that are on a path towards digital transformation and continuous improvement. To get the maximum value from DevOps, it is important to use recommended best practices, inclusive of people, processes and technologies. A gap assessment is a great way to efficiently evaluate an organization’s practices for DevOps, and determine a strategy for improvement. Gap analysis provides valuable inputs for formulating a strategy and roadmap to improve a topic including DevOps.

    My SlideShare, Gap Survey, Assessment and Analysis For DevOps, explains how my Nine Pillars Gap Assessment Tool greatly speeds up the gap survey, gap assessment and gap analysis process for DevOps.

    It is important to clarify a few definitions for this blog, because I have found that different people and organizations use these terms differently. In this blog, and in my consulting work, I use the following definitions.

    • Gap survey is a discovery tool used to collect information about the current state of practices for a topic.
    • Gap assessment is a process to determine differences (gaps) between current practices and recommended practices for a topic. Gap assessment is also used to refer to the entire collection of activities in this list.
    • Gap assessment workshop is a meeting in which the gap assessment is validated prior to gap analysis.
    • Gap analysis is the process of determining priorities for reducing gaps found during a gap assessment.
    • Gap assessment tool is software that helps perform gap surveys, gap assessments and gap analysis.

    For those that want to dig deeper on this topic, additional definitions and contextual information related to the way I do gap assessments are clarified in my book Engineering DevOps.

    I define a seven-step process for gap assessments. The steps are as follows:

    1. Pick a topic to analyze. The same processes and tools can be applied equally well to almost any topic.
    2. Determine recommended practices for the topic. Something that is worth doing a gap assessment for, such as DevOps, will normally be fairly complex with dozens, if not hundreds of practices. Categorize the practices into groupings (I call the groups “pillars”) to make the list of practices more understandable and meaningful. To do this will require some research and work. For example, I have a database of practices and pillars for DevOps, continuous testing, DevSecOps and SRE and some other topics such as DevOps.
    3. Enter the practices from step two into a gap assessment tool. The tool can be a simple survey tool you create, a commercial survey tool or something custom-made.
    4. Determine who needs to be included in the gap survey.
    5. Collect data using the gap survey.
    6. Perform a gap assessment.
    7. Perform a gap analysis.

    For the remainder of this blog, I explain the above seven steps for DevOps.

    Step one: Pick a Topic to Analyze
    In this blog, the topic is DevOps.

    Step two: Determine Recommended Practices for DevOps
    For an example DevOps assessment, I used the practices under the nine practices categories described in a blog that I co-authored called Nine Pillars of DevOps Best Practices. The nine practice category pillars are:

    • Leadership
    • Collaborative culture
    • Design for DevOps
    • Continuous integration
    • Continuous testing
    • Continuous monitoring
    • Elastic infrastructure
    • Continuous delivery
    • Continuous security

    Step three: Enter the Practices from Step Two into a Gap Assessment Tool
    A gap assessment tool, pre-loaded with a sample of DevOps practices, in a file called DevOps Assessment, can be downloaded for free from one of the resource pages found on my website Engineering DevOps. You can edit the practices categories and add/delete practices from each category if you prefer to make changes.

    Step four: Determine Who Needs to be Included in the Gap Survey
    For the gap analysis to be comprehensive, people in roles that are affected by DevOps practices need to be surveyed, or at least represented, to ensure their perspectives are included. The following are example roles that typically are included.

    • Business leaders – because they influence culture and budgets for DevOps.
    • Developers – because they need to design in accordance with DevOps practices.
    • Project owners – because they influence product work priorities.
    • QA testers – because they need to test in accordance with DevOps practices.
    • Ops – because DevOps practices affect operations.
    • Security – because DevOps affects security practices.

    The survey can be conducted for an individual application, a group of applications, or all the applications in the enterprise. However, it is important that the gap assessment and gap analysis be performed on the organization segment that is being targeted for improvement.

    Step five: Collect Data Using the Gap Survey
    A gap survey should allow each surveyed person to enter an importance level score, a practice level score and comments for each practice. All this information is essential for the gap assessment and gap analysis. In the gap survey included within my gap assessment tool, survey respondents are asked to score the importance of each practice as one of 0=not relevant, 1=not important, 2=nice to Have, 3=important, 4=very important or 5=critical. Practice level choices are 0=not sure, 1=rarely, if ever, 2= sometimes, 3=most of the time, 4=always or 5=We are really good at this. Comments that are relevant to qualify or explain the scores entered for each practice should also be entered, especially if there is any doubt or ambiguity.

    Step six: Perform a Gap Assessment
    The gap assessment process requires all the practices scores collected from the surveys to be collected and assembled to calculate an aggregate set of scores. Gap scores are calculated using a formula that weights each practice level score with the corresponding importance level score. A visual representation helps to identify practices areas and individual practices that have the highest, most important gaps.

    No matter how professionally written the practices and score definitions are, it is not unusual for some people to misunderstand and to enter scores that they would not have otherwise. For this reason, it is important to ensure that the data collected is validated before conducting the gap analysis. The preferred approach is to conduct a gap assessment workshop with key representatives from each role that participated in the survey. During the workshop, the scores for each practice that have a high deviation between survey responses are discussed and, if necessary, adjusted.

    Step seven: Perform a Gap Analysis
    The gap analysis process involves extracting the high gap practices and tagging and ranking each of them against solution categories determined by a consultant or topic expert.

    The results of the gap analysis indicate where solution strategies and implementation roadmaps need to be focused to reduce the most important gaps.

    What This Means

    While it is true you can easily stand up DevOps tools quite quickly, this alone will not realize the benefits of following effective and safe DevOps practices. Those who are serious about DevOps will do well to learn from others, and be serious about using recommended DevOps practices. This blog explained a gap assessment approach, which uses my nine pillars of best practices for DevOps. My free DevOps gap assessment tool facilitates getting a gap analysis for DevOps and priorities to improve DevOps practices for an organization.

  • A Benefit of Incompetence

    A Benefit of Incompetence

  • What To Do When Your DevOps Initiative is Failing

    What To Do When Your DevOps Initiative is Failing

    These days, it is difficult to find someone who hasn’t heard of DevOps. Statistics show that in 2017, just 6 percent of developers worldwide had never heard of it, and in 2018 this number decreased to 3 percent. You have to wonder who these 3 percent of developers are!

    Companies have been embracing DevOps initiatives with two core objectives: to reduce their time to market by delivering and deploying software faster, and to receive quicker feedback from customers. Many of these implementations have been successful. However, undertaking DevOps requires a major shift in terms of an organization’s culture and mindset, and companies often face difficulties.

    I want to examine some of the common challenges we see organizations facing and provide some best practices for overcoming them.

    Why do DevOps Initiatives Fail?

    Some of the core reasons DevOps initiatives fail include:

    • Not spreading a DevOps culture across the entire organization.
    • Choosing the tools without conducting research about what they do and evaluating whether they are a fit in your organization.
    • Setting unrealistic goals.
    • Trying to apply the same recipe that other companies have.
    • Not training your people.
    • Implementing a hybrid approach to DevOps where you attempt to keep your old culture.

    DevOps Initiatives Require Time and Patience

    Measuring your goals and monitoring your progress are the only ways to detect early warning signs that your DevOps initiative is not living up to expectations. But it’s important to note that patience is also required, as in most cases you will not achieve your objectives straight away.

    Getting to a full DevOps environment is not easy, particularly in large organizations, which are used to having siloed structures in their development process. So, start from the bottom up. Establish small goals that should be achievable, such as increasing the unit test coverage, improving the automation suit, reducing the time from code-commit to testing-ready. Measure these and use the results as possible warning signs or alert signals.

    There are also a series of steps you can take to put an initiative back on course. First, re-evaluate the process or processes you are having difficulties with and evaluate if:

    • The right metrics have been set and you are monitoring them.
    • The people applying the initiative are engaged with the DevOps culture, particularly if they are in different areas.
    • You have chosen the right tools.

    The Entire Organization is Responsible for Achieving DevOps Success

    In DevOps, the whole organization should be responsible if the DevOps initiative fails. There shouldn’t be one DevOps department or one person in charge of it; DevOps should be adopted by everyone, following the same path as Agile development. After all, they complement each other.

    Those companies that have successfully implemented DevOps have developed a strategy that fits the specific needs and demands of their organization. Although I’m currently mentioning some strategies that help companies achieve success with their implementation, there is no “magic” DevOps formula. A common mistake companies make when trying to fix a sub-par DevOps initiative is to simply copy what other companies do. Most of these companies end up abandoning the initiative because the outcome is not what they expected.

    Finally, if you’re planning to start implementing DevOps, keep in mind that it’s a process in which technical tools are as important as spreading a new culture based on collaboration, multidisciplinary, autonomy and open-mindedness. As this article points out, in many cases, the blockers to achieving success with DevOps are human, not technical.

    — Gabriel Vasquez

  • Understanding Pillar 1 of AWS’ Well-Architected Framework

    Understanding Pillar 1 of AWS’ Well-Architected Framework

    AWS’ Well-Architected Framework provides a wealth of incredibly valuable best practices and guidelines structured around five distinct pillars that more cloud-native organizations would do well to heed. The first of these pillars, Operational Excellence, recommends a number of particularly valuable techniques. However, successfully adopting the design principles and practices within Pillar No. 1 requires a strong foundation: a company culture that is willing to continuously work to improve.

    In practice, this can mean holding high-profile “Game Days” focused on advancing internal procedures, nurturing an environment where engineers and product leaders really collaborate to promote business value while pursuing rapid iterative release cycles, championing operational automation and related approaches. Fortunately, adopting the tenants of AWS’ Operational Excellence Pillar can naturally assist in implementing cultural changes where they are needed.

    Design Principles for Operational Excellence

    To achieve operational excellence within AWS, incorporate these six design principles:

    Perform Operations as Code

    The cloud makes it possible to apply the same rigorous practices used to reduce risks in software development (such as version control and automated testing) across your full application environment. Cloud infrastructure allows all aspects of operations to be defined, managed and implemented in software code, spurring a revolution in their automation and consistency.

    Annotated Documentation

    Without effective automation, maintaining documentation is tedious to keep in sync with reality—ultimately making it challenging to troubleshoot issues when they inevitably arise.

    Fortunately, the cloud makes it possible to create documentation automatically as an artifact of the build process. This documentation is detailed and annotated, always up-to-date and can be read by us humans (as well as systems). With automated documentation, engineers can react to issues more rapidly and accurately, and enlist automated systems to do the same.

    Make Frequent, Small and Reversible Changes

    While agile, iterative development methods have become commonplace in many areas, product-driven organizations tend to stick to large releases. However, operational excellence calls for a systems design that incorporates tightly focused, failure-resistant components in an environment designed for constant change and improvement. Implementing this flexibility allows for more rapid deployment of new iterations—but easy rollback if issues pop up.

    Refine Operations Procedures Frequently

    Operational excellence requires holding regular meetings to reflect, suggest and implement improvements to operations procedures, the same as those held when practicing any agile methodology. Game Days, where procedures are tested and iterated on, will help build and strengthen excellence as well.

    Anticipate Failure

    Rather than waiting for failures to strike and then studying the causes in a post-mortem, practicing routine “pre-mortem” exercises that recognize failure points before any harm is done. Game Days can then be used to investigate these scenarios and ultimately bolster resilience. For example, premier cloud-native organization Netflix demonstrates this design principle with its Chaos Monkey project, which is purpose-built to strengthen application resiliency against random instance failures.

    Learn From All Operational Failures

    Operationally excellent cultures have the openness to share post-mortem findings across the organization without pointing fingers. This broadly shared knowledge can then inform improvements in all areas of the business, not just in engineering and product departments but marketing, finance, and others as well.

    Critical Components of Operational Excellence

    Embracing the six design principles above paves the way for you to practice operational excellence in three key areas defined by the Well-Architected Framework: Preparation, Operation and Evolution.

    Preparation

    Operational excellence requires operations teams to possess a clear and complete understanding of the goals for each workload within the complete system, and how those goals will be achieved.

    Operational Priorities

    This clarity must include knowledge of the workloads the operations team is responsible for, the shared business goals and their role in achieving them and any relevant regulatory or compliance requirements. This information allows teams to correctly prioritize those workloads with greater importance or complexity.

    AWS offers these helpful resources for setting operational priorities:

    Businesses can also enlist AWS Certified Partners—experts selected by AWS to provide consulting, and professional and managed services as an extension of a company’s own operations team—to make sure priorities are set wisely.

    Design for Operations

    A well-architected workload is designed for operations from the beginning, with consideration for deployment, updates and similar needs. AWS provides the ability to model entire workloads as code, from applications to infrastructure, policy, governance and operations. It also offers a robust set of tools and services for operation design, such as CloudFormation and the AWS Developer Tools. Businesses can also create systems featuring the observability and insights needed to operate workloads by using AWS Cloudtrail, AWS VPC Flow Logs, AWS CloudWatch and more.

    Operational Readiness

    Beyond the technology used, operational excellence is also a function of elegant, consistent processes and procedures for deploying and operating workloads. This means maintaining accurate documentation, building a well-trained and resourced team of experts and having governance that makes sure everything is done the right way.

    It’s also smart to utilize automation when evaluating environments. CloudWatch events can be automatically addressed by scripting procedures created with AWS Systems Manager’s Run Command and Lambda. Configurations can be automated and benchmarked against best practices using AWS Config rules.

    Operation

    The first step to operational success is clearly defining it with key metrics based on shared business goals, so that events can be addressed with those goals in mind and success can be recognized when (or when not) achieved.

    Understanding Operational Health

    Because no two businesses are the same, the definition of operational health and success will vary. For example, one company might have workloads that demand low-latency and high-throughput performance to be prioritized over cost. Another might require a focus on cost-effectiveness and high availability. Operations teams must first understand those priorities and how they serve business goals to deliver what is actually needed.

    AWS offers tools and services for analyzing workloads to serve specific goals. CloudWatch Logs and Dashboards can provide system and business level views of key metrics. Amazon ElasticSearch and Kibana can offer further visualization of operational health metrics, as can outside tools and services like Logstash and Grafana.

    Responding to Events

    When events occur, operational excellence means having prepared and even anticipated failures and putting metrics, processes and procedures into action for a quick and effective response. At the same time, teams must have metrics to understand the business impact of workload components to resolve the highest priority issues first. AWS enables teams to script automatic event responses, such as automated rollbacks triggered by failures. Amazon CloudWatch offers a powerful service for these automated responses.

    Evolution

    Curiosity and a desire to learn and improve are essential traits for operations teams pursuing operational excellence. Every experience should be viewed as an opportunity to learn and to share that new knowledge base. The best teams enjoy learning from failure and by asking for the perspective of other business units.

    AWS offers an incredible platform for the analysis and experimentation needed to evolve practices. Amazon CloudWatch and CloudTrail can be combined with Amazon ElasticSearch with Kibana. Data exported to Amazon S3 can enable analysis with Amazon Athena and Amazon QuickSight. AWS also helps share best practices with CloudFormation templates, Chef Cookbooks, Ansible Playbooks, Lambda functions and more. 

    — Jonathan LaCour

  • 3 Tips to Build A DevSecOps Organization

    3 Tips to Build A DevSecOps Organization

    For organizations undergoing digital transformation today, modernizing the existing environment can present serious challenges when it comes to security. Whether you’re dealing with a transition from legacy to cloud or hybrid models, or shifting from ITIL and waterfall methodologies to DevOps, the increasing number of technologies and complexity mean you need to rethink your security posture. The traditional approach to security—that is, a static strategy that largely involves patching on piecemeal measures at the 11th hour—is simply not a winning formula anymore, especially when dealing with requirements such as privacy compliance or contractual agreements.

    DevSecOps represents a fundamental shift in which real business needs drive a dynamic, living/breathing approach to security based on continuously changing requirements. To evolve from DevOps to DevSecOps, an organization must focus on integrating security into the very fabric of the software development cycle, and work to increase intelligence, situational awareness, and collaboration.

    Here are three critical ways to consider to ensure your DevSecOps strategy is up to snuff.

    Create a Single Source of Truth

    When you have multiple teams trying to work at breakneck speed, having one absolute source of data is essential. Gone are the days when we could rely on static spreadsheets that lived locally on this or that person’s computer, and even communication mechanisms such as email are too manual and out of sync to be trusted. What’s more, it’s impossible to draw meaningful correlations and map trends if your data is sitting in silos across your organization.

    Creating a single source of truth will ensure the greatest accuracy of information for everyone. You need to pinpoint where your data is coming from, how it should be collected and how it should be shared. You’ll want to integrate your full tool stack and workflow, and harness automation to streamline hand-offs between collaboration tools, system updates, chatbots and more.

    Shift Left: Secure By Design

    Relying on firewalls and antivirus as your primary security measures is a bad, bad habit. The key is instead to shift left of these elements and work to embed privacy from the start. This is the new age of security, using a risk-based approach instead of a reactive one—that is, identifying what needs protection, why it must be protected and how you will do so. It’s also understanding that security should not be just an external threat perspective, but also having visibility into what’s happening internally.

    The key drivers of your secure-by-design approach for DevSecOps should be privacy compliance, contractual obligations and threat models, both known and emerging. The goal here is to be proactive and avoid spending valuable resources to address threats or vulnerabilities that could have been prevented. And in case you didn’t already know, properly written, patched and documented code accounts for about 80 percent of security!

    Understand Your Environment

    If your organization has embraced DevOps, then you’re likely aware of necessities such as process, collaboration and automation. However, these can sometimes come at the expense of other important things, including privacy and security. A lot of this is due to lack of oversight and poor visibility into change management.

    As organizations accelerate their adoption of cloud services, threat vectors are ever-expanding. As such, you need to have complete situational awareness of your organization. You need to know what to monitor for and when, and this cannot be limited to the events directly related to security. Instead, focus on extending your perimeter of knowledge beyond your DevOps pipeline and ensure you’re monitoring everything from operating system logs and directory systems to DNS and servers. Without all of this context, there’s simply no way to correlate security incidents with other data from your IT environment. This is the information you need to document processes, workflows and playbooks, and ensure your teams can communicate and collaborate rapidly to address issues before the business is impacted.

    Remember, when it comes to the ultimate big-picture goal of DevSecOps, it’s always about minimizing the financial impact to your organization. Whether we’re talking about your reputation or lost time and resources, the bottom line is dollars down the drain. And the data is compelling so far—in its “State of DevOps 2017 Report,” Puppet found that high-performing DevOps teams experience a 96X faster MTTR from downtime and a 5X lower change failure rate compared to lower-performing teams.

    This article was co-written by Domnick Eger.

    — Robert Hawk

  • Making the Database Key to DevOps

    Making the Database Key to DevOps

    Here are three best practices for doing DevOps with the database

    The speed of business today demands that the development and deployment of applications is fast-moving, with frequent yet error-free releases. That’s why the adoption of DevOps is trickling down from Amazon, Facebook, Google and the other usual suspects to every company that relies on technology to drive its communications or sales with users.

    The database is now entering the picture, too, because when front-end applications are changed, there often needs to be a change at the back end to accommodate it. Redgate’s 2018 “State of Database DevOps Survey” found that more than one-third (35 percent) of organizations are deploying database changes either daily or more than once a week.

    Some are working even faster, with companies such as online travel provider Skyscanner moving from releasing database changes once every six weeks to 95 times a day when it first introduced database DevOps—and increasing the frequency since.

    This accelerated speed of delivery is a key benefit of database DevOps, with 28 percent of respondents to the DevOps survey quoting it as the biggest advantage. But as releases become more frequent, so, too, does the chance of something going wrong, along with the difficulty of pinpointing the exact cause in a large, complex, fast-changing environment.

    Database DevOps Best Practices

    Even with the most comprehensive testing, issues can occur. And as more and more developers now work across both the application and database, and with larger, more disparate teams, companies need the ability for everyone to drill down quickly to identify and fix issues.

    Achieving this means bridging the database and application worlds by focusing on three key areas.

    Deployment Choice

    Different developers and database administrators will use specific tools to deploy database changes. This can depend on their experience, skills or simply that they have a preference for a particular way of working.

    You need to be able to allow this choice, but without affecting the ability to track down issues. Therefore, ensure that you have a monitoring tool in place that can pinpoint not just when and where changes were made, but which tool was used. This will help quickly find the root causes of any issues, by providing the full context around the problem.

    A Single View

    Data is the lifeblood of many businesses—companies now have more and more databases within their organization, often spread across different locations. However, no matter how complex your IT environment or the size of your estate, you need to be able to monitor database performance from a single place. That way, you can easily identify issues and pinpoint precisely which database is being affected.

    Wherever a database is, and whatever the distance it is being monitored from, the monitoring also needs to be in real time, with updates in seconds, not minutes, providing the reassurance that you always have the latest picture of what is occurring. This enables you to become more proactive, spotting developing trends and issues such as deadlocks or blocking processes, before they affect your users or overall performance. Having a single, up-to-date view is therefore essential when integrating the database into DevOps processes.

    Fast, Customizable Alerts

    The ability to make frequent, reliable, repeatable changes to the database as well as the application is integral to DevOps flows, and monitoring them means evaluating dozens of different parameters constantly.

    Clearly, it’s impossible to keep an eye on everything that is happening, even when events are condensed to a single monitoring screen. Therefore, ensure you’re able to easily create and customize alerts for specific parameters that are important to your business, so that they don’t get missed. This enables you to better understand DevOps processes and ensure that you have early warnings of potential complications, no matter where they occur.

    For the database to be an integral part of DevOps, organizations need a complete, interoperable view of their entire estate so  they can track down problems, monitor processes and spot issues before they affect users. That view should be equally available to the whole DevOps team, including developers, so they can see immediately what effect their latest change is having and take the appropriate action to fix any breaking change while it is still fresh in their mind.

    Only then can databases take their key place in DevOps flows, eliminating bottlenecks and speeding up development to meet fast-changing business needs.

    — Matt Hilbert

  • What’s the Impact of DevOps on Business Continuity?

    What’s the Impact of DevOps on Business Continuity?

    Organizational requirements change rapidly. The strategic plan you created in the past and previously only updated annually is now out of date within a quarter.

    Agile and DevOps practices have been adopted to help resolve complex business issues in these times of flux through technology improvement, providing organizations with an increased certainty of intent, quality and safety.

    However, integrating the potential impact of Agile or DevOps into business continuity practices isn’t always considered a sound and highly valuable idea. The continuity of an organization is often considered important only after a crisis has occurred—the proverbial closing of the gate after the horse has bolted.

    Below are a few ways that business continuity capabilities need to change in light of the impact of Agile and DevOps.

    First, a Business Continuity Primer

    Maintaining business continuity in your organization:

    • Provides the ability to adapt as required to meet competitive, regulatory, organizational, technology or new market opportunities.
    • Provides your stakeholders, suppliers and, most importantly, customers and staff with the assurance the organization will remain a sound, going concern.

    For example, many organizations have a data repository in an application such as SAP, Oracle or SharePoint. This repository is saved into the cloud or other media daily or as needed.

    That is great. But if you mapped the processes used inside a department, you might find that employees download this data into Excel or some other tool and manipulate it for their own purposes. This is where the real work is being performed, and if this data is lost, the business continuity impact could be quite severe.

    DevOps and Business Continuity

    Agile and DevOps methodologies suggest that you design, create, test and release your technology changes in rapid cycles of two weeks or less, with any technology change introduced probably having a business process—and thus business continuity—impact.

    Therefore, whatever you release must always be ready for production. But how can you provide this level of assurance? This is how:

    • All “epics”—high-level business needs that are broken down into smaller stories—should also have a simple statement highlighting the impact of request.
    • All user stories should have tests to confirm the work being performed is as requested but also that they would be usable no matter where needed.
    • Feedback is provided to the development, infrastructure, cloud provider and product owner regarding the success of the tests.
    • Any issues need to be resolved before the work/change is released.
    • Issues and other lessons become part of the backlog of new work.
    • If a continuity test was not performed, then this user story is prioritized for completion in near future sprints.

    DevOps and Business Continuity Maturity Assessment Questions

    There’s a relatively simple way to understand how your organization’s business continuity capabilities stack up: Ask a number of focused questions (to the people who should be able to provide suitable answers).

    Start by asking:

    • How often do you test your code as you create it? (“Oh yes, we do that.”)
    • Do you always test before you go live? (“Oh yes, we do that.”)
    • How many of you create your code—be it infrastructure virtualization, new applications, internet, etc.—in environments that replicate, as close as possible, the production environment? (“Oh, hmm, no our environments are all different.”)

    Step it up with the “killer” questions:

    • How can you ensure that what you’re about to release is safe?
    • How can you ensure that the work you’ve done is not a waste?
    • How do you know that your organization can be kept “in business” and will not be exposed to an adverse effect from the change, or a business continuity situation?

    Understand How the Organization Relies on Technology

    As mentioned, business continuity is ultimately all about keeping your business in business, and thus there’s a need to understand how technology is employed to maintain:

    • Regulatory compliance.
    • Competitive advantage.
    • A safe place for your people and suppliers to work.
    • The needed access to your products or services.

    And thus, you will need to:

    • Map the various value streams of work performed.
    • Map the technology used at each step, down to workstation applications.
    • Quickly assess the impact of that stream being blocked or unavailable by a facilities, people or technology issue.
    • Create a plan to make that value stream as available as required.

    Organizations that have mature business continuity practices will, of course, also use technology to automate these activities. But they will also ensure that changes needed are tested as soon as possible in the product life cycle.

    If your tests are not performed in a production-like environment continuously, then how do you know you are ready and safe to go live? Go-live should not just encompass the new code but also the processes, training, security, continuity testing, organizational change and supplier update, if needed.

    Fortunately, the DevOps tools that support these capabilities are readily available and thus need to become part of any development, testing framework or infrastructure.

    Ultimately, it’s important to sufficiently invest in keeping your business in business. DevOps practices are another element to factor in to your organization’s business continuity management policies, processes and practices.

    — Sarah Lahav

  • 3 Best Practices for Better Incident Response

    3 Best Practices for Better Incident Response

    It only takes a headline-grabbing mass internet outage or two to drive home the financial and brand incentives of keeping an application up at all times. That’s why in today’s DevOps organizations, incident response is a common topic of conversation.

    There’s a lot of talk within the DevOps community about organizational processes and designs for effective incident response such as the Incident Command model, as discussed by our friends from Fastly in this talk. We also frequently discuss technologies and tools for incident response, spanning monitoring, alerting, logging, mitigation and many other focus areas. At NS1, we make use of the Incident Command model and have invested in the usual array of technologies for incident response, along with developing many of our own.

    Even after investing significant time considering how to respond to an incident and the budget they allocate for related technologies, it’s still possible for organizations to fail to respond effectively when a real incident arises. As a provider of mission-critical DNS and traffic management services for many of the largest internet properties, we know there’s little room for failure. We’ve learned that augmenting our processes and technologies with three simple practices makes a huge difference in real-world incident response situations.

    The Value of a Good Checklist

    A good checklist is a favorite tool of mine when it comes to effective incident response. A checklist is a simple, concise, easy-to-understand tool to help an operator recall key steps to take in a particular situation. In an intense incident response scenario, it can be easy for an operator to skip or forget steps in their response. Rather than being overly prescriptive, a checklist’s job is meant to help an experienced professional who is familiar with the systems they’re operating make the best use of their knowledge in a high-pressure situation.

    A checklist that I have really come to appreciate doesn’t have anything to do with DevOps or networked systems operations. It is the Cessna 172S Emergency Procedure for engine failure during flight:

    In his book, “The Checklist Manifesto,” Atul Gawande discusses this checklist in depth. This example demonstrates some of the key principles behind a good checklist: it’s simple and short, not overly specific or prescriptive; it’s limited to only the most important actions the operator needs to recall during an incident mitigation; and, because of its simplicity and clarity, it helps the operator calmly execute those key actions.

    This Cessna checklist offers another great lesson for operators of today’s IT infrastructure. As Gawande writes, “Because pilots sometimes become so desperate trying to restart their engine, so crushed by the cognitive overload of thinking through what could have gone wrong, they forget this most basic task: FLY THE AIRPLANE.” In IT operations, it’s no different—we can sometimes become so distracted by root cause analysis and the desire to know why an incident has occurred that we forget to focus on mitigation first—analysis can come later, once the incident is resolved.

    Running Drills

    It’s possible to have well-vetted checklists, great tools and technologies and the best organizational models, but they all can be forgotten in an instant of panic if people aren’t used to using them. So, how do we get great at using them all in real-world situations? Through practice.

    Any incident response strategy needs to incorporate fire drills because they ensure we regularly exercise our processes, tools and checklists in simulated scenarios that approximate real-world incident response situations. There are a number of ways to create fire drills with respect to networked systems. One tool we often discuss in the technical operations community is one built by Netflix called Chaos Monkey, a software embodiment of what has become an emerging discipline: chaos engineering. It’s a methodology for improving the resilience of a distributed system by creating failure situations—for example, by randomly “breaking” virtual servers in a distributed application to force failure handling systems to engage. This is a kind of “automated” fire drill that drives resilient architecture.

    A “war game” that simulates DDoS attack situations is another drill we have benefited from. We build tools similar to those used by sophisticated attackers and regularly exercise our DDoS mitigation response by literally attacking production systems. In this type of fire drill, we split our team into attackers and defenders (similar to the red/blue team approaches used by information security organizations for penetration testing). We’ve learned that running good DDoS war games requires us to attack real systems with real attack techniques, which helps us understand the dynamics of our distributed systems and builds muscle memory in our team around key steps needed to respond to real-world incidents. We’ve also found that fire drills such as these are great tools for team-building; they’re almost like running football or soccer scrimmages at the end of a long practice session, helping everyone on the team understand how they can best contribute and what “position” they should play in a real-world event.

    Analyzing After the Fact

    Postmortem or retroactive analysis is a final practice for driving effective incident response.  After any incident, real or simulated, a thorough review of the incident and the response is critical. Asking and carefully answering some simple questions about the response to the incident is the key: What happened and why? What worked well in our response? How could we have responded more effectively?

    The reason to do this type of analysis is to apply learnings from the incident and create a feedback loop, so that in the future, your team’s response to incidents in general—and certainly to similar incidents—is more effective. Obviously, it’s critical to fix any issues that surface during an incident response in your systems, incident response processes and mitigation tools. You should also review relevant checklists after an incident: what was missed, or what was confusing? And finally, feed the outputs of real or simulated incidents into your fire drills so you can practice similar scenarios in the future.

    Be Prepared

    Like death and taxes, incidents are going to occur. Companies that deliver content or applications in networked systems need to be prepared for them. Best practices such as creating a simple checklist and conducting drills helps a team’s incident responses improve over time, helping to reduce stress during an actual incident and increasing the likelihood of a swift resolution.

    About the Author / Kris Beevers

    Kris Beevers is founder and CEO of NS1. He is a recognized authority on DNS and global application delivery, and often speaks and writes about building and deploying high performance, at scale, globally distributed internet infrastructure. He holds a PhD in Computer Science from RPI. Prior to founding and leading NS1, he built CDN, cloud, bare metal and other infrastructure products at Voxel, which was acquired by Internap in 2011.