devopsy.kinsta.cloud

Tag: devops culture

  • Reducing Friction in Developer Collaboration

    Reducing Friction in Developer Collaboration

    Given how the individual unit of deployment has gotten increasingly smaller, such that the unit of deployment is usually a microservice, it would be tempting to think that individual developers are more able to control how well the code they write (the service they deploy) will work in production. In fact, though, almost the opposite is true. Instead, as the number of services in an environment increases, whether or not one service will work correctly in a production environment has more to do with how well it works with other services rather than the quality of the code.

    This makes collaboration between developers even more critical than ever before, but the packaging of code in microservices actually makes collaboration more challenging. Developers don’t collaborate by working together on a shared codebase — instead they work completely independently on services that could easily be written in different languages, providing the illusion of complete independence.

    In most organizations, ensuring that services work together is a largely informal process. It’s managed by walking over to a colleague’s work station and asking what updates they’re working on or what version libraries they’re using. In a remote work environment, teams rely on Slack messages or Zoom calls to solve issues relating to services interaction. This creates a lot of friction for something that is a critical part of the software development workflow. Failures in service integration can and do cause problems in production.

    Pay Attention to Service Communications

    A perfectly crafted microservice is not going to run in isolation in the production environment. Compatibility issues between each service and upstream and downstream dependencies are just as likely to cause bugs, downtime and poor performance as problems with the code deployed in a container.

    The only way to catch compatibility issues before production is for developers to work together and communicate about the updates they’re working on and how those might impact interdependent services. In the current system, however, this level of collaboration introduces a lot of friction in the development workflow, reducing the individual’s and the team’s velocity. Since developers are generally evaluated based on velocity and code quality — but not necessarily on how well their services work with other services — if the friction remains, developers will be tempted to deploy without fully understanding how dependencies will be impacted.

    Since CI/CD pipelines are generally not set up to test how well services communicate with each other, problems with service fit that aren’t addressed at the development stage generally aren’t found until either a canary deployment goes wrong or there are problems in production.

    Replacing Tribal Knowledge

    One of the other reasons companies need to focus on developer collaboration is that remote work has made it even harder to communicate tribal knowledge. In my experience, no development team has been able to completely wean themselves off of tribal knowledge. Even teams with extensive technical documentation often had or have experienced team members who maintain a unique understanding of the application’s business logic and can share that knowledge when needed.

    Tribal knowledge also relates to service integration and communication — often, insights about how services work together isn’t well-documented, even if each individual service is well-documented. Often, the easiest way to see how they would work together is not to have the two developers responsible for each service work together, but rather to have a senior developer take a look. Especially as development teams work remotely, that type of review isn’t feasible.

    How to Help Developers Collaborate

    Most teams have ways for teams to communicate — they have a Slack setup, they have internal forums and video conferencing solutions. But they often don’t realize that these communication platforms aren’t really collaboration platforms — they are not providing a way for developers to actually work together on interdependent services the way they might if working on a shared codebase. Communication isn’t enough for true collaboration to happen — developers need a way to work together and see how colleagues’ changes impact their own services.

    Facilitating true collaboration, the kind you would get from two people looking at the same screen and coming up with solutions together, is more challenging than making it easier for developers to talk to each other. Without true collaboration, teams will continue to have trouble with communication between services — and there’s no Slack for that.

  • The Single-Sentence Postmortem

    The Single-Sentence Postmortem

    Do you have four seconds?  In the time it takes you to read this sentence you could be done with your incident’s postmortem.

    “But my postmortem has 8 sections!  I have to construct a critical timeline, restate my followup tasks, show the major milestones, evaluate root cause…”

    Stop right there.

    First of all, your incident management system should be automatically capturing your timeline, follow-up tasks and major milestones as they happen — so you shouldn’t be worrying about reproducing that content at all.

    Second of all, the most critical thing a responder can bring to a postmortem, in the moments right after the incident concludes, is the one-sentence personal context of what actually happened. It’s the same sentence they’d say to a co-worker in the lunch room when asked what went wrong. But to achieve organizational resilience, the knowledge in that sentence has to be captured and distributed. I can’t emphasize this enough – a single sentence can unlock 80% of the value. Writing something, anything, is so much better than writing nothing, and yet, too often, we see companies skip this part of the incident management process.

    Modern incident management, as a practice, was born on factory floors.  Instead of firing the person who was considered accountable for a major incident, having that person write up their view of what happened was more impactful at preventing future incidents. That’s because only the responder has the complete context to shed light on the contextual, systemic failures that actually caused incidents, as compared to the outside view of management and other employees who will often see problems as human-centered or due to obvious oversights (using 20/20 hindsight).

    When asked if he would fire an employee who made a $600,000 mistake at IBM, Thomas J. Watson famously replied, “No! I just spent $600,000 training him – why would I want somebody to hire his experience?”

    The reality is that any amount of your unique perspective as a responder is invaluable when compared to nothing, and without it, the chances of your company learning from these mistakes is slim. That’s why we encourage a unique approach: we tell our customers to spend four seconds on their postmortem.

    Especially for those new to incident response and management practices, it’s important to develop a practice of doing postmortems right away, as soon as the incident closes. As quickly as possible, open up your postmortem, jump to the ‘What Happened?’ section, and write one useful sentence. That’s it.

    Once you do that, we know that one of two things will happen:

    1) You’ll have other things to do, so you’ll publish the postmortem, and that one critical piece of knowledge will be shared throughout the company. That is infinitely better than nothing. 

    2) Or, you’ll realize you have more to say and get it done, right then and there.

    Like putting your running shoes on even if you don’t want to go for a run — at the very least, you’re up and active; at most, you’ll get out there and rack up some miles.

    Working with companies across industries and seeing thousands of incidents filed; hearing the stories of hundreds of companies’ incident management implementations has taught us that the alternative to a quick postmortem often isn’t often a more in-depth one, it’s no postmortem at all.

    So if you’ve just finished an incident– and if you’re about to pick up your coffee and move on to the next task at hand – instead, spend four seconds writing one sentence. It could make all the difference.

  • How MLOps Can Help Get AI Projects to Deployment

    How MLOps Can Help Get AI Projects to Deployment

    Did you know that most AI projects never get fully deployed? In fact, a recent survey by NewVantage Partners revealed that only 15% of leading enterprises have gotten any AI into production at all. Unfortunately, many models get built and trained, but never make it to business scenarios where they can provide insights and value. This gap – deemed the production gap – leaves models unable to be used, wastes resources and stops AI ROI in its tracks. But it’s not the technology that is holding things back. In most cases, the barriers to businesses and organizations becoming data-driven can be reduced to three things: people, process and culture. So, the question is, how can we overcome these challenges and start getting real value from AI? To overcome this production gap and finally get ROI from their AI, enterprises must consider formalizing an MLOps strategy.

    MLOps, or machine learning operations, refers to the culmination of people, processes, practices and underpinning technologies that automate the deployment, monitoring and management of machine learning models into production in a scalable, fully governed way. Laying an MLOps foundation allows data, development and production teams to work collaboratively and leverage automation to deploy, monitor, manage and govern machine learning services and initiatives within an organization. In short, MLOps can help enterprises see AI projects through to deployment at scale by addressing the three previously mentioned barriers: people, process and culture.

    People

    There’s a skill, motivation and incentive gap between teams developing machine learning models (data scientists), operators of those models (DevOps, software developers, IT, etc.), and the business leaders who look for the results of that work. By providing actionable visibility into the different steps of the machine learning model life cycle for these different roles, MLOps can increase collaboration across departments so that enterprises can streamline the process of getting models operational and finally meet strategic goals.

    Process

    Operations teams are geared towards ensuring overall quality of service, optimizing runtime environments on their cloud, resource management and role-based access, among many other tasks. Data science teams are often unaware of the considerations these dependencies require, and, hence, models they create do not take these tasks into consideration. Additionally, lack of a proper native governance structure as it pertains to machine learning models – with system, performance, life cycle, and user logs – stifles troubleshooting, as well as hinders legal and regulatory reporting. Organizations that don’t properly monitor their models end up introducing potential risk to their organizations. When production models don’t reflect the ever-changing patterns in data and user behavior, these factors may affect the accuracy and overall performance of the model. Machine learning-based applications are far more sensitive to production environments than typical software applications. Therefore, machine learning models need to be validated and updated over time to ensure that they’re performing properly and leveraging the most accurate and recent data. As part of implementing an MLOps strategy, by having the right monitoring processes in place to detect drift – and many other machine learning-specific metrics – enterprises can ensure their models are delivering the most accurate results, and that their projects are more likely to move forward.

    Culture

    Fear of failure (or worse) can delay organizations from collaborating to operationalize machine learning models, or prevent this entirely. Leadership needs to be clear about their commitment to the process and fully support teams in enduring the hardships involved. This is only compounded by the fact that machine learning applications typically tend to involve processes that might introduce material risk to the company. To stay on top of such transformations while still producing trustworthy value, enterprises must keep careful tabs on all model deployment processes and the updates they entail. Many operations professionals are not aware of the unique characteristics and sensitivities of machine learning. As their role is focused on managing mission-critical production environments, they are often solely concerned with the implications on their work that deploying machine learning into production might have. While these are justifiable concerns, in the final analysis, they serve as trivial reasons for significant delays in progress and deployment. The way to mitigate all these risks and get all teams aligned around these common goals is to implement a solid MLOps strategy that matches the goals and unique position of the organization and its various teams’ responsibilities.

    MLOps: All Systems Go

    MLOps allows organizations to alleviate the people, process and culture issues posed by machine learning. MLOps provides a strategy and a set of processes and best practices, a technological backbone, for managing and scaling machine learning-based services through automation. It also provides for seamless collaboration between the data science teams responsible for generating models with the teams that are accountable for running the services in production environments. This combination of process automation with role-based collaboration and governance capabilities helps pave the way toward strategic AI goals. Specifically, MLOps can help organizations move from anticipating the management and scaling of machine learning services into production and turn that into reality.

  • Best of 2020: 5 Project Management Skills Every Developer Should Have

    Best of 2020: 5 Project Management Skills Every Developer Should Have

    As we close out 2020, we at devopsy.kinsta.cloud wanted to highlight the five most popular articles of the year. Following is the first in our series of the Best of 2020.

    The days of the siloed project manager and “skilled” development team are becoming a thing of the past, thanks to the increasing complexity of dev projects. In many cases, the knowledge gap between the project manager and the developer is large, making it nearly impossible for any non-technical person to effectively manage the entire scope without significant input from the development team.

    These days, every member has to have a basic level of project management skills to be able to contribute to the overall direction. With the days of Gantt charts behind us and more companies using project management software as they work from remote locations, all team members have better access to all of the components of the project and an expanded sense of responsibility.

    When the developers take an active role in the management of the overall project, the benefits are numerous. Not only do the developers have a better sense of how long each step of the process will take—and their contribution to the larger project—but they can also more accurately prioritize the crucial elements needed to take the project across the finish line or recognize where errors, scope creep or delays may derail a project. In short, individual team members become more active collaborators in delivering better, tighter code and applications more quickly.

    Project Management Skills to Know

    That said, project management doesn’t happen magically, and it would be foolish to assume that just because someone is a skilled developer they know how to manage a project. But familiarity with a few critical PM skills and requirements can go a long way. Here are five aspects of project management every developer should understand.

    Estimation and Scheduling

    Like any sector of business, project management has its key skills and lingo. The most significant areas of focus for project managers tend to be scheduling and estimation. Scheduling is critical, as you have to assign tasks and accurately manage the workload across the team.

    Estimation is also necessary and important to get right, especially for the product development team. The estimation of a completion date triggers many other teams to perform their functions across the company and is often used by the executive team to target success or failure. But delivery dates are notorious for being missed. Working within date ranges versus single-point deadlines is highly encouraged. Ranged estimates capture and help manage the uncertainty so common in software development today, giving the entire team more opportunity for success and allowing for the unexpected roadblocks that occur with nearly any project.

    Critical Communication

    As a team, it’s essential to keep the lines of communication open. We’re not talking about unnecessary chit chat; we’re referring to critical updates that may affect the overall project. This includes having strong interpersonal skills, namely the ability to maintain clear team communication, complete document reviews and have succinct interactions about project deliverables and expectations.

    Technology and online collaboration skills are also important. Ensure all team members know how to effectively estimate their time and keep their progress updated in software the entire team can access. Ensure document sharing and team updates are enabled, so there are no excuses for missed or mixed messages.

    Problem-Solving

    You can tell a lot about a project team by how the team members solve the major and minor technology and business-related problems that occur—often daily. You might expect that Agile development and DevOps teams have institutionalized troubleshooting in development and engineering. However, teams still need to treat problem-solving as a skill that should be continuously strengthened, especially as technologies evolve. Appointing a lead to coordinate and direct others when problems arise can be an efficient way to take control of a spiraling situation quickly.

    Technical Documentation

    Writing technical documentation can be a neglected task but a valuable skill for project teams. Too often, documentation isn’t part of the team’s workflow, but it should be. Whether a project team has a staff or contract technical writer assigned to its project, documentation is an important skill. It’s also important to note that documentation doesn’t necessarily need to take place in traditional-type documents. Today’s project teams are often better served by wikis, enterprise social tools and online project management tools. What matters is that the documentation is centralized, online, secure, auditable and searchable by the team and by management.

    Risk Management

    Many of the above skills lead to an end-goal of managing risk—in this instance, the risk of not delivering a stable, feature-complete project on time. Risk management is a skill that is most effective when distributed among the project’s team members. This is especially important with teams that consist of cross-functional specialists and those with no clear oversight or direct leader. The key to distributing risk management across a project team is to make individual team members accountable and to explain the potential risks in their project tasks to the greater group. This “ownership” of their function and deliverables allows for the most realistic planning possible, requiring each team member to take accountability for precise timelines for their piece of the project. These timelines then roll up to the greater project plan and provide a more accurate estimation of completion date and potential pitfalls.

    The Well-Rounded Developer

    Project management skills are useful for all employees across an organization. There are no downsides to embracing proven tactics from the project management playbook as they’ll only serve to increase efficiency and promote better team organization and cohesiveness. Developers who have technical skills in collaboration with project management knowledge are always in high demand. Dedicating time to honing these skills is a worthwhile endeavor for any developer.

  • Antipatterns That Hurt DevOps Implementations

    Antipatterns That Hurt DevOps Implementations

    Avoiding these DevOps antipatterns can help organizations reap the benefits of implementations faster

    It’s a given that all organizations at some point will suffer from poor development practices when trying to find the perfect balance between software development and IT operations. The good news, however, is that implementing DevOps can help you improve these processes and get them synchronized and working together in harmony.

    However, understanding and mapping out the core issues and the common DevOps “antipatterns” that are holding you back from successful DevOps implementation is essential. This article will focus on the most common DevOps antipatterns that organizations tend to fall into and how to avoid or overcome them.

    What Are DevOps Antipatterns?

    At first glance, DevOps may sound like a pretty straightforward concept. At its core, DevOps is about bringing together separate but functional areas and bridges the gap between software development (Dev) and IT operations (Ops) with the ultimate goal of delivering software faster and with better stability.

    That said, DevOps antipatterns can best be defined as ideas that are counterproductive to the DevOps culture. They include:

    • Blame culture: In this kind of culture, blaming and punishing people is the norm when mistakes are done. It can happen at an individual or organizational level.
    • Root cause analysis (RCA): RCA is a process that involves identifying the “root” cause of a problem or an event, as well as the appropriate actions to take to avoid a recurrence. It’s an iterative and continuous process that must be performed until data is exhausted or all organizational factors have been identified.
    • Human error: Human error is often cited as a root cause in RCA. This is also common in “blame culture” and someone has to be held accountable for mishaps and mistakes.
    • Silos: An organizational or departmental silo defines the mentality of teams that don’t share their knowledge with other teams within the same company.

    4 DevOps Antipatterns and How to Break Them

    Wrong Perception of Your DevOps Team

    The perception you have of your DevOps team matters. It’s essential to consider the team as part of the entire process. Neglecting to provide them with structure, resources, and a positive working environment can increase your chances of DevOps failing. Furthermore, making DevOps teams more productive requires proper implementation of processes and tools.

    How to Break It

    A successful DevOps implementation requires more than just bringing people together. Structuring the team correctly and adding processes is vital if they are to succeed. Providing the necessary resources, technology and information security they need will get the best results.

    Practices such as continuous integration (CI) and continuous delivery (CD) are both concerted processes that can be used to bring development teams together in alignment so software products are delivered quickly, operations are more simple to manage and no downtime is present during deployments that are detectable to the end user.

    Poor Choice of Technology

    Technology plays a significant role in IT; however, the rapid development of new technologies has made it challenging to separate essential and overhyped tech trends. For instance, DevOps itself is a powerful digital transformation tool as it plays a critical role in new software development and problem-solving. Nevertheless, many of the latest IT trends should not be worth your attention unless you and your team are willing to implement it correctly.

    How to Break It

    This antipattern can be broken by top-level and project managers who make the decisions regarding technology choices. Asking questions such as, “What does the organization need?” or, “How well will this technology work for us?” will help when choosing what tech is right for your organization. With thousands of tools and technologies available, you need such a mindset to make the right decisions.

    Unclear Responsibilities in Teams

    You may have come across this problem in many other non-IT projects. However, in both IT and DevOps teams, the consequences can be severe if solutions aren’t implemented quickly. If team members aren’t clear on their roles and responsibilities, this can lead to poor decision-making and unnecessary interference in other people’s positions.

    How to Break It

    First of all, you need to clearly define roles and responsibilities for everyone in the team. A clear definition of roles boosts productivity. For example, having a data scientist in your DevOps team can play a huge role in business success due to their very specific data analytics skills, techniques and tools. In fact, 89% of business managers agree that hiring a data professional will help to improve success at their respective organizations. There just needs to be active and consistent communication between team members for everyone to conduct their own roles and responsibilities efficiently.

    Every team member should follow the strategies and plans in place so timelines can be monitored and effective top management decisions can be made. When implementing complex DevOps architectures that require teams and software to work together, poor decision-making will be an issue if roles are left unassigned.

    Wrong or No Code Deployment

    To put it bluntly, there’s no DevOps without code deployment. With this being the ultimate result, DevOps makes no sense if it fails. This is quite a challenge for several organizations, as having Dev and Ops teams working in harmony can seem impossible.

    How to Break It

    As mentioned above, code deployment is often a challenging task for DevOps teams. However, organizations can break the resultant antipattern by integrating the units correctly to enable a collaborative environment, a great strategy for successful DevOps implementation.

    Such an environment will also make code deployment easier by reducing the need for a series of meetings and reviews, which ultimately delay teams from getting code to deployment.

    NoSQL’s Key Role in Database DevOps

    DevOps processes need to work in a synchronized manner to be successful. Issues with the database can cause it to slow down and affect the delivery of database changes. However, NoSQL’s unique approach is designed to simplify the process of DevOps implementation in the following ways:

    Faster scalability

    Many small companies fail to maintain peak performance with greater workloads. A more significant workload calls for better tools that can help maintain peak performance, subsequently making it easier for organizations to scale out.

    More efficient technical support

    Having technical support that doesn’t hold you back is key to DevOps success. The right NoSQL solution enables you to solve problems before they become full-fledged, expensive and time-consuming.

    Faster adaptability

    The changes that new codes bring for databases can be challenging. NoSQL makes it easier to deliver solutions to client requests, as it organizes them using dynamic indexing. Therefore, it verifies all queries before answering them with the appropriate index.

    Platform testing

    Testing is one of the critical steps you need to go through before actual implementation. Databases should be flexible enough to allow you to test queries without interfering with the real data. Testing helps to determine how new changes will impact the development process.

    Conclusion

    As you can see, there are many ways to implement DevOps systems in your business to deliver products faster, offer more technical support and automate essential tasks.

    All of the requirements we have covered are necessary if a business is to scale and grow. Businesses need to ensure the correct strategies are in place, roles in the company have been clearly defined and common antipatterns have been identified and eliminated.

  • How Collaboration Can Optimize Product Development

    Strategic collaboration allows companies to build partnerships and seek solutions that are adaptable and specialized

    Systems engineers have faced complexity since the dawn of time. And while bringing complicated projects to market under old rules, methods and technologies was never a walk in the park, today’s highly competitive market presents new challenges to prove the old rules of product development are defunct. In fact, a recent report from Machina Research, now part of Gartner, estimates the number of connected machine-to-machine devices will increase to 27 billion by 2024, up from 5 billion in 2014. The sheer increase in machine devices alone provides a clear picture of the importance placed on systems engineers to stay the course when navigating product development’s complex nature.

    Within the constantly evolving modern product development system comes an increasingly intricate development process. Today’s system engineers are fighting to fit a square peg into a round hole by using old tactics to solve new challenges. This includes an accelerated rate of innovation, increasingly complex end user asks, tighter deadlines and ever-changing regulations. Alongside these challenges, Forrester Consulting found that product development is facing five main obstacles in keeping teams from true optimization:

    • The confusing and constantly changing nature of requirements and a lack of quick development for solutions.
    • Stakeholders providing conflicting priorities, assumptions and unclear objectives.
    • Difficulty collaborating across globally distributed teams.
    • Unnecessary handoffs and delayed decisions.
    • The increased need for collaboration across diverse roles.

    So what is the solution to these problems, you ask? One could argue strategic team collaboration is the best way to address both the challenges of the modern product development landscape and the obstacles systems engineers are facing. Let’s dig into that a bit deeper.

    Team Collaboration: An Enabler of Innovation

    Teams that are still operating with outdated strategies and systems find themselves unable to adapt and adjust quickly when the market changes. With rapid growth predicted to stay at a steady rate, strategic collaboration allows companies to build partnerships and seek solutions that are adaptable and specialized. This includes sharing data throughout your whole business, including distributed teams and partner organizations, which allows transparency to reach the entire supply chain. With the COVID-19 pandemic causing supply chain halts and the backup of product releases, it is now more important than ever to ensure communication throughout the entire development process.

    With the country leaning on a remote workforce more heavily this year, virtual meetings and emails are not sufficient enough as communication tools for a team that has to coordinate across departments, roles, companies and geographic boundaries. As a first step to meet these demands, system engineers should consider the following strategies:

    1. Ask your team the question, what does success look like? If each member has a different end goal, you will end up wasting valuable time. By defining what success looks like, you can align your teams on what you are building and clarify expectations upfront. This saves time and allows for better communication across teams.
    2. Support better decision-making through situational awareness. When you have communication upfront about a team’s end goal, they are far more likely to make decisions that work toward achieving that. Without situational awareness, employees are unable to comprehend the impact of their decisions or clearly define their roles and responsibilities. This makes it difficult to settle on the right choice when faced with a bump in the road.
    3. Ensure you’re up-to-date with your traceability practices. Especially for those who need to adhere to regulations, traceability analysis allows you to ensure your system holds up under regulatory demands. Further, it meets contractual terms before you encounter a problem.
    4. Create collaboration with a purpose. Ensure relevant data is accessible to everyone to whom it may be relevant, and keep them in the loop on decisions that happen outside the process as much as possible. When you keep your teams in the know, they are able to work collaboratively and with greater situational analysis.

    At the end of the day, your goal should be to empower your whole team through data sharing, transparency and a clear definition of success. When you are able to do this, your team will be better positioned to make decisions that benefit the project and work symbiotically across diverse roles.

    We may not be able to predict exactly where the product development industry is going, but know that there will be rapid innovation. It’s best to prepare your teams now; otherwise, it will be a bumpy ride.

  • Data and Tools Don’t Solve Problems, People Do

    Data and Tools Don’t Solve Problems, People Do

    When developing software, many managers believe that data and tools will solve all their problems. If they just have enough data or the right data analysis tools, development and delivery become automated and decisions are made without human input. But in reality, humans are a necessary part of the process. There are a few pieces of data you may want to start gathering, but you often don’t need a lot of data to make the right decisions. And sometimes, you don’t need data at all. You can use the data and tools you already have in place, but your most valuable asset is your brain. To illustrate this point, I’ll provide a few examples.

    If it takes too long for your team to fix a bug, you can dig into data from your bug tracker or ALM tool, but reasoning about the process is also a necessary step. It’s not sufficient to simply look at some reports from your tooling. In fact, reasoning about your process may be all you need. You may already know where your bottlenecks are and where the waste is. Nevertheless, when you need to dig into the data, the point solution tooling you use could be the best place to do so.

    If your epics are moving too slowly, you can use a report provided by whatever tool you are using to understand lead time, as well as the cycle time between states. You can drill down from there to look at stories as they flow from inception to completion and use whatever tool you are using to look for bottlenecks and wait states. Likewise, for build through deployment to production, you can look at data out of your tooling to understand how long it takes to go through each phase.

    To illustrate this further, relatively few companies today are deploying more than once per sprint, and many, less often. Until you get your lead time down to two weeks, measuring how much undeployed code inventory has built up is useful only for its shock value. And then, you don’t need to update it every month. Measuring lead time is sufficient, and easier.

    Whatever tool you are using, however, can only model what you have in the underlying systems. If the underlying systems don’t model the wait states, your analytics aren’t going to give you any insight into your delays. Include wait state columns in your scrum board or kanban.

    A great approach to continuous improvement is the Toyota kata, and there is a great book by that name. You should read it and then implement it in your organization. But know that this can’t be tooled-up or made automatic. You do need to be able to measure for the intended improvements, but it still takes humans to do the thinking. Toyota was successful because its approach ingrained in every employee the notion of improving their processes. Toyota never minded people seeing what it was currently doing because Toyota knew that its reason for success was its ability to change and to improve faster than the competition. This requires humans. Brains.

    Example: Maybe you are looking at metrics about builds from a tool. You might look and see that you have multiple builds per hour. But if you don’t really understand the metric or what it is you’re looking at, you might not realize that some individual builds may only be built once per day, or worse. Or you might not realize that your dev team’s builds only go to QA every other week, slowing down your overall flow of value.

    Gary Gruver, in his book, “Engineering the Digital Transformation,” says, “Most organizations don’t really think closely about how value flows through their organization,” (page 99). I agree. Moreover, they too easily buy into the promise of not having to think closely about how value flows.

    Gruver continues: “The biggest change for the industry is to create a culture of continuous improvement.” He doesn’t say “tooling for VSM” or tooling for metrics. He says culture change. The stuff of people and brains.

    Gruver also says, “Software flow is very hard to measure. … That is hard with software because the size of each feature is different, which makes it very difficult to have a quantifiable measure of flow.” Another way to say this is that you shouldn’t have unrealistic expectations that your tooling will make the decisions for you. You should expect to have to think. And then maybe dig into the data already provided by various tools in your IT or software dev value stream, and even modify some of those to capture additional data. You should expect to have to analyze each component value stream, each type of work, the mix of work types and the variability of each.

  • How To Build a Culture of Resilience Through Good Habits

    How To Build a Culture of Resilience Through Good Habits

    Good habits are hard to form. I’ve been listening to the audiobook “Atomic Habits“ by James Clear on my morning runs, and something struck me. At Gremlin, along with our software, what we’re trying to promote are positive new habits for our customers. According to the author, one of the primary reasons new habits don’t stick is because there’s often sacrifice without immediate gratification.

    Psychologically, we’re wired to want instant gratification. But not all habits give immediate rewards; in fact, many delay our gratification for some time. So how do we help ourselves pursue good habits? Put simply: We need to make them obvious, attractive and easy.

    One of the ways to make them easy is to focus on what the author calls “gateway habits”—the smallest piece of the habit that can reasonably be achieved in two minutes. So if your goal is to eventually run a marathon, the gateway habit is putting on your running shoes every day.

    To build a culture of resilience at your company, start small and create getaway habits. If a team runs a GameDay once a month (time dedicated to experimenting on your systems) or even simply runs their first single chaos engineering experiment, then award that person or team with immediate recognition.

    Here are some other ways to build a culture of resilience at your organization:

    • Recognize the change to new habits.
    • Create DNR (do not repeat) items.
    • Adopt “You build it, you own it.”
    • Track the four golden signals.

    Recognize the Change to New Habits

    Incentives are a great way of kick-starting a new habit, but they don’t necessarily sustain the good behavior. It’s identifying the improvements that result from the new habit that really makes it stick. We’ll get to some specific metrics you can track later in the article, but notice for now that identifying the improvements is when gratification starts to drive enthusiasm. Ideally, that enthusiasm grows until the new habit becomes part of your identity.

    In our example of running a marathon, the moment of most significant change is when the person starts to self-identify as a runner. Then, the habit is no longer a chore, but rather part of who they are. Ideally, we want all computer engineers to adopt a specific set of habits until they consider themselves site reliability engineers (SREs) as part of their identity. That’s when the habit is solidified. 

    Create DNR (Do Not Repeat) Items

    Engineers and product managers want to ship new products and features. There’s nothing quite as satisfying in software development as deploying new code and seeing what you built running out in the world. But, if what you built is consistently breaking or providing a bad user experience, then you are hurting your customers and ultimately your business.

    To make sure we are always learning and getting better, at Gremlin we have what we call “DNR,” or Do Not Repeat work. This work consists of action items from outages and incidents that must not ever be repeated, lest we fail to learn our lesson from these failures. Practically, what this means is all feature work is halted until the issues highlighted as DNR work are remedied and the fixes are verified. In other words, you don’t get to write new code until your old code is fixed. We all know that many teams struggle with the trade-offs of moving fast, but ultimately, if you don’t have strict guidelines in place, then more often than not engineers will prioritize shipping something over making sure it’s reliable.

    Creating a DNR item is an easy way to incentivize the behavior you want to see internally by appealing to the engineer’s desire to produce new features. We convince them to write better code because better code means they get to spend less time fixing things.

    Adopt “You Build It, You Own It”

    This is the driving principle of DevOps. It is the reason behind shifting left. When the team that develops the software is different from the team that operates it, then there’s a misalignment of incentives. If I am a developer being tracked (and promoted) solely on the amount of code I ship, my focus will be on getting more bits out the door and not on ensuring the features I release will withstand the burdens of operation. That’s another team’s concern.

    That’s the motivation behind the proliferation of the “you build it, you own it” mindset at top-performing organizations. Hell, my first day at Amazon, they tossed me a pager and said “Good luck.” And while that may sound daunting (and it was), I can tell you that it not only motivated me to make sure I built systems to last, but it also fundamentally changed the way I thought about architecting systems. Spoiler: I’m not a big fan of my pager going off at 3 in the morning.

    In other words, the person or team building the system needs to be the same person or team that feels the pain if that system is failing. But it’s not just about punishment and pain, that team also needs to be recognized and rewarded when their system is running reliably. This creates an alignment of incentives that promotes the kind of habits seen across top-performing teams.

    Track the Four Golden Signals

    In monitoring distributed systems, Google’s SRE book outlines the four golden signals of monitoring as latency, traffic, errors and saturation. If I could wave a magic wand and immediately improve the culture of an organization, then I would have service-level objectives (SLOs) tied to these four metrics. But going back to habit formation: If you make acquiring a habit too difficult upfront, then an engineering team will ultimately reject it. So if your organization isn’t mature enough to create SLOs, simply beginning to track these metrics will up-level your game tremendously. They will give you an understanding of what is not working well and help guide your priorities.

    To Summarize

    Imagine a world where, after a major incident happens, the points of failure involved become DNR work. The same failure is not allowed to happen again and no new feature work will be completed until the fixes are implemented. And more importantly, no new feature work will be completed until those fixes are verified via a chaos engineering experiment, which is then cataloged and run continuously against your system. Then you take that knowledge and share it with other teams so they can run the same experiments and make sure they are immune to those failures as well.

    This is how you build a culture of resilience.

  • Digital Transformation Is More About a BizDevOps Culture than Tech

    Digital Transformation Is More About a BizDevOps Culture than Tech

    Digital transformation is top of mind for every organization now. The problem is, many of these organizations take the phrase too literally and think it’s all about adopting new technical capabilities and tools. Tech is certainly a part of it, but real digital transformation is also about changing company cultures and ways of working. Tech and culture go hand in hand for digital transformation; you can’t change one without the other.

    Successful digital transformations mean redefining the way IT works, breaking down the silos between the development, production, UX and business teams in a way that cultivates greater collaboration, unlocks new business value and simply makes work better for everyone. All of that comes down to pivoting to a BizDevOps culture that creates more visibility, and positive reinforcement, for engineers and developers.

    Building a Cultural Transformation Around BizDevOps

    The goal for BizDevOps is to move development and production teams away from nursing applications and tediously staring at dashboards, and align them closely with the business side, so that they’re putting their time and energy toward delivering more business-relevant services. Shifting gears to a BizDevOps structure is its own culture transformation. Eliminating the silos between teams encourages better cross-collaboration and opens up new visibility for developers to see how their work is being received by users and impacting business value for the organization as a whole. 

    But, breaking down these silos to get engineers more engaged in production introduces another challenge: Those developers are already busy building products and features that they’re responsible for. Why would they want to get involved in production on top of all their other work? Overcoming this hurdle means making concerted efforts to give developers more visibility into production environments. More than that, it means exposing developers to more positive feedback.

    Encouraging Engineers Through Constant Feedback

    What makes engineers feel proudest about their work? When they see how it’s being used and enjoyed by their target audience. The problem is engineers rarely get that kind of feedback. The feedback that is typically sent back to the engineers are critical comments and complaints; they rarely see or hear the good things, just the problems they have to fix for the next iteration.

    When you consider that, who could blame engineers for feeling discouraged or unwilling to want to take on more work? From their perspective, it’s not just about adding more to their plate, it’s also taking on more negativity and more criticism.

    But it doesn’t have to be that way. What if engineers were provided with immediate feedback on all the new applications, features and services they were creating? Both the good and the bad, in real time? That’s part of the key to facilitating a BizDevOps culture: When engineers are seeing how the products they put out into the world are being enjoyed, as well as the positive impact they have on business value, they’ll naturally begin to feel more appreciated and also more reaffirmed that they are doing good, meaningful work and that they’re having an appreciable, positive impact on users and the whole business. This doesn’t mean putting on blinders to negative feedback, but it’s easier to act on negative feedback when it’s balanced out by positive feedback, too. Exposing engineers to more feedback about the impact of their work might also help inspire them to have new ideas and drive new innovations. The more information you’re armed with, the better position you’re in to innovate.

    Organizations need to prioritize building easy ways for developers to have this visibility in their environments where they’re getting immediate feedback from end users, both the good and the bad. The more feedback they receive, the more encouraged by, and invested in, their work they become and, consequently, the more responsibility they feel for their contributions to the business. With that responsibility comes the culture change you need to digitally transform.

    Top-Down Support

    This culture change can’t happen without executive buy-in. The C-level needs to provide the necessary tooling and clarity on new organizational structures to break down silos between the development, operations, business and support teams, and then shepherd those teams into a more collaborative and visible environment. That support creates the foundation for a wider culture change, one that encourages a transition from operations to development and enables a bigger shift to BizDevOps. 

    Imagine the silos between teams as vertical walls. Now, shift those walls 90 degrees. That changes the structure so that your platform and autonomous cloud teams are sitting on the same level together, providing services across the organization, delivering software into production at faster speeds while simultaneously achieving visibility across the whole environment.

    The democratization of data enabled by this level of visibility is helpful in another way. When all teams have access to the same data, and can see what each other is working on and how all that work moves toward a shared goal, it drives everyone to find new ways to make each team’s workload lighter–such as with automation. When teams are not just automating their own work, but automating each other’s because they see how lifting one group lifts the whole organization–and has a concrete impact on business value–that’s what BizDevOps is all about.

    A Transformed Culture

    In the end, every engineering team should think, feel and act like their own startup, responsible for everything from end to end. That spans from what they’re building, to how they implement it, to the impact it has on the end user and the business. Every engineering team needs to be working like this, keeping top of mind considerations like, how can a new application or feature or service be optimized so that it not only runs well, but also runs efficiently? Not just efficient in terms of performance, but in scalability as well.

    Think about how cloud providers have potentially endless resources to deploy code. Just because something might run well doesn’t mean it’s running efficiently, and just throwing more code at a situation will inevitably raise costs and hurt overall business value. That’s something that engineers and developers should be keeping in mind as part of their work–not just how to deliver value in terms of new features or services, but how to bring down the total cost of ownership as well.

    That’s what digital transformation looks like: A culture shift where new BizDevOps organizational structures, visibility across production environments and immediate access to feedback, both positive and critical, enables all teams to work together in a cohesive way, driven by what creates value for both the business and the end users.

  • Five Attributes of a Great DevOps Platform

    Five Attributes of a Great DevOps Platform

    As we move toward 2020, the expectations are high, the software industry is moving faster and the demand for rapid and continually improving digital experiences have raised the standard. To match the needs and demands of the market, firms around the world are embracing the DevOps culture to ensure and streamline the development, packaging, testing, deployment and maintenance of software at scale. 

    Every DevOps organization is defined by its own leadership, culture, practices, policies, leadership style, structure, talents, skills, technical capabilities, processes and tools. However, there are certain common characteristics that are important to consider before deciding a better DevOps platform to use in your organization. Today, we will discuss such important features that can help you while selecting your DevOps platform. 

    What is DevOps?

    DevOps is a mindset in an organization that enforces a set of collective practices to ensure easy collaboration between dev and ops so the business can run smoothly without any hassle. DevOps aims to fasten the time to market through frequent and incremental improvements to deliver quality software updates to customers. 

    The meaning and definitions of DevOps can vary from company to company. Automation being the focal point, firms want to focus on achieving zero-touch automation, with utmost security and governance as a priority in the delivery pipeline.

    Even though the meaning of DevOps varies, several inherent DevOps principles cannot be replaced, and they repeatedly insist on collaborative relationships, developing early, failing early, shared learning and continuous feedback. These DevOps principles enhance quality with improved deployment rates while simultaneously improving predictability, scalability and security of the production environment. Tech leaders in the company need to view DevOps as a holistic approach rather than a small set of guidelines for the teams, and DevOps guidelines should be aligned with the company’s long term technology vision and objectives. 

    Here are some facts to consider while deciding a better DevOps platform for your organization.

    A Mindset of a Unified Culture

    Culture plays a significant role in establishing the guidelines while embracing DevOps in any organization. Through DevOps culture, companies seek to bring dev and ops teams into harmony to promote collaboration, automation, process improvements, continuous iterative development and deployment methodologies. But above everything else, a sound DevOps culture fundamentally solves one of IT’s biggest people problems: bridging the gap between dev and ops teams to get them to stop working in silos and have common goals. 

    According to Gartner estimation, DevOps efforts fail 90% of the time when infrastructure and operations teams try to drive a DevOps initiative without nurturing a cultural shift in the first place. It is not just about the efficient tools or experts working; it is about the behavioral modifications and mentality necessary to effect cultural change.

    Hence, it is important for the firms to consider the culture of the company before selecting its tool as a potential DevOps tool for their development. 

    Automation Inside and Outside

    Automation is one of the key principles of a strong DevOps team because it has such a high return on investment. 

    Automation makes up as a large part of DevOps culture to standardize production platforms, and even firms believe it as true because standardization and automation help make deployments more predictable, easy and less prone to human errors. 

    Automate everything as part of the workflow process, including infrastructures, compliance and continuous delivery pipeline. For example, tools such as Artifactory can manage and control all your binary repositories to help your developers and ops collaborate and automate build and deploy tasks easily. Similarly, Docker allows development teams to containerize applications and automate application deployments with speed.

    By reducing manual, non-important and repetitive tasks, you will increase your development and testing speed while cutting down on human and unnecessary errors. Additionally, by automating tedious and repetitive tasks, firms free up team members to work on higher-level and more important tasks.

    Important to note here, JFrog Pipelines is an innovative DevOps automation solution that well integrates and connects all critical processes in the software delivery lifecycle in the company, from development to testing to production. Using minimum code, it allows you to create smart and event-driven workflows across tools and teams easily, to speed up the release of software applications. This helps you achieve continuous integration, continuous delivery, continuous deployment and continuous monitoring in DevOps.

    Support Both Hybrid and Multi-Cloud

    What is Multi-cloud?

    Multi-cloud is the use of cloud computing from two or more public cloud providers in single network architecture to meet their business objectives. A multi-cloud strategy has many advantages. It avoids vendor lock-in, decreases latency, enhances business continuity, overhead cost reduction and maximizes flexibility for development teams. However, multi-cloud management comes with some challenges that require careful orchestration, and needs to pay close attention to security and governance issues.

    What is Hybrid-cloud?

    A hybrid cloud is a mixture of on-premises and off-premises IT resources. Based on business and compliance requirements, the hybrid cloud allows firms to move data and applications between private and public environments. 

    Hybrid cloud computing allows instant transfer of information between public and private environments, allowing enterprises to experience the benefits of both environments efficiently. 

    Flexibility, scalability and cost savings are some of the top benefits of hybrid-cloud. 

    Considering a platform that supports both multi-cloud and hybrid-cloud is the best solution for the enterprises moving toward the digital transformation ladder.

    Continuous Updates 

    Frequently rolling out updates, in a continuous sequence, to bring service improvements to consumer devices more rapidly and efficiently is what every firm is trying to do these days. The customer wants new features as early as possible and continuously. That is one reason why companies embrace automation and follow practices like continuous integration, continuous deployment and continuous delivery through tools such as GitHub, Artifactory, Circle CI, Docker, Kubernetes, etc. 

    The DevOps platform should be able to manage your DevOps pipeline from a single pane of glass. A platform that can provide a universal end-to-end solution that integrates with your ecosystem to orchestrate and optimize all critical processes in your CI/CD pipeline. Boosting fearless updates continuously from code to the edge in self-managed, on-prem, hybrid and multi-cloud environments.

    A Robust Community 

    All successful companies have built their own communities through their products and services. DevOps is all about the culture shift. Changing existing culture and moving forward is one of the hardest things to do—despite the benefits that DevOps offers, such as speed and agility. No matter what changes a team implements, embracing DevOps often comes to a halt once the companies realize it requires a change in culture and mentality of all the teams involved. This is where the value of community comes in, by fostering the development of a more open communication, easy and supportive feedback, and collaborative culture within teams, companies can be able to drive this DevOps culture. This mindset should come from within and outside of the organization by tapping into this sense of community-based collaboration. Communities are required because they ensure the well being of many of the technologies we use daily.

    Your favorite DevOps platforms—Jenkins, Docker, Kubernetes, Artifactory—and their popularity is because of the communities they have created around the tools. 

    A well-focused DevOps platform gives power to people who listen carefully to what the market is saying, gets the feedback and utilizes it effectively. Hence, community is a very important feature of a DevOps organization. 

    Conclusion

    Companies around the world are embracing DevOps, and the demand for DevOps skills soars as organizations spend their time and money on digital transformation. DevOps has become a modern approach to IT that helps companies to think and execute broader business strategies. DevOps can drive significant brand value by accelerating speed to market, better collaboration, ensuring better testing and automation. A DevOps platform should have all the qualities mentioned above to succeed in this highly competitive market.