devopsy.kinsta.cloud

Category: DevOps Culture

From One-Click Deployment to One-Click Feedback
What makes a group of individual contributors a team?

You can look at today’s product development teams as an ensemble of craftspeople from different disciplines. Code builders, design artists, storytellers, and an orchestrator. Similar to an orchestra, their output is based on the ability of each and every one of them to work together in harmony, continuously and collaboratively.

Let’s dig a bit deeper into today’s development methodology to see what this means in a practical sense.

At a high level, we see three major steps: planning, development and release.
1. Planning = Empower design craft
2. Development = Empower code building craft
3. Release = Optimize time-to-market (since there is no more time for nonsense!)
Despite the fact that teams are supposed to work together and be very in tune with each other, when we look at the reality, it’s very different. We actually work in silos, nurturing black-box thinking. Instead of collaboration, often there are three separate units that comprise a whole, solely with interfacing endpoints.

In such a configuration, it’s unavoidable that one unit’s priorities will be disruptive to another’s work, or flow. Without better integration of the actual collaborative parts of the work, in the form of frictionless and ongoing feedback throughout and during the end-to-end planning, development and release processes, eventually, your own unit’s priorities will take precedence over another’s.

Engineering IRL

A typical engineering process, whether it’s a new feature being developed, a bug fix or any other kind of collaborative work, usually looks like this:
- A developer receives a design handoff and, from that moment on, is expected to deliver a working manifestation of the design specifications.
- The developer gets to work (whether solo or on the dev team; independent of product or design, at this point) based on the specifications and starts writing the code.
- Once completed, the code is deployed to a pre-release staging environment, where the designer actually experiences the result of their craft for the very first time.
If you’ve been through similar processes, you’ll agree that, by this point, there is rarely time or tolerance in the process to iterate or provide feedback, nor correct much at all. More often than not, the mindset in fast-moving organizations is to stick with what we have ready to ship and just move forward. The actual “collaborative” part of the work is essentially bypassed. It’s really just a relay race of passing the baton downstream.

But this engineering paradigm has its consequences. Ever tried to dissuade an artist from providing their feedback? Expect a pretty bad vibe in the room (or virtual room in today’s hybrid and remote team structure).

This example, of course, is actually a virtual utopia of the development process. In reality, most organizations today have a far more destructive culture of continuous disruptions.

Now, let’s spice up the scenario with some real-world interruptions that often break the ongoing individual flow.

When a craftsperson is in the zone and creating, don’t interrupt! But what can you do when there are so many things happening in parallel? In small organizations that are just ramping up, most people are juggling much more work simultaneously.

What often happens looks a lot like this:

Jolene from sales requires a specific version for customizing a demo for a super important customer; Marko from support has opened 31 tickets you have yet to handle since it is a production issue that affects end users; your beloved single staging environment broke and you can’t review any new feature—oh, and don’t forget to attend the three meetings scheduled for today.

Frustration, anyone? I like to call it the ‘multi-context bad habit.’

Okay, I can already sense some eyebrows being raised, and I can almost hear you saying, “Yeah?! So what? We’ve always worked this way—what’s the issue??” Allow me to show you some numbers that may surprise you.

“Finding flow is key, and interruptions are a drag. Minimal or no interruptions give developers an 82% chance of having a good day, but interruptions throughout the day decrease the chance of a good day to just 7%.”

Eirini Kalliamvakou

“Meetings are both awesome and terrible. Collaboration improves our work, but too many meetings can be a blocker; going from two to three meetings per day lowered the chances of making progress toward their goals by 60%.”

Dr. Nicole Forsgren

To summarize, these data points basically indicate that disruptions, interruptions and the “multi-context bad habit” ultimately have a tangible, detrimental impact on results and outcomes. They diminish successful outcomes and delivery times by orders of magnitude.

Zero-Click Deployments as a Culture

Fortunately, there are multiple practices and tools today that can be leveraged to dissipate the friction, break down the silos and improve the flow.

This can be achieved by evolving from one-click deployment to zero-click deployments. Allow me to explain.

Today, we have enough working examples that could (and should) be replicated more widely in engineering practices. Here are some examples.

Preview Environments

Preview environments (sometimes referred to as ephemeral or PR environments) provide, well, the opportunity to preview products. By allowing multiple stakeholders to review and experience the product throughout the development process, you shift-left the way you manage your development processes and are able to provide the relevant feedback and fixes earlier in the process. This basically means that for any PR you get a live environment that is shareable and independent from your staging environment. (I like to call this an in-development feedback loop).

The JAMStack

Another good example can be taken from JAMstack. When JAMstacking, consider using Netlify, Vercel or AWS Amplify to gain this capability for your team.

Infrastructure-as-Code (IaC)

If your project is non-JAMstack, you can consider building a similar preview environment through IaC solutions like Terraform or Pulumi. The downside here is that this comes with quite a bit of hassle, as you have to build automation for every single PR to have a dedicated shareable URL of the environment for review.

This is great, though, because it democratizes the PR approval cycle by being more inclusive. Project managers, designers—practically anyone can review and approve the PR in development without having to wait until the end of the process to provide input. It’s easy to manage as it is isolated and independent of staging, which also provides always-on issue reproduction ability in the event that some kind of bug was discovered or tests need to be run during the development phase.

But it also isn’t all roses and sunshine.

There are some gaps and challenges that still need to be overcome for this to work. Some examples:

How do we streamline feedback?

Developers are used to getting comments from peers during code review in a language they can understand easily (for the most part) and quickly merge the input into their project. They don’t lose context by navigating away from the tool they’re working in (e.g. to Slack or Acme Corp. Task Manager), as it keeps them in their source code management environment (SCM). We need to think about how can we offer the same developer experience for everyone in the process to minimize PR turnover.

How do we provide actionable and quality feedback?

For example, UI and behavioral issue reporting are time-consuming, since you have to document and explain what, exactly, needs to be changed in order for it to be effective. Any wanted change likely involves the developer, which adds yet another disruption and interruption to their day.

Taking Zero-Click Deployments to Single-Click Feedback

If we want to optimize the developers’ flow, the next phase after simplifying the deployment aspects is to see how we add in the feedback layer in a way that isn’t intrusive.

We need to now take this mindset and empower what I like to call one-click feedback.

This isn’t a new concept, of course. Let’s take some inspiration from collaboration tools we all love like Figma, Miro and Notion.

What’s magical about each and every one of these tools is the ability to provide contextual feedback. This is important as it enables contextual collaboration. Any user is basically one click away from sharing their feedback while remaining in-context, increasing the potential for feedback to be dealt with frictionlessly.

This makes possible the following core values:
1. Any person can easily point out something which requires the creator’s attention.
2. The creator can instantly understand the feedback within the context of the project and what they are working on, and reply or tag another relevant peer for their point of view.
3. The creator doesn’t leave their main work environment to prioritize the new task or give it the attention it requires. Just as designers do not leave Figma to apply feedback, PMs don’t need to leave Miro to apply theirs, and so on.
These tools welcome feedback in an inclusive manner while enforcing a frictionless workflow. No one is messing with the creator’s flow as it is also happening asynchronously—no need for meetings, syncs or coordination of multiple stakeholders. This is also much more inclusive and relevant for today’s remote work culture, which requires investment in team communication through asynchronous channels and processes to be effective. (Here’s a great talk to watch on this topic; yes, the irony of interrupting your flow to watch this video is not lost. No, Really, Don’t Chuck Everything in Slack: Communications for Distributed Teams – from Florian Haas)

Assuming you find some truth in this, I’d like to address the now apparent elephant in the room: Why don’t we introduce the same abilities for in-development collaboration?

One-Click Feedback-as-Code

We’re starting to see a trend where framework-coupled companies like Netlify and Vercel, for example, are adding collaboration aspects to their offering. Netlify, in fact, is acquiring a company, FeaturePeek, to embed collaboration features on top of their preview environments, and Next.js starting an early access rollout of their Next.js Live product.

Imagine developers getting inclusive feedback during the PR approval cycle; not only comments about their code from peers but also UI, design and behavior comments with all of the needed assets and context to move forward without leaving their work environment! Such a shift will have a measurable impact on time-to-market, quality and velocity when employing such a workflow.

The main premise here is that to achieve better metrics associated with flow (as noted above by Dr. Nicole Forsgren), we should not mess with the flow. The benefits of reducing the friction and interruptions in creators’ flow are exponential and will enable us to get closer to working in tune and in harmony (if we stick with the orchestral references) or simply more collaboratively, as a team where each individual is always maximizing the time they are able to be in creation mode, without silos or black boxes.

What do you think? What’s your view of how teams should work together? How do you handle flow within your organization?
July 27, 2021
Stop Hiding the Cloud From Your Developers

If you say a person or organization “goes to great lengths” to achieve something, it means they try very hard and perhaps do extreme things to accomplish their goal. One example I’ve seen is that of more traditional companies “going to great lengths” to “hide” the cloud from their pool of potential technical consumers doing work like development. Instead of saying, “Here it is,” they block or restrict users from direct consumption. Under this model, developers don’t directly login to Azure or AWS, they go to the “internal corporate portal” and fill out a request form for the resources they want and submit it. Then, someone eventually processes the request and creates what is needed.

In extreme cases, organizations create “cloud teams” and staff those groups with people who build cloud resources on behalf of downstream technical consumers, like developers. Potential consumers must submit a request with a detailed design document specifying everything they want. Then, the cloud team schedules the project and delivers it at a future point. The whole process mimics the historical waterfall approach to software development or systems implementation.

Why do things this way? It always seems to come down to four distinct drivers:

Costs: “If we give open access, we will end up with a huge bill!”
Compliance: “Everything built must meet our security standards.”
Complexity: “Most users will waste time trying to figure it out.”
Control: “Our group has always coordinated access to resources.”

In traditional companies, the reasons for this restricted access always seem to be some variation of those on the list. And when you rebut with something like “Do you think Facebook, Twilio, Netflix, etc. restrict their developers’ cloud access?” the answer is usually something like, “Well, those are elite tech companies; we’ve always done it the way we do it now …”

In most cases, there is no singular argument you can make to change someone’s mind, or expand the range of possibilities. But you can offer facts. So, let’s take on their arguments one at a time and see if we can rebut them.

Costs

Argument: “If we open up the cloud to everyone, then all the developers will endlessly create virtual machines (VMs) and associated resources and services, and we will end up with a huge bill.”

Rebuttal: There are certainly examples of this; do a search for some variation of “unexpected cloud costs,” and they’ll pop up, especially from a few years ago when cloud adoption was just taking off. But what about today? What can be done to control costs? Even though there are lots of tips and ideas on how to control cloud costs, I’ll offer one simple possibility that directly applies to developers and development groups that is not often mentioned in many articles: implement a cloud resource quota per developer or dev group.

In AWS, for example, you can allocate via IAM; with Azure, you can use resource groups. The upshot is that there is a way to restrict the number and amount of resources that a single user can consume. So, whether you have 10 or 1,000 developers, you can use these built-in mechanisms to limit and then estimate what your consumption will be. If I have 100 developers and I put a limit on the number of VMs, the number of GB of RAM they can allocate and the amount of storage each developer can consume, then haven’t I just created a “cap” on consumption? The quota concept gives you a path to a “predictable” bill every month.

Compliance

Argument: “If we open up access to everyone, they’ll create VMs that don’t adhere to all our internal security standards.”

Rebuttal: Aren’t developers just looking for a VM (or VMs) to do dev work? We aren’t talking about building production-hardened VMs to host applications. We are talking about giving developers the proper sandbox in which to do their work. So, from this perspective, requiring that developers only create VMs that adhere to strict compliance standards reserved for production servers doesn’t make sense.

For example, maybe one of your compliance standards is that all production servers have a monitoring agent installed on them. Fine, that makes sense; what about dev/test? In dev/test, I might be spinning up a VM or a container that only exists for a short time, maybe even less than an hour while some tests are running. Does it make sense to require that a monitoring agent be installed on a VM that will only have a one-hour shelf life? Just throwing out the term “compliance” as a catch-all is not a real solid objection. Perhaps for dev/test, there are minimum standards you would want to follow regarding OS levels, patches, etc., but the VM(s) you use in dev/test do(es)n’t necessarily need to have all the additional security requirements reserved for production servers.

The other part of this objection we hinted at earlier is that you have to transition to “cloud thinking.” Resources that exist for the next 10 minutes in the cloud could intentionally be gone shortly after that. In the cloud, you can quickly create and destroy VMs, or entire environments of VMs. That is what makes the cloud a game-changer versus traditional on-premises methods. You can’t apply the same on-premises considerations to the cloud one-to-one. You need to come up with a new thinking model that takes into account what it means to use the cloud versus how you did it before on-premises.

Complexity

Argument: “It’s too hard to figure out some of these cloud architectures, and developers will waste time trying to do so versus having a central team do all the work correctly the first time.”

Rebuttal: Without the commitment to being a “learning organization,” there can be no DevOps, no transition/migration to the cloud, no digital transformation. Check out item number six on this list of 7 Ways to Make Developers Happy. (Spoiler alert: it’s ‘promote continuous learning.’)

If “complexity of learning” is one reason why you restrict your teams’ access to the cloud, then you are missing the point of what it means to be a technical expert. Developers partially define themselves by the depth and the range of knowledge that they have accumulated, so continuous learning is a significant component of keeping developers “happy.” Not to mention that happy developers are more engaged and tend to stick around longer at companies that keep them that way; you’re risking losing the technical talent you do have, which has a host of detriments to your organization.

Control

Argument: “If developers are creating their own servers, what does our team do?”

Rebuttal: This one is the toughest of all to rebut. Many organizations have one or more infrastructure teams who are responsible for creating servers and resources and giving access to downstream consumers. These groups control the resources. Now, along comes the cloud where everything can be self-service. And the natural reaction is, “If developers are creating on their own, what do we do?”

For many traditional companies making the transition to the cloud, there will still be many opportunities for participation. Instead of “architect,” now we need “cloud architect.” Instead of “engineer,” now we need “cloud engineer.” Also, what other higher-value work could you be doing for the company if you weren’t spending all your time provisioning and maintaining on-premises servers? Do you think all of the existing on-premises applications are going to go away overnight? Will your company become “cloud-native” in the next six months? What about all of the AIX, IBMi, Solaris and other legacy components that are in the data center? There will always be plenty of work for everyone to do. And it’s always an opportunity to learn some new skills.

The reality is that giving developers direct access to the cloud (with controls) frees up many other corporate resources to work on things that are, strategically, more important. If your company is not cloud-native right now, then guess what? It won’t be for a long time. Maybe in the short term, you’ll be hybrid-cloud and then in the long term, cloud-native. Along the way, there will be opportunities for everyone to rethink how things get done and maybe learn something new along the way.

If you are hiding the cloud from your internal customers, ask yourself, “Why?” Are the original reasons still relevant, or are they holdovers from a time long past? Maybe it is time to re-examine your original rationale for why developers don’t have direct access to the cloud. If the goal of your organization is to “service the customer,” “do more,” “move faster,” etc., then removing internal barriers to accessing the cloud is one way to make progress.

July 26, 2021
Programmability is Key to Agile Transformation

Enterprise agility has rapidly become one of the most crucial variables for a business’s long-term resiliency. With the COVID-19 pandemic, never-ending disruptions to the global supply chain and nearly every industry’s typical processes flipped upside down, there has never been a more important time to prioritize agility than now.

While the rewards of agility are high, organizations are struggling to implement key technologies and adhere to the methodologies that can get them there. In fact, the 14th Annual State of Agile Report notes that 59% of organizations are following agile principles, but only 4% of these organizations are getting the full benefit. But the ability to adapt quickly, seize new opportunities, and reduce costs is critical for survival in the ever-evolving and hypercompetitive digital age.

Agility is about adapting quickly to changes and increasing productivity. Importantly, agile methodologies are not agile themselves and therefore will not magically transform enterprises into agile organizations. Instead, the focus needs to be on practicing the correct principles to become truly agile. With this in mind, let’s look at how two of these critical principles—programmability and observability—can increase productivity and allow for innovation across the business.

Programmability

Programmability describes the capability of hardware and software to change and accept a new set of instructions that alter its behavior. Let’s look at each of these two forms.

One way programmability is used is to connect analog devices with software, for example, software-supported network routers. More recently, programmability has been extended to use networking to control the hardware involved and bridge with software systems. The rising popularity of infrastructure-as-a-service (IaaS) has increased the demand for programmability and brought concepts, such as infrastructure-as-code (IaC), into the realm of reality.

The other use of programmability is to extend existing software capabilities. The two most widely known examples of this are application programming interfaces (APIs) and software development kits (SDKs). Using both approaches, a developer can extend the default capabilities of an existing program. The rebirth of APIs as business APIs has significantly extended the introduction of programmability into application software systems. The approach is particularly popular with application service providers since it enables end users to extend and customize the core capabilities provided by a product or service.

Connecting Agility and Programmability

So how are these two concepts connected? Agility requires a foundation and the right kind of environment to provide optimal results. Programmability is an enabler for creating an adequate path for agility. An organization with highly programmable components increases the ability to automate repeatable tasks. Let’s look at two scenarios: a business use case and a technical use case.

Programmability in Business

When a purchase order (PO) arrives, the customer care representative must verify the organization’s ability to fulfill the request and then create an order in the back-end customer relationship management (CRM) system. Traditionally, this process has required a good deal of manual effort: first, extract the PO data; then verify the details using other systems or spreadsheets and, finally, type the data into the CRM system.

By contrast, consider a person working in a modern environment with high programmability. The PO can be received in an electronic format, such as JavaScript object notation (JSON), extensible markup language (XML) or electronic data interchange (EDI). Then it can be validated by connecting to other systems and reference data repositories, prompting the user to verify it and feed the data into the CRM system.

Programmability in a Technical Environment

In a typical technical use case, software developers will finish writing code; quality assurance (QA) engineers have to test the release; release engineers have to verify it and the DevSecOps team has to deploy the released artifacts. Similar to the business use case, this technical workflow has traditionally required extensive, tedious, manual work.

However, the process becomes significantly more streamlined when the development organization makes the environment programmable. In this case, once developers commit the code, the automation flow will kick in, exert the committed code, run through the life cycle stages and deploy it into production.

Through automation, which is enabled and extended by programmability, organizations can streamline their repeatable tasks and provide more time and space for users to work on essential or innovative tasks.

Now, let’s look at the implementation details of the two use cases.

Programmability in the Digital Supply Chain

In the business use case, the automation of a workflow, such as PO processing, is treated as part of a modern digital supply chain initiative. The systems, messaging and data involved in such a flow are linked together using integration technologies.

Integration approaches first involved point-to-point connections and then evolved to the use of centralized enterprise service buses (ESBs). More recent, modern approaches have centered around API-centric, integration microservices and integration platform-as-a-service (iPaaS) solutions that use decentralized models and cloud-native technologies.

Now, organizations build or acquire digital platforms by combining APIs, integration and security technologies to create digital supply chain flows. Critical to the success of any digital supply chain is the ability to select systems with built-in API capabilities and enable existing systems to expose applications and services as APIs and make the entire enterprise programmable.

Programmability in a Secured Software Supply Chain

In the technical use case discussed earlier, DevSecOps automation provides the foundation for automating various aspects of the software life cycle. Here, development tools, testing tools and governance-related tools are all connected using a pipeline.

IaC and automated build servers are two popular technologies used in automating such flow, which we call a secured software supply chain. Puppet, Chef and Ansible are widely adopted IaC scripting languages while Bamboo and Jenkins are popular build servers. However, as the DevOps space has evolved with cloud-native and decentralized technologies, GitHub and GitOps are also playing a vital role in building effective pipelines.

Similar to the business use case, the components involved here need to connect to the build pipeline, and therefore the programmability of each element involved plays a significant role in automating the workflow. Furthermore, in order to build a comprehensive pipeline, the code quality, unit test, automated test, infrastructure provisioning, artifact building, dependency management and deployment tools involved have to connect using APIs and extend the required capabilities using IaC.

Observability

By analyzing both the business and technical use cases, we have a clear idea about the impact of automation and how it is enabled by programmability. However, we have missed one important element: observability. Let’s examine this here.

The environment created by automation and programmability provides a good playground for teams to be practice being more agile, experiment with new ideas and take them to production rapidly. While agility increases, the responsibility of running healthy systems that provide a continuous digital experience is vital.

Observability helps detect and predict issues using analytics, machine learning (ML) and artificial intelligence (AI). Modern cloud infrastructure provides a certain level of observability at the operational level. Meanwhile, some modern programming languages, such as BallerinaLang, have built-in support for application-level observability associated with the specific language. When working with languages that lack this functionality, developers will have to either use a third-party tracing library that is compatible with the programming model or directly insert observability instructions along with tracing and debugging instructions.

Enterprises today are on a journey towards agility as they seek to optimize their digital strategies. However, organizational agility needs to catch up with project agility—and then, with each individual’s agility—to be more effective. This is the concept of “structured agility,” which is being fueled by “two-pizza teams” or self-organized teams.

APIs are the foundation of programmability, which makes it possible to connect analog and digital systems, as well as extend software systems. Programmability increases the speed of delivery, enables an enterprise to become “composable” with high reusability and allows a flexible environment by opening it up for innovation—the core principles of agility.

July 22, 2021
Puppet Survey Suggests DevOps Adoption is Stagnant

Puppet today published an annual State of DevOps report that suggests overall adoption of DevOps practices has stagnated.

The survey of 2,650 IT, development and information security professionals finds 83% are implementing DevOps practices, to a degree. However, only 18% of respondents are working for organizations that have highly evolved, automated DevOps processes in place. More than three-quarters of respondents (78%) are somewhere along the path toward achieving that goal.

A full 90% of highly evolved DevOps teams have automated most repetitive tasks compared to only 67% of mid-level organizations, the survey found. In addition, 91% of highly evolved teams report a clear understanding of their responsibilities.

Not surprisingly, the biggest hurdles identified are more cultural than technical. The most common culture blockers at mid-level organizations include a culture that discourages risk (21%), unclear responsibilities (20%), deprioritizing fast flow optimization (18%) and insufficient feedback loops (17%). In contrast, less than 2% of highly evolved organizations report resistance to DevOps from the executive level, the Puppet report found.

Nigel Kersten, a field CTO at Puppet, said during an online panel that, after more than a decade of DevOps advocacy, people continue to be the biggest limiting factor when it comes to adoption. Specifically, from a compensation perspective, many IT operations personnel tend to feel undervalued compared to developers, noted Kersten.

Another challenge enterprises encounter is trying to balance the need to create a standardized IT environment that can be managed and secured consistently and the desire of DevOps teams to embrace new tools as they see fit. Senior IT executives often tend to make platform decisions based on cost criteria that don’t always account for the need to enable agile development methodologies. Most of those platforms are designed to be managed by an IT administrator using a graphical tool rather than via a command line interface (CLI) and application programming interfaces (APIs) that appeal to DevOps teams.

As a result, individual pockets of DevOps teams that select and maintain a platform of their choosing are emerging within enterprise IT organizations. Those teams often view their chosen platform as a better alternative for advancing developer productivity. Finding and retaining site reliability engineers (SREs), for example, to manage those DevOps platforms is a major challenge for most enterprise IT organizations. Most IT administrators lack the programming skills needed to become an SRE, and it’s not clear how many IT administers are inclined to acquire those skills.

Conversely, it’s not clear how many programmers want to programmatically manage IT infrastructure at the expense of time they would otherwise devote to writing code. Organizations that can only hire a limited number of developers may not want to shift responsibility for managing IT operations to the proverbial left simply because they need more code to be written faster. Finding and retaining developers, in many cases, is just as much a challenge as hiring and retaining SREs.

Advances in automation and artificial intelligence (AI) should make it easier for organizations to embrace DevOps best practices in the months and years ahead. However, it should be apparent to all that adoption of those best practices across an entire enterprise still requires a significant amount of time and effort, and will for many years to come.

July 20, 2021
Why IT Should Focus on Quality Over Velocity

Quality assurance (QA) isn’t a new phrase for developers. Most know it as a staple part of the software process that is crucial for producing quality code. But throughout the pandemic, with extreme pressure for businesses to digitally transform quickly, it’s a discipline at risk of being dismissed by organizations and tech leaders as a last-minute, box-ticking exercise – but at what cost? Many have neglected QA in search of that quick app launch or new digital service, leading to an unwanted surprise or customer disappointment when the apps and features don’t work.

We all know the old adage: ‘quality over quantity,’ but when considering software development, CIOs should have quality over velocity in mind. Although producing software at speed is clearly a necessity for business success–whether startups pivoting during a pandemic or enterprises keeping up with these nimbler competitors–QA must become more than a forgotten principle for internal IT teams. To achieve the necessary standards at the speed required, tech leadership must embed QA as a specified requirement of any software project, rather than an afterthought.

Quality Assurance = Customer Satisfaction

Quality assurance is about monitoring the software development process and method to ensure that the output adheres to the project requirements, as well as industry security and compliance standards. With QA at the core, businesses can gain a better understanding of the status of their product, which allows them to assess, improve and consistently enhance its output for better outcomes and know whether the output meets the business requirements.

In order to draw out a strategy that ensures quality at each step, businesses need to understand their customers’ needs, user journey, expected target markets, and how they expect the software to perform. Without this, businesses will struggle to not only meet and ensure customer requirements but also business goals and expectations.

Without QA, it is difficult for businesses to assess the functionality of the product or service and it becomes extremely difficult to track critical defects that become costly to address once found post-production. Poor QA can also lead to several other challenges, including delayed product releases, unhappy customers, reduced brand reputation and increased technical debt.

Quality Shouldn’t be Assumed

In its infancy, software development took on the best practices used in the hardware/manufacturing industry, which largely relies on assessing quality at the end of the product life cycle before release. As a result, much of the technology sector adopted outdated strategies which then became common practice. In fact, there are still hundreds of CIOs and CTOs I speak to that expect QA processes to be an add-on at the end of the software development process.

Many technical leads believe it’s developers who drive quality in a software project throughout its life cycle. And this is true, to an extent – there are some fantastic, talented developers out there! But this perspective can teach bad habits and can mean QA is an expectation of individual developers rather than a specified requirement for the entire project.

That’s because it’s easy to forget that developers aren’t quality engineers; those are, in fact, two entirely separate roles that should be working side-by-side throughout any digital project. A developer’s role involves writing and maintaining the source code, which usually includes implementing unit and integration testing to ensure that the code matches the requirements. However, they are not involved in performing functional and non-functional testing or reporting defects to the development team.

It’s a People Problem

Any business embarking on a digital project based on software needs both developers and QA testers—you can’t have one without the other. But tech leaders can struggle to hire both good developer talent and QA engineers—whether due to expense or lack of availability of skilled workers. Recruitment is a lengthy and costly process, especially when recruiting skills that are already in high demand.

In light of this landscape and paired with the impacts of the COVID-19 pandemic, which has caused many businesses to slim down their in-house teams, IT departments are increasingly stretched. This means processes like QA, which many companies don’t view as a necessity or a priority, can be neglected. But putting quality assurance at the heart of the software life cycle can actually save time, helping to avoid clearing code with errors that will end up coming back to be updated.

Ultimately, businesses don’t just need to build software, they need to improve it and make it work and evolve with its needs and goals. Quality assurance makes this possible.

July 19, 2021
Why You Should Embrace Incidents and Ditch MTTR

The cliché is that everyone in IT hates incidents, and the natural reaction when assembling incident response metrics is to look for numbers that you can lower over time. Fewer incidents and shorter incident response times must be better, we think. You might already be familiar with the common metrics associated with these goals, including mean time-to-resolution (MTTR) and total incident count.

As it turns out, however, these metrics actually work against your company’s resilience goals. To understand why, let’s look at the industry that birthed most of our modern incident management practices and has learned the most about how to properly apply them: the airline industry.

What the Airlines Can Teach Us

From the 1950s to the 1990s, the airline industry built and refined best practices for incident management that we have since simplified and that have been widely adopted across the tech industry. What’s often termed “modern incident management” simply means the predictable and repeatable approach to handling unexpected disasters and the learnings-driven remediation steps that are prized when building a DevOps or SRE practice.

One of the most important findings to come out of this research is the somewhat surprising fact that incidents weren’t being declared often enough. Instead, incident reports were created and the carefully assembled response and cataloging process was only followed after catastrophic events, which meant earlier opportunities to catch the underlying problem during a less severe incident were being missed.

So why did an industry with such incredibly high stakes take so long to realize that more incidents were actually a good thing? You guessed it: a misguided desire to decrease the incident count. In chasing this goal of “lowering the number,” pilots and mechanics were often reluctant to file incident reports for anything less than the most critical situations, which meant the powerful, learnings-focused response and remediation process was only triggered in response to relatively rare and critical situations.

This is a textbook example of Goodhart’s Law in action. To paraphrase, Charles Goodhart posited that any metric created will ultimately become the goal. In other words, if the incentives are aligned to reduce incidents, no matter how good your response and learning process is, it will only be executed in the most dire and unavoidable circumstances. After discovering this, the industry rapidly moved their focus towards expanding the use of the incident process by working to expand the spectrum of situations that could be classified as incidents with the goal of triggering the formal incident response process more often, distributing learnings for more incidents of lower relative severity and, yes, ultimately resetting the goal to increase the number of incidents.

This approach effectively turned Goodhart’s Law to the airlines’ advantage—any effort to game the metric(s) meant more learning, not less. Filing incidents became directly and heavily incentivized and these metrics were never used as a justification for punitive action. This resulted in the safest decade of commercial aviation on record, with over 12 billion passengers traveling and no fatal accidents.

What IT Can Learn About Successful Incident Response

As technology companies, there is much for us to learn from this exercise in achieving resilience versus chasing “lower numbers.” The primary learning is this: increasing your incident count by incentivizing the use of your incident management process for more non-critical incidents—likely your SEV2s and below—and focusing your metrics and incentives on that goal, is likely the best way to reduce your overall count of catastrophic failures.

For most companies, this means putting MTTR on the back burner and instead focusing on making the incident process itself more accessible throughout your company and across your organizations. It means leaders should talk about impactful and recent postmortems publicly and that postmortem distribution should be a priority; it should also be something that’s measured. Most importantly, it means opening up the incident management process more widely. Ideally, everyone at your company should have access to systems that allow them to file incidents, read incidents in progress and review postmortems from previous incidents.

I like to reference the “big red button” philosophy of factory floors, where the decision-making power to declare an incident is in the hands of every employee. An unsafe situation shouldn’t be up for discussion, debate or approval before being acted on, even if the situation is later determined to be less severe than originally thought. The context of why that incident was filed—why that big red button was pressed—may prevent larger, more catastrophic failures in the future.

So if you’re just getting started building out an incident management practice at your company, or you’ve been chasing MTTR metrics and focusing on pushing your incident count down as a primary goal, consider taking these learnings to heart and creating a positive incident culture with positive metrics meant to encourage and reward incident responders and reporters. Your company’s future depends on it.

July 16, 2021
Lessons Learned on a Low-Code Journey

The popularity of low-code/no-code tools are on the rise as organizations chase the promise of creating ‘citizen developers.’ The idea of using these tools to empower non-technical developers and/or to free development teams from writing code line by line is certainly compelling.

About a year ago, a friend reached out to ask if I would critically review a book he was writing about citizen development. At the time, the expression was new to me but after reading and reviewing the book, I was intrigued. It challenged me to embark on my own citizen developer discovery. I want to share several applicable lessons I learned along the way.

Intrigued: Citizen Developer Tools Versus Dev Teams

The curious part of me was trying to understand if these were tools that would remove the need for expensive development teams. The previous, well-intentioned attempts had fallen short of their promise—think Computer Aided Software Engineering (CASE) from the 1990s as an example.

Around this time, I discussed the topic with the CTO of a progressive client. His business users would not wait for technical resources when they needed apps to address persistent problems. He was quick to point out that low-code was preferable to shadow IT. The architecture for these low-code/no-code tools meant database administration was in his team’s hands. He was supportive of business units building small tactical pieces of functionality with a clear business case, and he supported them where possible through training and allocating resources to their projects for data integrations and other functions.

The Challenge: Write an App

Encouraged, I decided to challenge myself over one weekend to see if I could write a commercial application. I had studied programming languages like Fortran and C in the 1980s, but I’d never worked as a developer and had never written commercial code.

Start of Day One

I selected a low-code tool at random. I would recommend, however, that a citizen developer engage the IT team rather than choosing at random, as most low-code/no-code tools have an underlying programming language. If your organization is a Java shop, for example, then it would make sense to pick a tool that has the same language—in this case, Java—at its source.

I purchased a six hour training course on YouTube for $20. This would prove to be a wise decision. Learning the tool I selected was not straightforward, but the training course allowed me to pause the tutorials, rewind and repeat to ensure I was understanding concepts and could follow along with the ‘how to.’

First task: write ‘Hello World.’ I recommend that citizen developers start here. If you can do this, then it’s more than likely you can build something more complicated.

Three hours later (of which two were a system error) … it was done.

Clicking on the green button produced the screen below.

At this point, I still had more questions than answers. The GUI for the development tool was point-and-click within a Windows directory structure. There was no command line editor. I was even more encouraged. Could it be this straightforward?

A note: while it took less than an hour to complete the ‘Hello World’ task, I encountered a Windows system error which pointed to a server port error. The vendor’s help desk didn’t operate over the weekend, but their documentation was adequate; it took a further two hours of effort to fix the problem, which arose because my Windows Surface Pro was both a client and a server.

Would an ‘average’ citizen developer have been able to overcome this problem? I’m not sure. I recommend when trying low-code/no-code tools that you also trial the help desk, even if you need to pay for it. While many software tool companies try to force software engineers to work through wikis to solve problems, citizen developers may need more hand-holding.

The next step was to build an application. Most non-technical people tend to describe applications from a graphical perspective and as business functions. This is very much in keeping with how I worked for a further two hours, at which point the ‘Home’ web page looked like this:

Using data from our in-house spreadsheets, I created the data model below with relative ease. I believe a citizen developer would benefit from asking an in-house database expert to review this ‘design’ to save time later when maintaining the application.

This work was undertaken at the same time as I was building a second tier of pages to amend data. So, for example, the automatically generated customer page which allows a user to search, add, edit and delete customer records looked like this:

This was completed in four hours. Once the table was created with the correct columns, the creation of the web pages was automatic. In fact, it became repetitive and I found it was easy to lose my concentration and make errors.

I was learning to save every small change to ensure that the live web and mobile application builds were working. A professional developer would see this as a form of continuous integration for the citizen developer.

I don’t recommend having teams of citizen developers working together, at least at the outset, as it introduces branching and versioning and the concept of ‘breaking a build.’ Professional developers would need to be consulted at this juncture.

End of Day One

I had a real sense of achievement at the end of day one and was eagerly anticipating day two. Citizen developers should note that, at this point, I had produced what developers would call a prototype. It would be possible, upon reaching this point, to approach IT to assign a developer to finish the job. If IT does not have capacity, then keep going.

Start of Day Two

Having designed screens and a database, the next task was to add business logic.

Creating flows of work with rules was a more engaging task, but still accomplished via drag-and-drop or point-and-click. Boolean logic operators like if- then- else allowed me to quickly build logic and attach it to visual elements on the webpage.

This is an event registration flow where data is accepted from a web page and, based on a true or false response, treated differently depending on that response.

As before, the first workflow took some time to learn. There were many ways to both trigger an action and direct the subsequent work. After a further half day, I had produced enough working code that I felt I probably couldn’t learn a whole lot more and the application was roadworthy.

From there, creating the mobile version of the app was as simple as pushing a button. At the outset, I had set a switch for mobile deployment. The web and mobile editions were always in sync, and the GUI was adjusting automagically for the deployment platform. Citizen developers should check on in-house standards for web and mobile app deployment configurations and ensure their selected tool supports them.

End Day Two

Lessons from My Discovery

Organizations that want to adopt low-code or no-code tools will need to ensure IT support and training for business users that want to self-service their app dev needs. I recommend early engagement and frequent retrospectives with IT colleagues.

Business functions should consider ‘easy’ and tactical opportunities for piloting.

Allocate an uninterrupted period of time to focus on the task. Learning a new tool over a weekend was a really good idea, too. There was zero context switching, apart from my wife dropping in with coffee for a chat.

I recommend citizen developers try to be a little bit Agile for these engagements. Using a ‘Plan, Do, Check, Act’ methodology means that you are not ‘all in’ on this—you are simply taking small discrete steps to see if you can solve problems more efficiently than you would if you instead waited for IT.

Professional developers should ensure that any low-code/no-code tools produce code that can be imported into their standard IDE and that the citizen developer’s code is not locked into the tool.

As a DevOps Institute Ambassador, O’Reilly helps empower DevOps Institute community members with the SKIL framework to advance the Humans of DevOps with skills, knowledge, ideas and learning. Learn more about O’Reilly’s work in DevOps here.

July 6, 2021
The State of Continuous Delivery in 2021

One of the hallmarks of DevOps is to improve the delivery of software. Therefore, mature DevOps systems employ continuous delivery (CD), the process of releasing bite-sized software changes on a regular basis. Amid high demand for digital innovation, streamlining this release pipeline is becoming more and more crucial for enabling overall software fluidity.

So, what is the state of CD among today’s development teams? The Continuous Delivery Foundation (CDF) recently released its 2021 State of Continuous Delivery Report, a study that helps establish baseline continuous delivery metrics. Below, we’ll review some key takeaways from the report and discover mean rates for things like deployment frequencies, recovery times and average lead time for code changes.

Benefits of Continuous Delivery (CD)

Before we look at the broad metrics surrounding CD adoption, let’s consider: what are the benefits of CD? A big net benefit of CD, the report noted, is that it helps realize feedback earlier in the software development cycle. A lean, CD-driven development approach could help quickly discover features that won’t succeed, which avoids wasting precious development resources.

CD also increases overall delivery speed for small software chunks. In an era where microservices architecture is all the rage, this process is a necessity for continually initiating many disparate updates. For example, Netflix utilizes Spinnaker to deliver hundreds of microservices and thousands of daily deployments.

With these results, organizations can reap increased responsiveness and agility. CD can enable fast feedback to changes, which can help businesses adapt to unforeseen conditions, such as zero-day exploits. An investment in CD can also bring net positives to developer team dynamics, according to the report.

Continuous Delivery Statistics

Software performance can be indicative of overall organizational performance. So, how should an organization measure its software delivery success? Well, the book Accelerate outlines four crucial metrics to watch: lead time for changes, deployment frequency, time to restore service and change failure rate. The CDF report establishes current industry averages for three of them.

In terms of deployment frequency, the study found that the highest percentage of developers (31.3%) release once per week to once per month. A large percentage of developers, 27.3%, release every month to six months. Only 10.8% of developers are elite performers, releasing multiple times per day. This indicates that rapid release cycles are still maturing across organizations.

The average lead time for changes is more variable than you might think. It takes 28.3% of developers one week to one month to go from committed code to running code successfully in production. For 27.3% of developers, this lead time is increased, from one month to six months. Only 5.74% of developers say it takes less than an hour, indicating that a truly on-demand delivery mechanism remains out of reach at most organizations.

On the other hand, developers are much quicker to act when it comes to restoring services. The survey showed 34.4% of respondents said it takes them one hour to one day to respond to an unplanned outage. A significant number, 15%, respond to incidents in less than one hour. Still, about 50% of outliers are responding somewhere between one day and more than six months.

Decreasing the mean time to recovery (MTTR) is imperative to respond to broken applications, quickly address security vulnerabilities, and meet evolving customer expectations. As a result, MTTR has become a de facto metric for Site Reliability Engineers (SREs) to watch closely.

Factors That Affect Software Delivery

Other factors may influence software delivery agility, such as industry type and programming language preferences. Out of the many industries attempting to refine their delivery models, the report highlighted retail as an industry that ranks high in both speed and stability metrics. Other sectors, such as financial services, government and energy ranked high on stability yet received only average scores for speed. Industries like telecommunications received poorer overall scores.

Programming language choices may also dictate overall software delivery performance. The study found that shell scripting languages, Go, JavaScript, PHP and Scala were among the top five programming languages, based on the combined software delivery performance data for organizations using each language.

DevOps Aims to Improve CD

Continuous delivery is vital to responding quickly to changes and enabling iterative, progressive delivery of new features.

Another facet of modern CD that can’t be understated is the widespread prevalence and reliance on open source tools. CDF open source packages like Jenkins, Jenkins X, Spinnaker and Ortelius remain vital to enabling CD processes for numerous developer teams.

The 2021 State of Continuous Delivery Report provides a baseline of standard continuous delivery practices across companies and sectors. The findings are based on SlashData’s Developer Nation survey, which reached 19,000+ respondents from 155 countries from late 2020 to early 2021. One area where reporting is lacking is change failure rates, which we hope future studies will benchmark. For more insights, you can download the report here.

July 2, 2021
No More Status Quo: Disruption and Reinvention through The Entrepreneur’s Weekly Nietzsche

Everyday we’re reinventing ourselves. Whether we’re reinventing knowledge through learning, or technology through development, nothing stays stagnant and disruption makes sure of that.

In a TechStrong TV segment, MediaOps Co-Founder and Editor-in-Chief Alan Shimel sits down with Brad Feld, current managing director at Foundry Group and author of “The Entrepreneur’s Weekly Nietzsche: A Book for Disruptors” to discuss this reinvention and the importance of disruptors.

If you’re not familiar with Brad Feld, he’s a well known venture capitalist and author who’s had his hand in the startup community as an author and founder of TechStars, a capital market company. He teamed up with Dave Jilk, a former partner of Feld Technologies, Brad’s very first company.

To cycle back to the word “disruptor,” it’s one Brad wants you to hang on to. He says, “Anybody who is involved in being a disruptor of an industry whether you’re a founder or a leader or an employee of a company, we like to think this is applicable to what you’re doing.”

Because of their focus on disruptors, it’s only fitting that Brad and Dave chose to concentrate on one of the most disruptive philosopher’s in history: Friedrich Nietzche. As Brad says, “In the 1860s to the 1890s, when he wrote the body of his work, Nietzsche was very disruptive to the status quo – of everything.” Like Nietzche, if you’re an entrepreneur or in the tech community today, you’re challenging conformity and reinventing the world.

While this book is intended for entrepreneurs, you certainly don’t have to be one to enjoy this book. “Entrepreneurial thinking is valuable for everybody in all aspects of life. Doesn’t mean you have to be an entrepreneur but the kind of mindset of an entrepreneur is powerful to understand,” says Feld. If you’re a disruptor, or simply enjoy learning, you’ll get a lot from this book.

And if you’re worried about not understanding the philosophy component, fear not. “The Entrepreneur’s Weekly Nietzsche: A Book for Disruptors” is incredibly accessible. Brad uses very small quotes (one to three sentences) from Nietzsche and translates it into contemporary English. After each quote is a short essay intended to “provoke thought, not tell you the answer.” As Brad says, “One of the meta goals of this book is not to tell people what to do. It’s to stimulate thinking.”

As a bonus to most quotes and essays, there’s actually a narrative written by an entrepreneur (some famous, some not-so-famous) discussing their experience and what they thought of when Brad and Dave gave them the quote.

With 52 different topics, this book is not meant to be gobbled down. While you can, you’ll get a lot more out of it when you dip-in and dip-out of areas that you’re interested in. Feld and Jilk’s hope is to simply get people to think: “We’re trying to help people stimulate thought around these ideas. Presumably any one of these topics is something you can ponder and meditate on.”

“The Entrepreneur’s Weekly Nietzsche” isn’t an untouchable academic piece, rather a relatable reflection of the human experience. Talking about the importance of including his own experience in the book, Brad emphasizes, “I don’t write books for a living. I’m a venture capitalist for a living. I write books because I love to write, I love to read and I love to think. My motivation for the writing is part of my own experience learning.”

Whether you choose to read a chapter a week or every few weeks, Brad and Alan recommend you take each chapter on its own. Think about it and internalize it before moving on. And if you’re able to re-read chapters, that’s even better. “Today, what I find myself doing when I do these talks is I’m reading these quotes again and I’m discovering new things in the quotes and what they make me think,” says Brad. “Again, not trying to tell me the absolute truth from the quote, but the thought that it then stimulates.”

If you’re into learning, disrupting or the entrepreneurial mindset, check out “The Entrepreneur’s Weekly Nietzsche: A Book for Disruptors” and attend Brad’s session at TechStrong Con 2021 Virtual Summit, on June 14, to enter his book giveaway.

June 30, 2021
CircleCI CEO Sees Growing Need for Developer Experience Engineers

There is no shortage of titles to go with all the tasks that make up a DevOps workflow, but given the critical role developers now play within any organization, it might be time for organizations to start creating a developer experience engineer (DXE) function within DevOps teams that is specifically tasked with increasing developer productivity.

CircleCI CEO Jim Rose said it’s become apparent there is more friction than ever in the application development and deployment process, as more complex applications are constructed, thanks, in part, to the rise of microservices.

A DXE, as outlined in a report published by CircleCI, would be responsible for implementing a common set of DevOps principles, maintaining the right tools and putting in place a cohesive set of standard processes that ensure developer success. Today, there are simply too many languages, frameworks, engineering styles and processes being employed within the same organization, and that conspires to reduce overall developer productivity, said Rose.

While software engineers are, today, addressing subsets of these issues, Rose said the time has come to assign these tasks to an engineering team that views itself as an advocate for developers by making available more responsive infrastructure.

As organizations of all sizes increasingly realize how dependent they are on software to drive digital business transformation initiatives, Rose noted that developer productivity is becoming a major corporate issue. Revenue forecasts and profit margins are now being directly impacted by the rate at which software is being delivered and updated.

Software engineers today routinely build DevOps pipelines for developers, but not much time and effort is put into optimizing the developer experience, noted Rose. That’s become not only a major productivity issue, Rose said, it has a material impact on the ability of organizations to attract and retain developer talent. One of the reasons so many developers prefer to work for smaller startups than large enterprises is because their daily work experience is simply better. Large organizations need to reduce the amount of friction developers experience if they want to be able to compete for that same talent, said Rose.

A DXE would also be more inclined to solicit input from developers on tool selection to best optimize productivity, noted Rose. Many software engineers today construct pipelines with little to no regard for how simple or hard it might be for a developer to employ tooling.

Organization today are, of course, starting to become more obsessed with measuring application development velocity. However, in many cases, simply making someone responsible for achieving a goal is going to result in more gains at a lower cost. Analytics tools have a role to play, but obsessing over key performance indicators (KPIs) may not generate the same result as quickly as simply giving an engineer a specific mandate.

As responsibility for application management continues to shift left toward developers, it should not come as a surprise that there is a lot more interest in automating the underlying infrastructure. The issue, at this point, is determining to what degree software engineers are going to achieve that goal versus forcing developers to seek out yet another platform that promises to do it on their behalf.

June 28, 2021