devopsy.kinsta.cloud

Category: DevOps Culture

Open Source Vulnerabilities Were Up 50% in 2019 — How Will It Impact Software Development in 2020?

Open source vulnerabilities have been on the rise in recent years, but 2019 was truly one for the record books with a spike of nearly 50% over the previous year.

According to the recently released State of Open Source Security Vulnerabilities report by the WhiteSource research team, the number of vulnerabilities jumped from 4,100 in 2018 to 6,100 in 2019.

Figure 1: Number of reported open source vulnerabilities per year.

In hopes of better understanding what is behind this rise in the number of open source vulnerabilities, what it tells us about the state of open source usage in modern software development, and even where we are likely headed in the years ahead, let’s take a look at the report’s findings for insights on the current state of open source security.

More Code, More Vulnerabilities

According to the research report, the primary driver behind the rise in open source vulnerabilities is a function of the growth of open source. There are now more open source projects, code and members of the community than ever before.

All of these good folks are working hard to not only write more code, often with the backing of large tech giants, but also to seek out vulnerabilities in the code that can put the users of their components at risk. The combination of more code being written and the number of eyeballs reviewing the code for those inevitable human mistakes, comes together to produce a larger number of vulnerabilities being reported.

The increase in the number of vulnerabilities in open source components will have a considerable impact on software development. In recent years, we have seen open source components playing a bigger role in how developers build their applications. Most estimates believe that open source components comprise between 60-80% of the codebase in most modern applications. When a vulnerability is reported in a popular project such as Apache Struts or the Linux kernel, then huge swaths of developers can quickly find their software in need of patching.

The Good, the Bad and the Bazaar

WhiteSource’s research revealed the reassuring news that over 85% of reported open source security vulnerabilities have a fix available.

One of the reasons for this improvement is the fact that more enterprises are dependent on open source components for their own software, so they are investing in making it more secure, via bug bounty programs, backing foundations that support open source security or backing the projects themselves.

However, even as the situation in the field appears to be improving, there is still a way to go.

The fact that many published open source vulnerabilities are not indexed in one central location adds complexity to this challenge for development teams attempting to stay on top of published open source security vulnerabilities. If proprietary software is still considered to be the classical Cathedral, well-organized information and support under one roof, then open source is very much the Bazaar.

The WhiteSource report found that only 84% of known open source vulnerabilities appear in the National Vulnerability Database (NVD). This means that information about another 16% of the known vulnerabilities is out there, spread out among a wide range of resources. These might be security advisories, bug trackers and a laundry list of other locations. While 45% of reported open source vulnerabilities that were not initially published on the NVD eventually end up there, that could take months since their publication on other resources.

The challenge for organizations is that they run the risk of using a vulnerable component unaware of the security updates, while hackers are scouring all resources, looking for vulnerabilities that could be their next meal ticket.

Given the evolving security challenges facing organizations, what does the future hold for open source security in the year ahead?

Our Predictions for 2020

The easiest prediction that we can provide is that 2020 will have even more reported open source vulnerabilities than were published in 2019. The only question is: How big it will be? Chances are that the count will be more significant due to recent advances making it easier than ever to report and an open source vulnerability.

This past year saw the launch of GitHub Security Lab which aims to help researchers, maintainers and users easily report an open source vulnerability and publish a verified fix in one centralized location. By simplifying the reporting and documentation process, GitHub has gone a long way in helping their users to protect their software with better quality reporting.

Key Takeaways

For many, the rise in the number of reported open source vulnerabilities might be concerning, making us confront the fact that we probably have plenty of them in our software right now. However, the situation is actually improving, and even heading in the right direction. When vulnerable open source components are reported, we get to track them and move quickly to remediate. Only once you know you have a problem can you begin to fix it.

Hopefully, the increase in security awareness will keep pushing for a stronger, more organized and better supported open source community that will be resilient in the face of even more vulnerabilities that are likely to show up in next year’s research report.

April 20, 2020
The Way Forward: Digital Resiliency Wins

As we wrote in Digital Transformation: Digitally Resilient Organizations, digitally resilient enterprises have built themselves business-technology platforms that are elastic, agile and can correctly position themselves to securely meet current business demands. These enterprises are also constantly learning from their data and adjusting as markets, workforce and customers demand.

These enterprises have also eliminated the bulk of their technical debt and continue to innovate so they aren’t disrupted by competitors with a better idea, execution or understanding customer needs.

Consider the grocery delivery wars, and how the coronavirus pandemic may have shifted the balance. Today, $78 billion retailer Target has a chance to prove, to a delivery-eager customer base, its prior investments in its online services designed to compete with Amazon Prime. Will its new Drive Up pick-up service and Shipt delivery service meet customer demand during this period of social distancing— and keep those customers over the long run—while Amazon tries to manage its way through supply shortages and shipping delays?

Walmart and Instacart have also seen significant usage surges, due to social distancing mandates. According to app store intelligence provider Apptopia, comparing average daily downloads in February to Sunday, March 15, Instacart, Walmart Grocery and Shipt have seen surges of 218%, 160% and 124% respectively.

“We’ve also noticed people in the United States have been turning to Target to stock up on groceries, entertainment and everyday essentials. March 15’s downloads saw an increase of 98% over the average number of daily downloads in February,” Apptopia wrote in its analysis.

While certain industries, such as online food and grocery delivery, may have seen years of adoption growth condensed into weeks, many enterprises are seeing downturns in demand.

Research firm IDC previously estimated that real GDP in the U.S. would reach 2.4%, but downshifted that to a worst case 1.4%. While IT spending dropped from an expected 5.1% year over year growth to worst case 1.3%. IDC pegs the base scenario for GDP growth now at 2.0% and for IT spending at 4.3%.

This has thrown CIO priorities into the air. According to a recent poll conducted by McKinsey & Company, CIOs are pressed to solve immediate challenges. For example, 32% cited increased demands for collaboration services and operating norms for working from home at scale, and facing increased strain on budgets (22%), a medium- and longer-term consideration.

McKinsey & Company advised CIOs to keep their focus on stabilizing emergency measures by strengthening remote working capabilities, improve cybersecurity, adjust ways of working with agile teams and prepare for a breakdown of parts of the vendor ecosystem (supply chain). In the interim, CIOs need to address immediate IT cost pressures to reduce costs, and creatively redeploy the IT workforce, while also pivoting to new areas of focus in the future.

According to McKinsey & Company, many organizations are successfully digitally engaging with customers, and cited a government in Western Europe who embarked on an “express digitization” of quarantine-compensation claims to deal with a more than 100-fold increase in volume.

“Sometimes this effort is about taking loads from call centers, but more often it addresses real new business opportunities. To engage with consumers, for example, retailers in China increasingly gave products at-home themes in WeChat,” McKinsey & Company wrote.

“Technology departments must anticipate and prepare to offer more of these kinds of digital services, products, and channels,” McKinsey & Company advised. “The key to reaching customers will be creating suitable access interfaces between internal IT systems and external social platforms and accelerating the integration of new vendors and distributors.”

In the months ahead, the organizations to move beyond the economic downturn will be those that built the platforms that enabled them to efficiently endure the recession and position themselves to adapt to whatever the new business as usual turns out to be.

April 20, 2020
Tips For Managing Development Teams Remotely During Coronavirus

Thanks to the global coronavirus pandemic, around a third of the world’s population is on lockdown in their homes, which means a huge number of businesses are currently trying to navigate remote working solutions. Trying to collaborate on software development is difficult at the best of times when your team is all in the same office, but with your team split across various home offices communication and collaboration is made all the more difficult.

Remote development work is an altogether different challenge, one that needs careful thought and new solutions. Luckily, there are ways to get around it, and they don’t all require huge investment or innovation. Just tweaking the way you think and work will help you continue to deliver on your development goals.

Spread Knowledge

Many teams run into problems with their remote workforce when they realize knowledge is not equally distributed amongst team members. I don’t mean the company log-in details (though you might have that problem too), but that the company’s vision and culture have not been effectively communicated. In an office setting this isn’t too much of a problem, as developers can be sent off on projects and the team can come together to implement them with one voice at the end, but when the whole team is divided it’s a lot harder to achieve this final cohesion.

To avoid having to piece your projects together like a jigsaw puzzle, make sure all of your developers are clear on what goals the company wants to achieve as a whole, and what culture and tone the company is wishing to promote. That way any development decisions made in private working sessions are still geared toward the overall company aim, and individual projects will slot in well together.

Communicate

To spread knowledge effectively, you also need a major focus on communication. You won’t realize how much the office dynamic contributes to communication until it’s gone. Working from home eliminates many of the little interactive scenarios in which team members can discuss ideas and clarify details.

To counteract this, you need to communicate far more than you previously did. Send more emails for clarification and have more regular meetings (over video chat) than you otherwise would. This might seem annoying or even counterproductive at first, but having a clear flow of information across the whole team is important for accountability.

Take Note

Related to communication, you may find it really helps to note more things down. You’d be surprised how much information is transmitted from being physically in the same space as someone, so a lot of workers find it a lot harder to pay attention to video of audio meetings and can easily forget details. Whether it’s minutes of a Zoom meeting or a brief one-on-one phone conversation, jot down a note of important tasks or details to remember, and circulate them throughout your team.

Be Humble And Accept Feedback

If you’re managing a development team that’s just gone full remote, accept the fact that you might not have all the answers. This might be as new to you as everyone else, and even if you have experience of remote working before it probably wasn’t during a global pandemic. By all means, show a strong sense of leadership and implement a solid plan of action, but be humble enough to accept that it might not be a perfect plan.

Show your humility by being open to feedback from your team members. They are having to adjust to a new lifestyle that may require new strategies to deal with. You might consider flexible working hours, or reassessing your tools or communications strategy. Figure this out as a team and you’re much more likely to keep morale up and not sacrifice any productivity.

Embrace The New Normal

Most importantly, don’t spend your time waiting for things to go back to normal. Treating this period as a brief interruption to your normal working schedule will stop you from fully committing to new strategies and tools to help you produce good work right now. It will show in your products, and if your remote working period lasts longer than you’d expected it will mess up your production schedule. For now, this is the new normal: embrace that and start with your best foot forward.

April 17, 2020
The Key to Embracing DevOps: A Real-Time Roadmap

While the term “DevOps” has become somewhat of an industry buzzword, it represents an important shift in the way teams think about planning, building and deploying software. Drawing its inspiration from the Agile movement, DevOps extends many of Agile’s core principles beyond development to how teams deploy, monitor and inform their next round of product planning.

Because DevOps touches so many areas of the software development lifecycle, more stakeholders than ever before are now involved in the design, production and release of software projects. With more moving parts and people, the continuous delivery of valuable software depends on these stakeholders being aligned at all times and moving in close coordination with each other.

In reality, however, that’s much easier said than done.

The problem with most projects is that they face a lag time based on the manual nature of updates required from the people doing the work. Sometimes this delay may be considerable. At the same time, accuracy becomes a problem because it involves people manually making and changing entries. This only compounds as the number of teams and dependencies involved in a project grows.

Another significant issue is the disruptive nature of context switching on development teams. Ideally, developers can concentrate on coding and not have to contend with a myriad administrative issues. In reality however, most developers are constantly forced to leave where they’re working to provide status updates for the team. Every “start-stop” slows the process and can even impede creativity.

While it’s tempting to just do away with updates, they’re incredibly important for the business. Stakeholders and customer facing teams need to understand if projects are on track or running behind schedule. Similarly, management needs to understand what projects are at risk in order to step in and provide additional resources or work with the team to determine ways to reduce scope.

So, how can teams stay aligned without compromising the developer experience?

While there’s no one right way to achieve total team alignment, one solution is maintaining a real-time roadmap that is shared between development teams, ops teams and the rest of the business. In a true, closely coupled DevOps environment, data from testing, deployment, integration, monitoring or other steps can be fed into roadmaps to reflect new realities and considerations.

If such input comes fluidly and automatically, the roadmap will not only reflect the exact status of development but also the practical matter of having software work as intended in a production environment. Again, not only can business owners know the true status of a project, adjustments and decisions can be made along the way to ensure that goals and expectations can be met.

A by-product of accurate, fluid, real-time roadmaps is the trust that stems from all being on the same page and having joint ownership for the entire process. The transparency and currency of such a vantage lessens finger-pointing and lessens “us-them” tendencies, as the work and the results can be jointly held. What results from this is accountability across the entire software development lifecycle.

Real-time, integrated, roadmaps perfectly fit with the spirit of what DevOps tries to achieve.

They operationalize work being completed and allow for user feedback and telemetry to continuously flow back into the next rounds of iteration planning. Real-time roadmaps accurately reflect what’s happening at each stage in the DevOps lifecycle, creating a single source of truth for developers, business owners and operations teams working at each stage of the process to ship value to customers.

The spirit behind DevOps calls for tighter integration leading to increased levels of efficiency and effectiveness. Adding real-time, integrated roadmaps to the DevOps discipline makes it even better.

— Aaron Upright

April 16, 2020
Business Conferences Amid the COVID Crisis

We’re all hunkered down in quarantine, so major trade shows have been cancelled. Virtual events will get us through this patch–but what comes next?

It is said that in times of crisis, we adapt. We modify our behavior and we persevere. And when the immediate crisis has subsided, we look at what worked, and often adapt much of it into our lives as we move forward.

We’re obviously in a crisis now, and among the many changes we have had to make is one that impacts our professional lives. For years, we’ve regarded major conferences and trade shows as an integral part of our technical learning curve–and yes, as a major part of our social interaction with our industry peers.

That’s all changing before our very eyes, and the COVID crisis is only one element. To be sure, the restrictions placed on our travel and ability to assemble have directly impacted the conference industry. A recent article places the loss at more than a billion dollars, and that number has only increased since then.

O’Reilly’s decision to shutter its physical conference business also hit the industry like a bombshell. But our ability to adapt and change presents us with other opportunities. O’Reilly’s CEO acknowledged as much, saying, “With large technology vendors moving their events completely online, we believe the stage is set for a new normal moving forward when it comes to in-person events.”

The past few years have seen accelerated growth in smaller regional conferences, along with a greater emphasis on virtual conferences. The shift has already begun, and we believe it will continue long after we are back to business as usual, which we believe will not be “business as usual.”

For openers, CFOs and others responsible for the bottom line will be more active in questioning the value of spending significant sums of money on a conference in Las Vegas or elsewhere, especially in lieu of the cancellation of conferences this spring. They will ask: What is the ROI for going to these large get-togethers? Can the money be more efficiently and effectively spent to attract and retain prospects and customers? Are there smaller, regional alternatives that cost less and pose fewer risks?

Conference attendees ask the same: Can I spend less money and achieve similar results closer to home? Would I prefer to be among a smaller public gathering? Can I simply attend an online forum of some sort?

Regional and virtual summits will meet those needs on both counts; in fact, they have already begun doing so. A recent Los Angeles-based physical conference that had signed up 300 participants quickly pivoted to an online virtual summit when the “stay at home” order was issued. In the process, the event more than doubled its registrations. Even when things get back to normal, companies may well see regional conferences as delivering the same educational and business benefits as the large mega-conventions. They will be treading lightly, with even less inclination than before to attend and spend at the bigger shows.

I’m not suggesting that all of the big conferences will shrivel up and fade away. Those sponsored by major vendors, such as RSA, Microsoft, Tableau and Gartner, will still be held and will still be successful. But the COVID crisis has forced companies, big and small, to take a more focused look at the conventions they may have previously taken for granted. And it’s not only in the tech sector, of course; events such as hardware shows, auto shows, even PizzaExpo, have been delayed or cancelled.

Our current crisis has forced us to adapt and to alter our thinking in any number of areas. The purpose of the conferences we attend to learn and interact with our peers are, and will be, no different than in the past. However, their physical nature (where and how they are held) will. The change has already begun.

— Michael Hiskey

April 15, 2020
2020 DevSecOps Community Survey: Of 5,045 Developers, Guess How Many Are Happy?
We just released the seventh annual DevSecOps Community Survey. We captured the pulse of over 5,000 developers in 102 countries. What we learned about trends in DevSecOps will surprise you—especially the parts about how happiness influences company culture, software security and business competitiveness.

When we tallied up the results we quickly saw some interesting correlations. Organizations with mature DevSecOps practices innovate faster and produce more secure software. Why? It boils down to having happier developers. Developers in mature practices were 1.5x more likely to say they enjoyed their work. Mature DevSecOps practices are also 1.8x more likely to use automated security tools, which certainly makes developer’s lives easier.

Here are just a few interesting tidbits we discovered.

Happiness Influences Culture

“We’ve always known that DevSecOps is about culture. The 2020 DevSecOps Community Survey, for the first time, reveals clear and convincing empirical evidence that developers are happier and more productive when security is part of the digital transformation and DevOps journey,” said James Wickett, head of research at Verica.io.

Just like you can’t buy happiness, you can’t buy good culture. You have to cultivate it. Here’s what the survey showed:
- Developers who received secure coding and programming opportunities are five times more likely to enjoy their work.
- By contrast, only 19% of unhappy developers get similar training. Unhappy developers say they aren’t satisfied at work and are actively seeking other opportunities. You can see why they are grumpy cats.
- Happy developers don’t feed off rumors—at least, not for security incidents. Instead, they focus on empirical evidence from better integrated tooling and security teams. Developers working in mature practices are 3.8x less likely to rely on a rumor.
Happiness Influences Security

“Mature DevOps practices are constantly testing, deploying and validating that software meets every requirement and allows for fast recovery in the event of a problem. As a result we can easily say, ‘DevSecOps is DevOps done right,’” said Hasan Yasar, technical director and adjunct faculty member at the Software Engineering Institute, Carnegie Mellon University.

Happy developers build more secure code. Here’s how they do it:
- Mature DevOps practices are 1.7x more likely to have a complete SBOM (Software Bill of Materials), including dependencies. This makes finding and fixing open source components faster and easier.
- Code quality? Check. Happy developers are 3.6x less likely to neglect security. This is because, with proper tools, they can focus their efforts on what is important.
- Grumpy developers are 2.6x more likely to ignore a security warning compared to happier developers working in more mature DevSecOps practices.
Happiness Influences Business Outcomes

“We cannot achieve higher levels of DevOps maturity until we understand how tightly woven people are into the transformation process. More than anything, DevSecOps success is tied to human effort,” said Jayne Groll, CEO of DevOps Institute.

Businesses depend on secure code built by happy people. Look at these findings:
- Happy developers are 1.4x more likely to follow their company’s open source governance policy. This is an important part of risk management. Ignoring policy and lessening software supply chain security can have catastrophic business implications.
- Happy developers are 1.8x more likely to recommend their employer to peers and friends, a real boon for attracting and retaining talent in a competitive environment.
- Happy developers are 1.7x more likely to get the job done. If productivity is the name of the game (and it often is) then businesses with mature DevSecOps teams outcompete the rest.
Download the full report here. Also, be sure to register for All Day DevOps | Spring Break on April 17th. I’ll be doing a session that goes over the report in depth and you won’t want to miss it.

— DJ Schleen
April 15, 2020
In IT, Purity Is Not Necessarily a Virtue

Throughout the history of IT, we’ve had our share of purists. OS purists, language purists, etc. The stark one was the “That is not Object Oriented Programming” crowd. While the poster-children of IT purity, we don’t really hear from them anymore—and thank goodness for that.

But a new crowd has reared its head in DevOps specifically, development more generally. They don’t believe there is ever a reason to modify a running container. Call them deployment purists, though maybe one of you has a better term.

Like OO purists, this approach ignores the reality of the environment we work in. Yes, the best option most of the time is to fix an issue and roll out a replacement container. But, the fact is that claiming anything else is not using containers “correctly” is patently wrong. My go-to example for why I say this is the easy one, because explaining complex infrastructures while trying to make your point is a waste of time.

Rolling feature releases in most organizations that do them regularly are achieved with environment variables on feature tags. But changing an environment variable is changing an instance. So, are all these organizations wrong, or perhaps the idea of purity is the problem when it comes to IT? Certainly, the driver for DevOps in many organizations was that purity had created islands of toolsets with varying delivery timelines and quality controls.

I contend it is past time to let purity be a guideline instead of a mantra. Yes, in most situations rolling out a new instance is the correct response, but in many–live applications that will drop connections and lack enough redundancy to bleed users off, for example–the correct solution is to modify the instance. “Never change production” is a good mantra, but it is a terrible purity test. We should avoid it, while acknowledging that not all situations allow us to make that choice.

Keep on rocking it. We’re weeks into COVID, and I can hear the stress in my operations friends’ voices. Know that the rest of us are grateful to have you keeping things running for us and bending over backward to make certain we have the access we need. Thank you, from me, for all of the people relying upon you.

— Don Macvittie

April 14, 2020
Development Versus QA: Ending the Battle Once and for All

It’s an accepted norm in software engineering that quality assurance (QA) engineers and developers will be perpetually at odds. We are so used to this dynamic that some organizations have decided it’s a good thing. They believe some animosity is beneficial, because QA has to hold developers accountable for good quality, and developers need to pressure QA to not impede time to market with too many tests.

I disagree. I think infighting between development and QA is a sign of a larger, more endemic problem: They can’t align on who is accountable for quality at what stage, and more importantly, what is mission critical.

Developer Versus QA Incentives

Developers are incentivized to ship as much code as possible, as fast as possible. This is because the KPIs they care about most are temporal in nature (time-to-market, feature velocity, sprint velocity, etc.).

However, when it comes to QA engineers, the things that incentivize them are somewhat counter-intuitive. You might think the most important goal for QA engineers would be to ensure quality, which is at odds with time-to-market and velocity, because more quality assurance will consume more time and resources. QA engineers will tell you (because they believe it) that this is their incentive and the source of the conflict. What nobody’s able to admit is that a QA team’s incentives are actually not to ensure quality but to ensure they are not blamed. They get rewarded by having fewer bugs or other issues that can be blamed on them.

The reason why minimizing blame is the number one priority for QA engineers is that in the QA realm, there is a general acceptance that bugs are always going to make it to production, no matter what. This is something we expect because a 100% guaranteed bug-free product would take years to ship rather than weeks, and would therefore be economically unviable. Since they know there will be problems to deal with no matter what they do, they want to show that they did everything in their power to prevent those problems. Naturally, they want to write as many tests as possible to minimize the risk of bugs that they should have caught. But since it’s impossible to write an infinite amount of tests, they have to prioritize what to test for.

A QA team is given no data by which to prioritize what to test, so this prioritization is essentially a guessing game. It may be an educated guessing game based on experience and expertise, but it’s still predicting what users are most likely to do on an application without objective data as to what they really care about and how they really will use the application. QA teams may add a good amount of structure to their brainstorming and guessing so that they feel like it’s a rigorous process, but without hard data, it never will be.

The fact that there is not an objective, data-driven method by which testing priorities are set means two things. First, it means potential quibbling between QA engineers and developers over what should and shouldn’t have an end-to-end (E2E) test built around it. Second, there’s likely to be disagreements over test suite size and what that will do to run time. Since the QA engineers’ incentive is to not get blamed when things go wrong, they naturally want to write as many tests as possible. The more tests you write, the higher the chances of bugs being detected by one of them. But as we discussed, developers are incented to maximize speed, and want the product out the door as fast as possible. The more tests that have to be run, the longer it takes the product to get to market.

Without data, your QA strategy devolves into an argument between developers and QA engineers. Different organizations resolve this argument in different ways. In my experience, I see some teams that don’t have any E2E tests at all. These organizations have much more of a speed-centric, DevOps-first mindset. At the other end of the spectrum, I see companies that have over a thousand E2E tests, and it takes two days to run a regression cycle. The former, where developers have won the argument, accepts a substantial amount of risk when they deploy. The latter, where QA engineers have triumphed, is unwilling to sacrifice the feeling of security that comes with thousands of tests to achieve fast deployments.

Testing Is Too Subjective

Too often, organizations burn out and break down due to constant infighting between silos; sometimes between QA and development. This is a common management problem that consultants (including myself) often work on. Yet, this battle between QA and developers continues to play out over and over again in software organizations, with seemingly no end in sight. But I don’t believe it’s destined to go on forever. I envision a future in which QA and development live in harmony.

So, how do we end the eternal battle between development and QA? The way to resolve this conflict is to realign both sides’ incentives and departmental goals. And the way to do that is by basing decisions on objectivity instead of opinion.

When arguments over different opinions on testing strategy arise, those opinions are subjective; they are informed by individuals’ own personal incentives and perspectives. An objective way of defining a target or scope eliminates the space for argument. Obviously, two departments working together toward a shared target are going to operate better than two departments that have competing incentives. This alignment requires an objective standard by which we decide what tests are important—what we should be testing and what we shouldn’t.

A lot of people reading this might be thinking, “Well, we have an objective standard. We have a process that determines our testing priorities.” But let me ask you this: If your QA engineers were swapped out with a group of equally smart and capable professionals, would they come up with the exact same testing strategy and set of tests to prioritize in the testing suite? The answer, of course, is no. It wouldn’t be exactly the same. And if it’s possible for a different person to look at the same data and come to a different conclusion, then the process is by definition not objective.

Bringing Objectivity to Testing

When prioritizing what should be tested, you need to know what will be most important to your user. So instead of having people guess what’s important to users, why not simply let your users tell you?

Machine learning has unlocked the ability to bring objectivity to testing. By analyzing real user behavior on your app, you can prioritize and orient your test suite to what your users actually care about. You will objectively know what to test based on actual user behavior, you’ll catch more bugs that actually matter to your users, and you’ll catch them much sooner, with fewer tests than if you’d come up with them yourself. The quality of your applications will greatly improve without having to create unnecessary tests that clog up JIRA boards and deployment pipelines. Both sides win.

A New Day for Features Testing

This also enables a seismic shift in features testing. With new features, we don’t understand what’s important to users yet because we don’t know how they’re going to use them. So how do we test for it? In the near future, machine learning and autonomous testing technologies are going to be able to solve this problem simply by leveraging experience.

We label many things “new,” but truly new things rarely enter the world. This is especially true of user interfaces (UIs). UIs should be similar to other UIs. They should be intuitive to brand new users. They must be fairly familiar to customers so that they can use them without having to relearn them. Machine learning and autonomous testing technologies can easily compare new features to feature sets that have been live in other applications before. When the datasets are similar, such as with UIs, login pages, checkout pages, etc., you can reasonably predict from past experience how people are going to use them, and leverage that data to create new tests. The new tests won’t be perfect, but they’ll be a lot more accurate than QA engineers taking their best guesses.

One Team, One Mission

You can imagine this future where your users are constantly informing a machine as to what should be tested in regression, and the amassed experience of thousands of other applications that have similar feature sets is available for analysis. This mass collective experience, which humans couldn’t possibly create, is going to be used to decide what your future test suites should look like. The battle between developers and QA engineers will be over, because the testing strategy will be aligned and there won’t be any argument over what should and shouldn’t be tested.

Now, you’ve got a team that has an objective, data-driven balance of speed and testing, with much greater confidence that development can move forward quickly while ensuring quality. Both sides can coexist peacefully and finally work as one team with one mission.

— Erik Fogg

April 10, 2020
Uncover the Secrets in Your Apps Through Observability
When it comes to your applications, the question to ask yourself is: “How helpful is monitoring if you aren’t proactively capturing the data you need to understand what is causing the errors you find?” The answer: Not very. That is why it is important for us to go beyond monitoring and work on expanding our capabilities in terms of observability.

When applied to the world of software, observability is how well you can understand what’s going on inside your application based on accessible outputs. Looking at this from outside the realm of errors and application failures, etc., observability might describe how well a developer understands how a certain feature works the way it does.

This example clearly shows that the metrics and context needed to attain what we can call a reasonable level of observability depends on the requirements of the individual system. We wouldn’t necessarily expect a developer at one company to understand how the features of another company’s application work behind the scenes. Likewise, an e-commerce application and a healthcare system will require different metrics to be observable.

Bringing Observability to the Team

Every company that operates software has at least the most basic level of observability thanks to log files. Unfortunately, in most cases we don’t know ahead of time what’s going to break, meaning no data gets logged. Occasionally, we may have a hunch that a certain method will be more likely to fail but even then, the data that’s written to the logs is usually shallow and requires additional investigation and context.

A key measure of observability is how well you can answer “why this happened” without needing to ask additional questions. Teams that rely heavily on log files to understand and troubleshoot issues generally have low observability, as we all know that the logs often lead to hours (if not days) of follow-up questions. To improve observability, it’s important to focus on proactive solutions.

Organizations, especially at the enterprise-level, use much more than log files to collect data around application errors and slowdowns. Application Performance Monitoring (APM) tools also play a significant part in most teams’ tooling stacks. Traditional APM tools provide additional context by helping teams identify when and where their application is experiencing performance or availability issues.

These tools, and more, are commonly used by companies to monitor application behavior and to investigate the root cause of issues. Each tool that provides additional context into the internal function of the application increases our observability. Once your tool stacks provide the full context needed to understand the root cause of any issue, you’ve reached a reasonable level of observability.

Creating a Culture that Supports Observability

Just like with every other buzzword in the industry–CI/CD, DevOps, etc.–achieving a reasonable level of observability means cultivating a culture that values and supports such a goal. There are several layers that need to be addressed and integrated together, but let’s break it down.

The first step is to create a quality-focused, team-wide engineering mindset toward system hygiene. From beginning to end all members of the team should be trained to build and employ code with quality, first and foremost. The mindset of engineers must go beyond the desire to write high-level code, but also involve the planning process and understanding the purpose behind code changes. It is using foresight to plan for the unpredictable as much as we can.

We all know that releasing code to production is basically “sending it out into the wild,” and we can all do a better job of making sure we have all the equipment we need to understand its behavior. That includes our mindset, as it forces us to consider our goal much earlier in development, namely by answering the following questions:
- What are we trying to improve?
- How can we measure success or failure?
- What metrics do we need to measure this?
Once this has been established, radical transparency and open communication follow naturally and further enable improvements in this area. When standards are established and issues are clearly communicated, problem areas in development can more easily be identified and resolved.

Without the human aspect in place, no tool in the world can give an organization a reasonable level of observability.

Conclusion

Observability is about more than just monitoring your system–it’s about understanding it. In order to achieve a higher level of observability, tooling and culture are two major factors that should be addressed. There are many different ways to attain the best level of observability for your organization, so understanding what you need in order to be proactive is crucial.

— Chen Harel
April 10, 2020
Seven Effective Tips to Get the Most Out of your DevOps
It seems that every firm is trying to include DevOps in their software delivery cycle these days. This is understandable. According to the 2019 Accelerate State of DevOps Report, companies using DevOps are more likely to achieve elite performance in software development and delivery.

Companies such as Google and Amazon can be deploying thousands of updates a day. These rapid updates improve company performance and result in the provision of better software solutions for the end-user.

In this article, we will look at seven effective tips to get the most out of your DevOps so you can maximize success. Before we look at some of these tips, let’s briefly define DevOps.

What is DevOps?

DevOps is a term for a group of management practices that aim to quicken the deployment of changes to a system while ensuring high quality delivery. It incorporates management theories such as agile infrastructure and agile development. One of the most influential books on the topic, which resonated with a lot of influential business leaders in Silicon Valley, is “The Lean Startup” by Eric Ries.

At its heart, DevOps boils down to the idea that you should be able to implement changes quickly, so you can release a superior product to your customers faster. This ability to bring a superior product to market fast, gives companies that can achieve this edge an advantage over their competitors.

Review Your Organizational Structure

Collaboration is the bond that allows software companies to deploy and execute changes fast. Companies that are experts at DevOps classically reduce the obstacles to collaboration.

In traditional IT organizations, departments are siloed. They interact with each other only when necessary. Hence, they have limited knowledge of each other’s work. This model presents a linear process of developing, testing and deployment.

To reduce these silos, you need to first analyze your organizational structure, and identify obstacles to collaboration. This starts by ensuring management is on board with your vision. You need to:
- Review the job description of managers and create KPI that support collaboration.
- Create remuneration packages that foster closer collaboration between departments.
Addressing structural issues that impede collaboration between departments will help. You also need to put in place a system of communication between departments.

Fortunately, there are useful tools to smoothen continuous collaboration between different lines of business. Channels of communications should not be limited to sending emails or conference calls. Remember, DevOps requires continuous and fast collaborations to ensure fast deployment.

Using unified communications (UC) is an effective way to foster collaboration in a company. Unified communications integrates phone calls, email, chat, document sharing, conferencing and other platforms necessary to exchange information within the organization. Importantly, you can also access much of this information in one place.

Reduce Operational Hurdles for Deployment

The way an update is deployed will differ by company. There is usually a system in place to discuss targets and a series of checks in place before deployment. For example, department heads will come up with an idea. They will be developed, then reviewed by quality assurance before operations deploy it.

This can be a cumbersome and time-consuming process, which slows down the speed of development. The problem for many companies is that changes, both large and small, need to go through the same process of quality assurance before deployment.

Source: Tech Beacon.

While it makes sense to have this process in place for major updates, it should be possible to push smaller updates faster. Two things need to happen to achieve this:
- Empower staff to make decisions independently while giving them the opportunity to learn from mistakes without risk.
- Reduce the administrative hurdles to deployment by creating a ranking system for reviewing updates. This will allow developers to push small updates independently, or with minimal oversight.
Empowering staff to push changes enables you to speed up DevOps.

Plan in Small Batches

Planning in small batches is important in promoting DevOps in an organization. With small batch testing, the cycles of testing hypotheses, acquiring feedback and going at it again can be done within a very minimal time frame. It is fast and efficient—simple.

This feedback might feel obvious. Perhaps the better question to consider when planning in small batches is: How small is small? Let the INVEST concept in agile development help you out.
- Independent: Planned small batches should be independent of other parts.
- Negotiable: Every batch should be iterable and negotiable upon receipt of feedback from deployment.
- Valuable: Each one should add value to the project and the end-users.
- Estimable: Batches should contain estimable information for scope determination.
- Small: Small should be small enough to be processed within a few hours up to a few days.
- Testable: A planned batch should be testable and can be monitored continuously.
Results of deployment, testing and monitoring of each batch should then be deployed at least once a day. Also, in planning small batches, different departments should be kept in the loop. This will greatly benefit DevOps efforts in the organization.

Create Space for Collaboration

Even without DevOps, fostering collaboration between departments is imperative. You can formalize collaboration through regular meetings. However, departments are usually situated in separate office areas, floors or even different buildings. Think about having to walk to a conference room on a different side of the facility just for a meeting.

This kind of setup is an example of a bottleneck for collaboration in an enterprise. And with collaboration being the heart of DevOps, creating a company culture and office space conducive to teamwork is of utmost importance.

SpaceX is an example of a company that promotes collaboration throughout the organization through company culture and physical space management. Offices are open, and each person can see what everyone else is working on. This promotes motivation for employees to work passionately. Also, everyone has access to camera feeds from different parts of the factory, which inspires curiosity about what others are doing. Because of this culture, employees become dedicated and excited to accomplish results.

Take Advantage of Microservices

In processing complex IT systems, it is more beneficial to break bulks to smaller, more digestible pieces. This is where microservices come in.

Microservices or microservice architecture is an architectural style wherein a complex system is broken down into smaller services. Rather than treating a system as a whole, it is perceived as a collection of manageable pieces.

Teams simultaneously working on parts of a system as a whole makes it more prone to breakdown. Utilizing microservices makes monitoring, maintenance and testing easier. They are also loosely connected to each other which means fixing wouldn’t much affect the other parts. Moreover, each microservice is releasable independently.

As an example, Wix started out with a monolith which was useful at first when they were only handling 1 million sites. However, interdependencies and poor quality coding resulted in downtime which is ultimately bad for business. Now that they shifted to using microservices architecture, they quickly scaled up reaching 70 million sites.

Maintain Continuous Monitoring

DevOps promotes fast and continuous integration and deployment. A fast-paced environment is more susceptible to problems. Something at any point can fail within the software development life cycle. Continuous monitoring helps you identify mistakes fast. It ensures the optimal performance, reliability and health of the software.

Setting monitoring targets, like development milestones, can improve this process. A development milestone is an overarching category that can give an insight into your team’s actual performance. Monitoring tells you the performance of your DevOps implementation.

A method to consider is synthetic monitoring wherein your application undergoes simulated customer interaction. This way, you can monitor the exact action and performance of your software before they get released to actual users.

Practice Change Management

DevOps employs continuous integration, continuous deployment and continuous monitoring. It is a cycle of fast-paced changes. Change management is essential to keep the system run like clockwork.

It might be easier said than done because changes have effects on various stakeholders. This includes your customers as the end-users, security, regulatory compliance and company policies. With this, changes within the development, testing, deployment and monitoring should be cohesively managed.

Change management is intrinsically reactive. It usually happens because of feedback received after deployment. But with DevOps, change management should be proactive. Instead of managing large-scale changes, processes are broken down into smaller parts with clearly defined scopes. Furthermore, anticipated changes are scheduled, scaled and communicated across the board. Possible risks and problems are efficiently managed because of this.

Summary

No system is perfect nor exactly the same. The key to a successful DevOps implementation is assessing what works best for your organization. DevOps is not a standardized process. It is not a system that will work for every company. Rather, DevOps is a theory of management.

Improving DevOps at your firm will involve a system of continuous review and improvement. You need to understand what is working, the problems and what you can improve. This can all be done through continuous processes with the help of automation and collaboration. No matter the size of your software development projects, these simple tips and tricks will help you make the most out of your DevOps.

— Sam O’Brien
April 9, 2020