devopsy.kinsta.cloud

Category: Enterprise DevOps

AIOps 2020: IT Talent Is the Game-Changer

Artificial intelligence for IT operations (AIOps) is a fairly new catch-all term for any multi-layered development initiative involving big data analytics, machine learning and/or AI to automate and solve business and IT problems. Definitions vary, but most support the notion that AIOps improve IT performance, infrastructure, cloud and operations.

This is a dramatic shift in how IT operations have been traditionally managed, requiring a digital transformation commitment from the organization and teams involved. Any enterprise, in any industry, with a complex systems infrastructure can benefit from AIOps. However, differences in AIOps strategies can result in benefits realized in unique ways.

Whether AIOps solutions are homegrown or commercial, they typically fall into two categories: domain centric and domain agnostic. Domain-centric AIOps focus on specific data types and sources, while domain-agnostic AIOps are more open and ingest a variety of data points.

Kentik is a domain-centric AIOps solution focusing its platform on large volumes of network data, such as NetFlow, SNMP, streaming telemetry and BGP routing data. It can provide companies with AI capabilities to automatically recognize traffic anomalies, predict cost overages, alert on traffic spikes and more.

Moogsoft, a domain-agnostic example, provides AI for detecting anomalies and alerting ops teams to potential issues, as well as reducing mean time to discovery (MTTD) and mean time to respond (MTTR). It can monitor system events and is a good illustration for how AIOps not only frees IT resources for other important tasks, but does so effectively.

No matter which option is right for your organization, it is important to start small when testing the AIOps waters. Pick a specific use case, implement it and then expand out as the use case proves itself.

Any Software or Web Property Can Benefit from AIOps

AIOps benefits run deep within an IT infrastructure. With the COVID-19 outbreak, the internet and systems running businesses globally are more strained than ever, given work from home mandates, increased traffic and higher digital asset volume.

This bandwidth stretch hits people personally, too (e.g. Comcast, Netflix, YouTube, etc.). Greater usage results in an expanded attack surface for cyber criminals to target. AIOps consistency, reliability, security and self-sufficiency directly affect a company’s viability and the amount of time required of support teams to address issues that arise.

Certainly, an organization is in an optimal position when IT teams can use AIOps to proactively address issues before a red flag needs attention. However, when the fires on systems around the world start, AIOps is there to put them out with its ability to quickly inspect and act.

For example, in mid-March when the COVID-19 outbreak resulted in widespread school/work from home mandates, online services from ABCMouse to InstaCart to Microsoft Teams saw site complication and outages. Any CTO knows that IT operations and AIOps teams surely answered the call to quickly detect and resolve root causes.

However, tools or knowledge only provide half of the equation necessary to be successful. Many organizations either don’t have access to the right resources to quickly remediate issues, or don’t have the technology necessary to get the job done. For instance, the State of N.J. has an urgent need for COBOL programmers to support IT efforts. Even if N.J. knows the way to solve its problem, valuable time is being wasted trying to find the means for it to be accomplished.

Top Tech Talent Is AIOps’ Secret Weapon

Automation is the backbone of technology, but true AIOps panacea is found when the right technologists know how to use the information AIOps provides.

Think of a medical MRI: Once the technology identifies issues, human intervention steps in to determine how and when to improve and fix things. The same is true with AIOps. It frees an IT team to prioritize the most important items and who should tackle them, instead of combing through endless logs and wasting every developer’s time on break fixes.

How a highly qualified, on-demand workforce makes a tangible, measurable difference here is a factor not to be underestimated. Talented technologists who can immediately ramp up on teams and systems—because it’s what they do day in and day out—seamlessly augment internal development staff and can address/remediate issues.

The future of AIOps goes beyond the technology itself and rests in the hands of dynamically built, virtual teams that can address issues just as quickly as the AIOps platforms identify them.

April 27, 2020
How 200 CIOs Responded to a Fully Remote Economy

The COVID-19 global pandemic has ushered a significant priority shift for the digital workplace. As development teams become completely remote, executive priorities are rapidly adapting to meet the gravity of these unanticipated times.

But, were IT leaders prepared for such a dramatic shift? What are they doing to evolve? To answer these tough questions, let’s turn to the data. A March 2020 study, conducted by Adobe and Fortune, surveyed 200 chief information officers (CIOs) on how their priorities are changing to meet the needs of a global crisis. Here’s what they found.

Remote Work

As it turns out, the majority of enterprises (84%) report they were prepared for remote work, with a slight increase for SMBs, at 94%.

“Like most business leaders, CIOs now have to rethink priorities, or at least reorder them, and we must reinvent ourselves now as virtual leaders,” said Adobe CIO Cynthia Stoddard.

We are living in a golden age of virtual collaboration tools—teams have Slack, GitHub, Zoom, Microsoft Teams and many other options. Yet, 53% of CIOs still cite communication as the biggest hurdle to remote work. Nevertheless, half of the CIOs surveyed reported their organizations are still actively hiring, indicating steady growth for IT.

New Technologies

Regarding technological adoption, some organizations are struggling to keep pace. Twenty percent of CIOs report serious shortfalls in technology tools. Interestingly, less than 25% of CIOs estimated increased financial spending for areas such as public cloud, infrastructure and artificial intelligence/machine learning (AI/ML).

“Indeed, nearly all CIOs (90%) surveyed said they use a public cloud service for at least some of their data, but most is housed on-premises,” said Giselle Abramovich. “Only one in three organizations store half or more of their data in a public cloud.”

One thing most enterprises agree on is the need for better security—seven in 10 anticipated increased financial investments into cybersecurity.

Artificial Intelligence

Only 50% of enterprise CIOs said they use AI in one or more projects (25% for SMB CIOs). However, it seems to be a growing discipline, as 90% of the organizations that have implemented AI have done so in the past year.

Though AI has garnered significant interest, there are hurdles to actual implementation. According to the report, “the biggest issues related to AI include getting data in order, funding and finding the right talent.” With more people online and the importance of distanced communication, it is likely that investments into AI will increase, especially for use in IT support and customer interactions.

“We’re making AI a strategic priority at Adobe. AI implementation is difficult for many organizations because it’s still new to most businesses, and there is a shortage in talent that has a deep understanding of it,” said Stoddard.

Analysis: Impact on DevOps

CIOs are in a unique position. Leaders are reevaluating business practices and also reassessing their technological investments. As the Adobe/Fortune study shows, many companies were prepared for 100% remote work, albeit some more than others.

To endure a COVID-19 economical drought, CIOs are taking action. The report indicates increased investment into remote tools, AI and cybersecurity, while aligning hiring and new infrastructure to reach these goals.

Institutions with a previous investment into cloud infrastructure as well as scalability practices, such as serverless, are likely well-equipped for increased online interaction and unanticipated usages. In this virtual realm, automation is also key to streamline deployments and monitor applications in remote fashions.

Though the report doesn’t differentiate between company types, it’s likely that digitally native enterprises are well-equipped as well. In addition, the trend from physical storefront to “brick and click,” has only been magnified by the current climate, forcing retail and banks to invest heavily in digital technologies and DevOps.

From a DevSecOps standpoint, increased end-user online action means an increased number of threats. Thus, security strategies for containers and service meshes should be reevaluated.

By repositioning priorities, IT can adapt to the new state of interaction. With deliberate execution, CIOs and all executives alike could find themselves on the right side of the river once the economy resumes.

April 22, 2020
Incident Resolution for Remote Teams
People working in IT support and incident management right now are faced with unusual difficulties supporting large remote workforces and managing unpredictable workloads. On Reddit, system admins and other IT pros are bemoaning the hiccups and hassles of working in isolation while trying to resolve issues and maintain high SLAs. You can’t go grab your indispensable SME for troubleshooting, because that person is also home and inundated with messages and alerts from many different tools.

Networks, VPNs and servers are being hammered extra hard right now from higher usage of SaaS apps and collaboration tools. User expectations are even more grandiose than normal: The home-based IT environment is a lifeline to whatever remains of human interaction. IT organizations are expected to make applications and services work flawlessly–overnight. Undoubtedly, IT pros are working with assiduity to keep the wheels of work moving for their employers during a time of crisis. Yet, working longer hours isn’t the answer. We need new ways of working—ones which may prove better than the current status quo for the long term.

In many organizations, the process of incident management and resolution begins with people looking at screens full of metrics. You see a spike in something, such as network I/O, and immediately begin to investigate it, assuming the worst. A more adverse scenario is when users call out problems on Twitter, Reddit or other popular tech communities, creating a noxious PR situation which IT and the business must swiftly respond to and manage. This ad hoc, reactive fire drill takes time, and resolving that particular issue may not be the most important item of the day. In software development, it’s akin to the bad practice of designing the architecture of a system first and then the user experience.

Given everything that’s at stake in the foreseeable future of stay-at-home mandates, SREs and IT support technicians need to flip the incident management process. Rather than beginning with the metrics and working backwards to uncover the problem, if there even is one, start by looking for the most important problems to solve. Then, discover the resources and components which support that end-user process and finally, analyze the metrics which are most relevant. For example:
1. A B-to-C provider of a streaming video service is now seeing a 200% increase in daytime traffic. This is resulting in sluggish load times for users and glitches.
2. IT has tagged the components beneath that service, so IT operators know exactly which VMs, databases and load balancers come into play.
3. Next, the operations team can identify the most relevant metrics, for instance CPU idle time, to measure and take steps to correct.
The Problem-First Approach

It sounds so simple: flip the process. Yet, this is not the case. IT operators and SREs are not accustomed to thinking in terms of which problems to solve or prevent. Being proactive or predictive hasn’t been the common workflow.

Here is how IT operations and DevOps teams can take a proactive approach to managing the availability and health of enterprise services.

Coordinate Across Departments

Identifying the most critical problems to solve right now is the hardest step and the one which will take the longest. Common problems these days are related to cost and scale: How do I right-size my environment to reduce costs or handle more workloads? Broaden your field of inputs to narrow down to the specific issue, such as slow page loads on the financial reporting website. Product managers, account managers and business unit leads who are the closest to the customer experience can deliver feedback on the top issues affecting customer/user satisfaction. Teams should also review recent support tickets to identify common themes of pain.

Fine-Tune Your Approach to Metrics

The Utilization Saturation and Errors (USE) Method is one way to approach the problem-first incident management process. As detailed by Brendan Gregg, a senior performance architect at Netflix, this methodology begins by posing questions, seeking answers and working backwards to the metrics. For each resource that you want to measure, identify three metrics: one for utilization, one for saturation and one for errors.

“The USE Method has made you aware of what you didn’t check: What were once unknown-unknowns are now known-unknowns,” Gregg explained.

Standardize Workflows

Create a common process for incident management across all your teams. Without the advantage of having almost everyone in the same room to huddle together in an ad hoc fashion when big issues crop up, it’s imperative to institute clear steps and roles. Doing so will prevent the frustrations, confusion and oversights that needlessly delay resolution. Since most incidents are composed of multiple contributing factors, teams need to adopt a small number of user-friendly tools to document and organize the information. At our company, we’re now using tools such as Miro, an online whiteboard application, to replace our physical whiteboarding sessions. Of course, there’s also Zoom, Slack, Jira and a host of other cloud-based tools already in place at many organizations. Mandate which tools everyone should use, with some guidelines on how to use them.

Increase Automation

In some organizations, scaling requirements in response to demand have increased tenfold. Automation is playing a critical role now; moving away from a web GUI, for example, is more scalable and aligns with modern tools such as Chef and Puppet. User tickets can be auto generated, for instance, from emails and linked to code management systems like GitHub. Modern development and operations teams are also expanding automation in unit testing and provisioning.

Watch for Burnout

Whether there’s more work and/or a need to fill the hours during long quarantine days, many software engineers, testers and architects are working longer days right now. Yet, exhaustion and burnout can lead to errors and oversights along with low morale. It’s up to managers to make sure that employees are taking breaks, working reasonable days and having the time and energy to attend to personal needs.
April 21, 2020
Five Attributes of a Great DevOps Platform

As we move toward 2020, the expectations are high, the software industry is moving faster and the demand for rapid and continually improving digital experiences have raised the standard. To match the needs and demands of the market, firms around the world are embracing the DevOps culture to ensure and streamline the development, packaging, testing, deployment and maintenance of software at scale.

Every DevOps organization is defined by its own leadership, culture, practices, policies, leadership style, structure, talents, skills, technical capabilities, processes and tools. However, there are certain common characteristics that are important to consider before deciding a better DevOps platform to use in your organization. Today, we will discuss such important features that can help you while selecting your DevOps platform.

What is DevOps?

DevOps is a mindset in an organization that enforces a set of collective practices to ensure easy collaboration between dev and ops so the business can run smoothly without any hassle. DevOps aims to fasten the time to market through frequent and incremental improvements to deliver quality software updates to customers.

The meaning and definitions of DevOps can vary from company to company. Automation being the focal point, firms want to focus on achieving zero-touch automation, with utmost security and governance as a priority in the delivery pipeline.

Even though the meaning of DevOps varies, several inherent DevOps principles cannot be replaced, and they repeatedly insist on collaborative relationships, developing early, failing early, shared learning and continuous feedback. These DevOps principles enhance quality with improved deployment rates while simultaneously improving predictability, scalability and security of the production environment. Tech leaders in the company need to view DevOps as a holistic approach rather than a small set of guidelines for the teams, and DevOps guidelines should be aligned with the company’s long term technology vision and objectives.

Here are some facts to consider while deciding a better DevOps platform for your organization.

A Mindset of a Unified Culture

Culture plays a significant role in establishing the guidelines while embracing DevOps in any organization. Through DevOps culture, companies seek to bring dev and ops teams into harmony to promote collaboration, automation, process improvements, continuous iterative development and deployment methodologies. But above everything else, a sound DevOps culture fundamentally solves one of IT’s biggest people problems: bridging the gap between dev and ops teams to get them to stop working in silos and have common goals.

According to Gartner estimation, DevOps efforts fail 90% of the time when infrastructure and operations teams try to drive a DevOps initiative without nurturing a cultural shift in the first place. It is not just about the efficient tools or experts working; it is about the behavioral modifications and mentality necessary to effect cultural change.

Hence, it is important for the firms to consider the culture of the company before selecting its tool as a potential DevOps tool for their development.

Automation Inside and Outside

Automation is one of the key principles of a strong DevOps team because it has such a high return on investment.

Automation makes up as a large part of DevOps culture to standardize production platforms, and even firms believe it as true because standardization and automation help make deployments more predictable, easy and less prone to human errors.

Automate everything as part of the workflow process, including infrastructures, compliance and continuous delivery pipeline. For example, tools such as Artifactory can manage and control all your binary repositories to help your developers and ops collaborate and automate build and deploy tasks easily. Similarly, Docker allows development teams to containerize applications and automate application deployments with speed.

By reducing manual, non-important and repetitive tasks, you will increase your development and testing speed while cutting down on human and unnecessary errors. Additionally, by automating tedious and repetitive tasks, firms free up team members to work on higher-level and more important tasks.

Important to note here, JFrog Pipelines is an innovative DevOps automation solution that well integrates and connects all critical processes in the software delivery lifecycle in the company, from development to testing to production. Using minimum code, it allows you to create smart and event-driven workflows across tools and teams easily, to speed up the release of software applications. This helps you achieve continuous integration, continuous delivery, continuous deployment and continuous monitoring in DevOps.

Support Both Hybrid and Multi-Cloud

What is Multi-cloud?

Multi-cloud is the use of cloud computing from two or more public cloud providers in single network architecture to meet their business objectives. A multi-cloud strategy has many advantages. It avoids vendor lock-in, decreases latency, enhances business continuity, overhead cost reduction and maximizes flexibility for development teams. However, multi-cloud management comes with some challenges that require careful orchestration, and needs to pay close attention to security and governance issues.

What is Hybrid-cloud?

A hybrid cloud is a mixture of on-premises and off-premises IT resources. Based on business and compliance requirements, the hybrid cloud allows firms to move data and applications between private and public environments.

Hybrid cloud computing allows instant transfer of information between public and private environments, allowing enterprises to experience the benefits of both environments efficiently.

Flexibility, scalability and cost savings are some of the top benefits of hybrid-cloud.

Considering a platform that supports both multi-cloud and hybrid-cloud is the best solution for the enterprises moving toward the digital transformation ladder.

Continuous Updates

Frequently rolling out updates, in a continuous sequence, to bring service improvements to consumer devices more rapidly and efficiently is what every firm is trying to do these days. The customer wants new features as early as possible and continuously. That is one reason why companies embrace automation and follow practices like continuous integration, continuous deployment and continuous delivery through tools such as GitHub, Artifactory, Circle CI, Docker, Kubernetes, etc.

The DevOps platform should be able to manage your DevOps pipeline from a single pane of glass. A platform that can provide a universal end-to-end solution that integrates with your ecosystem to orchestrate and optimize all critical processes in your CI/CD pipeline. Boosting fearless updates continuously from code to the edge in self-managed, on-prem, hybrid and multi-cloud environments.

A Robust Community

All successful companies have built their own communities through their products and services. DevOps is all about the culture shift. Changing existing culture and moving forward is one of the hardest things to do—despite the benefits that DevOps offers, such as speed and agility. No matter what changes a team implements, embracing DevOps often comes to a halt once the companies realize it requires a change in culture and mentality of all the teams involved. This is where the value of community comes in, by fostering the development of a more open communication, easy and supportive feedback, and collaborative culture within teams, companies can be able to drive this DevOps culture. This mindset should come from within and outside of the organization by tapping into this sense of community-based collaboration. Communities are required because they ensure the well being of many of the technologies we use daily.

Your favorite DevOps platforms—Jenkins, Docker, Kubernetes, Artifactory—and their popularity is because of the communities they have created around the tools.

A well-focused DevOps platform gives power to people who listen carefully to what the market is saying, gets the feedback and utilizes it effectively. Hence, community is a very important feature of a DevOps organization.

Conclusion

Companies around the world are embracing DevOps, and the demand for DevOps skills soars as organizations spend their time and money on digital transformation. DevOps has become a modern approach to IT that helps companies to think and execute broader business strategies. DevOps can drive significant brand value by accelerating speed to market, better collaboration, ensuring better testing and automation. A DevOps platform should have all the qualities mentioned above to succeed in this highly competitive market.

April 21, 2020
Metropolis Now: The Two Worlds of Employment in the Age of Automation

Current events have revealed that there are two worlds in the employment landscape. One is digital, the other analog. For the most part, those in IT—the digital workers—have been unaffected by the spike in unemployment due to the economic side-effects of the COVID-19 pandemic. But, it doesn’t mean they’ll be untouched in the long-term. As current events demonstrate, when it comes to the essentials in life, we’re all connected. Yet, the divide will continue to separate us to the detriment of all unless we do something about it.

When observing current events, I can’t help but think about Fritz Lang’s 1927 silent movie, Metropolis. It’s a science fiction film that depicts a world in which a small number of people live a privileged life in skyscrapers that tower high above the city. The rest of the population toils away below ground, doing the mundane work required to keep the city running. It’s a stunning piece of cinematography. The sets and special effects are breathtaking. What’s also impressive is that, for better or worse, so much of what Lang presented nearly 100 years ago has come to be.

You can view the full-length version of Metropolis here.

Figure 1: Scenes from the 1927 silent movie, Metropolis by Fritz Lang.

Lang’s vision of the future has a strange, almost eerie prescience for me, particularly as someone who has spent most of his adult life working in IT. It’s one that’s entirely possible, yet one I hope we avoid.

Allow me to elaborate.

Life Is A-OK in the Cloud

There’s a good argument to be made that today we live in Lang’s Metropolis, a world in which those who are doing really well live in the light of society while those who are just getting by live below ground in darkness.

The widening gap between prosperity and economic hardship is a trend that’s been unfolding over the last 30 years. We’ve become accustomed to it. Sadly, it’s no longer newsworthy.

What is newsworthy is the dramatic spike in unemployment due to the business shutdowns caused by the COVID-19 pandemic. The uptick in unemployment reveals a troubling set of circumstances: If you’re a digital worker chances are you’re still working. If you work in the analog world, that place where factories, restaurants, airlines, movie theaters, hotels, cruise ships and warehouses exist, there’s a good chance you or someone you know is out of a job, for now anyway.

Take a look at the graph from the US Private Sector Job Quality Index, a project from Cornell University Law School and others, by way of vox.com. It lists the jobs that are most vulnerable to layoffs due to the COVID-19 pandemic. IT workers don’t even make the list. It’s as if we live in another world.

Figure 2: Information workers are absent from the list of jobs at risk due to COVID-19.

The long and the short of it is that for developers, sysadmins, SREs and all the other people working in information technology, the recent economic downturn is but a blip in the daily news feed. For those manning a cash register, taking a ticket at the local cineplex, cutting hair or driving for a rideshare service, it’s a life-altering event. Or, to put it terms of Lang’s film, those of us who live in the clouds are doing well, and for those on the ground, it’s a different story.

Now, consider this: What if all the people who are presently laid off from their jobs find out there are no jobs to go back to? What if the economic recovery is slower than anticipated or society gets so accustomed to doing without that consumption doesn’t resume? Is such a scenario possible? Yes. Is it probable? Dunno. Will those of us in IT who work remotely all over the planet, from the comfort of our wired office, suffer? I doubt it. But, as for the rest, what then?

Let’s go back to the movie.

A Workforce Divided

There’s a point in the film when the protagonist, Freder (the son of the industrialist who built Metropolis), goes to his father to inquire about a catastrophic underground explosion shown earlier in the film.

The son asked, “Your magnificent city, Father, and you, the brain of this city, and all of us in this city’s light … and where are the people, Father, whose hands built your city?”

The father responded, “Where they belong.”

The son asked, “Where they belong? In the depths?”

His father nodded his head yes. (See the scene here).

There you have it. With that one physical gesture, Lang foreshadows a future which might very well be at our doorstep. As I mentioned earlier, it’s eerie, it’s prescient and it looks as if it’s coming to be.

What Does This All Have to Do with DevOps?

Those of us in DevOps have made valuable contributions to the world. We’ve done dramatically more good than harm. We’ve also been well compensated. The average salary for a SRE in the USA is $131,340, with a low of $60K and a high of $192K. Yet, the top salary for grocery clerk at Ralph’s supermarket is $22 an hour, which comes out to $44K a year over a 50 week year.

Do SREs deserve such a high salary? Of course, they do. Does a grocery clerk deserve only $44K a year in comparison? At the end of 2019, it would have been easy to say, yes. It’s a low skill, low education job. But that was then, and this is now. As of this writing, toilet paper is a premium commodity and the shelves need to be stocked with it. I’ll wager that in some circles, paying someone $50 an hour to get the rolls on the shelf is reasonable compensation. It all depends on how much people want toilet paper.

Am I being a bit dramatic? A little, but still, it’s worth thinking about.

So, what’s the important take away? Let’s go back to Metropolis one last time.

At the beginning of the film, there’s a scene in which the protagonist, Freder, was frolicking about in a lush arboretum-like greenhouse atop a skyscraper. He was surrounded by pretty maidens dancing. An elevator door opened into the garden. Freder observed as a group of raggedy waifs from beneath the city exited out of the elevator. A woman leading the waifs was at the head of the group. The waifs looked around in amazement at such luxury. Then, Freder’s servants ushered the group back into the elevator to exit. Freder was obviously affected by the waifs. He went into the depths of the city to investigate.

Shortly thereafter, Freder went to see his father to inquire about the explosion I described earlier. His father asked, “What were you doing in the machine halls, Freder?”

The son responded, “I wanted to look into the faces of the people whose little children are my brother, my sisters…”

It’s a powerful scene. It makes me wonder how many of the faces I’ve seen of those who were working in February are now sitting home, unemployed and unable to pay the rent in April.

The takeaway is this: We’re all connected at a level much deeper than the trivial interactions of the given social network. Current events have demonstrated the fact in spades. COVID-19 was first reported in China around November 17, 2019. On April 4, 2020, less than six months later, there were 240 cases reported in the state of South Dakota in the United States. If that’s not connection, I don’t know what is.

Those of us who work in the cloud are doing quite well, despite the setbacks caused by the global pandemic. But, there’s a growing number of us who are not doing so well and won’t be for the foreseeable future. It’s easy when everything is peaches and cream to be unaware of a world in which six-figure salaries are not the norm. But, when things go south and there’s no toilet paper to be had, the realization that we’re all in this together smacks us square in the face, like it or not.

The question then becomes, given that we’re all connected, what type of world do we want to share, the one in Lang’s Metropolis or something better?

It’s something worth thinking about. The pandemic will pass, but the underlying disparities will remain unless we do something about them.

Don’t forget to check out my new, free book, “The Impact of Automation.”

This book takes a serious examination of the history, impact and results from automation on our society and culture. Hope you enjoy it.

April 21, 2020
The Way Forward: Digital Resiliency Wins

As we wrote in Digital Transformation: Digitally Resilient Organizations, digitally resilient enterprises have built themselves business-technology platforms that are elastic, agile and can correctly position themselves to securely meet current business demands. These enterprises are also constantly learning from their data and adjusting as markets, workforce and customers demand.

These enterprises have also eliminated the bulk of their technical debt and continue to innovate so they aren’t disrupted by competitors with a better idea, execution or understanding customer needs.

Consider the grocery delivery wars, and how the coronavirus pandemic may have shifted the balance. Today, $78 billion retailer Target has a chance to prove, to a delivery-eager customer base, its prior investments in its online services designed to compete with Amazon Prime. Will its new Drive Up pick-up service and Shipt delivery service meet customer demand during this period of social distancing— and keep those customers over the long run—while Amazon tries to manage its way through supply shortages and shipping delays?

Walmart and Instacart have also seen significant usage surges, due to social distancing mandates. According to app store intelligence provider Apptopia, comparing average daily downloads in February to Sunday, March 15, Instacart, Walmart Grocery and Shipt have seen surges of 218%, 160% and 124% respectively.

“We’ve also noticed people in the United States have been turning to Target to stock up on groceries, entertainment and everyday essentials. March 15’s downloads saw an increase of 98% over the average number of daily downloads in February,” Apptopia wrote in its analysis.

While certain industries, such as online food and grocery delivery, may have seen years of adoption growth condensed into weeks, many enterprises are seeing downturns in demand.

Research firm IDC previously estimated that real GDP in the U.S. would reach 2.4%, but downshifted that to a worst case 1.4%. While IT spending dropped from an expected 5.1% year over year growth to worst case 1.3%. IDC pegs the base scenario for GDP growth now at 2.0% and for IT spending at 4.3%.

This has thrown CIO priorities into the air. According to a recent poll conducted by McKinsey & Company, CIOs are pressed to solve immediate challenges. For example, 32% cited increased demands for collaboration services and operating norms for working from home at scale, and facing increased strain on budgets (22%), a medium- and longer-term consideration.

McKinsey & Company advised CIOs to keep their focus on stabilizing emergency measures by strengthening remote working capabilities, improve cybersecurity, adjust ways of working with agile teams and prepare for a breakdown of parts of the vendor ecosystem (supply chain). In the interim, CIOs need to address immediate IT cost pressures to reduce costs, and creatively redeploy the IT workforce, while also pivoting to new areas of focus in the future.

According to McKinsey & Company, many organizations are successfully digitally engaging with customers, and cited a government in Western Europe who embarked on an “express digitization” of quarantine-compensation claims to deal with a more than 100-fold increase in volume.

“Sometimes this effort is about taking loads from call centers, but more often it addresses real new business opportunities. To engage with consumers, for example, retailers in China increasingly gave products at-home themes in WeChat,” McKinsey & Company wrote.

“Technology departments must anticipate and prepare to offer more of these kinds of digital services, products, and channels,” McKinsey & Company advised. “The key to reaching customers will be creating suitable access interfaces between internal IT systems and external social platforms and accelerating the integration of new vendors and distributors.”

In the months ahead, the organizations to move beyond the economic downturn will be those that built the platforms that enabled them to efficiently endure the recession and position themselves to adapt to whatever the new business as usual turns out to be.

April 20, 2020
Keys to the Kingdom: Protect Your Server from Hackers

Automation and DevOps undeniably improve the agility and responsiveness of IT. Most of us have taken up at least a bit of the wave of automation that DevOps and Agile have brought to our door, and many are far along on their DevOps journey.

But every once in a while, we need to look back and lend a hand to those still coming. The fact of the matter is that sometimes we forget to warn those who are new to massive automation about some serious issues.

It is intuitively obvious, if you think about security at all, that putting together a command and control server—we’ll use Ansible in this blog, but any tool that can roll out applications to a collection of servers—creates a big screaming target for any hacker that gets into your infrastructure. You cannot assume that will never happen, so organizations need to think about protective systems/architectures early, or spend more time patching things up. Remember, the more you build, the more weight there is to making major changes.

For starters, severely limit access to the command and control machine. It belongs on a management subnet/VLAN with only a few access points. The servers it is managing absolutely should not have access back to the command and control server. If they do, any one of them getting controlled will provide a pathway to all of them getting controlled.

Next up, early on there were quite a few debates/discussions about which user accounts should be able to run command and control scripts (Playbooks in Ansible). Eventually, the bulk of the community settled on each individual running them, not a single generic user with a name like “ansibledeploy.” Some still use a central user, but the majority wants the ability to track down the leak that individuals running commands allow—searching for strange/uncommon command execution in logs will cough up which account got compromised and allow for rapid response—which outweighs centralization.

Third, learn and use the security features available in your application. Ansible offers Ansible Vault to allow encryption of playbooks. Since many playbooks require root access, encryption allows those playbooks to hide root access. Use of Ansible Vault has grown over time, and there are excellent tutorials out there to help use it to lock down command and control. Your product will have a similar tool, because putting root passwords in plain text, all over the disk, is just not acceptable.

In the Ansible Galaxy/Github repositories, you can find excellent roles to help in your quest to lock things down. Geerlingguy is one of the most popular, and his Ansible Role – Security is excellent if you follow the caveats he provides in the readme.

This is simply an advisory, not a step-by-step solution set. Overall, just remember that putting together a command and control server makes a tempting target. And like Geerlingguy points out in his readme, your security is your concern.

Look at where the server can be accessed from, who can access it, where sensitive data bits such as passwords are stored—hint: plain text passwords in files went out of fashion when mainframes were hooked up to LANs—and how to protect all of the above. It’s been done, there are plenty of resources to help in your given environment. Take the time to look them up and implement relevant suggestions.

After all, I won’t get a call in the dead of night if your servers are all taken over, you will. Do everything you can to avoid getting that call. And keep rocking. Everyone remote working has made many organizations more susceptible to all forms of attack, don’t let your guard down.

April 17, 2020
2020 DevSecOps Community Survey: Of 5,045 Developers, Guess How Many Are Happy?
We just released the seventh annual DevSecOps Community Survey. We captured the pulse of over 5,000 developers in 102 countries. What we learned about trends in DevSecOps will surprise you—especially the parts about how happiness influences company culture, software security and business competitiveness.

When we tallied up the results we quickly saw some interesting correlations. Organizations with mature DevSecOps practices innovate faster and produce more secure software. Why? It boils down to having happier developers. Developers in mature practices were 1.5x more likely to say they enjoyed their work. Mature DevSecOps practices are also 1.8x more likely to use automated security tools, which certainly makes developer’s lives easier.

Here are just a few interesting tidbits we discovered.

Happiness Influences Culture

“We’ve always known that DevSecOps is about culture. The 2020 DevSecOps Community Survey, for the first time, reveals clear and convincing empirical evidence that developers are happier and more productive when security is part of the digital transformation and DevOps journey,” said James Wickett, head of research at Verica.io.

Just like you can’t buy happiness, you can’t buy good culture. You have to cultivate it. Here’s what the survey showed:
- Developers who received secure coding and programming opportunities are five times more likely to enjoy their work.
- By contrast, only 19% of unhappy developers get similar training. Unhappy developers say they aren’t satisfied at work and are actively seeking other opportunities. You can see why they are grumpy cats.
- Happy developers don’t feed off rumors—at least, not for security incidents. Instead, they focus on empirical evidence from better integrated tooling and security teams. Developers working in mature practices are 3.8x less likely to rely on a rumor.
Happiness Influences Security

“Mature DevOps practices are constantly testing, deploying and validating that software meets every requirement and allows for fast recovery in the event of a problem. As a result we can easily say, ‘DevSecOps is DevOps done right,’” said Hasan Yasar, technical director and adjunct faculty member at the Software Engineering Institute, Carnegie Mellon University.

Happy developers build more secure code. Here’s how they do it:
- Mature DevOps practices are 1.7x more likely to have a complete SBOM (Software Bill of Materials), including dependencies. This makes finding and fixing open source components faster and easier.
- Code quality? Check. Happy developers are 3.6x less likely to neglect security. This is because, with proper tools, they can focus their efforts on what is important.
- Grumpy developers are 2.6x more likely to ignore a security warning compared to happier developers working in more mature DevSecOps practices.
Happiness Influences Business Outcomes

“We cannot achieve higher levels of DevOps maturity until we understand how tightly woven people are into the transformation process. More than anything, DevSecOps success is tied to human effort,” said Jayne Groll, CEO of DevOps Institute.

Businesses depend on secure code built by happy people. Look at these findings:
- Happy developers are 1.4x more likely to follow their company’s open source governance policy. This is an important part of risk management. Ignoring policy and lessening software supply chain security can have catastrophic business implications.
- Happy developers are 1.8x more likely to recommend their employer to peers and friends, a real boon for attracting and retaining talent in a competitive environment.
- Happy developers are 1.7x more likely to get the job done. If productivity is the name of the game (and it often is) then businesses with mature DevSecOps teams outcompete the rest.
Download the full report here. Also, be sure to register for All Day DevOps | Spring Break on April 17th. I’ll be doing a session that goes over the report in depth and you won’t want to miss it.

— DJ Schleen
April 15, 2020
In IT, Purity Is Not Necessarily a Virtue

Throughout the history of IT, we’ve had our share of purists. OS purists, language purists, etc. The stark one was the “That is not Object Oriented Programming” crowd. While the poster-children of IT purity, we don’t really hear from them anymore—and thank goodness for that.

But a new crowd has reared its head in DevOps specifically, development more generally. They don’t believe there is ever a reason to modify a running container. Call them deployment purists, though maybe one of you has a better term.

Like OO purists, this approach ignores the reality of the environment we work in. Yes, the best option most of the time is to fix an issue and roll out a replacement container. But, the fact is that claiming anything else is not using containers “correctly” is patently wrong. My go-to example for why I say this is the easy one, because explaining complex infrastructures while trying to make your point is a waste of time.

Rolling feature releases in most organizations that do them regularly are achieved with environment variables on feature tags. But changing an environment variable is changing an instance. So, are all these organizations wrong, or perhaps the idea of purity is the problem when it comes to IT? Certainly, the driver for DevOps in many organizations was that purity had created islands of toolsets with varying delivery timelines and quality controls.

I contend it is past time to let purity be a guideline instead of a mantra. Yes, in most situations rolling out a new instance is the correct response, but in many–live applications that will drop connections and lack enough redundancy to bleed users off, for example–the correct solution is to modify the instance. “Never change production” is a good mantra, but it is a terrible purity test. We should avoid it, while acknowledging that not all situations allow us to make that choice.

Keep on rocking it. We’re weeks into COVID, and I can hear the stress in my operations friends’ voices. Know that the rest of us are grateful to have you keeping things running for us and bending over backward to make certain we have the access we need. Thank you, from me, for all of the people relying upon you.

— Don Macvittie

April 14, 2020
SRE (Part 2): A Practical Approach
In my last piece on cultural SRE, I covered the basics of defining what SRE means for your team. However, some teams may not need dedicated SRE support. This is where the central organization comes in handy. Dedicated resources are focused on building guidelines, standards, services and platforms for teams to consume that can operate autonomously in your environment. Having a broadly communicated central team also allows for these other teams to know where to go when they need assistance or would like some recommendations on their technology stack.

Application/Service SRE Versus Infrastructure SRE

Application SRE includes:
- Embedded for application/service teams.
- Architecture guidance for new services and infrastructure.
- Design and implementation for modern technologies.
- Support within the application/service team.
- Automation for apps and services for the team.
- Automation for operational work.
- Monitoring/metrics for application/service team.
- Benchmarking and performance assistance for code.
- Infrastructure deployment and architecture for applications and services.
- Write documentation and runbooks for alerts issued by application/service stack.
Infrastructure SRE includes:
- Build and management of “plumbing” technical infrastructure (provisioning, OS, dns, dhcp, networking, central auth, etc.).
- Automation of infrastructure services (telemetry, monitoring, log aggregation, configuration management, anomaly detection, orchestration, etc.).
- Build and management of consumable services and tools (message queues, databases, distributed compute farms, API services/integrations, containers, etc.).
- Infrastructure as Code (IaC).
- Implement Global IR&IM process.
- Support for Application SRE teams.
- Architectural guidelines and best practice documentation.
Organizational Layout

Organization structures, whether we like to believe it or not, really help in the flow of communication throughout the team. It allows for the team leaders to form a single mission and have that mission carried out by their team members. I mentioned in Part 1 of my SRE Narnia guide that I believe this organization must be centralized to be successful. Below is a high level overview that shows the separation between application SRE and infrastructure SRE.

The leads can handle multiple teams and disciplines. This can shrink and scale as you see fit within your organization. Application SRE is straight forward. The infrastructure SRE could be embedded across existing infrastructure teams or form a new team focused on a specific project.

Example SRE Interaction

Below is an example interaction between an application team (this one runs a ruby app) and an SRE hierarchy. You will see a team called “Frontline Support” as the initial interaction. This team is not required but is a nice to have. It helps in any environment big or small, and can really offload a significant amount of the operational workload along with having a global view of issues coming in.

The duties for frontline support are to follow runbooks for alarms flowing in from the monitoring systems. It is similar to a traditional NOC but with more expertise across the stack they are watching. It also allows for trend analysis and bringing data to conversations around reoccurring issues or bugs in the system.

I do recommend using this as a starting point for junior level team members. It allows them to learn the system quickly and buddy up with more seasoned members that are either on this team or have the pager for the week. It is a great way to train and onboard your technical employees. Nothing works better than showing them what is broken.

Summary

Overall, everyone’s journey will be different. There are quite a few books and examples that exist out there, but please only use them as ideas to form your own opinion on how these organizations should work. There is no prescription and nothing is ever perfect. I have left out quite a bit of information in this post but I am always interested in having conversations about this topic and others alike. There is one consistent theme you’ll see across the board and that’s shaping the culture. Focus on your culture and hire great talent along with great leaders. Good luck and happy hacking.

— Anthony Caiafa
April 10, 2020