During these crazy times, I can’t help but wonder, how would I even survive being on lockdown for so long if it wasn’t for technology? It is the one thing that has kept most of us sane while stuck inside during this crisis.
Apps like Instacart and Zoom have been a hit during the pandemic. Why leave the house to go grocery shopping and risk exposure to the virus when you can get your groceries delivered at your doorstep? And thanks to video conferencing services like Zoom, you can have face-to-face meetings with your boss and colleagues, or socialize with your friends and family during happy hour.
Of course, it doesn’t stop there. Tech companies, such as Google and Facebook, have done a great job in providing information on COVID-19 from trusted sources, tips on how to take care of yourself and ways to deal with staying home.
Let’s not forget all the entertaining dance moves we have learned from apps like TikTok, and all the nights we’ve spent binge watching our favorite shows on Netflix. According to Netflix’s quarterly earnings report, the streaming service gained more than 15 million subscribers between January and March.
Technology has helped businesses to carry on working and it has helped all of us to stay strong during these hard and uncertain times.
To celebrate the tech community, MediaOps is hosting the first-ever TechStrong Con, a virtual event that brings together some of the most prestigious industry leaders from companies including Microsoft, JFrog, Red Hat, IBM, GitLab and many more for 12 hours of immersive learning and networking.
Taking place on June 4, starting at 9 a.m. EDT, the event will feature a wide range of tech-related topics, from hardware and applications to processes and business of tech. The sessions are divided into three tracks: Emerging Technologies, Security/DevSecOps and Digital Transformation.
Become a part of the event, get downloadable resources and interact with tech leaders and other attendees. Registrants will also be able to participate in TechStrong raffles for a chance to win one of the following prizes:
An Xbox One X 1TB Console – Star Wars Jedi: Fallen Order Bundle
One set of Bose 700 Noise Cancelling Wireless Bluetooth Headphones
One pair of Apple AirPods Pro
10 Amazon gift cards (valued at $50 each)
Strengthen your skills, improve your best practices, empower your teams and become stronger than yesterday. Together we are strong, together we are TechStrong!
As enterprises move from the first phase of the pandemic to the reopening phase, they’re going to have to determine what tech investments going forward will provide the most value in the their new normal.
Emad Georgy, founder and CTO at consulting firm Georgy Technology Leadership, said that while overall tech spending will remain down, that pain won’t be evenly distributed. Georgy claimed that investments will continue to remain high in the obvious industries, of course, such as healthcare, retail and grocery chains.
“However, multiple adjacent and dependent businesses investments will also remain high. Suppliers of materials required by these industries, companies involved in their supply chain, companies supporting remote collaboration tools will continue to thrive,” said Georgy.
Coming off the heels of the economic shutdown, Alex Milowski, platforms researcher and evangelist at database management software provider Redis Labs, said that both people and systems have experienced significant disruptions, and this has hurt their productivity and strained technology platforms in some ways, while for other job functions platform and infrastructure are at over capacity.
Milowski said organizations needed to strike a balance between the immediate needs of staff, so that they could adjust to the changes in their work and lives and still access critical work systems while also ensuring that customer needs are also met.
However, now that enterprises are looking forward, how do they ensure that their digital transformation efforts remain in place?
As enterprises get their footing as the new economic reality settles in, Milowski advised organizations to focus on small wins.
“Prioritizing projects where one can increase the confidence of the team to function under uncertainty may be the best strategy. New projects may need to be created to address surges in demand via caching or acceleration systems,” said Milowski. “A team might use this time to address auto-scaling systems so they can scale-down as a cost-saving measure. The increased collaboration and creative problem solving will build team confidence in their ability to deliver. With each win, ensure readiness is improving.”
Within those early successes, teams can then troubleshoot their processes with less disruption to their business.
“As their confidence grows, they can ease into picking more business-critical and complex projects. In the end, they are more likely to succeed, with less disruption, as their team is now ready as they know how to deliver new digital initiatives in very different contexts. Successive small wins turn into big wins,” said Milowski.
Georgy indicated that such uncertain times require a sharp focus and prioritization.
“Financial insecurity often leads to sharpened discipline which one could argue we should have anyway in an organization. Unfortunately, many companies are plagued by ‘management by assumption.’ That is the practice of managing and deciding based on opinions or second-hand knowledge because they don’t have time to [dig into] the weeds,” he said.
It’s time to prove concepts out. Georgy added that it’s more important than ever to make data-driven decisions based on experiential learning.
“Opinions are no longer welcome. Every person at the decision table must justify their decisions with data, with real-world experience and with action,” he said.
That can be done by CIOs as they track their changing customer base.
“Customers are changing how they spend and re-evaluating what is important to them and this will be reflected in roadmaps as well. Now, more than ever, feedback loops, usability groups and touchpoints with customers to understand this changing dynamic is key to evaluate new approaches and priorities in the roadmap,” said Georgy.
John Belden, project execution advisory services practice leader at UpperEdge, stated that now is the opportunity for CIOs to lead in a way that hasn’t been readily possible in more than 20 years.
Belden advised CIOs to:
Take the lead: CIOs can win back the seat at the management table that they enjoyed during the Y2K era. By proactively reaching out to work with counterparts in HR, marketing and supply chain, CIOs can shift from order-taker to problem-solver, which will also help in prioritizing projects.
Determine an approach to road-mapping: There are likely to be three different strategies to road-mapping and navigating this time:
Reinvent: This approach establishes a clean sheet of paper and takes a more traditional approach. Companies that could never get the cash or attention to build a roadmap, might have this work thrust upon them if corporate boards demand a pandemic risk mitigation plan.
Revise: This would involve the shuffling of the existing portfolio, some reprioritization and confirmation of direction from the various vendor’s strategies.
Accelerate: Some companies may have already charted a course that accounts for assumptions of a new world order. In these cases, it is likely that CIOs will be asked to accelerate to either leap-frog competitors or solidify their competitive position.
Transform vendor engagement strategy: The onslaught of vendor marketing initiatives and client targeting strategies could end up being like the wild west with cloud providers bypassing traditional IT channels to market their wares. CIOs need to work proactively with their business counterparts to set the ground rules for engagement and craft a strategy that motivates suppliers to assist clients in developing their own strategies that can be realized quickly and efficiently. Developing a long-term view in managing the short-term downtime will, in the end, pay dividends in executing during this economic downturn.
Short-term survival with an eye on meeting long-term goals is key to survival, and in the months ahead those enterprises that successfully meet that balance will set themselves apart.
If you live in the downtown areas of Pittsburgh, Palo Alto, Miami or other AV hot spots, you’ve probably been missing the Ford Fusion Hybrids with the bulky lidar sensors in your neighborhoods. The COVID-19 pandemic has resulted in many job cuts as well as work from home orders from the government, and this has forced big driverless car technology companies such as Argo AI, Waymo, Aurora, etc., to halt on-road testing of their vehicles and work remotely.
This would impact many automobile companies partnered with these firms that had plans to mass produce and release their fully autonomous vehicles by the end of 2021, including the Ford Motor Company, Volkswagen Group, BMW, etc. The only way to bring back the accelerated development and testing of these technologies is by realizing their current value in the consumer market, especially when humans need resources to be delivered to them without spreading the virus further.
This article will shed insight on some perks that autonomous vehicles provide in mitigating the widespread impact of this virus, as well as whether these advantages make us realize their value in everyday life.
Home Delivery Without a Human Delivery Worker
We’ve all tried delivery when in a rush or simply when we’re too lazy to go out and buy food or groceries. This used to be an occasional thing for the most of us; something we wouldn’t do every time we purchased an item because the delivery charge was an additional expense. But COVID-19 has changed our entire lifestyle dramatically, and now most of us order our groceries online or through shopping apps such as Instacart or Amazon, so that we can avoid having to physically go to the store. While this seems like a better alternative, it also puts millions of delivery workers at risk of contracting and spreading the virus, making the outbreak worse than it originally seemed.
Autonomous delivery vehicles are a potential solution to this issue, since they allow for driverless home delivery of items that customers have purchased online, limiting physical interactions with the items to two point-contact (store shopper and customer). Nuro, a leading self-driving vehicle startup in the U.S. founded by former Google engineers Jiajun Zhu and Dave Ferguson, recently became the first driverless car manufacturer to obtain permission from the Department of Transportation (DOT) to produce and put driverless cars into action for home delivery. The R2, Nuro’s second generation driverless car after the original R1, is a low-speed vehicle that can interpret and adapt to a wide variety of weather and road conditions, while not posing considerable risks for pedestrian safety. Its maximum speed is sealed at 25 mph with self-adjusting acceleration and braking control in response to traffic flow and obstacle detection.
Like most driverless cars, R2 does not possess a steering wheel, rear-view mirrors and actuator pedals, placing it in the exemption category for vehicle production according to the DOT’s Federal Motor Safety Standards (FMVSS). This is an exemption that is not easily approved by the DOT, with many automobile companies trying their luck at getting their mass driverless car production proposals approved without success, until Nuro recently hit the jackpot.
Although this exemption has only been approved “with regulatory certainty” as of now, due to the low speeds that R2 can travel, it has enabled Nuro to get more R2s out working in Houston, delivering customer orders from Walmart, Domino’s and Kroger. The R2 is a start to transitioning to a fully driverless delivery experience, which in our current state, is a great way to prevent the spread of the virus from exacerbating further.
Could Autonomous Vehicles Be Our New Mail Trucks?
Every week, thousands of mail workers across the U.S. hop into their mail trucks and drive to multiple neighborhoods and communities that they serve, in order to deliver mail to residents. Receiving mail in a timely manner is very important for the most of us, since we need to pay our bills for water, electricity, gas, etc., every month and most of these statements arrive by mail. During this pandemic, mail workers have been working long hours to get all of our mail delivered to us without delay. Through this process, they haven’t been able to take many days off to spend with their own families or rest at home if they get sick, making them even more vulnerable to catching the coronavirus.
Autonomous mail trucks are a solution that USPS has been considering since late 2017, and to fulfill this dream the Postal Service has partnered with the University of Michigan and TuSimple, a self-driving truck startup. The plans to put an autonomous mail truck to work have only been extended as test runs to a couple of cities across the U.S., such as Phoenix and Dallas, and about 10 rural routes in Michigan as of May 2019. Although these remain a viable solution to be used in conjunction with traditional mail delivery by mail workers, efforts to develop autonomous mail trucks has declined over the COVID-19 crisis due to other priorities that have sprung up.
Counter to this thought, introducing autonomous mail delivery vehicles during this time to assist mail workers (as originally planned by USPS) or operate independently with robotic assistance in at least a few areas that have already been tested and approved, would highly facilitate the mail delivery process and reduce the stress and workload of mail workers across the United States.
The latter option would allow mail to be delivered directly to customers, with robots inside the vehicles serving as mail workers that sort and place mail in each resident’s mailbox. Residents can pick up their mail directly from their mailboxes without needing to worry about social distancing, spreading the virus to the workers or, counteractively, getting the virus from them. Even if mail workers did accompany the autonomous mail trucks, their work would be halved in intensity, since they would only have to sort and place mail in each resident’s mailbox, without needing to worry anymore about physically driving to every neighborhood that they serve.
Shifting Supplies from One Location to Another on Pedestrian-Free Routes
We’ve long considered autonomous vehicles to be an option to use in factories, truck yards and other less congested routes, primarily to move inventory to different locations. This is a great use for AVs since it involves minimal, if any, contact with humans or pedestrians, so it doesn’t pose a high risk for safety.
The pandemic has raised a lot of new challenges to implementing this idea across the board, in terms of the inability to commit much time at all to these efforts at the current moment, as well as the difficulty in monitoring test vehicles within facilities with very few workers on site to overview the actual process.
Despite these challenges, Phantom Auto, an automotive safety solutions company based in California, recently announced their decision to deploy their new driverless trucks — which they call productivity vehicles — across their truck yards to load/unload inventory on their own, without needing a worker to drive the trucks. Phantom Auto calls these trucks “almost” autonomous in terms of their navigational capabilities, since they may still need a little input from remote workers to guide them to the right docking/undocking locations within the yard, but the workers do not have to be physically present on site or inside the vehicle to drive it over themselves.
This is possible due to new technology called HILT (Human in the Loop Technology), a telecommunication software advancement that allows telemonitoring, remote assistance and, in emergency scenarios, teledriving performed by a worker remotely. Performing these labor-intensive tasks autonomously during normal times would be a great relief on workload for workers employed at truck yards, but during this crisis, this is an absolutely essential step that more factories and truck yards should invest time and efforts on, in order to prevent the spread of the virus amongst their workers.
Mayo Clinic, an American non-profit organization and academic medical center, has also been working to get coronavirus tests and medical supplies delivered to hospitals on pedestrian-free routes in Jacksonville, FL, to prevent workers from getting exposed to the virus. The driverless vehicles being used to transport these supplies are autonomous shuttles from Beep, Mayo Clinic’s new corporate partner from early April of this year.
Currently, about four of these shuttles are working independently on different routes in the Jacksonville area, while being monitored by Beep, Mayo Clinic and the Jacksonville Transport Authority Officials, to ensure that the vehicles don’t steer off route. With these constraints and monitoring methods in place, autonomous vehicles can be deployed across a variety of platforms to perform simple, supply shifting tasks that human workers have been undertaking in addition to their primary responsibilities for years, during the crisis and post-pandemic.
Everything Considered…
With their day to day work and responsibilities increasing multifold during the pandemic, frontline workers have been doing a lot of overtime, not being able to spend time with their families during and further exposing themselves and their families to the virus, so that the majority of us can stay safe at home and still get our necessities delivered to us. It’s now time for technology to take over, and one possible solution to help ease their work is automating tasks that are laborious, such as grocery and mail delivery, supply shifting, etc.
Autonomous vehicles that have already been approved for deployment, or in the final stages of testing, can help with performing these tasks since they don’t require a driver. These vehicles can be supervised by a physically present or remote worker to ensure that they follow the right routes and operate at the appropriate speeds
A few firms have already started taking the initiative of deploying these vehicles to help prevent the spread of the virus amongst their workers and customers, and are witnesses to the overall simplification and ease of the automated process over the manual labor that they had originally employed. The shift to a more automated future that we had originally planned at the end of 2021 should start when we need it most, which is during this time of crisis. So, at this time, are we willing to make the change and invest more efforts in developing and testing autonomous vehicles to perform our everyday tasks?
It’s Spring 2020, and even in challenging and unprecedented times, love continues to bloom. That said, data pipelines are never far from my thoughts and the confluence of those two things got me thinking about data engineers and how their initial love for creating data pipelines often turns to loathing.
Data pipelines are the backbone of modern data systems, yet there’s still much to be desired when it comes to connecting and orchestrating the movement of data between data systems, including business intelligence, data processing, data warehousing or other applications. As an industry, we haven’t paid as much attention to the happiness of the people who build and maintain data pipelines as we have to the volume or velocity they support. As a consequence, today’s data engineers are building increasing resentment toward the technologies and tools that previously offered so much promise.
What is happening here, and can it be fixed?
Love at First Sight
When we first met, everything was magical. We traveled the data world, far and wide, no longer bound by the constraints of relational databases and ETL tools. Vertical scaling was a thing of the past, and with horizontal scaling, we felt we could reach the end of the earth together.
The introduction of Hadoop, Spark, Kafka and more was a truly magical evolutionary stage for data pipelines. We could store and process more data than ever before, derive insights never thought possible, and do it all at a fraction of the cost.
What was there not to love?
Definitely, Maybe
As our courting phase came to a close, we were happier than ever. We were spending more and more time together, eagerly building a future. We were both getting accolades at work, accompanied by exciting promotions and responsibilities. We were also growing increasingly comfortable with each other, and while at times it felt you were being a bit clingy, it was nice to be needed.
With pipeline technology continuing to advance, it felt scale and speed had few, if any, limits. We built more and more pipelines, linking outputs and inputs, tuning performance and performing increasingly complex transformations, all as part of an increasingly interconnected data ecosystem.
Our data teams grew, and with more data engineers we built even more pipelines. We also started to notice that these pipelines are quite a bit higher maintenance than originally thought. They require frequent tuning and optimizations, yet still break far more often than they used to, usually at the least desirable times–like at 3 a.m.
This is definitely turning into a love/hate kind of thing.
10 Things I Hate About You
I miss the old days. What happened? I’m just trying to hang out with my friends and you won’t stop calling–it’s suffocating me. When we do spend time together, the romance is gone. Horizontal scalability has lost its magic.
Most data engineering teams we encounter are firmly planted in this stage of relationship with data pipelines. They’re honestly feeling a bit trapped. They’ve built a ton of remarkable pipelines that do incredible things. These pipelines also require a tremendous amount of maintenance, and each new one they build only compounds the problem. As a result, we see most data engineering teams actively distancing themselves from data pipeline technology, building increasingly large layers of abstraction that sit between them and the raw technology.
Surely, all is not lost?
Something’s Gotta Give
Right when I thought we were at our tipping point, we had a breakthrough. We weren’t communicating effectively, our languages of love were entirely different. I wanted to focus on the destination, the end result, and you wanted to focus on the journey, the path to get there. There clearly was a gap, and we needed help bridging it.
Fortunately, for the data engineer and their muse, the data pipeline, there are advances in pipeline technology, bridging the gap and offering a path toward reconciliation. One such path combines declarative programming with an underlying control system that understands the declarative model. Such control planes translate declarative models into imperative instructions for the underlying infrastructure, making pipeline development far less plumbing, and far more architecting. Declarative programming allows data engineers to focus on the big picture, drawing on their experience and creativity to design what the optimal data pipeline should look like, rather than spending time repairing failures, tuning parameters or manually tracing code.
The underlying control system, or control plane as it’s called, imbues the pipeline with autonomy, data awareness and intelligent persistence. Autonomous data pipelines are less needy and can handle data changes on their own (rather than making the data engineer responsible for everything). Data awareness allows pipelines to intuitively know what changed and what subsequently needs to change. Intelligent persistence allows pipelines to be considerate of cost and resources and makes sure everything is fast, efficient and never duplicated.
Love Actually
I think it’s safe to say those among us who chose to enter data engineering didn’t do so imagining ourselves, infrastructure plumbers, spending hours upon hours maintaining data pipelines that fuel someone else’s data-driven creativity. No, data engineers want to create digital services and contribute business value, too–they’re just trying to keep everybody else out of the data muck in the meantime.
This is the dawn of a new era in data engineering, and many of the early adopters of this latest approach to pipeline development have found it enables creation with far less time, code and maintenance. By embracing this new method of development, data engineers can again see a happy future where more of their time and skills are devoted to the things for which they–and the business–have real passion.
Although 99.999% availability may be IT’s ambition, it’s far from reality. In March 2019, Google’s Gmail experienced a 4.5-hour global outage. In the same month, Facebook suffered a 14-hour outage, its most massive outage to date, crippling app-accessibility worldwide.
Downtime and low availability will likely occur to all systems eventually, especially when introducing code changes. When outages occur, it’s truly the detection systems and smart mitigation processes that separate quick rebounds from long recoveries.
LogicMonitor has released its IT Downtime Detection and Mitigation Report, a survey of IT professionals to uncover trends and tactics impacting availability in 2020. The report surveyed over 300 IT design makers at organizations with 2,500 or more employees within the U.S., U.K., Australia and New Zealand.
Below, we examine the LogicMonitor report’s key findings. We’ll discover, on average, what sort of projects are causing the most outages and brownouts. We’ll also see what strategies top IT teams are using to prevent, detect and mitigate disruptions.
Which Transformation Initiatives Hurt Availability the Most?
Ninety-six percent of organizations surveyed had experienced at least one outage in the past three years. But why? One reason LogicMonitor conducted this report was to pinpoint which digital transformation initiatives and IT trends are the leading contributors to high-profile outages and brownouts.
When determining causation, answers revolve around novel cloud IT initiatives. Fifty-nine percent of respondents felt that mobile computing was causing more brownouts/outages. Fifty-seven percent found AI and edge computing were making outages more common. Other high-ranking reasons included digital transformation (57%) and IoT (53%).
According to the report, “LogicMonitor’s research suggests that IT decision-makers hold IT transformation initiatives responsible for increasingly frequent outages and brownouts.”
Though we generally view digital transformation initiatives in a positive light, they do come with caveats. The move from private to cloud infrastructure, for example, can bring unforeseen costs. Also, some authorities note a lack of talent specializing in cloud and hybrid environments. The report also suggested that accelerated cloud technologies may require time to mend before they realize positive business returns (and higher availability).
Critical Strategies for Preventing Outages
Another goal of the study was to discover what sort of tactics IT professionals are actively using to prevent outages.
The study exposed many strategies that IT folks are currently undertaking. The top three tactics were performing preventative maintenance (75%), reviewing system logs (71%) and increasing the capacity of systems (71%). Other top strategies included designing redundancy into systems, and keeping an eye on customer support tickets.
With the rise of a completely remote workforce, some IT executives do anticipate higher expenditures due to AI and cloud architecture, which could bring added pressure. However, it’s worthy to note the cloud hasn’t negatively affected all groups, or even reached all companies. According to another study conducted by Adobe and Fortune, only one in three organizations store their data in a public cloud. Of course, one could argue that in doing so, such companies substitute innovation for less breaking change.
Another necessary aspect to keep a vigilant eye on the digital ecosystem is proactive monitoring. According to the report, 74% of teams rely on proactive monitoring to detect outages. Continuous monitoring has been a philosophy trumpeted time and time again by DevOps professionals to enable a better release frequency and a window to detrimental interruptions.
Careful analysis ahead of time, active monitoring systems to spot problems and continuously reviewing system logs were the most important tactics found being applied by IT leaders to mitigate found outages and brownouts.
Best Practices: Prevent, Detect, Mitigate
In terms of takeaways (other than the ultimate LogicMonitor product sell), the IT Downtime Detection and Mitigation Report does reveal some interesting trends on what IT leaders view to be leading causes of system outages. It will be interesting to see how this evolves and what technologies mature (or remain headaches) into the future.
The report leaves IT professionals with some common, albeit helpful, advice to consider going forward:
Digital transformation is top of mind for every organization now. The problem is, many of these organizations take the phrase too literally and think it’s all about adopting new technical capabilities and tools. Tech is certainly a part of it, but real digital transformation is also about changing company cultures and ways of working. Tech and culture go hand in hand for digital transformation; you can’t change one without the other.
Successful digital transformations mean redefining the way IT works, breaking down the silos between the development, production, UX and business teams in a way that cultivates greater collaboration, unlocks new business value and simply makes work better for everyone. All of that comes down to pivoting to a BizDevOps culture that creates more visibility, and positive reinforcement, for engineers and developers.
Building a Cultural Transformation Around BizDevOps
The goal for BizDevOps is to move development and production teams away from nursing applications and tediously staring at dashboards, and align them closely with the business side, so that they’re putting their time and energy toward delivering more business-relevant services. Shifting gears to a BizDevOps structure is its own culture transformation. Eliminating the silos between teams encourages better cross-collaboration and opens up new visibility for developers to see how their work is being received by users and impacting business value for the organization as a whole.
But, breaking down these silos to get engineers more engaged in production introduces another challenge: Those developers are already busy building products and features that they’re responsible for. Why would they want to get involved in production on top of all their other work? Overcoming this hurdle means making concerted efforts to give developers more visibility into production environments. More than that, it means exposing developers to more positive feedback.
Encouraging Engineers Through Constant Feedback
What makes engineers feel proudest about their work? When they see how it’s being used and enjoyed by their target audience. The problem is engineers rarely get that kind of feedback. The feedback that is typically sent back to the engineers are critical comments and complaints; they rarely see or hear the good things, just the problems they have to fix for the next iteration.
When you consider that, who could blame engineers for feeling discouraged or unwilling to want to take on more work? From their perspective, it’s not just about adding more to their plate, it’s also taking on more negativity and more criticism.
But it doesn’t have to be that way. What if engineers were provided with immediate feedback on all the new applications, features and services they were creating? Both the good and the bad, in real time? That’s part of the key to facilitating a BizDevOps culture: When engineers are seeing how the products they put out into the world are being enjoyed, as well as the positive impact they have on business value, they’ll naturally begin to feel more appreciated and also more reaffirmed that they are doing good, meaningful work and that they’re having an appreciable, positive impact on users and the whole business. This doesn’t mean putting on blinders to negative feedback, but it’s easier to act on negative feedback when it’s balanced out by positive feedback, too. Exposing engineers to more feedback about the impact of their work might also help inspire them to have new ideas and drive new innovations. The more information you’re armed with, the better position you’re in to innovate.
Organizations need to prioritize building easy ways for developers to have this visibility in their environments where they’re getting immediate feedback from end users, both the good and the bad. The more feedback they receive, the more encouraged by, and invested in, their work they become and, consequently, the more responsibility they feel for their contributions to the business. With that responsibility comes the culture change you need to digitally transform.
Top-Down Support
This culture change can’t happen without executive buy-in. The C-level needs to provide the necessary tooling and clarity on new organizational structures to break down silos between the development, operations, business and support teams, and then shepherd those teams into a more collaborative and visible environment. That support creates the foundation for a wider culture change, one that encourages a transition from operations to development and enables a bigger shift to BizDevOps.
Imagine the silos between teams as vertical walls. Now, shift those walls 90 degrees. That changes the structure so that your platform and autonomous cloud teams are sitting on the same level together, providing services across the organization, delivering software into production at faster speeds while simultaneously achieving visibility across the whole environment.
The democratization of data enabled by this level of visibility is helpful in another way. When all teams have access to the same data, and can see what each other is working on and how all that work moves toward a shared goal, it drives everyone to find new ways to make each team’s workload lighter–such as with automation. When teams are not just automating their own work, but automating each other’s because they see how lifting one group lifts the whole organization–and has a concrete impact on business value–that’s what BizDevOps is all about.
A Transformed Culture
In the end, every engineering team should think, feel and act like their own startup, responsible for everything from end to end. That spans from what they’re building, to how they implement it, to the impact it has on the end user and the business. Every engineering team needs to be working like this, keeping top of mind considerations like, how can a new application or feature or service be optimized so that it not only runs well, but also runs efficiently? Not just efficient in terms of performance, but in scalability as well.
Think about how cloud providers have potentially endless resources to deploy code. Just because something might run well doesn’t mean it’s running efficiently, and just throwing more code at a situation will inevitably raise costs and hurt overall business value. That’s something that engineers and developers should be keeping in mind as part of their work–not just how to deliver value in terms of new features or services, but how to bring down the total cost of ownership as well.
That’s what digital transformation looks like: A culture shift where new BizDevOps organizational structures, visibility across production environments and immediate access to feedback, both positive and critical, enables all teams to work together in a cohesive way, driven by what creates value for both the business and the end users.
The COVID-19 global pandemic has ushered a significant priority shift for the digital workplace. As development teams become completely remote, executive priorities are rapidly adapting to meet the gravity of these unanticipated times.
But, were IT leaders prepared for such a dramatic shift? What are they doing to evolve? To answer these tough questions, let’s turn to the data. A March 2020 study, conducted by Adobe and Fortune, surveyed 200 chief information officers (CIOs) on how their priorities are changing to meet the needs of a global crisis. Here’s what they found.
Remote Work
As it turns out, the majority of enterprises (84%) report they were prepared for remote work, with a slight increase for SMBs, at 94%.
“Like most business leaders, CIOs now have to rethink priorities, or at least reorder them, and we must reinvent ourselves now as virtual leaders,” said Adobe CIO Cynthia Stoddard.
We are living in a golden age of virtual collaboration tools—teams have Slack, GitHub, Zoom, Microsoft Teams and many other options. Yet, 53% of CIOs still cite communication as the biggest hurdle to remote work. Nevertheless, half of the CIOs surveyed reported their organizations are still actively hiring, indicating steady growth for IT.
New Technologies
Regarding technological adoption, some organizations are struggling to keep pace. Twenty percent of CIOs report serious shortfalls in technology tools. Interestingly, less than 25% of CIOs estimated increased financial spending for areas such as public cloud, infrastructure and artificial intelligence/machine learning (AI/ML).
“Indeed, nearly all CIOs (90%) surveyed said they use a public cloud service for at least some of their data, but most is housed on-premises,” said Giselle Abramovich. “Only one in three organizations store half or more of their data in a public cloud.”
One thing most enterprises agree on is the need for better security—seven in 10 anticipated increased financial investments into cybersecurity.
Artificial Intelligence
Only 50% of enterprise CIOs said they use AI in one or more projects (25% for SMB CIOs). However, it seems to be a growing discipline, as 90% of the organizations that have implemented AI have done so in the past year.
Though AI has garnered significant interest, there are hurdles to actual implementation. According to the report, “the biggest issues related to AI include getting data in order, funding and finding the right talent.” With more people online and the importance of distanced communication, it is likely that investments into AI will increase, especially for use in IT support and customer interactions.
“We’re making AI a strategic priority at Adobe. AI implementation is difficult for many organizations because it’s still new to most businesses, and there is a shortage in talent that has a deep understanding of it,” said Stoddard.
Analysis: Impact on DevOps
CIOs are in a unique position. Leaders are reevaluating business practices and also reassessing their technological investments. As the Adobe/Fortune study shows, many companies were prepared for 100% remote work, albeit some more than others.
To endure a COVID-19 economical drought, CIOs are taking action. The report indicates increased investment into remote tools, AI and cybersecurity, while aligning hiring and new infrastructure to reach these goals.
Institutions with a previous investment into cloud infrastructure as well as scalability practices, such as serverless, are likely well-equipped for increased online interaction and unanticipated usages. In this virtual realm, automation is also key to streamline deployments and monitor applications in remote fashions.
Though the report doesn’t differentiate between company types, it’s likely that digitally native enterprises are well-equipped as well. In addition, the trend from physical storefront to “brick and click,” has only been magnified by the current climate, forcing retail and banks to invest heavily in digital technologies and DevOps.
From a DevSecOps standpoint, increased end-user online action means an increased number of threats. Thus, security strategies for containers and service meshes should be reevaluated.
By repositioning priorities, IT can adapt to the new state of interaction. With deliberate execution, CIOs and all executives alike could find themselves on the right side of the river once the economy resumes.
People working in IT support and incident management right now are faced with unusual difficulties supporting large remote workforces and managing unpredictable workloads. On Reddit, system admins and other IT pros are bemoaning the hiccups and hassles of working in isolation while trying to resolve issues and maintain high SLAs. You can’t go grab your indispensable SME for troubleshooting, because that person is also home and inundated with messages and alerts from many different tools.
Networks, VPNs and servers are being hammered extra hard right now from higher usage of SaaS apps and collaboration tools. User expectations are even more grandiose than normal: The home-based IT environment is a lifeline to whatever remains of human interaction. IT organizations are expected to make applications and services work flawlessly–overnight. Undoubtedly, IT pros are working with assiduity to keep the wheels of work moving for their employers during a time of crisis. Yet, working longer hours isn’t the answer. We need new ways of working—ones which may prove better than the current status quo for the long term.
In many organizations, the process of incident management and resolution begins with people looking at screens full of metrics. You see a spike in something, such as network I/O, and immediately begin to investigate it, assuming the worst. A more adverse scenario is when users call out problems on Twitter, Reddit or other popular tech communities, creating a noxious PR situation which IT and the business must swiftly respond to and manage. This ad hoc, reactive fire drill takes time, and resolving that particular issue may not be the most important item of the day. In software development, it’s akin to the bad practice of designing the architecture of a system first and then the user experience.
Given everything that’s at stake in the foreseeable future of stay-at-home mandates, SREs and IT support technicians need to flip the incident management process. Rather than beginning with the metrics and working backwards to uncover the problem, if there even is one, start by looking for the most important problems to solve. Then, discover the resources and components which support that end-user process and finally, analyze the metrics which are most relevant. For example:
A B-to-C provider of a streaming video service is now seeing a 200% increase in daytime traffic. This is resulting in sluggish load times for users and glitches.
IT has tagged the components beneath that service, so IT operators know exactly which VMs, databases and load balancers come into play.
Next, the operations team can identify the most relevant metrics, for instance CPU idle time, to measure and take steps to correct.
The Problem-First Approach
It sounds so simple: flip the process. Yet, this is not the case. IT operators and SREs are not accustomed to thinking in terms of which problems to solve or prevent. Being proactive or predictive hasn’t been the common workflow.
Here is how IT operations and DevOps teams can take a proactive approach to managing the availability and health of enterprise services.
Coordinate Across Departments
Identifying the most critical problems to solve right now is the hardest step and the one which will take the longest. Common problems these days are related to cost and scale: How do I right-size my environment to reduce costs or handle more workloads? Broaden your field of inputs to narrow down to the specific issue, such as slow page loads on the financial reporting website. Product managers, account managers and business unit leads who are the closest to the customer experience can deliver feedback on the top issues affecting customer/user satisfaction. Teams should also review recent support tickets to identify common themes of pain.
Fine-Tune Your Approach to Metrics
The Utilization Saturation and Errors (USE) Method is one way to approach the problem-first incident management process. As detailed by Brendan Gregg, a senior performance architect at Netflix, this methodology begins by posing questions, seeking answers and working backwards to the metrics. For each resource that you want to measure, identify three metrics: one for utilization, one for saturation and one for errors.
“The USE Method has made you aware of what you didn’t check: What were once unknown-unknowns are now known-unknowns,” Gregg explained.
Standardize Workflows
Create a common process for incident management across all your teams. Without the advantage of having almost everyone in the same room to huddle together in an ad hoc fashion when big issues crop up, it’s imperative to institute clear steps and roles. Doing so will prevent the frustrations, confusion and oversights that needlessly delay resolution. Since most incidents are composed of multiple contributing factors, teams need to adopt a small number of user-friendly tools to document and organize the information. At our company, we’re now using tools such as Miro, an online whiteboard application, to replace our physical whiteboarding sessions. Of course, there’s also Zoom, Slack, Jira and a host of other cloud-based tools already in place at many organizations. Mandate which tools everyone should use, with some guidelines on how to use them.
Increase Automation
In some organizations, scaling requirements in response to demand have increased tenfold. Automation is playing a critical role now; moving away from a web GUI, for example, is more scalable and aligns with modern tools such as Chef and Puppet. User tickets can be auto generated, for instance, from emails and linked to code management systems like GitHub. Modern development and operations teams are also expanding automation in unit testing and provisioning.
Watch for Burnout
Whether there’s more work and/or a need to fill the hours during long quarantine days, many software engineers, testers and architects are working longer days right now. Yet, exhaustion and burnout can lead to errors and oversights along with low morale. It’s up to managers to make sure that employees are taking breaks, working reasonable days and having the time and energy to attend to personal needs.
As we move toward 2020, the expectations are high, the software industry is moving faster and the demand for rapid and continually improving digital experiences have raised the standard. To match the needs and demands of the market, firms around the world are embracing the DevOps culture to ensure and streamline the development, packaging, testing, deployment and maintenance of software at scale.
Every DevOps organization is defined by its own leadership, culture, practices, policies, leadership style, structure, talents, skills, technical capabilities, processes and tools. However, there are certain common characteristics that are important to consider before deciding a better DevOps platform to use in your organization. Today, we will discuss such important features that can help you while selecting your DevOps platform.
What is DevOps?
DevOps is a mindset in an organization that enforces a set of collective practices to ensure easy collaboration between dev and ops so the business can run smoothly without any hassle. DevOps aims to fasten the time to market through frequent and incremental improvements to deliver quality software updates to customers.
The meaning and definitions of DevOps can vary from company to company. Automation being the focal point, firms want to focus on achieving zero-touch automation, with utmost security and governance as a priority in the delivery pipeline.
Even though the meaning of DevOps varies, several inherent DevOps principles cannot be replaced, and they repeatedly insist on collaborative relationships, developing early, failing early, shared learning and continuous feedback. These DevOps principles enhance quality with improved deployment rates while simultaneously improving predictability, scalability and security of the production environment. Tech leaders in the company need to view DevOps as a holistic approach rather than a small set of guidelines for the teams, and DevOps guidelines should be aligned with the company’s long term technology vision and objectives.
Here are some facts to consider while deciding a better DevOps platform for your organization.
A Mindset of a Unified Culture
Culture plays a significant role in establishing the guidelines while embracing DevOps in any organization. Through DevOps culture, companies seek to bring dev and ops teams into harmony to promote collaboration, automation, process improvements, continuous iterative development and deployment methodologies. But above everything else, a sound DevOps culture fundamentally solves one of IT’s biggest people problems: bridging the gap between dev and ops teams to get them to stop working in silos and have common goals.
According to Gartner estimation, DevOps efforts fail 90% of the time when infrastructure and operations teams try to drive a DevOps initiative without nurturing a cultural shift in the first place. It is not just about the efficient tools or experts working; it is about the behavioral modifications and mentality necessary to effect cultural change.
Hence, it is important for the firms to consider the culture of the company before selecting its tool as a potential DevOps tool for their development.
Automation Inside and Outside
Automation is one of the key principles of a strong DevOps team because it has such a high return on investment.
Automation makes up as a large part of DevOps culture to standardize production platforms, and even firms believe it as true because standardization and automation help make deployments more predictable, easy and less prone to human errors.
Automate everything as part of the workflow process, including infrastructures, compliance and continuous delivery pipeline. For example, tools such as Artifactory can manage and control all your binary repositories to help your developers and ops collaborate and automate build and deploy tasks easily. Similarly, Docker allows development teams to containerize applications and automate application deployments with speed.
By reducing manual, non-important and repetitive tasks, you will increase your development and testing speed while cutting down on human and unnecessary errors. Additionally, by automating tedious and repetitive tasks, firms free up team members to work on higher-level and more important tasks.
Important to note here, JFrog Pipelines is an innovative DevOps automation solution that well integrates and connects all critical processes in the software delivery lifecycle in the company, from development to testing to production. Using minimum code, it allows you to create smart and event-driven workflows across tools and teams easily, to speed up the release of software applications. This helps you achieve continuous integration, continuous delivery, continuous deployment and continuous monitoring in DevOps.
Support Both Hybrid and Multi-Cloud
What is Multi-cloud?
Multi-cloud is the use of cloud computing from two or more public cloud providers in single network architecture to meet their business objectives. A multi-cloud strategy has many advantages. It avoids vendor lock-in, decreases latency, enhances business continuity, overhead cost reduction and maximizes flexibility for development teams. However, multi-cloud management comes with some challenges that require careful orchestration, and needs to pay close attention to security and governance issues.
What is Hybrid-cloud?
A hybrid cloud is a mixture of on-premises and off-premises IT resources. Based on business and compliance requirements, the hybrid cloud allows firms to move data and applications between private and public environments.
Hybrid cloud computing allows instant transfer of information between public and private environments, allowing enterprises to experience the benefits of both environments efficiently.
Flexibility, scalability and cost savings are some of the top benefits of hybrid-cloud.
Considering a platform that supports both multi-cloud and hybrid-cloud is the best solution for the enterprises moving toward the digital transformation ladder.
Continuous Updates
Frequently rolling out updates, in a continuous sequence, to bring service improvements to consumer devices more rapidly and efficiently is what every firm is trying to do these days. The customer wants new features as early as possible and continuously. That is one reason why companies embrace automation and follow practices like continuous integration, continuous deployment and continuous delivery through tools such as GitHub, Artifactory, Circle CI, Docker, Kubernetes, etc.
The DevOps platform should be able to manage your DevOps pipeline from a single pane of glass. A platform that can provide a universal end-to-end solution that integrates with your ecosystem to orchestrate and optimize all critical processes in your CI/CD pipeline. Boosting fearless updates continuously from code to the edge in self-managed, on-prem, hybrid and multi-cloud environments.
A Robust Community
All successful companies have built their own communities through their products and services. DevOps is all about the culture shift. Changing existing culture and moving forward is one of the hardest things to do—despite the benefits that DevOps offers, such as speed and agility. No matter what changes a team implements, embracing DevOps often comes to a halt once the companies realize it requires a change in culture and mentality of all the teams involved. This is where the value of community comes in, by fostering the development of a more open communication, easy and supportive feedback, and collaborative culture within teams, companies can be able to drive this DevOps culture. This mindset should come from within and outside of the organization by tapping into this sense of community-based collaboration. Communities are required because they ensure the well being of many of the technologies we use daily.
Your favorite DevOps platforms—Jenkins, Docker, Kubernetes, Artifactory—and their popularity is because of the communities they have created around the tools.
A well-focused DevOps platform gives power to people who listen carefully to what the market is saying, gets the feedback and utilizes it effectively. Hence, community is a very important feature of a DevOps organization.
Conclusion
Companies around the world are embracing DevOps, and the demand for DevOps skills soars as organizations spend their time and money on digital transformation. DevOps has become a modern approach to IT that helps companies to think and execute broader business strategies. DevOps can drive significant brand value by accelerating speed to market, better collaboration, ensuring better testing and automation. A DevOps platform should have all the qualities mentioned above to succeed in this highly competitive market.
Current events have revealed that there are two worlds in the employment landscape. One is digital, the other analog. For the most part, those in IT—the digital workers—have been unaffected by the spike in unemployment due to the economic side-effects of the COVID-19 pandemic. But, it doesn’t mean they’ll be untouched in the long-term. As current events demonstrate, when it comes to the essentials in life, we’re all connected. Yet, the divide will continue to separate us to the detriment of all unless we do something about it.
When observing current events, I can’t help but think about Fritz Lang’s 1927 silent movie, Metropolis. It’s a science fiction film that depicts a world in which a small number of people live a privileged life in skyscrapers that tower high above the city. The rest of the population toils away below ground, doing the mundane work required to keep the city running. It’s a stunning piece of cinematography. The sets and special effects are breathtaking. What’s also impressive is that, for better or worse, so much of what Lang presented nearly 100 years ago has come to be.
You can view the full-length version of Metropolis here.
Figure 1: Scenes from the 1927 silent movie, Metropolis by Fritz Lang.
Lang’s vision of the future has a strange, almost eerie prescience for me, particularly as someone who has spent most of his adult life working in IT. It’s one that’s entirely possible, yet one I hope we avoid.
Allow me to elaborate.
Life Is A-OK in the Cloud
There’s a good argument to be made that today we live in Lang’s Metropolis, a world in which those who are doing really well live in the light of society while those who are just getting by live below ground in darkness.
The widening gap between prosperity and economic hardship is a trend that’s been unfolding over the last 30 years. We’ve become accustomed to it. Sadly, it’s no longer newsworthy.
What is newsworthy is the dramatic spike in unemployment due to the business shutdowns caused by the COVID-19 pandemic. The uptick in unemployment reveals a troubling set of circumstances: If you’re a digital worker chances are you’re still working. If you work in the analog world, that place where factories, restaurants, airlines, movie theaters, hotels, cruise ships and warehouses exist, there’s a good chance you or someone you know is out of a job, for now anyway.
Take a look at the graph from the US Private Sector Job Quality Index, a project from Cornell University Law School and others, by way of vox.com. It lists the jobs that are most vulnerable to layoffs due to the COVID-19 pandemic. IT workers don’t even make the list. It’s as if we live in another world.
Figure 2: Information workers are absent from the list of jobs at risk due to COVID-19.
The long and the short of it is that for developers, sysadmins, SREs and all the other people working in information technology, the recent economic downturn is but a blip in the daily news feed. For those manning a cash register, taking a ticket at the local cineplex, cutting hair or driving for a rideshare service, it’s a life-altering event. Or, to put it terms of Lang’s film, those of us who live in the clouds are doing well, and for those on the ground, it’s a different story.
Now, consider this: What if all the people who are presently laid off from their jobs find out there are no jobs to go back to? What if the economic recovery is slower than anticipated or society gets so accustomed to doing without that consumption doesn’t resume? Is such a scenario possible? Yes. Is it probable? Dunno. Will those of us in IT who work remotely all over the planet, from the comfort of our wired office, suffer? I doubt it. But, as for the rest, what then?
Let’s go back to the movie.
A Workforce Divided
There’s a point in the film when the protagonist, Freder (the son of the industrialist who built Metropolis), goes to his father to inquire about a catastrophic underground explosion shown earlier in the film.
The son asked,“Your magnificent city, Father, and you, the brain of this city, and all of us in this city’s light … and where are the people, Father, whose hands built your city?”
The father responded, “Where they belong.”
The son asked, “Where they belong? In the depths?”
His father nodded his head yes. (See the scene here).
There you have it. With that one physical gesture, Lang foreshadows a future which might very well be at our doorstep. As I mentioned earlier, it’s eerie, it’s prescient and it looks as if it’s coming to be.
What Does This All Have to Do with DevOps?
Those of us in DevOps have made valuable contributions to the world. We’ve done dramatically more good than harm. We’ve also been well compensated. The average salary for a SRE in the USA is $131,340, with a low of $60K and a high of $192K. Yet, the top salary for grocery clerk at Ralph’s supermarket is $22 an hour, which comes out to $44K a year over a 50 week year.
Do SREs deserve such a high salary? Of course, they do. Does a grocery clerk deserve only $44K a year in comparison? At the end of 2019, it would have been easy to say, yes. It’s a low skill, low education job. But that was then, and this is now. As of this writing, toilet paper is a premium commodity and the shelves need to be stocked with it. I’ll wager that in some circles, paying someone $50 an hour to get the rolls on the shelf is reasonable compensation. It all depends on how much people want toilet paper.
Am I being a bit dramatic? A little, but still, it’s worth thinking about.
So, what’s the important take away? Let’s go back to Metropolis one last time.
At the beginning of the film, there’s a scene in which the protagonist, Freder, was frolicking about in a lush arboretum-like greenhouse atop a skyscraper. He was surrounded by pretty maidens dancing. An elevator door opened into the garden. Freder observed as a group of raggedy waifs from beneath the city exited out of the elevator. A woman leading the waifs was at the head of the group. The waifs looked around in amazement at such luxury. Then, Freder’s servants ushered the group back into the elevator to exit. Freder was obviously affected by the waifs. He went into the depths of the city to investigate.
Shortly thereafter, Freder went to see his father to inquire about the explosion I described earlier. His father asked, “What were you doing in the machine halls, Freder?”
The son responded, “I wanted to look into the faces of the people whose little children are my brother, my sisters…”
It’s a powerful scene. It makes me wonder how many of the faces I’ve seen of those who were working in February are now sitting home, unemployed and unable to pay the rent in April.
The takeaway is this: We’re all connected at a level much deeper than the trivial interactions of the given social network. Current events have demonstrated the fact in spades. COVID-19 was first reported in China around November 17, 2019. On April 4, 2020, less than six months later, there were 240 cases reported in the state of South Dakota in the United States. If that’s not connection, I don’t know what is.
Those of us who work in the cloud are doing quite well, despite the setbacks caused by the global pandemic. But, there’s a growing number of us who are not doing so well and won’t be for the foreseeable future. It’s easy when everything is peaches and cream to be unaware of a world in which six-figure salaries are not the norm. But, when things go south and there’s no toilet paper to be had, the realization that we’re all in this together smacks us square in the face, like it or not.
The question then becomes, given that we’re all connected, what type of world do we want to share, the one in Lang’s Metropolis or something better?
It’s something worth thinking about. The pandemic will pass, but the underlying disparities will remain unless we do something about them.
