Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.
This week: Okta and Sitel under fire over Lapsus$ hack, Greenpeace and others call for bitcoin change, and Europe still hates encryption. (more…)
DevOps and cloud-native are key for organization looking to accelerate their digital transformation journeys. Techstrong Live! Days, a two-day, in-person event, explores the latest DevOps, cybersecurity and cloud-native technologies and trends that are shaping the way we do business. The first stop of the Techstrong Live! Days conference tour will take place in Washington, DC on April 13-14 at the Convene Event Center. The in-person event offers educational sessions, demos and hands-on training sessions (certification available) led by top industry experts.
DevOps/Cloud-Native Live! is day one of the conference. The agenda for day one features a mix of topics, including API security, Kubernetes, DevSecOps, cloud security, observability and more.
The sessions include keynote presentations, educational talks and hands-on training with leading DevOps and cloud-native tools.
DevOps/Cloud-Native Live! features an outstanding lineup of innovators, practitioners and thought leaders, including:
Cybersecurity Live! is day two of the conference. The agenda for day two features a mix of topics, including AppSec, software supply chain security, SBOM adoption, cloud security, DevSecOps, cybersecurity threats, 5G security and more.
The sessions include keynote presentations, educational talks and hands-on training on forensics/hacking (certification available) and API security. By attending and participating for a full day at Cybersecurity Live!, attendees will receive a certificate in-person or via email granting seven contact hours for the appropriate CEU/CPE Credits.
Cybersecurity Live! features an outstanding lineup of security experts, including:
Attendees will also have the opportunity to attend networking receptions each day to connect with other IT professionals, exchange ideas and learn from one another. Breakfast, lunch and snacks will also be provided to all attendees.
Join us at Techstrong Live! Days in Washington, DC to level up your existing skills, discover new cloud-native and DevOps tools, improve the security of your APIs and network with experts and peers.
For more information on DevOps/Cloud-Native Live! and to register, please visit the DevOps/Cloud-Native Live! website.
To learn more about Cybersecurity Live! and to register, please visit the Cybersecurity Live! website.
Today, most DevOps teams do not think too much about their approach to risk. Usually, the typical attitude focuses on what to add to reduce risk further. In other words, “Can we add more tests? Can we deploy more carefully?” More tests, more processes and less risk are all obviously good things. Or are they?
As a programmer, I’ve seen a lot of projects where there are so many tests and so many processes that they actually become counterproductive. Those ‘overheads’ presumably do reduce risk, but nobody knows by how much. I call this attitude ‘blind risk avoidance.’ This line of avoidant thinking usually builds up over time as we react to failures. It’s a very natural reaction.
But when risk avoidance becomes your standard modus operandi, your delivery cycle gets slower and slower over time. Your team starts to feel they are trapped in molasses but, as time goes on, they not only gradually train themselves to tolerate this but to actively justify that it’s a good thing. In the meantime, nobody pays attention to the fact that your software is losing a competitive edge and that you’re being out-innovated and out-executed by others. Nobody pays enough attention to engineering team attrition; losing people as they drop off, one-by-one to move to more modern software development teams with less baggage. The ultimate conundrum lies here: If you move too fast, you break things. But if you move too slowly, your business and your team are doomed.
So what is the optimal approach for DevOps teams to take when it comes to risk? The truth is, neither extreme is desirable. Blind risk avoidance leads to a failure of value delivery. On the other hand, blindly ignoring risk means crashing and burning.
The ideal balance lies in not avoiding risk entirely or ignoring the reality of risk. Instead, a sensible middle ground can be found in taming risk.
Moving forward, DevOps teams and developers must focus on taming risk to deliver software confidently with data-driven actions, speed and quality.
Taming risk should become a larger part of DevOps culture. One of the most famous ways this approach has been adopted is by Google’s site reliability engineering (SRE) team.
Google’s SRE team embraces two key principles when it comes to taming risk: Automation for reduced operational overhead and structured and rational decision-making for better, data-driven choices.
To explore this mindset of risk-taming at Google, let’s dive into how the SREs use automation to slash waste, save time and reduce repetitious, low-value tasks and tests.
At Google, SREs are software engineers with an extremely low tolerance for repetitive, reactive work. (Aren’t we all?) But here’s the difference at Google: Thanks to a workplace culture of taming risk, operations that do not add value to a service are avoided or automated away.
The vast majority of DevOps teams hold the same beliefs as Google: That automation cuts down on waste and saves everyone time.
We all know automation allows for lower operational overhead, frees up software engineers to proactively assess and improve the services they support and boosts team morale. No one wants to spend their time or energy on repetitive, low-value tasks. Automation frees up DevOps teams to expend their energy on what matters most: Continuous quality.
Bottom line? Automation is integral to the idea of taming risk. Automation of repetitive, low-value tasks is the perfect way to embrace a mindset of taming, not avoiding or ignoring risk entirely.
Another way that Google’s SRE team tames risk is through structured and rational decision-making based on data.
Data-driven decisions are crucial to the success of any software engineering team, project or mission. Without proper data and feedback, there’s no way for software engineers to make the best, most calculated decisions and mitigate risks.
A data-driven decision methodology can—and should—be applied to DevOps testing cycles.
Another key aspect of data-driven decisions that speaks directly to the concept of taming risk? Service level objectives (SLOs).
A service level objective is an element of a service level agreement that measures the performance of the service provider. SLOs are created (and agreed upon) between the service provider and the customer and help to mitigate disputes between providers and customers.
Now, let’s take the concept of SLOs and apply them to engineering and QA teams.
In traditional organizations, engineering and QA are forced to interact intentionally with a healthy amount of tension. One team produces software and the other team tears this software down. While this kind of necessary and healthy conflict between teams can lead to better results, the inherent push and pull between engineering and QA can create an excess of interorganizational infighting.
Engineering teams will always advocate for more velocity or blind ignorance of risk, above all else. On the flip side, QA will always advocate for quality, or blind avoidance of risk, above all else.
Of course, neither side is entirely correct (nor incorrect).
Instead, the solution to this tension lies somewhere in the middle: The taming of risk, not ignorance or avoidance of risk. This is where an SLO, or the numeric target that represents ‘somewhere in the middle,’ comes into play. For engineering teams focused on velocity and QA teams focused on quality, creating an SLO—a happy medium to work toward with regard to risk—will create a more balanced workplace culture. In short, this approach can be considered a taming of risk.
Now, we’re finally going to get to the donuts. To demonstrate how embracing a mindset of taming, not avoiding, risk can create better, happier and faster DevOps teams, let’s take a look at Krispy Kreme Donuts.
Krispy Kreme churns out millions of delicious, hot, glazed donuts every single day. However, during production, it’s inevitable that one or more of those donuts will be flawed. (I know, you’re asking, “Is there really such a thing as a flawed donut?!” For the purposes of this analogy, let’s say yes, there is.)
One common donut defect occurs when the donuts bunch up on the production line and clog it. Now, let’s assume this common error is caused by the speed of the line. If the Krispy Kreme team attempts to solve this problem by pushing the line even faster, tolerance gets smaller and the risk of even more smashed, defective donuts only rises. That’s certainly not a solution to this issue.
On the other hand, if the Krispy Kreme team slows down the donut production line speed, there will be a smoother—albeit slower—operation.
Is Krispy Kreme better served by slowing down its entire production line and sacrificing its velocity to eliminate the risk of a potential defective donut? Of course not!
If that’s not true, then is Krispy Kreme better served by cranking up their production line speed to go as fast as possible, regardless of the risk of defective donuts? Again, of course not!
For sweet success, Krispy Kreme must embrace a mindset of taming risk. By baking careful measurement and an iterative feedback loop into their donut production process, Krispy Kreme is better able to create delicious, high-quality donuts at an optimal speed, producing some flawed donuts. This balance of speed with quality creates the best economic outcome and the mindset and measurement allow for a constant evolution to risk-taming in the future. As the production line changes, the optimal speed also needs to evolve.
DevOps teams should take this same approach. Tame the risk without sacrificing speed and quality. ‘Donut’ slow velocity to fix a few minor issues!
If you’re still not convinced, here’s another way to think about taming risk that is decidedly less delicious than warm, glazed donuts: Taxes.
The IRS balances velocity and accuracy during tax season as they process millions of tax reviews. While suspect returns are flagged for audits, the IRS simply cannot catch every single flawed tax return. It’s not economical or even possible to run detailed audits on every single problematic tax return, every single year.
So, instead, the IRS tames its risk with automation and data-driven decisions. The IRS focuses on creating and using methods to detect patterns that correlate with flawed tax returns. Then, once these data-driven problem returns are detected by automation, experts can devote their precious time and mental energy to tackle the biggest of the problematic returns.
Again, this approach to taming risk creates an inherent balance between quality and speed and allows for an evolution of the process as economic activities and laws evolve.
Whether you love eating donuts or doing your taxes, the bottom line is clear. Most organizations need to balance potential risks with maintaining speed, quality and team morale. DevOps teams are no different.
So how can we, as DevOps teams, tame the risk?
The basic idea here is to do smarter testing, not just more testing. The good news is that if you have a reasonable level of automation, the data coming out of your existing automation can make your testing smarter. Approaches like predictive test selection can let you spend your computer hours more meaningfully, allowing you to improve speed and quality at the same time. Approaches like flaky test insights can let your precious engineering resources focus on the most impactful effort to improve the quality of tests.
What is the right balance between speed and quality? This is the constant question DevOps teams face. But, here’s how I answer this riddle: Taking calculated risk is the most logical and rational approach in the workplace.
It’s easy to avoid risk. Anyone can skirt around potential pitfalls by avoiding new solutions and sticking to the same old methodologies. But it’s so much more satisfying to take a calculated risk rather than ignore or avoid the risk outright.
If you want to improve the velocity and the happiness of your DevOps team, it’s time to embrace a new mindset: Tame the risk, gain control.
One of the biggest headaches IT leaders face today is managing cross-functional teams to develop and deploy high-quality software faster. Not only does their work have to be continuous to stay productive, but there’s the need to save the business money, time and improve end user and customer satisfaction while fixing a seemingly endless parade of bugs.
Without a robust DevOps process in place that benefits the entire organization, IT management and tech leaders often feel like they are chasing the impossible.
DevOps is a set of practices that help you plan how both development (Dev) and operations (Ops) teams can work cohesively to deliver your company’s products and services, meet customer needs and stay ahead of your competitors. In short, it is a set of tools that leverages people, processes and technology to enable safer, faster handling of any IT software delivery initiative.
Yet, according to Gartner, as of the beginning of 2022, 75% of DevOps initiatives have not met these expectations because of inefficient collaboration, lack of buy-in from employees, trying to do too much too quickly and unrealistic expectations. For a methodology that is so critical to businesses today, it’s still surrounded by uncertainty and doubt. Let’s look at how you can make DevOps work for you and why it’s so important that you do so.
Because detecting and addressing software bugs is a major priority of any IT department, disruptions need to be handled quickly and efficiently. This can only be done through immediate action and a tried-and-testing workflow in which all departments involved can act fast. The beauty of DevOps is that the process can be continuous rather than reactive and thus improve the quality of your software when deploying new features, as DevOps allows you to make changes rapidly.
Without DevOps, productivity is slowed or even brought to a halt which leads to an often-damaging customer experience.
A significant benefit of adopting DevOps practices is that it lowers the failure rate of new features while improving recovery time. The continuous deployment, testing and feedback loops ensure faster service delivery and happier customers. The development team can focus on creating superior products by automating the software pipeline while the operations team can improve business delivery.
Faster delivery of your products and services should not be a pipe dream. Those who understand and implement DevOps should feel confident they can stay ahead of the competition.
DevSecOps builds on the DevOps pipeline to integrate security from the very beginning of the development cycle rather than at the deployment stage when flaws and vulnerabilities are more difficult and costlier to fix. Combining development, security and operations across your products and services will improve quality, security and your ability to meet client expectations. Building security into your pipeline seamlessly means bugs and other issues can be addressed more easily and remain almost unnoticed to the end users.
DevOps and DevSecOps make this possible so that outdated security practices are not integrated into your software and don’t interfere with customer demand. So, if increasing customer satisfaction and positively impacting your bottom line by delivering high-quality, secure and effective software is on your business priority list, then implementing a DevOps process can not only save IT headaches but deliver better business outcomes.
Competing in the gaming industry is becoming increasingly costly. To attract new players, millions of dollars are spent in user acquisition (UA), development and cloud infrastructure. As the space becomes more expensive, revenue and cost optimizations can become a matter of survival.
The cost to businesses because of server outages, game response lag or generally poor performance can equate to billions of market cap of gaming companies. The value of observability in modern games is about ensuring that revenue opportunity is not missed in a costly operating landscape. Observability assures that all investment spent on user acquisition is protected and that gaming user experience (UX) is maximized. And user experience depends not only on how the game’s UI is performing but also on the entire gaming environment performance and reliability.
A modern gaming environment is usually highly distributed—a lot of pieces have to come together. The engineering team develops code for the gaming application, which can consist of many microservices, containers, managed cloud services, third-party APIs and serverless functions. Game designers create game levels, the art team produces visual effects and animations and the audio department creates game audio. Once these moving parts are packaged together, the game service has to be deployed on cloud infrastructure and the game itself shipped as executables for consoles, mobile devices and laptops.
All these elements generate massive amounts of observability data, particularly if a game is popular and played by millions of users. Thus, making sure that all gaming elements are working as expected means that gaming companies have to implement efficient ways of collecting and analyzing the massive amounts of that data. That data comes from gaming services, consoles and other sources in real-time. For instance, it is not uncommon to deal with hundreds of terabytes of ingested log data per day. As a result, improving the gaming experience can be quite challenging for gaming companies with large-scale operations. The companies’ success depends on how efficiently they can harness all the data generated by gaming environments.
This is where observability comes in. Observability provides critical insights into what is—or what may be—wrong with the game, all its moving parts and user experience. Powerful observability capabilities are essential, including processing vast amounts of data (logs, metrics, traces, etc.) in real-time, managing fluctuating observability data from variable user loads and running concurrent real-time queries on a massive scale.
So, what, exactly, needs to be measured for a great gaming user experience?
Gaming companies are under continuous financial pressure to deliver new features that attract and retain new users in order to offset mounting operating expenses. Attracting new users is a balancing act from the gaming companies’ perspective. If they move too slowly, they will lose gamers. If they move too quickly, the gaming user experience can deteriorate due to newly introduced bugs. Gamers can be very vocal on social networks, especially if their intent is to express their dissatisfaction with compromised gaming quality. As a result, companies are forced to balance the rate of feature introduction with user satisfaction. That’s why they need to measure user experience continuously.
To improve gaming user experience, gaming organizations must establish measurable key performance indicators (KPIs), which can drive improvements and overall quality. KPIs include, but are not limited to:
Gaming applications are often bursty when it comes to user load, especially during the introduction of new games or new features. One of the critical indicators to monitor is service latency to manage fluctuating user demand during those times.
DevOps teams and SREs need to collect logs at a massive scale to analyze the performance of services, infrastructure and individual host utilization metrics such as high CPU utilization. This information gives engineers clues about the sources of bottlenecks and potential deterioration related to service latency. And logging tools need to be able to deal with this fluctuating demand, too. This level of insight into the state of the gaming application helps engineers troubleshoot issues before gamers face increased service latency that can impact their gaming experience.
Game developers can create custom logs and metrics and combine those with observability data from the gaming engines. Both must be analyzed to assure a high-quality user experience. Some of the common gaming data parameters available for analysis are:
The more data collected, the better the chances to understand any issues that can turn players away from the game. Observability data management should evolve in parallel with game development. As new features are added, the code should be instrumented with KPIs to ensure the quality of the newly introduced code in the long run.
To store time-sensitive gaming data, DevOps teams must use fast databases. The database performance and how fast the game’s properties are retrieved from a database directly impact user satisfaction. Slow database performance negatively affects the gaming user experience. For DevOps teams to avoid any service issues, it is a must to measure key database performance indicators such as response times metrics, out-of-memory error logs and others.
Fast databases could be expensive in the extremely competitive market where gaming companies are always looking for cost optimization to survive in the business. It’s non-trivial for DevOps teams to monitor user experience related to database performance and, at the same time, to look for ways to optimize the cost of fast databases.
For a large-scale video game, new code is often pushed many times per day. Builds can be significant and deployed across many gaming consoles and end devices. Continuous build pipeline and delivery monitoring are crucial for frequent and commonly large software deployments.
With large builds deployed daily over the network to thousands of gaming platforms—where each platform receives a massive amount of data—monitoring of build deployments should be automated. Errors from log data should trigger alerts and drive redeployments. QA automation is necessary to reduce the burden of QA efforts and make build deliveries less error-prone and less tedious.
Insights into user behavior are as important as monitoring possible issues affecting user experience. Observability tools can provide key insights into the overall engagement of all game users. High engagement then drives more excitement. Higher user excitement may increase the success of ad campaigns and accelerate in-game purchases.
When thinking about observability tools that can be instrumental in driving continuous improvement of user experience in the gaming industry, the following functionalities are important:
Observability is key to protecting gaming revenue opportunities and assuring the success of video games, as long as you know what to look for in observability tools that are used to gather data from gaming environments.
For cloud architects, a deep understanding of cloud tools and technologies is only the starting point in terms of skills that need to be mastered. Enjoying a successful career as a cloud architect requires much more, including certain skills that are typically not associated with the tech industry.
The cloud architect needs to be an expert on end-to-end system design since their primary responsibility is solving a customer’s problems through the use of technology. The architect’s ability to develop a design that solves a business’s problem is where their true value lies. Because design is architecture, the cloud architect must have expert knowledge of all the current and legacy systems that exist in the network and in the data center and also understand how to migrate them to the cloud, matching them to the proper cloud-based systems.
Networking and data center skills are critical for the cloud architect because one of their major responsibilities is designing a plan to take the systems that were on the network and hosted in a data center and migrate them to the cloud. To do this successfully, cloud architects must be experts on networking and data center technologies.
Digging deeper, they must be experts at routers, load balancers, switches, routing protocols, IP addressing and WAN connections whether those are private line-based, VPN-based or software-defined networking. They must be experts on servers and server virtualization and containers and storage, too: Block storage, object storage and file storage. The cloud architect also needs to be an expert at file-sharing protocols such as Server Message Block for Windows and NFS for Linux. Security, too, is an important skill; they must understand how firewalls, IDS/IPS systems and how VPN concentrators work. The cloud architect needs to have full knowledge of databases, understand how web applications work and understand ERP applications and supply chain applications. Good, strong network and data center knowledge is the foundation for architects who are overseeing the migration of tech from the data center to the cloud.
Cloud architects are digital transformation specialists. They must craft end-to-end business solutions for customers with the goal of improving their businesses. If they do not have a thorough understanding of customers’ business, they cannot effectively engineer solutions that will solve their business problems.
Business skills include the ability to calculate an ROI model based upon the expected value of an opportunity, to read a financial statement and to read a balance sheet. It includes knowledge and understanding of what the CEO, CFO, CIO and CTO care about and being able to communicate with them in ways they understand. Without an understanding of business, cloud architects limit themselves to simply adding or removing technology, which might make business outcomes better or might make them worse.
Accomplishing anything great requires a team, which means being a great cloud architect requires leadership skills. When working for a client, a cloud architect typically will need to set a baseline of current systems. Accomplishing that effectively might require a team of 50 cloud engineers working together to evaluate the customer’s systems. Leadership skills allow you to effectively manage that team of engineers.
And leadership skills go beyond effectively managing people as they work on proofs-of-concept and deployments. Leadership skills empower the architect to make sure teams are motivated, are fully engaged with projects and are contributing to success.
When on the job, a cloud architect is constantly selling. The solutions they designed need to be sold to the customer. The plan to implement the solution needs to be sold to management before the necessary resources can be obtained. When it is time to get people to work with us on a project, we need to sell them on its value. Sales skills are employed in a wide variety of ways. Cloud architects sell to customers, internal stakeholders and to management.
Cloud architects also need exceptional presentation skills because they will be presenting to customers. Those customers could be technology professionals like engineers; they could be presenting to management and/or to C-level executives. Additionally, cloud architects will be presenting information at executive briefings and at conferences. In general, presentations are part of everyday life for the average cloud architect.
Determining the business requirements for any cloud architecture project requires asking a lot of questions. They’re never just handed a piece of paper by a client that says, “Build this.” Without stellar communication skills, it’s difficult—if not impossible—to determine the business’ requirements, meaning the solutions that are designed could be incorrect.
Even once the business requirements are well understood, the next step is to take those requirements to a team of technology professionals, which means communicating effectively with tech people as well as business people. Communication skills, which include communicating to customers, teammates and various other stakeholders in the organization, are critical.
Emotional intelligence is a critical skill for those who want to increase their professional value and propel their cloud architect career forward. Emotional intelligence increases the capacity for leadership, understanding and communication. When used regularly in the job, it is the best predictor of career success and typically results in about a $30,000 increase in the average person’s salary. For a cloud architect to have a successful career, building on the skills outlined above is critical.
Harness this week announced it acquired ChaosNative as part of a plan to more deeply integrate chaos engineering with DevOps workflows.
At the same time, the company added a Harness Service Reliability Management (SRM) and Harness Security Testing Orchestration capability to its continuous integration/continuous delivery (CI/CD) platform.
Chaos engineering, as a discipline, refers to experiments on a software system that test its resiliency in the face of an unplanned outage. The goal is to deliberately make resources unavailable to see how the application environment either adjusts or whether it fails.
ChaosNative’s LitmusChaos platform, which manages these chaos engineering tasks, was donated to the Cloud Native Computing Foundation earlier this year.
Scott Sanchez, chief marketing officer for Harness. said the company plans to shift chaos engineering further left as part of an effort to enable developers to test the resiliency of their applications before they are deployed in a production environment.
It’s still early days where adoption of chaos engineering within DevOps workflows is concerned. Many IT leaders’ performance is measured based on the stability of application environments. The idea that they should deliberately break something as part of an effort to make an application more resilient requires a shift in mindset, noted Sanchez.
However, once IT leaders start to appreciate how chaos engineering actually improves the availability of applications it’s only a matter of time before the practice becomes more widely implemented, he added.
In the meantime, Harness is looking to automate more of the lower-level tasks that today’s DevOps teams often manually manage. Harness SRM enables DevOps teams to automatically track service level objectives (SLOs) that are attached to specific pipelines within the Harness CI/CD platform.
The Harness STO module, meanwhile, makes it simpler to orchestrate security testing across multiple pipelines.
In general, there is now more focus on the productivity of DevOps teams as the number of applications that are being built simultaneously increases. The challenge, however, is not just the number of applications but also the dependencies that exist between them.
Harness is making a case for a CI/CD platform that automates DevOps tasks by employing, for example, machine learning algorithms to identify which applications are most likely to fail before a build is completed. The goal is to reduce the amount of time DevOps teams waste waiting for builds to complete only to discover that the application almost immediately failed.
The time, effort and cost of maintaining a DevOps platform will naturally vary as the overall environment becomes more complex. However, as DevOps processes become more automated, the platforms themselves become more accessible to a wider range of organizations. Today, the effort required by a DevOps to set up and then support a CI/CD environment is often considerable.
Nevertheless, as more organizations rely on custom applications to drive digital business processes, it’s only a question of when and to what degree DevOps best practices will be more widely adopted. After all, the total cost of application downtime is only going to rise in the age of digital transformation.
In recent years, our reliance on software has exploded. It has been more than a decade since Marc Andreessen made his prophetic observation that “Software is eating the world.” Today, nearly every interaction is mediated through a screen and backed by software. With the massive shift to remote and digital interactions due to the pandemic, enterprises that were already in the process of acclimating to a digital-first future had to quickly pick up the pace to survive today’s increasingly complex tech landscape, putting an inordinate strain on IT teams.
Developers obviously deliver the most crucial component of the software development life cycle; however, ownership of their developed services and applications is no less important. The idea of ‘full-service ownership’ means taking responsibility for supporting the software and services you deliver at every stage of their life cycle. Excellence in continuous delivery and innovation on them is about speed and velocity while maintaining quality.
At its core, service ownership is about connecting humans to technologies and services and understanding how they map to critical business outcomes. Achieving this level of ownership requires an understanding of what and who delivers critical business services. A clear understanding of what the boundaries and dependencies are of a given service along with what value it delivers is the starting point. And once it’s in production, a clear definition of who is responsible for it at any given time and what its impact is if it isn’t running optimally or, worst case, fails altogether. Empowering developers with this information brings DevOps teams much closer to their customers, the business and the value they create which, in turn, leads to better application design and development.
Building a culture around a full-service ownership model keeps developers closely tied to the applications they build and, therefore, closer to the value they deliver. Within the organization, this type of ownership breaks down long-established centralized and siloed teams into cross-functional full-service teams. For enterprises with deeply ingrained processes, this may be easier said than done—but the outcome is well worth the investment. Following these guidelines will allow even legacy IT teams to create a culture of full-service ownership.
With the pressure to innovate already weighing heavily on software teams, you may be wary about the burden more responsibility might cause. However, the end goal is mutually beneficial: Create clear ownership at real-time speeds. Not only does this allow engineering teams to ship features more quickly but it also empowers them to stay engaged and connected when something goes wrong—reducing hand-offs and mean-time-to-resolve (MTTR). In other words, no panic, no uncertainty and minimal customer impact.
Reworking team structures and technology for full-service ownership may seem overwhelming, but it doesn’t need to be. Traditional function-based silos are a feature of many legacy IT departments. Provisioning infrastructure and resources, developing code, quality assurance, owning the customer relationship, supporting software in production and so on are too often delegated across segmented, centralized teams. Allowing decentralized teams to own the entire life cycle of their services affords them the ability to add value in new ways. The key is to start small and implement changes gradually.
When beginning the process of full-service ownership adoption, pointing to small wins will gain the support of key organizational stakeholders who can then facilitate larger-scale change. Start with non-critical production systems or greenfield applications. Agile methodologies will also be useful in adapting to dynamic development needs and unpredictable events. Remember to be flexible with the timeline in the initial phases and be patient with your team—mastering the life cycle takes time!
Keeping long-term goals in mind, your teams should consider not only “build and deploy” but also “run.” Craft your tech stack in a way that comprehensively shows the entire infrastructure, leverages machine learning to reduce false alarms, and uses automation to conserve the manpower and resources involved in going on-call to support services in production.
A key requirement here is building trust and an accurate system for identifying who is responsible when something goes wrong. Adjusting to a DevOps approach will fail if developers receive false alarms, too many interruptions or are constantly escalated to repetitive fixes. Building and executing on a strong service directory and supporting their preferred schedules is critical. As is ensuring when they are needed, you contact them via the channels they expect. Failing to do this can result in minutes and hours lost and worse yet, frustrated and burnt-out developers. Recent data shows that developers who get called in to support services too often or incorrectly are more likely to leave their jobs, and that’s only going to stall your progress further.
With an accurate service directory in place, it is important to remove the false alarms. Teams must make sure “alerts and alarms” are correlated and an incident is confirmed before you interrupt your valuable time-strapped developers. This involves the right observability and monitoring tools but must also involve strong centralized machine learning augmentation to reduce noise and false alarms. As teams mature, automation plays a critical role here. First, by cutting out the common diagnosis and investigation steps that can burn minutes or even hours. Then, reducing escalations to developers (by over 50%) by putting common resolutions in the hands of first responders or even customer service teams. These collections of technologies are commonly referred to as AIOps and as digital business scales, they become a foundational requirement.
Today’s digital ecosystem is more advanced and more complex than ever before. Maintaining quality digital services is not just a competitive differentiator, it’s the backbone of your corporate reputation. Recent high-profile outages and cyberattacks have catapulted the fragility of the software supply chain into the international spotlight, meaning the margin for error is slim. Resolving issues proactively and in real-time before they impact customers is critical.
Transparent accountability and decentralized ownership ensure that teams have a deeper knowledge of issues when trouble strikes and can take swift action. Revealing to technical teams and developers the direct impact they have on business outcomes enables high-quality, innovative code that delivers on its promises.
When employee turnover spikes, IT in general and DevOps teams in particular need to be prepared.
At the time of this writing, the stock market is roiling, and DevOps compensation is bouncing up. This has an impact on turnover because many in DevOps (and nearly all senior positions) have compensation that includes stock—stock that may be down significantly or that is currently at high risk of going down. This causes two types of movement—those leaving an organization to avoid further loss in their compensation and those who are leaving their current gig to join an organization whose stock is down. Combined with those who leave for a role with a larger base salary, there are those who are leaving to earn more and those who are leaving to avoid more loss. IT budgets ballooned during the pandemic but those, too, are deflating, so while we have a worker, not a position shortage, there will be pressure from both sides.
So, why is this geek going on about employment, compensation and available positions?
Simple: Because DevOps is not any better at documentation than any other flavor of IT. In tightly-run shops, documentation is available and maintained. In most shops, it is spotty at best. In some, it is downright nonexistent. This has forever been true in IT and, if anything, it is more true because the early stages of DevOps implementation result in a massive amount of change.
People leaving in an environment with less-than-stellar documentation can be a problem. What tools are used? Where are they located? What’s the login for these SaaS tools? Etc., etc., etc. If turnover means you’re losing entire DevOps teams in relatively short order, this can become a nightmare relatively quickly. And that nightmare is very real for everyone left behind and those that are coming in. Someone has to go figure it out; IT management has to let the rest of the business know that retention issues are causing backlogs. Everyone not tasked with figuring it out has to pick up slack for those who are … you get the picture.
Take the hour or four and document what you have. It doesn’t have to be comprehensive, it just has to list what tools, where they are, who at the organization is responsible for credentials at any SaaS vendors and under what circumstances the tools are used. If each team documents their own stuff, even in a shared toolchain environment, comparing the documents will uncover little details one missed that another did not.
Let’s face it, we have—for better or worse—moved largely to a world where loyalty is based on compensation. Much like mercenaries, most IT employees are always listening for a better offer. So the organization, while acknowledging this fact, needs to take steps to minimize the negative impacts of turnover. New ideas and suggestions to change things are all good and you don’t want to interfere with that aspect of turnover. But anything that impedes the organization’s ability to deliver should be addressed beforehand. And that means documentation so remaining and new employees can get up to speed quickly and take ownership rapidly and with confidence.
And keep rocking it. Whether you are looking for a new gig or not, keep the apps you are responsible for running—and document them. You’ve contributed to the success of your organization; don’t slack off while moving along. And don’t hate those that are moving along. Not all of us base our loyalty on compensation, but we do all allow compensation to influence our decisions (unless you are working for free and I missed it?) Keep rocking, no matter your status. And don’t hesitate to tell interviewers that you helped your last organization achieve success.
Brian Lavallée from Ciena talks with Mike Vizard about how the work-from-anywhere movement has impacted DevOps, networking and security. The video is below, followed by a transcript of the conversation.
Mike Vizard: Hey, guys. Thanks for the throw. We’re here with Brian Lavallée who is senior director for solutions marketing at Ciena. Brian, welcome to the show.
Brian Lavallée: Thanks. Nice to be here.
Mike Vizard: You guys just put out this study talking about what’s happening on the technology impact from working from home. A lot of folks are accessing cloud resources. They have home networks that may or may not be up to the task. Are we looking at some sort of networking crisis coming up as we move to working from anywhere? Do people understand that bandwidth is going to be an issue?
Brian Lavallée: I wouldn’t say it’s a crisis, but it is a work in progress. Just over the last year, year and a half, the global pandemic, our homes have turned into the new branch office, the new classroom. People are learning, playing, and working from home, and it’s put additional stress on the network. One of the main challenges is speeds, because now you have two, three, four people accessing the same broadband connection from home at the same time, and then you have the whole security aspect, where if your home is the new branch office, very often you need additional security to do work, not only with your head office, your colleagues, but also with your cloud applications.
Mike Vizard: I have found myself trying to fight for bandwidth with an 11-year-old, so I know what the issues are and what the problems can be. Do you think companies, as we go forward, are going to turn around and say to their employees, “Here is the router we want you to use”? Are they going to give them that router, or the access point at home? How far will they go to make all this happen and maybe provide that additional layer of security?
Brian Lavallée: Yeah, I think, for sure, adding additional security, even availability, a more resilient connection is necessary going forward. Companies or enterprises are going to implement this differently. Some may do a lot of their security capabilities in the cloud. This allows you to have a centralized location to do all your security, you know, talking about SASE, _____ connectivity, and because it’s centralized, it’s easier to implement and control, rather than having specialized devices or specialized software sitting on everybody’s laptop. If you’re a large enterprise, you can have thousands to many thousands of people working from home, so centralizing your security aspect with a high-speed connection is probably the way moving forward.
Mike Vizard: What impact does that have back on the core networks? I mean, if everybody has suddenly got more bandwidth at the edge, does that have a cascading effect up through the rest of the network infrastructure? Do I need additional core routers or networks? How will this play out in your mind?
Brian Lavallée: Yeah, _____. Content is created and consumed at the edge, and that’s from humans or machines ‒ think sensors, cameras, and so on. Definitely, as we open up the access ramps, 5G is a way of opening up the wireless access ramp, plus you have a variety of wireline broadband technologies for the home, even the offices. For sure, it has a knock-on effect into the core network because, if you think about it, most of the content and applications that you access, whether it’s on your smart phone, your laptop, tablet, or whatever device you’re using, it’s sitting on a datacenter somewhere. That could be a datacenter close to you, an edge datacenter maybe sitting in the same city as you. It could be a datacenter on the other side of the world sitting in Norway or Sweden or Finland.
Mike Vizard: I’m feeling like a lot of the applications that people are starting to build, consume data and analyze data at the point at where it’s created more, it’s more happening out at the edge. Is there a lot more sensitivity around latency? Will the laws of networking physics come back and bite us soon, and are we seeing new classes of applications?
Brian Lavallée: Yeah, and that’s a good question because it leans into the topic of discussion, one of the hotter ones today, which is edge computing or edge cloud. By putting storage and compute access closer to the end-users ‒ again, those end-users could be humans or machines ‒ it does inherently reduce the latency only because you have a shorter physical distance to travel. But, we have to remember latency is not just about distance, it’s about the amount of network nodes that are sitting between you and your content, so the more network devices you go through ‒ switches, routers, and so on ‒ the more latency is incurred and accumulated. Pushing the storage and compute closer to the edge with high-speed connections is a very good way of reducing latency, and it also opens up a whole bunch of new applications where latency is a critical factor beyond just increased speed.
Mike Vizard: I feel like right after the pandemic, we sent everybody home, we gave them their little wireless network, we hooked them up with a VPN, and they accessed the applications in the office. Then, they went out to the cloud and they had this kind of backhauling effect of the network traffic flow. You mentioned SASE. Are we going to change the way we think about how applications are deployed and how we access them, whether they’re up in the cloud? Is the routing concept that we’ve known all these years changing? How do you think we’re going to go forward from here?
Brian Lavallée: Yeah, so I think the increased adoption of cloud-based services is an unstoppable juggernaut. It is happening. There are security concerns. Our survey that we did, the study, we did show that there was a security concern of moving more and more into the cloud, but the cloud providers, the telecom operators, they’ve done a lot of excellent work over the last few years to address security, make increased security for their end-users. The cloud-based service providers, they realize that security is top of mind above everything else, so I think that will continue going forward. As more and more end-users are accessing cloud-based applications and content, the network becomes a critical factor because the network now is sitting in between you and the content you’re trying to access.
Mike Vizard: Do you think the bad guys have figured all this out and they’re targeting people at the end point now and looking at these slightly misconfigured networks? Sometimes we’re using fraud RDP protocols. Are they getting it? Are they after us?
Brian Lavallée: They’ll always be there, nefarious actors, bad actors, whatever you want to call them, so they will always be there, but the cloud providers, the telecom operators, they know this, so they are consistently upgrading, modernizing, and putting increased security capabilities into their network. It’s always going to be a battle of people who want to do not so good things with the network and content, and the people that actually run those datacenters and networks. I think that will continue going forward, it’s kind of human nature, but we’ve made great strides as an industry, and that’s across IT, telecom, and cloud-based services in terms of security. With more and more people from home, I mean right now we’re living in probably the largest working-from-home experiment of our species, and, if you think about it, the network, it has bent in a few places, right?
Most of the networks are running hotter, but it hasn’t broken. The Internet has not broken. A lot of people early on thought that if everybody moves home, it’s not that you have more capacity coming to the home, but the traffic patterns have changed. During the day, when a lot of people went to school and work, the traffic was to the schools and the head offices, but now the whole traffic pattern has moved into the home. Although a lot of people were concerned, I think the telecom operators deserve a lot of credit. The network has bent in a few places, but it hasn’t broken, and they’ve just learned a lot over the last year, and the networks will just be that much more reliable going forward.
Mike Vizard: Prior to the pandemic, the carriers were trying to encourage people to offload from the network and use WiFi. Now we have 5G, and people are talking about 6G and who knows what comes after that. Are we getting to a point where we can just run 5G into the home? Will that get cost-effective at some point and we won’t have all these WiFi routers, or will it always be some sort of a balance between 5G, 6G and WiFi and we have to figure it out?
Brian Lavallée: Yeah, I mean my personal opinion on that, I think it will always be a mix of technologies. I’ve been in this industry long enough to know that a technology network completely disappears, at least not in my career. It always has places of use. The big difference with 5G over 4G LTE and previous technologies, and even WiFi, previous generations of mobile technologies, it brings a lot of new benefits to the table, right? If we look at the faster speeds, this is what gets the most press coverage because that’s what people will see right away, but it also brings much lower latency. We’re talking order of magnitudes lower latency. It connects to a wider variety of connected devices, such as sensors and smart devices, and then we have this whole concept of network slicing, which can guarantee end-to-end performance.
You can guarantee the speed, latency, and resilience. We haven’t had that in the past with any mobile network generation, and, to a large extent, we haven’t even had that on the wireline technologies. So, 5G coming right into your home, if you think about it, if you’re a network operator, it’s much easier just to send a 5G hotspot, it could be a router, residential gateway to somebody’s home, they turn it on, they plug it in the wall, and it discovers the mobile network and then BOOM, you’ve got one-gigabit-per-second capacity right to your home. That is, I think, the way of the future in many cases, but there are some technical challenges that still need to be overcome. But, most operators are looking at 5G fixed wireless access to the residential, but even businesses as something in their toolkit going forward to compete with wireline technologies.
Mike Vizard: Do you think there is a rebellion coming among the end-users who are using this stuff? When you hear them, they’re kind of complaining a lot about the experience, so do you think that at some point they’re going to start picking what companies they want to work for based on what kind of work-from-home, work-from-anywhere experience they can get? Will that factor into hiring and retention more than it has in the past?
Brian Lavallée: Yeah. I mean, the network is critical infrastructure today, especially for the younger generation who were born and raised on having always-on connectivity. I think that will continue going forward. But, as more people work from home, I know we hear a lot of talk about the new normal, what ire the workplace, for instance, or the school systems going to look like after the pandemic subsides? We don’t really know. But, what I think we do know is more people will be working from home. I see it in my personal life, I see it with my colleagues, I see it with friends and family. Their companies are looking at implementing working from home at least a couple of days a week. There are a lot of benefits to the company to do this, as well, so the network will become that much more important.
I think people will shop around if the provider they have, the network provider, is not giving them the performance that they need, and it will become a highly competitive marketplace. I mean, it is pretty competitive now, but I think it will be that much more so going forward in terms of speed, latency, and availability, the three main things that you need with a high-speed connection to your home.
Mike Vizard: I might even one day factor networking costs into my compensation packet, but we’ll see what happens.
Brian Lavallée: There you go.
Mike Vizard: What is your best advice to people who are on the IT side of this equation, they’re the network managers, they’re security people. What is the conversation that they should be having higher up the food chain to the board of directors about what needs to happen to really enable this next wave of work?
Brian Lavallée: Yeah, I mean, the higher speed to the home is something that a lot of people talk about. We just talked about it earlier, more people working, playing, and learning from home, so capacity will always be at the forefront of discussions and having competitiveness for an operator with another operator. But, I think increased security features, this could be wrapped around a broadband connection to your home, so rather than just buying a high-speed pipe to your service provider, they would also provide you with SASE type of capabilities, so security, firewalls, and so on. I think looking at bundling, IT services with the connection is definitely the way to go. It is a differentiator, and it is what people will be looking for because if you’re the average home user, whether it’s for playing, learning, or working, you really don’t care what technology you’re buying, as long as it’s cost-effective and it meets your needs.
I think my advice for the operators is to look at what is available recently, some of the newer technologies ‒ virtualization and SASE technologies and the SD _____, a whole bunch of other technologies ‒ and start to wrap these into a bundle that you solve most, if not all of the needs of the subscriber, and then you could differentiate far above and beyond just a higher speed.
Mike Vizard: All right, guys, you heard it here. Faster things that are insecure do not necessarily represent progress. Hey, Brian, thanks for being on the show.
Brian Lavallée: Happy to be here. Thank you.
Mike Vizard: All right, guys, back to you in the studio.
[End of Audio]