Linux founder Linus Torvalds, today at the KubeCon + CloudNative + Open Source Summit China conference, warned attendees that managing software is about to become a lot more challenging, largely because of two hardware issues that are beyond the control of DevOps teams.
The first, he said, is the steady stream of patches being generated as new cybersecurity issues related to the speculative execution model that Intel and other processor vendors rely on to accelerate performance. That model is the root cause of malware such as Spectre and Meltdown that have roiled the IT industry. Additional bugs in speculative execution with colorful names such as Fallout and ZombieLoad are showing up more frequently. Each of those bugs requires another patch to the Linux kernel that, depending on when they arrive, can require painful updates to the kernel, Torvalds told conference attendees.
Short of disabling hyperthreading altogether to eliminate reliance on speculative execution, each patch requires organizations to update both the Linux kernel and the BIOS to ensure security. Turning off hyperthreading eliminates the patch management issue, but also reduces application performance by about 15 percent.
The second major issue hardware issue looms a little further over the horizon, Torvalds said. Moore’s Law has guaranteed a doubling of hardware performance every 18 months for decades. But as processor vendors approach the limits of Moore’s Law, many developers will need to reoptimize their code to continue achieving increased performance. In many cases, that requirement will be a shock to many development teams that have counted on those performance improvements to make up for inefficient coding processes, he said.
In the meantime, Torvalds noted updates to the Linux kernel are still coming at a rate of every three months, and the Linux team is basically working on a six-month planning cycle—there is no master five-year plan the Linux team is working from. Roughly 1,500 developers work on contributions to the Linux kernel, with 100 maintainers overseeing the implementation of those contributions.
Naturally, cybersecurity patches at the kernel level have significant implications for all of DevOps. Changes to the kernel need to be absorbed by all the various distributions of Linux, which in turn impacts all the stacks of software that depend on Linux. Jim Zemlin, executive director for The Linux Foundation, said that in the wake of the rise of these hardware issues and previous cybersecurity issues involving open source software such as the Heartbleed vulnerability, cybersecurity is the top priority for The Linux Foundation. As part of that effort, The Linux Foundation is researching various DevSecOps approaches to better securing the global open source supply chain, he said.
In the meantime, organizations large and small alike will need to up their DevSecOps game significantly if they want to continue to push application performance limits.
We are at an inflection point in the rise of big data. Enterprise IT infrastructure is producing data at a scale like the world has never seen before. IT operations (ITOps) management has accelerated past the notion of data storage and processing (vis a vis the Hadoop model) and into the world of analytics-as-a-service. We now need systems to drive insights and manage change on-demand. In fact, we need a new operating model for IT in the face of these new effects.
Can a New System Save Us?
The key to transforming into a responsive, data-driven enterprise—while avoiding a flood of unnecessary processes and over-staffing—is to adapt automation to intelligently consolidate information flows. At the same time, the dynamic nature of today’s IT environment requires IT teams to become literate in a completely new approach to technology, one that spans all functions of the business. Consequently, risks that previously stayed inside organizational units now leak across domains, influencing decision-making for the entire company. These factors will try to pull the organization in multiple directions at once; only a unified data management system with the intelligence to extract operational insights can drive positive business change.
Culture Has Got to Change, Too
Agile processes such as DevOps and AIOps offer mechanisms to control the flow of data and turn it into actionable results. For example, continuous application development and deployment gathers data about functional use, user sentiment, efficiency and various other metrics to inform the overall process and improve the product. Such rapid dev cycles let organizations identify and fix issues more quickly and with fewer resources, ultimately delivering greater business value.
The resulting streamlined collaboration among IT, development and customer teams improves capabilities but also introduces dependencies that must be managed effectively. It changes the culture to adapt to the system, and thereby, the system of service delivery will fundamentally affect how organizations are structured, built and run. Eventually, there will be increased digitization of every human process, and that means human impact slowly will be replaced by AI impact. This doesn’t mean that machines are coming to take our jobs; instead, it means the modern enterprise IT organization must address a new future. How will it look?
Retraining and Reskilling Is a Fact of Life: In a world of dynamic IT, requirements are no longer static, and teams need to be prepared. Organizational units must be able to work in different ways with new technology, with new skills that include robotic emotional intelligence, cloud-native literacy and infinite adaptability.
System operators become stewards of the business because system operations directly impact business outcomes. It won’t be enough for workers to merely deploy, configure and manage IT while isolated from the goals of the company, because those goals are inextricably linked to how objectives are achieved through innovation. Employees will need to develop new skills for functioning in a dynamic system. Training staff in agile methodologies, service-oriented software design and six sigma-style approaches for process improvement will become critical for success.
Automation Is Everything: Automation by policy won’t be enough; even runbooks will become obsolete. Data and results are always changing, so simple automation will not drive meaningful change. AI can be used to drive automation policies and frameworks, with the potential to reduce errors and integrate disparate systems, which previously required multiple points of oversight.
The trend toward using AI for data understanding is driving intelligent services into other parts of the business, as well. Rather than just for correlating system and operational data, AI is being adapted to business process optimization. Moving forward, IT operations can begin to run themselves, led by deep data analytics. Alerting and response processes automatically will use their own feedback to update the system’s intelligence, identify emergent trends and take actions to deliver improved operational results.
Risk Becomes Universal: Increasing connectivity and generalized frameworks (such as DevOps and site reliability engineering) mean that risk is no longer siloed to one department or function. It’s spread everywhere, and cascading effects become commonplace.
In general, successful engineering outcomes rely on coordinated processes and policies that dictate operational needs, even after deployment. Introducing connectivity between formerly disparate systems can create instability that potentially affects operations in other unintended areas. Thus, broad risks arise if different systems can’t interoperate once unified into a single IT ecosystem. Addressing requirements under a generalized framework helps prevent gaps between application implementations.
The System Drags the Company Forward: Insights-driven action will transform digital business well in advance of executive priorities. Change will happen before anyone is ready. Indeed, with so much complexity, so many tools and processes and so many competing business demands, IT changes now impact culture in entirely new ways.
Business impacts will come from system intelligence rather than manual processes. Efficiency in operations management is derived from the entire system, including the people themselves. What was previously the IT culture is rapidly being replaced with a combination of self-adapting processes and workers who focus more on value than on plumbing.
Transforming the enterprise from silos into integrated platforms will propel the business forward. The resulting system will also drive cultural change at an accelerated rate, perhaps faster than the worker community is prepared to accept, but informed by real data instead of executive intuition.
October 2019 will mark the 10th anniversary of the first DevOpsDays, where the term “DevOps” began to catch on. In the world of information technology, 10 years is a long time and it is quite evident based on the amount of change that has transpired since then. To name a few, we have seen the maturing of CI/CD automation and many innovations such as application release automation, containers, cloud infrastructure orchestration, DevSecOps, DevOps database, DevOps governance and micro-services.
These innovations have become part of “mainstream DevOps.” This evolution is continuing with AIOps, machine learning, value stream management and others. Each of these innovations has made DevOps not only more powerful but also more complex.
As a consultant working on DevOps projects through their life cycle, I have learned that complexity can plague the success of DevOps within an organization. Complexity cannot be avoided, as DevOps is complex and will likely continue to be. However, the key to avoiding failure through your DevOps journey is to engage the complexity by using DevOps tenets to implement DevOps. Do not try to boil the ocean. Instead, at each leg in the journey, take inventory of where you are in terms of current goals, state and best practices. Fine-tune your direction and build your solution using proven continuous delivery methods.
The 27-Factor Assessment Model
A comprehensive yet efficient assessment of your current state is key for ensuring complexity has not gotten out of hand and the organization is focused on success criteria that meet the needs of the business.
I use a 27-factor assessment model. The factors are the cross-product of current best practices for three dimensions (people, process and technology) with nine pillars.
A Scope of the 9 Pillars
Leadership: Gain insight and understanding of your leadership practices and how leadership can improve DevOps adoption and performance.
Culture: Identify opportunities to improve cross-functional collaboration and communication across your organization.
Application Development(or Design for DevOps): A practical analysis of application development practices identifies improvements for your DevOps performance.
Continuous Integration: Analysis of branching and software integration workflows and tools identifies bottlenecks and improvements for builds and packaging processes.
Continuous Testing: Analysis of end-to-end software testing, verification workflows and tools identifies bottlenecks and improvements for accelerating testing while improving the coverage and quality of tests.
Infrastructure on Demand (orElastic Infrastructure): Analysis of systems and services for orchestrating and hosting applications and DevOps workloads provides insights to improve performance and utilization of resources.
Continuous Monitoring: Analysis of end-to-end monitoring workflows and tools for applications, releases metrics and infrastructure provides improved visibility for all business and technical stakeholders.
Continuous Security: Analysis of security practices and tools identifies opportunities to integrate security practices into the continuous delivery pipeline.
Continuous Delivery/Deployment: Analysis of delivery and deployment practices identifies improvements for safer, faster and more efficient releases.
Each combination of pillar and dimension has associated best practices used to score relevance and current state relative a DevOps goal agreed by the organization. Here is an example of the continuous security pillar.
The 27-factor assessment for an application is an important input to a future state value-stream map workshop. The value-stream map is then transformed into a road map and implementation plan.
This assessment approach is thorough and yields results that are usually accepted by stakeholders because they are involved in the process at each step. The result is aligned to their priorities and goals and the solution road map is directly traceable to stakeholder inputs. The approach is flexible enough to evolve over time and can be customized to suit the priorities of an organization. For those that argue that 27 factors are too many and too complex, I must say the complexity of DevOps itself requires a comprehensive assessment approach.
Conclusion
In this article, I described a comprehensive, efficient and evolvable assessment approach that uses 27 critical success factors for assessing DevOps applications. While DevOps offers immense value for software development, delivery and deployment, a rigorous and thorough assessment with up-to-date best practices are essential to accomplish DevOps goals. Each organization is different and has different DevOps goals and requirements. The changes to the three dimensions vary according to the results of each DevOps assessment.
Facing challenges is an inevitable part of life, but how we deal with them can make a big difference. When problems show up, it’s important to focus on the things that unite us—it’s better to have a common enemy than fight internally. Experience in complex software development has taught me that it’s crucial to stop, think and act at regular intervals.
The feedback loop is the concept on which the agile mindset is built. Instead of having a project that is one year long, we divide the project into smaller mini-projects of two to four weeks. As the saying goes: There is only one way to eat an elephant–one bite at a time.
To inspect and adapt quickly, I recommend a time frame of just two weeks. During these two-week sprints, the team must work hard to achieve the goal. At the end of each two-week period, the team meets to inspect progress and creates a plan for improvements during the next sprint.
The purpose of the retrospective is to:
Inspect how the last sprint went with regards to people, relationships, process and tools.
Identify and order the major items that went well and potential improvements.
Create a plan for implementing improvements to the way the team performs its work.
These retrospectives can be general or they can be focused on one issue that occurred during a sprint. Sometimes a retrospective is about de-escalating or resolving conflict and tension. If the sprint didn’t go as planned, the facilitator needs to be cautious about how they run the retrospective. They need to keep in mind how they say certain things, because team members might already have low spirits coming into the meeting.
I stay away from using harsh judgments, blaming specific members or showing my frustration. The goal is to create positive energy for the retrospective. Facilitators should motivate the team to have fun while they participate in the retrospective. I like to have high energy, be positive and try to get the others on the same boat.
As a scrum master myself, I was shocked to find out that only 81% of scrum masters hold retrospective meetings. The retrospective is an essential part of any given sprint and provides valuable information on how to improve.
Preparation
Before the retrospective meeting, I think about the format and tools. If you work in a distributed team, like me, a digital whiteboard is recommended. Using the digital whiteboard, we can have a retrospective no matter where we are in the world.
Here is my process for a zombie retrospective meeting. It’s divided into three steps: check-in, input and output.
Figure 1: Zombie Retrospective board.
Step 1: The Check-In (15 Minutes)
The check-in stage is kind of an ice-breaker. It should be fast, like the starter before the main dish—which in this case would be the deeper and more constructive discussion. Check-ins are treated as an exercise at the beginning of the retrospective to gather information on the just-completed sprint. One nice starter for more in-depth discussion is the happiness index: a quick way to check-in with the team and to see how everybody experienced the last sprint.
I ask the question: “On a scale of 1 to 5, specify how happy are you with the last sprint? No.5 is super-happy, and 1 is totally unhappy. Put the sticky note with one keyword next to the number on the whiteboard.”
Figure 2: Icon made by Freepik from www.flaticon.com
The Scale Is:
5 = Super-happy! Don’t want to change anything!
4 = Pretty happy, but there are some things that need to be fixed.
3 = I can live with this, but there are many things that need to be fixed.
2 = Not feeling so good about this right now.
1 = Totally unhappy, I want out.
I give the team three minutes to five minutes to gather the input, then we discuss the issues. It’s a perfect moment to turn the problem into opportunity. You can use solution-focused questions, such as:
What needs to be done to raise your happiness index?”
Let’s imagine we have an ideal world, what would it look like?
How would you like to have it?
Example:
The team members identified that they are moderately happy because they spend a huge amount of time preparing for the release. The work is boring and repetitive. I then turn the problem upside down and ask how we would like to have it. The team then starts with something they want.
Goal: As a team we want to have automatic regression tests, so that we can have faster releases and use our time for value-added work.
Next, we define action items that will help us achieve the goal—we plan to invest time and implement the automated tests. The product owner will be responsible for managing the product backlog in a way that we would achieve the goal within six months.
The scrum master needs to keep in mind which team members are more open to sharing their opinion and those who are more reserved. Shy employees may have lots of great input, but if they don’t feel comfortable sharing it, they will simply sit back and listen. It’s up to the scrum master to understand the team well and know which team members may need an extra push to be more vocal, to collect the input from the colleagues that are more reserved. The first step is to create an inclusive workplace with a team culture of trust, openness and transparency. You want to reach a point where everyone feels like they’re really a part of the team, and know how their efforts contribute to the team’s success. It’s important to remember the importance of tone of voice and body language and the positive influence this can generate.
Step 2:Team Input (45 Minutes)
There are plenty of techniques to collect input from the team during the retrospective. When I prepare the whiteboard for a retrospective, I look into the different techniques and choose the one that I believe is most fitting for the current context. When we face prevalent problems and challenges, I particularly like the zombie retrospective.
Figure 3: “The Zombie Retrospective” by Agata Matynia. Image representing the zombie retrospective used to support the team input stage. This tool helps teams to visualize problems as part of the journey to aid creative solutions.
With this poster, I explain the following to the team: When a virus turns the human race into bloodthirsty mutants, we need to save ourselves from the zombie attack. We need to think clearly, so our discussion would be split into three groups.
The zombies in the illustration represent the problems that we have.
The blocked chair represents the actions we can take to stop the problems affecting us.
The last place is our hiding place. The TNT explosives, gun and hand grenade represent the team strengths.
Once the team has finished contributing their input, we then go through every idea on a case-by-case basis and discuss it. This stage should take a maximum of 45 minutes.
Following discussion, we then move on to the final stage of the retrospective. This step takes all the input and turns them into concrete, measurable goals with action items, responsible people and clear timelines.
Step 3: Team Output (15 Minutes)
The output stage outlines the goals the team would like to achieve the actions they will take, who will do it and when. The tasks are handed out to the team, and the scrum master needs to delegate people to specific areas with deadlines to ensure that things get done. Otherwise, everyone will agree that something needs to be done, but nothing will be changed.
I use the following format to achieve this:
Goal: The end result the team wants to achieve.
Actions: The action points that need to be done to achieve the goal.
Who: The owner of the goal.
When: The specific date.
Example:
Goal: As a developer team member, I want to gather analytics on how the mobile app is used so that it can focus on things that matter.
Action point: Deciding what information to gather and who will do it.
Who: Anna is the owner of this topic.
When: In the next sprint.
The retrospective at the end of a sprint should never be left out. It allows for the continuous improvement of people, relationships, process and tools. It makes the team great.
Optional Step: Inspect and Adapt the Retrospective in the Plus/Delta Format (10 Minutes)
From time to time, it is a good idea to evaluate the retrospective meeting. The goal is to be able to learn from the current retrospective to do it better next time. For this, I recommend using a quick, simple technique such as Plus/Delta to make improvements.
In this final step, the team identifies the strengths of the retrospective and what changes can be made for the next one. I ask the team to use the sticky notes to share their thoughts.
Plus: What can the team change or add to bring more value to the retrospective? How we can repeat it?
Delta: Is there anything that we should change in terms of how we run the retrospective as a team?
The answers to these questions can give scrum masters valuable feedback on how to bring the team together, solve problems more efficiently and make the best use of the time spent in the retrospective. Up until this point, team members have only been discussing the previous sprint, so giving them some time to reflect on the meeting itself sometimes helps them unwind.
More Ideas to Spice Up Your Next Retrospective
The retrospective meeting is a great opportunity to inspect and adapt. If, like me, you would like to run it every two weeks, you will have about 26 of these meetings in a year. The zombie retrospective is great, but you should avoid using the same approach every time. Get as creative as you like—there are many variations and possibilities worth exploring.
For check-in, I also like to utilize pain scales, a common communication tool used in a variety of medical contexts and settings. Trust me, it also works perfectly in business. Try the LEGO pain assessment tool as a starter.
Marketing professionals and developers together are critical in advancing customer experience
More than a decade ago, in the early age of digital transformation, industry analysts and insiders began discussing the need for an increased focus on the collaboration between CTOs, CIOs and CMOs to achieve business goals. It was determined that CMOs would spend more on technology than their technical c-suite counterparts, thus creating an increased need for cross-functional alignment and cooperation.
Today, we find the same kind of alignment is still required, but on an even deeper technical level—between developers and marketers. They are charged with the day-to-day operations of highly scaled and complex marketing operations driven by increasingly complex martech stacks. Companies across industries and markets are becoming more digitally oriented, so the need for seamless collaboration between business functions is mission-critical.
There are many reasons for the perception that marketers and developers share completely different business priorities—from the trope of left brain versus right brain to completely different modes of operation, it has been assumed that marketers are practitioners of soft science and developers are entrenched in code. However, when you consider the nature of today’s modern marketing technology, you realize that both skill sets and sets of expertise are crucial to engaging, retaining and growing a company’s customer base.
To marketers, developers can come across as unapproachable or highly focused internally on technical tasks that distance them from customers. To developers, marketers may seem less technically savvy and unsympathetic to the time requirements in technical projects and the juggling act and trade-offs that ensue for competing priorities. The result is an environment and dynamic in which one party often can feel the other has more control and say over the other. However, what we’re learning is that most of these sentiments are exaggerated or are changing as the business landscape continues to evolve. The truth is, marketers and developers have a lot more in common than they think.
A recent survey of 500 marketers and developers at companies of all sizes uncovered the real truth behind this relationship and understand its inner workings. We looked at identifying their priorities, challenges and where they see success when it comes to their working relationship. Two things in particular stood out:
Business alignment is overwhelmingly the key objective that both parties agree they need to effectively achieve to succeed.
Customer experience is by far the highest priority where the two overlap.
Broken down, 79% of marketers and 83% of developers agree that being aligned on business goals and priorities is a huge factor in being able to succeed in their roles. Developers cultivate this partnership in large part by recognizing that the work of marketers is becoming more data-driven and technical—a fact that 73% of marketers report being true.
Meanwhile, customer success serves as the factor bringing marketing and DevOps closer together. Customer experience represents a critical driver to the success of any business, and on that, both marketers and developers passionately agree. Somewhat surprising, more developers (89%) than marketers (86%) agree with the notion that every touchpoint and interaction with a customer resolutely matters.
The area where marketers and developers face their greatest disconnect is time. Forty-three percent of marketers feel they cannot get requests done promptly, while an overwhelming majority of developers (86%) are confident in their ability to deliver work within a requested time frame. When it comes to the development of new products or software, this is the biggest pain point for both, leaving each side equally frustrated when trying to appease the demands of the other. But with strategic marketing efforts, better collaboration and a more innate understanding of each others’ roles, marketers and developers can drive powerful business results.
Here are just a few simple tips that can foster a more harmonious relationship between marketers and developers that results in real business value:
Marketers should involve developers in software evaluations as early as possible.
Developers should communicate the work required to accomplish a given request and any trade-offs that need to be made to curb marketers’ expectations.
Both parties should reconnect outside of specific projects to frequently share context about their goals, challenges and opportunities.
That last tip is key, as the survey results show marketers and developers are more aligned on those factors than they realize. As the role of marketers continues to evolve in digital organizations and developers take note of this shifting tide, the two have the potential to wade each other through this massive wave of digital disruption. All it takes is walking a mile in each others’ shoes and realizing that both pairs fit.
Gone are the days when a gut-feeling was a major decision-making driver in business. Instinct has ruled business leaders’ mind for a very long time, but as companies expand their business globally, instinct should give way to data as the best measure to remain competitive.
We live in a data-driven world. Today, number-crunching algorithms decide who we should hire, where we should invest and even who we should date. In fact, data-driven organizations that harness insights to create a competitive advantage are growing rapidly—at an average of 30% annually. What’s more, data-driven organizations are on a track to earn $1.8 trillion by 2021.
Data in Businesses
Technology is tucked inside every nook and corner of modern business. Smart leaders use data generated to understand what’s happening in the organization, and this analysis helps them bring agility to their business.
Top companies around the world use data to add value to their business. They gain a strategic advantage over their competitors putting data—not instincts—to good use.
Being data-driven offers multiple advantages:
Data is the new gold. A long-term relationship with data helps organizations set trends.
Data-driven decisions offer a better customer experience. Data reveals customer patterns and their journey with products.
Data-driven organizations respond better to market demands with innovative solutions.
Data-driven decisions hone key business areas such as finance and operations.
Data offers insights and executives turn those insights into key decisions. But you cannot ignore that fact that data is complex, let alone its analysis. We’re all familiar with excel sheets full of thousands of rows and hundreds of columns of numbers. So, how can this turn into insights? That’s where data visualizations come into action.
Data Visualizations Make Hidden Insights Obvious
Interpreting a voluminous dataset for actionable insights is tough. Data visualizations make voluminous data accessible and comprehensive. How?
Data viz, as it is often called, is the transformation of any unstructured data into visuals. Visuals add a component to human understanding.
For example, the World Bank releases its doing business project data every year, providing insights about countries with conditions for establishing businesses. It was launched in 2002 to measure the regulations applied to various small and medium-size organizations.
A total of 190 economies were surveyed on 12 topics including employing workers, tax benefits and resolving insolvency. A visual summary of data on doing business in 2018 was produced to make these crucial insights digestible for businesses, which then could put them to use.
The data science company that created the visualization took the data and plotted it on a world map. It added shades from red to green to indicate the least and the most favored economies for doing business. The data is made easy, appealing and comprehensive. Any business manager or executive can easily identify emerging economies for investment.
Using visual data, insights such as the best and worst country for doing business and economy rankings are now obvious. Furthermore, companies can turn these insights into key decisions regarding expanding business across boundaries.
Take a look at the example above: The United States is an economically powerful country; however, data tells that New Zealand tops the list of economies with ease of doing business in 2018 with 86.55 DTF*. Many countries, including Singapore, Demark, Korean Republic and Hong Kong, also rank favorably in the index.
Three major reasons that data visualization can be crucial for any data-driven organizations are:
Data visualizations amplify a story with pictures and visuals. Its impact on the audience is more powerful than bland numbers and text. Moreover, human minds are hardwired to remember visuals. Data visualizations tap into pictorial strengths to drive executive decision-making.
Data visualizations make difficult data comprehensible. Business executives and key decision-makers are not masters in every field. They need comparative measures to drive their decisions. Data visualizations provide just that.
Data visualizations help in decision analysis. It offers a platform to interact with the facts. Jeff Bezos said that not all the decisions are made by factual analysis. But whichever decisions can be made on facts, should be.
Conclusion
Data should be an influence to any major business decision. And so, data visualization is a powerful tool to bring dramatic changes to organizations. It can provide answers to many key questions, and make the data visually appealing, easy to play with and interesting to explore. CEOs and managers should emphasize the importance of the democratization of their data to add value to their processes.
*DTF: Distance to Frontier. It is a score between zero to one hundred, which illustrates the worst and best performance of an economy observed on each doing business parameter.
Protecting critical data and applications is a challenge under any circumstances, but it’s especially daunting when resources reside in the cloud. Most organizations today operate a significant portion of their workloads in the cloud, which adds to the complexity of the security problem—a security team can’t fully control cloud environments but is responsible for securing workloads and applications running there.
Cybercriminals are exploiting the situation. They’re becoming more aggressive and ingenious in their efforts, taking advantage of the fact that there is confusion about who is responsible for which aspects of security under the Shared Responsibility Model. Adding to the chaos is the differentiation between cloud providers’ security offerings and capabilities. Quite simply, in most cases, cloud providers are responsible for the security of the cloud and consumers are responsible for security in the cloud, meaning that security and networking teams still need to ensure their organizations’ workloads and applications are free from malware or other tampering.
To accomplish this, security and networking teams can’t rely solely on firewalls, threat detection and vulnerability patching to secure vital digital assets. Security tools developed for traditional data centers are not effective at securing workloads in the cloud due to the cloud’s dynamic nature. Security and networking teams must rethink their approach.
Unfortunately, many organizations are still relying on old and ineffective strategies—further hardening the perimeter, investing in additional threat detection and adding new tools and controls that focus on identification rather than prevention. This security approach has its place, but it does nothing to prevent malicious actors from moving laterally across the network once they achieve a foothold, which leaves cloud-hosted apps and workloads extremely vulnerable.
Instead, security teams need to enable zero trust in cloud environments, which treats all internal communications as untrusted by default. With zero trust, only authorized applications and services are allowed to send and receive communications, and only in specific ways governed by the principle of least privilege.
The Second Layer
Traditional tools aren’t effective in enabling zero trust in the cloud because security and ops teams have a very limited ability to manage all layers of the cloud’s open systems interconnection (OSI) model. Specifically, the lack of Layer 2 controls in infrastructure-as-a-service (IaaS) environments makes tools developed for discrete networks all but useless for locking down ports and controlling or scrutinizing IP addresses.
Even if an organization manages to implement these tools in the cloud, it cannot provide the level of granularity required to stop malicious lateral movement. For example, when on-premises tools are repurposed for cloud environments, they usually rely on subnets to define the boundaries of communication. Because cloud subnets are designed to handle elastic workloads, they need to be far larger than those in on-premises environments. As a result, cloud workloads are more exposed than is necessary.
Securely migrating workloads and applications to a cloud environment is a huge and cumbersome job when relying on traditional tooling. The best way to make workloads portable and secure is to decouple workload protection from the underlying infrastructure. In this way, organizations can be certain that any policies applied on-premises can migrate to the untrusted environment of the cloud, without the complexity of mapping changing network addresses, writing policy exceptions or losing fine-grained control during the process.
Decoupling workload protection from the network requires abandoning a network address-based approach in favor of one based on identity. By using the immutable properties of workloads, security teams can create cryptographic identities for applications and services, which then can be used to determine what is allowed to send and receive communications. Not only does this model provide stronger security, but it also works regardless of where applications are located, because these identities are unaffected when network elements change. What’s more, they can be constructed to tolerate upgrades. By building identity-based policies, security teams can create microperimeters around applications that follow workloads wherever they go, and the policies verify communication across boundaries every time an application attempts to communicate.
Implementing network-agnostic policies doesn’t just harden security. Software developers benefit as well. Without the roadblock of translating application-speak to network-speak, developers can build software and applications in any environment favorable to their workflow while defining policies to protect their work. Once the software is built, they can securely deploy the finished product to the production environment—even if it is hosted by a third party.
No organization should have to compromise security to benefit from the efficiencies offered by the cloud and ephemeral auto-scaling containers. Building identity-based policies that enable zero trust and are independent of the underlying network ensures that companies never have to consider making that trade-off.
The immense pressure for enterprises to deliver new applications and insights that exploit new business opportunities has been a driving force for digital transformation. This, in turn, has led IT leaders to embrace a multi-cloud strategy—whether that’s deploying a combination of cloud platforms such as AWS, Azure and Google, or extending private data centers to the public cloud—to achieve greater agility, optimized performance and cost savings resulting from deploying workloads across multiple cloud platforms.
However, maintaining multiple cloud platforms and services adds complexity and even confusion to enterprise IT environments already crowded with a range of technologies, applications and processes. While a recent Forrester study shows that enterprises are increasing investments in using multi-cloud strategies for business-critical applications and workloads, the majority of these enterprises also report that they experienced issues with deploying and using multi-cloud environments.
The difficulty in managing IT environments made more complex by multiple cloud services involves three major issues: ignoring the importance of cultural transformation, not addressing the widening skill set gap and implementing a DIY approach to cloud management.
Here are three ways to deal with these challenges, so you don’t end up sabotaging your multi-cloud strategy.
Make Culture Change a Priority
The cultural transformation required to support digital transformation and to fully take advantage of multi-cloud’s power is more than shifting the organizational mindset from capex to opex. It’s also more than picking one or a combination of the big three cloud providers and simply saying, “Let’s do this.” It requires building trust and breaking down the silos of technology knowledge—whether vendor, legacy or emerging technologies—across disparate IT teams to achieve the business goal of speedier application development and faster time to market.
Specifically, this culture change requires better collaboration between teams and arming central IT with up-to-date processes and management platforms that are focused less on managing infrastructures and more on achieving a balance between visibility and resource control, the linchpin for IT, and creativity, the true north for developers. It will have parameters in place for provisioning resources while helping developers get the resources they need—compute, storage, network services and more—as quickly as possible.
Culture change also enables the maturing of DevOps processes toward streamlining and bringing order to the clutter and smooth partnerships with other IT teams, especially security and operations. These include processes to eliminate bottlenecks in workflow, to advance automation and self-service, to consolidate resources and to make security integral to—rather than an afterthought in—the development process.
Invest in Building Expertise in Multi-Cloud and Emerging Technologies
The more cloud environments an organization has, the more skills and tools are needed to be able to deploy and optimize these environments efficiently. Despite increasing commoditization in the industry, there are still differentiators between cloud vendors and their services, and ways to leverage these to align with an organization’s specific needs. To do this, it is crucial to train the internal DevOps team or hire experts in multi-cloud environments and in key technologies for advancing multi-cloud such as containerization, microservices and serverless computing.
Additionally, a cloud management platform that provides self-service IT and enables any user to provision, manage and orchestrate IT resources, will allow both end users and admins more time to focus on getting high-quality work done.
Select a Cloud Management Platform that’s Right for You
Cloud management platforms address the complexity of hybrid and multi-cloud environments by providing a central platform that allows a selection of the clouds and services that best fit an organization’s needs. They also enable IT to connect to any third-party resource, whether that’s VMware on-prem, AWS in the public cloud or container orchestrators such as Kubernetes, to gather the inventory of cloud computing resources and to provision new resources to be used by anyone in the organization from one place. Self-service IT can also be enabled when appropriate.
In addition, by ensuring cloud management platforms are built on a single pane of glass, IT can gain a holistic view of hybrid and multi-cloud environments. It also allows effective monitoring of resource utilization across those environments, thus ensuring that cloud spend is not only reined in but also optimized.
Finally, a cloud management platform should connect both legacy and new technologies within cloud environments. Although the complexity will not go away, it gets easier to manage. Instead of having multiple teams from different departments managing and accessing resources in separate environments, a central team handles complexity with enterprisewide visibility and control.
The way to succeed is to embrace the complexity by configuring it and managing it behind the scenes, so developers and end users consume IT resources from one consistent way, regardless of where it comes from. Leave the complexity to the cloud and IT architects and let your end users and developers get to work.
In today’s information economy, data is a primary raw material and a source of value to both providers and consumers. For many companies, entire business models are built on the exchange of information.
Consider a ride-sharing business that owns no vehicles of its own. What the company does possess, however, is a database of private vehicle owners who are willing drivers, a roster of people who will gladly pay for a ride and an application that can instantly connect a rider with a driver via an application programming interface (API). The ride-sharing company makes its money by taking a cut of the fare.
Another example is a company that collects and refines financial data and makes it available to investors who might want to buy stock. Companies such as E*TRADE, Merrill Lynch and Fidelity buy that investment data to make it available to their clients. APIs are at the heart of these important data exchanges.
How Prominent are APIs?
The information economy is synonymous with the API economy. APIs are increasingly the way business partners—internal and external—exchange and monetize data and services. APIs allow for machine-to-machine data retrieval, essentially removing barriers and accelerating access to data. There is hardly a modern application today that doesn’t provide an API to integrate with other applications and data sources. The typical mobile app today uses an average of 10 to 15 APIs to move meaningful data into and out of the app. Modern web apps called SPAs (single-page apps) do the same. APIs have proven to be so efficient and useful that many companies have begun creating new business ventures by mobilizing their data to third-party developers and the public.
DevOps teams heartily embrace APIs to advance their ability to produce applications much more quickly. APIs, however, are largely a mystery to security teams that must ensure that this means of rapid data exchange does not create unnecessary risk and negative exposure for the company or its brand. Enterprise security teams are right to question the vulnerabilities and threats posed by APIs. Insecure APIs have led to some very serious data breaches, some of which are outlined in the Dark Reading article, “Expect API Breaches to Accelerate.”
For example, the mobile payment app Venmo exposed sensitive details about hundreds of millions of transactions through a poorly secured API. In another case, an insecure Panera Bread API used by a mobile app exposed users’ private details through a searchable API that required no authentication to access. As a result, as many as 37 million records, which included customer personal information identifiers (PII) such as names, email addresses, physical addresses, birthdays and the last four digits of credit cards—all in plain text—were exposed. T-Mobile, McDonald’s, Symantec, Instagram and numerous other companies have suffered through significant data breaches traced back to insecure APIs. The damage is high because most APIs are designed to serve data without security in mind.
DevOps and security teams can agree that a data breach coming from their applications is completely untenable. Thus, designing and applying a security program that spans across all APIs across all applications is an essential function in protecting data and brand. Just exactly how can this be achieved? Let’s look at the steps of automating an API security program without needing to add additional security staff.
The Challenge of Securing APIs
The very nature of identifying all APIs makes it a challenge to secure them. A developer can create an API in a matter of minutes and publish it on the internet with almost no resistance when using public cloud services such as Amazon, Microsoft and Google. What’s more, she can make changes to that API as often as she wants, and every change cycle presents an opportunity to introduce new risk. It’s not unusual for a developer to change her API weekly or daily.
Let’s go back to that example of the financial services company that sells data to stock brokers. Suppose the developer makes a change to her API which unintentionally enables the brokers to see the client lists of every other broker that consumes data via the API. That would be a catastrophic breach, yet this is precisely the type of API breach suffered by the U.S. Postal Service a year ago.
APIs today increasingly are built on serverless infrastructures such as Amazon Lambda, Azure Functions or Google Cloud Functions. Traditional firewalls, gateways and agents can’t protect this type of API built on ephemeral infrastructure. Now consider that a large enterprise might develop and/or use hundreds or even thousands of APIs. The magnitude of the challenge to secure every API becomes apparent. Manual security oversight and enforcement are simply out of the question. Automation of security assessments done on a continuous basis is the only scalable approach.
Start With an API Security Framework
Every organization that creates or uses APIs needs an API security framework that consists of three steps:
Continuous API discovery and specification creation.
Continuous API specification analysis and inspection.
API policy enablement and enforcement.
Let’s have a look at each step and its impact.
Know What APIs Are in Use and What They Are Intended to Do
Gathering API specifications is an important first step in defining what APIs are intended to do. However, many APIs exist without any specification. A service that creates API specs helps to deal with those APIs void of documentation. However, this spec creation service also needs to continuously monitor and discover new APIs to remedy the problem of not knowing what APIs exist and their intended function. It’s common for a developer to document what her API is designed to do initially, but updating that documentation every time she makes a change to her API is rare. To compensate, the organization needs to have an automated tool that gathers this information and notes each time an API changes, then accordingly updates the API specification.
For example, consider an API that exists to transfer customer information between a company’s back-end server and a mobile application. The developer adds a feature or two and the API changes to support these new features, resulting in the API doing things it wasn’t originally expected to do. How would the organization know about these changes? There is a need to continuously analyze the API for such changes to understand the current state of specifications for what the API is doing.
The organization needs to create specifications of all the APIs, especially publicly facing APIs hosted in the cloud. This service needs to operate continuously to look for changes in the APIs constantly. There are tools that can automate these various activities, including discovering new APIs and changes to existing APIs. These tools also can uncover other cloud resources and data services related to these APIs; enumerate API domains, functions and associated methods; and generate a standards-based (Swagger, OpenAPI v3) spec if one does not exist.
Analyze the APIs for Security Threats
The next step is to perform a security inspection on each API operation every time it changes. The security team needs to know: Does it have the right data encryption? Is proper authentication in place? What type of authorization policy is being applied? What is the API’s level of availability? What kind of data sources does it access? Knowing each API’s current security posture is critical if the organization wants to avoid a data breach.
This API analysis needs to take place continuously and at scale since manual processes are ineffective for organizations using numerous APIs. Fortunately, there are tools and technologies that automate this process and perform tasks such as finding potential vulnerabilities within the authentication and encryption layers of the API. The tools also can generate security tasks with recommended changes for developers to remedy their respective API problems and alert security personnel when there are discrepancies between the API specification and its functional operations.
Create and Enforce Security Policies
No two companies have the same appetite for risk. If there’s one aspect of creating an API security program that requires a manual investment of time, it’s policy creation. Automation then can be used to enforce the policy. A security policy for APIs should start with two primary questions: Who should be able to utilize the API? And what level of sensitivity, regulatory oversight and/or privacy concerns does the API have? Based on the answers to those first two questions, you can form the basis for the security policy.
For example, a bank has an API that provides transaction statements for a specified account. That API has access to extremely sensitive and private information for each of its customers. One customer should never be allowed to access another customer’s account statements. Therefore, the authentication, authorization, encryption and availability of that API would be set to the most restrictive security standards available to ensure data exposure is tightly controlled. Another API from this same bank may provide the current interest rates of savings and money market accounts. This is data the bank likely would share with anyone potentially interested in opening an account with them. In this case, the level of security restriction would be lowered to support a wide variety of systems, browsers, SDKs and apps to ensure this information is highly available.
Historically, API policy enforcement for authentication, encryption and availability was often done at the network gateway layer. As applications have migrated to modern architectures such as mobile and cloud, the gateway approach has increasingly become less practical, less scalable and more restrictive. Application developers are often leveraging authentication, encryption and availability zones provided to them through SDKs and/or cloud services. In these cases, an automated service that integrates with these security services is a better approach for API policy enforcement versus traditional network gateways.
Automation is Key
Security teams are notoriously overworked and short-staffed. Worldwide, there’s a dire shortage of skilled security professionals available for hire, so adding additional security staff to build and execute the API security program isn’t a practical option. Bringing in consultants is expensive and can be budget-busting. Thus, automation through tools and technology is a sensible approach.
Automation provides the benefits of saving time and money when it comes to executing the three steps outlined above. What’s more, an automated program provides consistency in the repetitive tasks necessary to discover, document and analyze the APIs in use, and enforce corporate policies to control risk.
APIs are critically important to modern applications. DevOps people heartily embrace them, and security teams can learn their value as well once they understand how a robust and continuous API security program will benefit their organization.
Enterprises are embracing DevOps underpinned by automation to ensure the quality of software across all digital touchpoints. In fact, automation has become the crux of continuous integration and delivery, the vital DevOps testing goals. Without automation, DevOps would become a distant dream, and businesses would struggle to keep pace with seamless delivery and integration. However, DevOps entails more than the implementation of the latest automation tool and demands a more holistic approach. It does so by bringing together processes, people and technologies under a shared vision. As defined by Microsoft, DevOps is the coalition of products, people and processes to facilitate the seamless delivery of value to the end customers.
DevOps is considered to be the latest advancement of the agile movement. It includes a greater number of stakeholders in the process of delivering value by linking the silo-based teams. It may not be the best practice, but it’s a discipline of people-centric management. Continuous delivery is the technical foundation of DevOps automation, which extends to the agile foundation as well. The close collaboration between the operations and development teams is known as DevOps, which is used to automate the delivery pipeline.
Role of Automation in DevOps to Increase Productivity
Since digital transformation has taken over the current business model, more and more companies are embracing the DevOps practice to drive the automated deployment of software. We are shifting toward an organizational culture in which development and testing take place simultaneously, quickly and more often. This helps maintain consistently high ratings on the quality of software with faster turnaround time. The main agenda of DevOps testing is to derive quality outcomes all along the software development life cycle (SDLC) and beyond. So, with the help of DevOps culture, a new code of development, testing and delivery is introduced in the overall SDLC.
The automation in DevOps empowers companies to:
Enhance the reliability, re-usability and quality of all components.
Save costs of upgrades and maintenance.
Improve the rate of success for standardized processes to ensure the rapid delivery of applications.
Automate Everything — Is It Even Possible?
With the help of the DevOps approach, developers automate and accommodate operational activities, from the source code commit to operational productivity. The primary goal of DevOps automation is to remove bottlenecks in the pipeline by implementing automation in a single level of development. However, DevOps is not a fully automated process and is not operational without human intervention. It is the DevOps testing specialists who are responsible for the process to run flawlessly. Governance at every stage of DevOps testing is still indispensable in various processes undergoing automation. Importantly, human administration is still crucial in handling exceptions to complete the tasks of automation.
The Strategic Role of Humans in DevOps to Ensure a Continuous Pipeline
Remove the Boundaries of Roles
The primary feature of DevOps automation culture is about an enhanced collaboration between the operations and development teams. It has led to a cultural transformation at the organization’s level and inside the teams as well, to encourage continuous delivery. Since shared collaboration is an important aspect of the DevOps culture, it strengthens the collaboration within teams. Before the DevOps approach, the perceived lack of interest by the development team regarding maintenance and operations used to affect DevOps testing and business goals. However, this scenario has since changed due to an organizational shift toward automation and DevOps testing.
The gap between the silos of operations and development no longer exists with the arrival of DevOps. It has removed the boundaries of several organizational roles. Teams are required to focus on the shared goals of DevOps to enhance the end user experience, so the team members need to rethink every investment and decision around this shared goal, which requires human intervention. Automation cannot substitute the decision-making process of humans, for the latter has become vital across the organizational ecosystem as, and when, needed.
Omit One-Size-Fits-All Approach in DevOps Testing
Automation cannot be viewed as the eliminator of roles. The organization still requires expert knowledge and skill set inside your team. The focus should be on developing agile and cross-functional teams that are recognized for removing long-established stumbling blocks between the operations and development teams, and for facilitating collaboration. Developing efficient and functional teams would ensure the delivery of superior quality products.
Continuous Monitoring, Testing and Improvement
One of the key agendas of DevOps testing is to carry out a deeper level of monitoring. Since rigorous monitoring is needed at every layer of the testing process, from integration to infrastructure and transactions to components, it has become a crucial part of the road map of the DevOps approach. In this digitally connected world, monitoring and testing are no longer distinguishable. Continuous monitoring by testers offers a positive response to the user experience while eradicating pre- and post-production obstacles.
Conclusion
With the impetus in the growth of DevOps and agile practices in the past few years, testing has taken a new structure and form. It has evolved into a crucial discipline, and it’s not restricted to a phase of the development cycle anymore. It has become an essential part of the operational activities of production. So, is the role of a tester transformed radically from only a checkpoint to a crucial active participant in the DevOps testing? Teams need to cooperate based on the shared vision of enhancing augmented user experience. We need to understand the fact that automation is not an elixir to deliver on DevOps goals, but merely a part of the DevOps strategy.
