devopsy.kinsta.cloud

Category: Enterprise DevOps

Enable Multi-Cloud Networking With Automation
As we enter a new era of technology, we’re experiencing a “third wave” of cloud innovation that brings new complexities into the IT landscape, mainly centered around multi-cloud networking. With every wave of cloud technology advancement, networking components are becoming more complex and distributed across multiple groups. Further, multi-cloud networking brings new challenges around connecting and managing across dispersed environments comprising private and public clouds, along with physical data centers. Organizations must unify their network operations in a way that enhances their capabilities into a new level of skill and efficiency.

Augment Strategies

Looking forward to new capabilities offered by automation tools, technology leaders can augment their 2022 IT strategies. How?
- Support a more seamless transition to cloud within hybrid IT infrastructures.
- Cultivate new values brought to an organization from network engineers.
- Enable executive leaders to easily align operations across multiple teams – spreading innovation across the entire organization.
Connecting Hybrid IT Infrastructures

As more IT leaders invest in an assortment of digital tools to streamline network operations – some engineering teams might feel more reluctant to leave behind old processes or feel intimidated by the multitude of new capabilities at their fingertips. With modern tools and processes gradually being incorporated into the workflow of IT departments, it’s critical to find the right balance in this “hybrid” environment, which will include a combination of both traditional on-premises systems with new cloud applications. Automation can help smoothen the transition to new technologies by simplifying manual processes previously relied on, while also bridging together the plethora of systems for easier management.

Leveraging Cloud-native automation

When bridging together disparate systems, leveraging cloud-native automation solutions that provide API capabilities can make all the difference. These solutions not only build the appropriate connectivity required between different applications but also provide the flexibility to manage networks in a cloud-friendly way. Additionally, automation can be deployed to ensure compliance of device configurations that span both on-premises network devices, as well as those connected to cloud-based infrastructures. For example, with automation it’s possible to quickly perform a configuration back-up, check for compliance and execute “intelligent remediation”. This translates to a network device or group of devices that can be backed up and the current configuration compared against the existing golden configuration standard. Changes can be automated to ensure the network is adhering to the current compliance standard. Further, automation is changing the way that IT teams approach daily operational challenges.

Evolving Value in Network Teams

Network engineers play a unique role in organizations by performing tasks that require years of training, and an understanding of the complex ins and outs of their internal systems. Although the performance of these manual tasks might be assumed to be the value-driver for engineers – the adoption of new technologies has pushed the boundaries of what network professionals are familiar with. In turn, this has evolved expectations for what constitutes true success and impact. Organizations are looking for engineers to harness the power of these new technologies and execute on a modern strategy – utilizing automation to support this new mindset.

Streamline the Adoption of Integrated Multi-cloud Environments

Breaking down the silos between technologies and teams is critical in order to understand how to get the most out of their investments. For network teams to fully embrace multi-cloud networking, they need to practice deeper management, segmentation, security and control of cloud networks. As with segmentation – automation can help network teams lean away from using overlays, and instead use the native software tools in AWS, Azure, etc. Additionally, many network teams might be under the false misconception that automation requires them to learn to program and write code – when in many new tools, network elements can be managed and orchestrated directly from within the code of an application. This will make network programmability part of the cloud software fabric, and streamline the adoption of integrated multi-cloud environments. Network teams can implement both compliance and automation to standardize both on-prem or new cloud programs, simultaneously building their automation skills without the need to write a single line of code.

Aligning Operational Efficiency

When investing in new IT infrastructures across organizations, IT leaders typically build up their programs “vertically” within each department. However, this creates a gap between teams, as each is operating in separate models and functionalities – driving up unnecessary costs and inefficient business practices. To successfully align multi-cloud initiatives for increased operational efficiency in 2022, IT leaders will be required to invest more “horizontally” across their organizations – and in doing so, incorporate automation tools to reduce time and stress.

Implementing NaaS

Achieving operational efficiency can reduce the time and stress experienced by IT teams – this is where the adoption of networking-as-a-service (NaaS) tools can play a beneficial role. IT leaders can invest in networking services directly from the cloud rather than building a new network. Over the next year, as cloud infrastructure expands, NaaS implementations will in tandem grow in importance – enterprises and other organizations (including service providers) will have opportunities to leverage cloud infrastructure without building new networks themselves. Automation tools can help accelerate the time-to-complete and automate the highest percentage of tasks in the activity, eliminating as much idle time as possible.

Moving Models and Making Improvements

When considering the investment in these types of tools, it is important to keep in mind how they impact the entire process versus individual tasks. By automating an entire process, not just individual tasks, the operator gains the flexibility to do more in much less time with fewer resources — making innovation possible. With increased flexibility and more time available, engineering and operations teams can focus their attention on strategic activities of greater value instead of performing routine, time-intensive activities. Operations are improved when IT leaders can move from a “human-centric” paradigm to an “automation-centric” model. For example, for maintenance checks, the team can let systems run an increasing number of pre-checks in seconds – enabling the process to execute more efficiently and safely. The key is adopting a model that automates the entire process versus individual tasks.

Moving Forward

In the next year, as more workplaces operate in hybrid models, IT leaders will need to evolve their processes to not only unify operations/approaches for remote and in-person employees – but move forward with this new wave of cloud modernization. In doing so, automation can help mitigate and manage the challenges that come with multi-cloud networking, bolstering skill and efficiency for long-term business impact.
January 10, 2022
Intel Expands Scope of OneAPI Toolkit Initiative

Intel has made available a 2022 edition of its oneAPI toolkit through which it makes processor extensions accessible to DevOps teams.

James Reinders, chief evangelist for Intel software products, said the goal is to make it simpler for DevOps teams to expose extensions to any company’s processors in the runtimes developers use.

At its core, the oneAPI toolkit is a collection of APIs that Intel created for x86, graphics processing units (GPUs), application-specific integrated chips (ASICs), field programmable gate arrays (FPGAs) and other classes of processors.

Reinders noted that developers have never had as much ready access to so many different processor classes before. The challenge organizations face, however, is making it simpler for developers to take advantage of those capabilities in a way that doesn’t require them to write low-level code, he added.

The oneAPI 2022 toolkit adds additional capabilities including the world’s first unified compiler implementing C++, SYCL and Fortran, data-parallel Python for CPUs and GPUs, advanced accelerator performance modeling and tuning and performance acceleration for artificial intelligence (AI) and ray tracing visualization workloads. Collectively, Intel said it has made 900 new and enhanced features available via the oneAPI toolkit in the last year, including deeper Microsoft Visual Studio Code integration, support for Microsoft Visual Studio 2022 and Microsoft WSL2 for Linux development on Windows.

As IT continues to evolve it will become more routine for developers to invoke heterogeneous classes of processors within the same application, said Reinders. Those capabilities will drive an explosion of software development innovation through the rest of the decade and beyond, he added.

Intel is especially focused on making it easier for developers to invoke inference engines that take advantage of processor extensions to run AI workloads, Reinders said. Going forward, most applications will take advantage of machine learning algorithms that run inference engines optimized by Intel via extensions to its processors. As those workloads are constructed, integrating AI models within DevOps workflows will become the next major challenge, noted Reinders.

It’s too early to say how simplified access to heterogeneous processors, also known as XPUs, might change the application development landscape. The one thing that is clear is IT organizations now have more options than ever. The issue now is finding a way to invoke all those capabilities in a way that doesn’t overwhelm developers. DevOps teams will play a key role in deciding which processors and associated extensions to make available to their development teams.

In the meantime, the battle for processor dominance will only intensify in the months ahead. DevOps teams will need to decide how comfortable they are using processors manufactured by a single company like Intel or whether they want to employ a mix of processors provided by multiple vendors. Regardless of what they decide, it appears APIs will play a critical role in making it easier to benefit from advances in processor technologies.

January 7, 2022
Crafting an Agile Enterprise Architecture
Digital transformation, business agility and enterprise architecture are frequently heard terms in this era of digital technologies and rapid change. The appropriation of digitalized applications, solutions and processes has enabled organizations to increase efficiencies. In turn, this allows them to respond to shifting customer needs and trends in a more flexible and agile manner.

Digital transformation, the rise in mobility and a remote workforce have created a shift in the business model paradigm. The adoption of a more agile approach has been gathering steam, and businesses that are building agility into their processes are seeing the payoffs. The results are greater productivity, faster time-to-market (TTM) and increased consumer satisfaction.

The Culture of Agile

Digital transformation succeeds in a business environment that embraces a more collaborative and flexible philosophy. Moving from simply doing agile business processes to being agile across the board requires a shift in the corporate mindset. When a business develops a truly agile architecture, everyone in the enterprise is involved in delivering solutions; from business operations and tech to marketing, finance, compliance and security. The agile culture moves from a company-oriented one to a people or customer-oriented culture, embracing the tenets of collaboration, co-creation, ownership, flexibility and communication.

Short-Term Cycles and Fluid Planning

Agility is redefining enterprise architecture from the traditional schematic of centralized governance, lengthy development cycles, scheduled quarterly reviews and reliance on a waterfall delivery method. Instead of long-term operating and strategy plans, agile enterprise architecture involves short-term cycles and fluid planning that is customer and market-centric. The language changes from top-down project mandates and management to continuous product development, experimentation, testing and monitoring within collaborative product teams.

Essentials for Crafting Your Agile Enterprise Architecture

The blueprint for a truly agile architecture requires fundamental shifts in the business dynamic. There are three essentials that stand out as requirements for attaining agility across the entire enterprise.
- Collaboration – The bedrock of successful enterprise-wide agility is collaboration. Innovation will flourish when it is decentralized, and isolated silos give way to cross-functional, agile and self-organizing teams. An isolated IT team leads to delayed projects, overrun budgets, productivity that is hard to measure and disconnects between business and operations. Every department must be involved in supporting and achieving key business goals.
Teams containing a mix of business line and IT professionals accelerate development and delivery, greatly reducing time to market. Based upon a shared customer-centric goal and vision, there is shared ownership of outcomes and a deeper level of engagement throughout the enterprise. Daily communication and collaborative feedback nurtures creativity, problem-solving, and drive continuous integration and continuous development. Collaborative input from colleagues, inside and outside the organization, is essential for enterprise, technology, and solution architects.
- Current state mapping – Adopting agile practices and principles throughout an enterprise entails potentially massive change. Without a clear roadmap for navigating the process, a business can flounder. Instead of achieving a leaner simplicity, there is increased complexity and a muddied vision. Before a business can successfully embrace an agile architecture, the current state of operations must be fully understood. Essentially, a blueprint of functionalities, capabilities, applications, and the tech stack must be crafted. Everything, from the data catalog to business process workflows must be mapped. Business leaders can ask the following questions to analyze the current state and assess impact points:
- Which projects should we prioritize and what capabilities will they require?
- Which processes rely upon what software applications?
- If we remove one or two apps, how will that impact processes?
Once the blueprint of the current state is mapped out, a business can work in a structured way to create the future state of its architecture with ease and at its own pace.
- Balance agile transformation with daily operations – Continuing to work with some of the current architecture for a time can allow the top level of the organization to ease into adopting the agile mindset and culture. In the iterative process of developing agile architecture, the architects, executives, and tech leaders will need to strike a balance between transforming initiatives and effectively maintaining daily operations. They will have to discern how much direction is needed versus support and how much they will set the pace.
If the focus is first on value delivered to customers, then the architecture will be prioritized according to the most strategic projects, capabilities and applications. An agile architecture will adopt a ‘just enough’ approach to project design and delivery.

Agile Architecture Optimizes Collaboration

Enterprise agility facilitates more direct communication and involvement of enterprise architects within the cross-functional teams. Through daily discussions and various touchpoints, architects can provide guidance, advice, and feedback on IT decisions and assist teams in jointly owning the solutions.

Working together, enterprise and solution architects can develop a complete view of the IT landscape and how it will need to be modified to fit into future needs; minimizing waste and optimizing resources. Solution architects can view the architectural map across all apps, solutions, and systems and collaborate on improvements. They can also help enterprise architects scale standardization across the entire business. That cornerstone of collaboration at the core of agile transformation enables architects and non-tech leaders alike to gain more clarity and alignment.

The Value of Agile

Business value is greatly improved when enterprise architecture combines agile practices. According to a report from McKinsey & Co., businesses transitioning to more agile models can improve financial performance by 20-30%, with 30-50% improving operational performance, TTM, and customer satisfaction scores. Embedding an agile model within processes, people, and technology will cultivate ongoing support for CI/CD, groundbreaking innovation, synergistic collaboration — and ultimately maximize customer outcomes.
January 7, 2022
Open Source Observability Comes of Age in 2022

Observability is growing in importance to meet envisioned business objectives. Despite the benefits to performance data, organizations have faced challenges in realizing the full potential of observability based on implementation, distributed software tools, and other technological hurdles. But with new innovation, the era of true observability is upon us.

Innovative changes

The open-source observability landscape, in particular, is going through a phase of innovation. Powered by distributed tracing “coming of age” with OpenTelemetry, and eBPF taking center stage in terms of restructuring as the next frontier of observability and monitoring on Linux, innovation is at an all-time high.

Kernel technology

Extended Berkeley Packet Filter (eBPF) has presented a fascinating technology native to the Linux kernel. Born with the intent of having a programmable way of dealing with networking, eBPF has evolved into a rather generic and performant facility to have safe, dynamic access to the Linux kernel, that enables, among many other things, to extract telemetry for observability and security purposes. eBPF is powerful and growing in popularity, and Microsoft has been working on a port to Windows.

Versatility

In terms of observability, eBPF proved to be remarkably versatile in 2021. For example, there are tools that use eBPF to perform distributed tracing, like the Pixie project that was open-sourced in mid-2021 by New Relic. But the most promising aspect of eBPF for 2022 is the fact that it is possible, with eBPF, to implement continuous profiling of applications in production. Examples of this are the aforementioned Pixie project and Parca. As proven by tools like Google’s Cloud Profiler and various commercial observability tools, having a low-overhead, always-on profiler in production is a powerful way to troubleshoot latency and memory issues in production environments, which are notoriously hard and time-consuming to reproduce in test labs.

Clear differences

While production profiling with eBPF is happening now, it is not yet a general capability for use with most applications. The way profiling works is surprisingly different across different runtimes and programming languages. While pretty much all of the languages behave similarly in terms of using the CPU and allocating memory (and even in such foundational computing matters, there are differences), the way process concurrency is notably different, pretty much across the board.

For example, Java is using threads inside the Java Virtual Machine that are mapped to threads in the Operating System (to be fair, there are also other options, like NIO and libraries like RxJava, Reactor, Vert.x, plus Project Loom will eventually ship). In Node.js, on the other hand, concurrency is mostly dealt with the Event Loop, and “real” threads at the Operating System are abstracted away from the developer. Differences like these matter for a profiler, because the data shown to the person troubleshooting an issue must be “translated” to the model for concurrency used by the programming language, and this requires explicit support in the profiler for specific runtimes.

Open source observability comes of age in 2022

Fortunately, 2022 is going to be the year that brings us full-featured, open-source, eBPF-based production profiling tools that understand the specifics of the many runtimes that are prevalent in today’s cloud-native applications, like Java, Node.js, Python and .NET, as well as compiled languages like Go and Rust. And that will be a game-changer for improving software and solving issues faster, for all DevOps and SRE and operators.

As we look out to the year ahead, revolutionary technology like eBPF means the rate of innovation happening at the operating system level is only picking up and setting the stage for the year of true observability.

January 6, 2022
IoT Developers: Get Familiar With LoRaWAN for 2022

IoT Analytics forecast the global number of connected IoT devices would grow to 12.3 billion active endpoints in 2021. That number is expected to grow exponentially in the near future as more and more industries make use of long-range wireless capability. For example, president Biden’s infrastructure plan promised major investment in smart cities and smart municipalities across the United States. 2022 will also see IoT growth in the manufacturing and utility sectors as ‘brownfield industries’ look to update their existing machinery for Industry 4.0. Both of these factors will bring about a massive surge in demand for low power wide area network (LPWAN) solutions using the long range wide area network (LoRaWAN) standard next year.

LoRaWAN is the standard protocol for LoRa wireless technology. LoRa was built specifically for long range use cases (it’s in the name). This means that technology using LoRaWAN requires significantly fewer repeaters and gateways to cover a large area, like a city, than would be required using traditional technologies. Additionally, LoRa is robust in how it responds to interference, and LoRaWAN sensors use less energy than other sensors, making them more efficient. In fact, ABI Research forecast that, by 2026, LoRa will be the leading non-cellular LPWAN technology.

In the grand scheme of IoT, LPWAN technology is relatively new. LoRaWAN became an open standard just six years ago when the LoRa Alliance was formed in 2015. This means the main challenge for IoT developers in 2022 and beyond will be educating themselves on the technologies needed to put LoRa-enabled plans into action. There are two big reasons why you should begin that process now: The rising demand for smart cities and Industry 4.0.

Building Cities Smarter

When it comes to infrastructure, roads and bridges might come to mind first, but in reality, infrastructure is much more holistic. Utilities like electricity, gas and telecommunications should all be included in any infrastructure discussion. These industries have already come to rely on IoT technologies to collect data and function in the modern world.

The government IoT endpoint electronics and communications market is expected to reach $21.3 billion in spending in 2022, according to projections by Gartner. As governments continue to modernize and implement more wireless technology, I expect a good amount of this money to go toward the development of solutions using LoRaWAN to create flexible networks on a very large scale for smart cities.

In fact, this movement has already begun. In September, the city of San Jose, California, announced its partnership with a blockchain-based provider to extend the city’s broadband access. In exchange, the city is now hosting and expanding its peer-to-peer IoT network infrastructure.

Updating Brownfield Industries for Industry 4.0

2022 will also see IoT advancements in ‘brownfield’ industries. These are industries with legacy, stand-alone machines and processes, such as in the manufacturing and utilities sectors, that weren’t originally created with internet connectivity in mind. Players in brownfield industries already have large install bases of machines that have no way of reporting status or other pertinent data.

These industries have long been looking at developing IoT solutions to solve this issue. Most are prototyping now with the goal of getting data to flow between large, legacy manufacturing machines with the end goal of improving manufacturing processes. However, unlike other industries where modern updates are often needed to improve efficiency, these manufacturers aren’t going to replace equipment that already gets the job done and has done so for decades. Instead, they need to find an alternate way to gather data and insights from the equipment they already have in place.

For many, LoRa-enabled sensors will be the answer. Often, upgrading to new technology requires equipment be substantially changed. That isn’t the case with LoRaWAN solutions, which allow manufacturers to add LoRa-enabled sensors to existing machines. LoRa-enabled sensors are durable, don’t require much maintenance and have low battery footprints. They also allow manufacturers to keep their data on-premises and on their own internal networks, a common preference for these types of industries. These characteristics make LoRa-enabled sensors ideal for manufacturing purposes.

In the new year, we will continue to see more pilot deployments go into production to help these industries gather the data they need to improve their processes and, in return, their bottom lines.

LoRaWAN Resources for Developers

The intention of the Biden administration’s high-tech infrastructure plans in the United States and the benefits of LPWAN (and LoRaWAN in particular) for private, long-range networks are converging to present a lot of great opportunities for this technology this year. That’s why 2022 will be the year IoT developers need to brush up on their knowledge of LoRaWAN.

If you are interested in learning more about LoRa, check out the LoRa Alliance or LoRa Developer Portal communities to get started.

January 4, 2022
The Future of OpenTelemetry for Observability

The observability landscape is changing. Today’s digital transformation journey has highlighted the need for observability as organizations look to capture, analyze and correlate all data across every level of the application stack including logs, metrics, traces and events. With the rampant rise of data, collecting these signals is imperative to both business insights and system optimization.

Over the last several years, open source data collection has given software and engineering teams the freedom to decide which technologies they want to standardize on for data collection and which vendors they would like data sent to. While open source data collection brought a lot of freedom and advantages, it also created some new problems. The lack of a single, comprehensive framework for all the critical signals observability tracked created a lot of friction for developers. As we move into the new year, businesses are taking the opportunity to reflect on how they can better use their data and apply intelligence to help customers make decisions and sift through all the noise. Enter OpenTelemetry.

What is OpenTelemetry?

OpenTelemetry is a collection of tools, APIs and SDKs that instrument, generate, collect and export all machine data; it was born from the need to develop an industry-wide standard for software instrumentation. By leveraging a single cloud-native framework to complete observability, OpenTelemetry is becoming the gold standard for collecting machine data.

OpenTelemetry consists of three primary components: Specification, collector and instrumentation. The specification uses an API, SDK and the semantic conventions and protocol of OpenTelemetry to determine the cross-language requirements for all implementations. The collector removes the need for multiple data collection agents. It is a vendor-neutral method for receiving, processing, transforming and exporting data. The instrumentation libraries allow users to integrate their applications with OpenTelemetry using auto-instrumentation and support from multiple frameworks and languages. Together they provide a robust framework for companies to disrupt their existing observability practices and amplify them even further.

It’s All About the Customers

Most modern enterprises have complex distributed microservices consisting of tens, hundreds or thousands of components that are all working together to deliver on customer experience. If organizations can’t correlate between those components or the different signals easily, it will slow down a business’s ability to effectively monitor, diagnose and troubleshoot issues.

For years, organizations have leaned into observability by relying on multiple infrastructure vendors to gather, enrich and ship data to vendors. Still, many still do not maximize observability’s most pronounced ability: Correlating complex systems and signals.

As end users desire deep insight into their data, they also expect vendors to offer the best experience to identify and resolve issues within their systems with the help of observability. With true observability, organizations can give customers back their time to focus on what matters, which is managing the digital experiences for their customers and empowering them to meet their customers’ needs.

A Similar Road to Adoption

Much like Kubernetes has been essential for the growing adoption of software containers, the industry is already witnessing the same adoption approach around OpenTelemetry for observability. Kubernetes won the war for becoming the standard for container orchestration. Now, OpenTelemetry will bring the same standards needed for observability with comprehensive visibility into the health and performance of every layer of the stack.

OpenTelemetry has become widely adopted by the software development community, but it’s still at the beginning of its journey. The developer community made Kubernetes a success, and we will see the same with OpenTelemetry as customers demand better integration and less lock-in. For vendors involved around applications and software—from more traditional approaches like application performance management to software development pipelines and management tools and through to cloud-native services—supporting OpenTelemetry will be table stakes in the future.

With extensive portability, greater developer control and strong vendor and cloud provider support, OpenTelemetry has opened a new world to customers by helping to improve achievable results across the application infrastructure, from the software development process to areas like security.

How can businesses continue to move the needle on observability while balancing growing customer demands? Quite simply: If organizations aren’t thinking about OpenTelemetry now, they are losing competitive advantage by lacking the proper visibility to deliver on the service level objectives that customers expect in this day and age.

January 3, 2022
New Relic: Observability Tops 2022 DevOps Agenda

If 2021 was the year when IT organizations became aware of the need for observability, then the coming year should see observability aspirations realized.

A report published by observability platform provider New Relic postulates that there are no less than 11 factors converging in a way that will make observability an absolute requirement for organizations.

At the top of that list is a simple fact: Observability is now required to drive digital business transformation initiatives that typically span multiple processes. In fact, a global survey of 1,300 IT leaders, software engineers and developers conducted by New Relic found 90% of respondents view observability as critical to the business.

Second, as a result, observability is now starting to be applied in both pre- and post-production environments as organizations take a more holistic approach to managing the software development life cycle, the report noted.

Third, the report suggested that IT teams are looking to make better-informed decisions based more on facts than intuition. To do this, teams are using data collected from across the enterprise application environment. In fact, the fourth factor advancing observability in the enterprise is that the cost of instrumenting application environments is dropping, which the report notes is making it easier to collect relevant data.

The fifth and sixth factors reflect a desire to rationalize the number of monitoring tools that IT teams currently use and the fact that usage-based pricing has simultaneously made it less expensive to leverage observability platforms as needed.

The seventh factor is the overall shift toward open source software that enables observability platform providers to collectively leverage innovations faster. The eighth factor, closely linked to the open source trend, is the rise of more complex open source platforms such as Kubernetes that require much higher visibility to successfully deploy and manage.

The ninth trend identified in the report is increased adoption of DevOps best practices that encourage IT teams to view observability as a core tenet of any IT management strategy. The tenth factor cited by the report is the need to drive increased collaboration among IT professionals.

Finally, the eleventh factor driving further adoption of observability is the need for increased application reliability as more organizations appreciate their dependence on software to deliver essential services.

Buddy Brewer, a group vice president and general manager for New Relic, said now that observability platforms are becoming more accessible, the next big challenge will be teaching IT professionals what questions to ask as they troubleshoot both legacy monolithic applications and a wide range of emerging microservices-based applications. The results of those queries then need to be easily shared with both developers and internal IT operations teams that must work together to maximize application performance and availability, he added. The latter requirement is why New Relic decided to acquire CodeStream to make telemetry data available with integrated development environments (IDEs), noted Brewer.

There is, of course, no shortage of observability platforms for IT teams to adopt. In fact, the biggest challenge now is getting every application stakeholder to agree to standardize on an observability platform—no small feat, as many DevOps teams may already have a preference for one approach over another.

January 3, 2022
Best of 2021 – Hiring When You’re Not FAANG

As we close out 2021, we at DevOps.com wanted to highlight the most popular articles of the year. Following is the twentieth in our series of the Best of 2021.

Most of us are not FAANG, and we shouldn’t act like we are. Instead of using that, I’m going to use the term we used to use to refer to them: “Web Monsters”, which isn’t 100% accurate, but includes more than just the five FAANG members (Facebook, Amazon, Apple, Netflix and Google) for this article. At some point, we are going to need a new term. Web Monsters doesn’t clearly include Apple or Citi, but FAANG clearly excludes Citi and eBay. But this blog is about something else, so please accept my apologies for that lack of clarity. Assume that ‘Web Monster’ means “Big, popular high-tech company that pays well and looks good on a résumé.”

There are several angles to “Not Web Monsters” that we can address, but I’m mostly going to focus on DevOps hiring, with a nod to irrelevant sales/marketing techniques. Hiring is an easy target because Web Monsters offer resume-building, pay and benefits that are super attractive in the industry, so they can (right now, at least) reject 99% of applicants and still have a good selection of candidates to choose from. Most organizations cannot compete in even one of those areas, though a few can when it comes to salary and/or benefits.

We in IT have long had a bad habit of asking our applicants to be more qualified than the average employee. It’s pretty ridiculous at times; things like asking for a candidate to have more years of experience with a technology than the technology itself has actually existed. What happens when you ask for the impossible? Honest, qualified applicants self-select out, and only those who don’t realize the technology hasn’t been around that long and liars end up applying. That’s just bad hiring policy, even if you are a Web Monster. But at smaller organizations where each hire has a larger impact on the organization, it can be devastating.

It seems worse in DevOps. In fact, all of the hiring items listed here seem worse in DevOps because DevOps is actually a nightmare mish-mash of skill sets. No two people define the job the same because “the job” covers everything from cradle to grave for an application. There is no part of app dev that “DevOps” couldn’t touch. That’s problem number one. Define the responsibilities of the role clearly, don’t ask for everything that it could be. As a rule, an Ansible specialist is not going to be a DevSecOps expert, and you shouldn’t expect them to be. I think, at a minimum, we need to split hiring along my traditional emphasis lines… DevOps and DevOps. If you are writing code and using Checkmarx while GitLab is building the results and you configured it to roll out to test … you’re not a DevOps engineer, even though there’s a bit of ops in there. Your strengths need to be in development and app configuration. Your job description needs to focus on that.

Could there be roles that need it all? Yes, particularly in the SRE or chaos engineering space; I could see that. But they are certainly not entry-level roles, or even close. Knowledge in the breadth of technologies required to touch the entire DevOps toolchain is developed over years and with a healthy curiosity. And, of course, the person you are hiring should have a similar skill set to those doing the job today. Which is a good sanity check against the “Throw in the kitchen sink” syndrome.

Next up is take-home tests and shared coding during interviews. We know that you have a vested interest in landing the best people, but if you are doing coding exercises with people who are primarily DevOps, you are wasting everyone’s time. Most DevOps jobs are done with PowerScript and/or bash with YAML files. Other languages used occasionally have no more place in a core job interview than other occasional bits. “It’s about how you think” is a common refrain. I would argue that an assignment that takes an applicant hours to complete (or in several cases I know of, days) is not an efficient way to evaluate the way a prospective employee thinks. And, in at least one case, I’m pretty certain the interview solved an actual problem that employees didn’t—or couldn’t—solve. Ask the candidate questions about how they’d solve problem X. Ask them convoluted questions. They don’t need to get to code level to show you how they think.

And finally, my nod to bad marketing: Unless you actually are doing the volume of one of the Web Monsters, the fact that Facebook is a customer of vendor X is totally irrelevant. In fact, like the government, most vendors can mark one or more of the Web Monsters as their customers, just because Web Monsters are so big they try a lot of different tools across many different teams. So having them as a customer should be irrelevant, but if it isn’t considered irrelevant, should be expected.

The team you have today is rocking it. Hire more people like them. You know what works and what you need. Get back to customizing hiring processes to deliver more of what you need and stop over-asking. There are more great people out there looking for roles; don’t scare them off by asking for irrelevant experience. Stick to the core. What are they going to do daily? Require that, then probe for other useful items at interview time. And get more good peeps in the seats!

December 29, 2021
Best of 2021 – Transform Legacy Java Apps to Microservices

As we close out 2021, we at DevOps.com wanted to highlight the most popular articles of the year. Following is the sixteenth in our series of the Best of 2021.

Velocity is one of the major tenets of DevOps, and anything that maintains or improves velocity benefits the overall process. Yet, there are still many speed bumps in the DevOps process that threaten to derail progress and slow DevOps processes to a crawl. One of the biggest speed bumps arises when transforming legacy monolithic applications into something more cloud-friendly—namely, microservices.

Microservices are clearly the future of applications developed for the cloud; research giant IDC predicts that 90% of all new apps deployed by 2022 will feature microservices architectures. IDC also notes the primary benefits of microservices are the improvements to developers’ ability to design, debug, update and leverage third-party code. Although DevOps tends to be about developing internal code, the benefits offered by a microservices architecture remain much the same.

Microservices: Getting There

Moving toward a microservices architecture can be straightforward or complex, depending on the path an organization is taking and where they begin. For example, if the goal is to transform legacy applications into microservices, the process can be complex and tedious. Transformation of legacy applications often means manually disassembling legacy code and breaking it up into standalone code elements that will run as microservices. That greatly complicates things for the DevOps pipeline, where fast iteration of existing code is the norm, not the exception.

However, there are tools coming to market that can ease the process. One such tool is vFunction, which recently came out of stealth with the mission to transform monolithic Java apps into microservices.

vFunction has an interesting approach to that process. The vFunction platform is deployed locally, where the vFunction Java virtual machine (JVM) agent examines legacy applications and gathers data which is then collected on the vFunction server. From there, software architects can further investigate the collected information about the legacy applications, and start designing a microservices architecture to modernize the codebase. “Once developers are satisfied with the design, vFunction generates a service specification file, which contains all the information required to create the microservice,” said vFunction CEO and co-founder Moti Rafalin. “That file is used by our automation engine, which scans the original code and copies the relevant artifacts to create the new service.”

Rafalin said that vFunction leverages supervised learning techniques, graph theory and clustering algorithms to identify dead code and code anomalies that could prevent a clean breakdown, or “decomposition,” of the legacy application. Ultimately, vFunction has the potential to automate many of the manual steps that DevOps engineers used to perform to break monolithic applications into microservices. According to Rafalin, vFunction makes the process as much as 15 times faster and can achieve somewhere between 80% to 95% automation. Currently, vFunction is focused solely on Java, which it deems the biggest opportunity today. However, the company has plans to support applications written in other languages in the future.

Beyond newcomer vFunction, there are other ways to deal with moving legacy Java applications into the cloud. One example is Anthos, which is a managed application platform that extends Google Cloud services into DevOps environments. Another is IBM’s Mono2Micro, which is an AI-based, semi-automated toolset to assist with refactoring monolithic Java applications.

For those looking to build microservices-based applications from scratch, numerous tools exist. Most interesting are the development environments that use rapid application development (RAD) techniques and leverage low-code models. There are dozens of RAD platforms available; prime examples being WaveMaker , OutSystems and AgilePoint.

In most cases, those moving legacy applications to a microservices model will usually select numerous tools, based on the particular use case. For some, it may be easier to rewrite smaller legacy applications; for others, refactoring may be the only way to capture the intellectual property in legacy applications. Either way, carefully selecting the appropriate tools goes a long way towards maintaining velocity in the DevOps realm. Maintaining that velocity proves critical for building cloud-friendly applications on budget and on time.

“Unless one modernizes those monolithic applications to be cloud-native, they end up requiring very large machines and often end up paying more to the cloud providers,” Rafalin said. “Organizations now understand that, if they want the true benefits of the cloud, they need to modernize these apps. And this starts with adopting an architecture that includes microservices, APIs, and modern design principles.”

The lesson is to choose wisely, and select toolsets that support scale as well as growth, and that can support the requirements of microservices.

December 27, 2021
Best of 2021 – Nine Pillars of DevOps Best Practices

As we close out 2021, we at DevOps.com wanted to highlight the most popular articles of the year. Following is the fourteenth in our series of the Best of 2021.

In a prior blog I explained my 27-factor DevOps assessment model which includes nine pillars of DevOps and three dimensions of people, process and technology.

The nine pillars represent categories of practices. In this blog, I include examples of practices for each of the nine pillars. In a future blog I will explain how to use these practices to conduct a gap assessment for your DevOps practices.

Leadership Practices for DevOps

• Leaders demonstrate a long-term vision for organizational direction and team direction.
• Leaders intellectually stimulate the team by encouraging them to ask new questions and question basic assumptions about the work.
• Leaders provide inspirational communication that inspires pride in being part of the team, says positive things about the team, inspires passion and motivation and encourages people to see that change brings opportunities.
• Leaders demonstrate support by considering others’ personal feelings before acting, being thoughtful of others’ personal needs and caring about individuals’ interests.
• Leaders promote personal recognition by commending teams for better-than-average work, acknowledging improvements in the quality of work and personally complimenting individuals’ outstanding work.

Collaborative Culture Practices for DevOps

• The culture encourages cross-functional collaboration and shared responsibilities and avoids silos between Dev, Ops and QA.
• The culture encourages learning from failures and cooperation between departments.
• Communication flows fluidly across the end-to-end cross-functional team using collaboration tools where appropriate (for example Slack, HipChat, Yammer).
• The DevOps system is created by an expert team, and reviewed by a coalition of stakeholders including Dev, Ops and QA.
• Changes to end-to-end DevOps workflows are led by an expert team, and reviewed by a coalition of stakeholders including Dev, Ops and QA.
• DevOps system changes follow a phased process to ensure the changes do not disturb the current DevOps operation. Examples of implementation phases include: proof of concept (POC) phase in a test environment, limited production and deployment to all live environments.
• Key performance indicators (KPIs) are set and monitored by the entire team to validate the performance of the end-to-end DevOps pipeline, always. KPIs include the time for a new change to be deployed, the frequency of deliveries and the number of times changes fail to pass the tests for any stage in the DevOps pipeline.

Design-for-DevOps Practices for DevOps

• Products are architected to support modular independent packaging, testing and releases. In other words, the product itself is partitioned into modules with minimal dependencies between modules. In this way, the modules can be built, tested and released without requiring the entire product to be built, tested and released all at once.
• Applications are architected as modular, immutable microservices ready for deployment in cloud infrastructures in accordance with the tenets of 12-factor apps, rather than monolithic, mutable architectures.
• Software source code changes are pre-checked with static analysis tools, prior to commit to the integration branch. Static analysis tools are used to ensure the modified source code does not introduce critical software faults such as memory leaks, uninitialized variables, and array-boundary problems.
• Software code changes are pre-checked using peer code reviews prior to commit to the integration/trunk branch.
• Software code changes are pre-checked with dynamic analysis tests prior to committing to the integration/trunk branch to ensure the software performance has not degraded.
• Software changes are integrated in a private environment, together with the most recent integration branch version, and tested using functional testing prior to committing the software changes to the integration/trunk branch.
• Software features are tagged with software switches (i.e., feature tags or toggles) during check-in to enable selective feature-level testing, promotion and reverts.
• Automated test cases are checked in to the integration branch at the same time code changes are checked in, together with evidence that the tests passed in a pre-flight test environment.
• Developers commit their code changes regularly, at least once per day.

Continuous Integration Practices for DevOps

• A software version management (SVM) system is used to manage all source code changes. (Git, Perforce, Mercurial, etc.)
• A software version management (SVM) system is used to manage all versions of code images changes used by the build process. (Git, Perforce, Mercurial, etc.)
• A software version management (SVM) system is used to manage all versions of tools and infrastructure configurations and tests that are used in the build process. (Git, Perforce, Mercurial, etc.)
• All production software changes are maintained in a single trunk or integration branch of the code.
• The software version(s) for supporting each customer release are maintained in a separate release branch to support software updated for each release.
• Every software commit automatically triggers a build process for all components of the module that has code changed by the commit. The system is engineered such that resources are always sufficient to execute a build.
• Once triggered, the software build process is fully automated and produces build artifacts, provided the build time checks are successful.
• The automated build process checks include unit tests.
• Resources for builds are available on-demand and never block a build.
• CI builds are fast enough to complete incremental builds in less than an hour.
• The build process and resources for builds scale up and down automatically according to the complexity of the change. If a full build is required, the CI system automatically scales horizontally to ensure the builds are completed as quickly as possible.

Continuous Testing Practices for DevOps

• Development changes are pre-flight tested in a clone of the production environment prior to being integrated to the trunk branch. (Note: “production environment” means “variations of customer configurations of a product.”)
• New unit and functional regression tests that are necessary to test a software change are created together with the code and integrated into the trunk branch at the same time the code is. The new tests are then used to test the code after integration.
• A test script standard is used to guide test script creation, to ensure the scripts are performing the intended test purpose and are maintainable.
• Tests are selected automatically according to the specific software changes. CT is orchestrated dynamically, whereby the execution of portions of the CT test suites may be accelerated, or skipped entirely, depending on how complex or risky the software changes are.
• Test resources are scaled automatically according to the resource requirements of specific tests selected and the available time for testing.
• Release regression tests are automated. At least 85% of the tests are fully automated and the remaining are auto-assisted if portions must be performed manually.
• Release performance tests are automated to verify that no unacceptable degradations are released.
• Blue/green testing methods are used to verify deployments in a staging environment before activating the environment to live. A/B testing methods are used together with feature toggles to try different versions of code with customers in separate live environments. Canary testing methods are used to try new code versions on selected live environments.
• The entire testing life cycle, which may include pre-flight, integration, regression, performance and release acceptance tests are automatically orchestrated across the DevOps pipeline. The test suites for each phase include a predefined set of tests that may be selected automatically according to predefined criteria.

Elastic Infrastructure Practices for DevOps

• The data and executable files needed for building and testing builds are automatically archived frequently and can be reinstated on demand. Archives include all release and integration repositories. If an older version of a build needs to be updated, then the environment for building and testing that version can be retrieved and reinstated on demand and can be accomplished in a short time (for example, minutes to hours.)
• Build and test processes are flexible enough to automatically handle a wide variety of exceptions gracefully. If the build or test process for a component is unable to complete, then the process for that failed component is reported and automatically scheduled for analysis, but build and test processes for other components continue. The reasons for the component failure are automatically analyzed and rescheduled if the reason for the failure can be corrected by the system; if not, then it is reported and suspended.
• System configuration management and system inventory is stored and maintained in a configuration management database (CMDB).
• Infrastructure changes are managed and automated using configuration management tools that assure idempotency.
• Automated tools are used to support immutable infrastructure deployments.
• Equal performance for all. The user performance experience of the build and test processes by different teams are consistent for all users, independent of location or other factors. There are SLAs and monitoring tools that ensure the user performance experience is consistent for all users.
• Fault recovery mechanisms are provided. Build and test system fault monitoring, fault detection, system and data monitoring and recovery mechanisms exist. They are automated and are consistently verified through simulated failure conditions.
• Infrastructure failure modes are frequently tested.
• Disaster recovery procedures are automated.

Continuous Monitoring Practices for DevOps

• Logging and proactive alert systems make it easy to detect and correct DevOps system failures. Logs and proactive system alerts are in place for most DevOps component failures, and are organized in a manner to quickly identify the highest-priority problems.
• Snapshot and trend results of each metric from each DevOps pipeline stage (for example, builds, artifacts, tests) are automatically calculated in process and visible to everyone in the Dev, QA and Ops Teams.
• Key performance indicators (KPIs) for the DevOps infrastructure components are automatically gathered, calculated and made visible to anyone on the team that subscribes to them. Example metrics are availability (uptime) of computing resources for CI, CT and CD processes, time to complete builds, time to complete tests, number of commits that fail and number of changes that need to be reverted due to serious failures.
• Metrics and thresholds for DevOps infrastructure components are automatically gathered, calculated and made visible to anyone on the team that subscribes to them. Example metrics are availability (uptime) of computing resources for CI, CT and CD processes, time to complete builds, time to complete tests, number of commits that fail and number of changes that need to be reverted due to serious failures.
• Process analytics are used to monitor and improve the integration, test and release process. Descriptive build and test analytics drive process improvements.
• Predictive analytics are used to dynamically adjust DevOps pipeline configurations. For analysis of test results, data may indicate a need to concentrate more testing in areas that have a higher failure trend.

Continuous Security Practices for DevOps

• Developers are empowered and trained to take personal responsibility for security.
• Security assurance automation and security monitoring practices are embraced by the organization.
• All information security platforms that are in use expose full functionality via APIs for automation capability.
• Proven version control practices and tools are used for all application software, scripts, templates and blueprints that are used in DevOps environments.
• Immutable infrastructure mindsets are adopted to ensure production systems are locked down.
• Security controls are automated so as not to impede DevOps agility.
• Security tools are integrated into the CI/CD pipeline.
• Source code for key intellectual property on build or test machines are only accessible by trusted users with verified credentials. Build and test scripts do not contain credentials for access to any system that has intellectual property. Intellectual Property is divided such that not all of it exists on the same archive and each archive has different credentials.

Continuous Delivery Practices for DevOps

• Delivery and deployment stages are separate. The delivery stage precedes the deployment pipeline.
• All deliverables that pass the delivery metrics are packaged and prepared for deployment using containers.
• Deliverable packages include sufficient configuration and test data to validate each deployment. Configuration management tools are used to manage configuration information.
• Deliverables from the delivery pipeline are automatically pushed to the deployment pipeline, once acceptable delivery measures are achieved.
• Deployment decisions are determined according to predetermined metrics. The entire deployment process may take hours, but usually less than a day.
• Deployments to production environments are staged such that failed deployments can be detected early and impact to customers isolated quickly.
• Deployments are arranged with automated recovery and self-healing capabilities in case a deployment fails.

What This Means

DevOps is a powerful tool that enables many benefits for organizations that use it. Achieving performance efficiently with DevOps depends on following best practices. By following the nine pillars of practices enumerated in this blog, organizations can achieve the performance potential that DevOps has to offer.

December 26, 2021