devopsy.kinsta.cloud

Category: Enterprise DevOps

IoT Security Concerns

The Internet of Things (IoT) is being hailed as the next step in our technological development. It’s hard to argue with that. The abilities it provides are as captivating as they are numerous: smart cars and homes, automatized agriculture, advancements in medicine that help with after-stroke recovery and the list goes on.

However, while the IoT is truly a huge topic nowadays, we must not let our fascination with it overcome us. It’s too easy to forget about security and privacy concerns when you hear about exoskeletons that can increase a person’s strength and dexterity–trust me, I know. But we can’t afford to forget about them.

It does seem, though, in our pursuit of the interconnected device environment, the security of those devices was somewhat left behind. Let’s take a look at what the primary IoT cybersecurity hindrances are and what can be done to overcome them.

When Security Struggles To Keep Pace

One example of the quantity of devices outrunning their quality safety-wise is smart homes. They are very popular now, and according to the IDC forecast this trend is not going away:

Data source: IDC Worldwide Quarterly Smart Home Device Tracker.

While undeniably convenient, smart homes can pose a potential threat to their users. To do their job, they must collect information about those users. And the more interconnected the entire system gets, the more trouble can a single weak link cause.

Take smart door locks, for example. Some of them come with facial recognition algorithms. That in itself is a pretty big privacy concern. Facial recognition can be used with an alarming accuracy to identify and track individuals. It can have various purposes, but very few of them (if any) are benevolent.

To add to that, within the smart home system, a smart lock is going to be connected to other devices. Maybe it is safe from cyberattacks on its own, but is the smart thermostat safe? What about the smart water kettle? It only takes one vulnerability to get access to a lot of one’s private data.

This risk can be somewhat mitigated by users taking certain precautions: installing AV software to protect their devices, updating the said devices regularly to prevent exploits, setting up a VPN on their routers to safeguard all of their internet traffic, etc. But how many users are actually going to go through with all this trouble?

Physical Problems

It should be noted that data breaches and cyberattacks are not the only threats the IoT faces. Some problems arise from poor design.

For example, among other considerations presented in the June NIST publication, a non-descriptive appearance of an IoT device is mentioned. If it’s just a black box with no markings on it to indicate what software and hardware it runs on, updating it (as well as keeping track of its activities) becomes a very unpleasant chore. It is especially true if there are thousands of such devices located on the premises of a company.

Alternatively, the support of a certain device may get discontinued well before the company plans to replace it. It is also possible the news of its discontinuation can be missed by the company, thus creating a possible vulnerability.

Sheer Scale of the IoT Makes Threats Bigger, Too

Another concern with the IoT is it’s so overwhelmingly difficult to maintain secure. If there is the need to install crucial updates on hundreds of millions of devices, the task becomes all but impossible.

The hundreds of millions of devices example is not an exaggeration, either. Recently, several critical vulnerabilities were found in the VxWorks real-time operating system. Thankfully, this finding was not made by hackers but by a security firm. Some might think it should, therefore, be easy from now on for those vulnerabilities to be patched.

Here comes the problem. There are 2 billion devices that run on VxWorks. Although not all of them can be affected by the vulnerabilities found, hundreds of millions still can. Moreover, there are difficulties in finding all the devices that need a patch, as it’s not obvious what devices even run on this RTOS.

Many of the VxWorks devices are used in medicine. To update them, it is required they undergo the certification and testing processes, and that can take a long time. And before all the necessary testing is done, those devices remain vulnerable.

Making the IoT Safer

That’s why the approach to IoT device security must become more holistic than it is now. It is necessary for the IoT market itself to develop, as security and privacy issues were named one of the biggest problems hindering the implementation of smart devices.

According to the recent Deloitte webcast, one of the biggest threats of the IoT today is security measures and systems are not incorporated into the core design of devices and networks. Another top risk is the lack of security awareness and training among engineers. Yet another one is the lack of proper incident response processes due to the wanting skill levels of security professionals.

It’s obvious the IoT field can benefit immensely from the implementation of DevOps and, especially, DevSecOps. To ensure safety in the IoT, it is not enough to start worrying about it only after the fact, as an addition to other functions. On the contrary, it must become secure by design, and its safety must be a concern on all levels.

— Dean Chester

September 19, 2019
IBN’s Success is Driven by Formulating the Right Test Validation Strategy
Intent-Based Networking (IBN) is widely considered to be the next big innovation in networking industry. IBN is all about simplifying the process of implementing new upgrades and changes in any networking environment to ensure the results match the original intent. IBN can be considered a successor to software-defined networking (SDN) as it seeks to automate networking operations while better aligning networks with business intent.

IBN helps to automate operations that were traditionally performed by network analysts or network support engineers through an intent based model. This is achieved by creating a level of abstraction that through certain orders, points out the service or application to be added, which automates all network configuration and management activities.

The Challenges Posed by SDN and NFV Based Networks

A number of organizations are making long-term investments in the cloud and are just beginning to realize the benefits of NFV and SDN. As a result, we’re still in a transitionary phase until the transformation is complete. As businesses continue to make this transition, there are a few key considerations from an INB architecture point of view that need to be considered before embarking on this journey:
- Analyze each networking program in your roadmap and define if intent-based networking can be added to these projects well before integrating IBN into the network architecture.
- Ensure robust and thorough monitoring and assurance systems are in place.
- Identify and examine the application programming interfaces (APIs) in the network architecture to determine if they will work with IBN. For instance, Nemo API in ODL/ONOS Controller.
- IT network teams should have a single pane of view across architectures, data centers, campuses, branches and cloud platforms.
- Ensure the network architecture has considered the necessary security elements required.
How IBN Ensures a Holistic Approach to Network Intelligent Verification

Unlike the traditional interpretation of network verification, the intelligent network verification I’m alluding to is able to consider network design, configurations and prevailing or existing network states to accurately assess network activity. It does not consider live traffic flows and pre-defined testing scenarios hence aiding in the verification of a negative scenario–something traditional testing can seldom accomplish.

As IBN possess the ability to consider the complete network as a single entity, it facilitates a comprehensive approach to network verification. As a result, organizations are more effectively able to validate the behavior of the network, based on its configuration and state, with the higher-level intent.

IBN Necessitates a Fundamental Shift in Network Design and Testing

IBN is shifting the network IT model from a reactive approach to a more proactive software-driven approach where an automated analysis of current network designs can virtually eliminate human errors and misconfigurations. The automation verification enables is helping to replicate and augment the network expertise of the IT staff in troubleshooting and triaging issues during outages, verifying fixes provided by vendors and documenting procedures and workflows.

There are newer testing challenges popping up due to factors arising from multi-vendor software platforms, protocols and APIs operating together in this environment.

One good practice to be adopted before designing a new testing approach would be to look deeply at the critical dependencies in designing an IBN Solution. However, while migrating from an existing network to a virtual network, both networks must be re-validated thoroughly.

Designing a Testing Validation Strategy that Delivers Value

To tap into the full benefits, it is clear SDN and IBN must be integrated together to exploit their potential. Hence, it is critical to have a testing approach focused on five important areas.

These areas along with the relevant tests that need to be considered are highlighted below:
1. Business Policy: Policy conversion to technology and map test, service design and logical topology test, username to host mapping test and REST API functionality test.
2. Automation: Service placement test, service functional test, service performance test, policy validation test (user intent and policy change validation).
3. Assurance: End-to-end monitoring test, performance of user intent and usage of resources test, performance of SLAs and service profiles test (traffic throughput, latency, jitter).
4. Programmable Network Infrastructure: Southbound API test (NEMO API), OpenFlow test, Ovsdb test, network management test.
5. Security: Service convergence test, conformance test (IBN customer specs conformance, Intent vulnerabilities reporting).
Testing is Critical to Ensuring the Success of IBN Automation

With constant transformation of the network and associated technical complexities faced by network engineers, verification becomes an important step for enabling IBN to achieve its objectives in an SDN-based IBN network architecture. While companies are still allowing software to drive their IT upgrades completely, a focus on a well-defined testing approach is critical to provide the much-needed predictability and assurance required for a successful IBN implementation.

Adopting a layered and integrated approach provides significant benefits related to early detection of bugs, improved test cycles through automation, predictability of outcomes, increase in confidence levels, enhanced scalability, improved security guidelines and adequate testing to ensure smooth delivery.

After all, when the Intent is right and more importantly validated, all the pieces come together to drive business agility.

— Jason Chandralal
September 18, 2019
Legacy Systems Prove to be Digital Transformation Headwind
In a survey of 450 digital transformation leaders, a sizable 73% of respondents said that, through digital innovation, they have made significant or better improvements to the end-user experience. Twenty-two percent report much more significant results than that, they self-reported as having transformed or completely revolutionized end-user experience. That’s considerably higher than the 15% in the same survey in the previous year.

These results, from the Couchbase 2019 Digital Transformation Report conducted within the U.S., U.K., France and Germany, are in stark contrast to results of a recent Kony survey which found, despite $5 trillion having been spent by enterprises on digital transformation, the vast majority of digital transformation initiatives fall short of customer expectations. Still, when it comes to digital transformation success, at least as far as its impact on customer experience, success is in the eye of the beholder.

Of the $5 trillion spent over recent years, much of it has been invested in backend system improvements. If the Couchbase survey is any indication, there remains considerably more spending on back end and legacy systems. Despite the feeling of success cited in the survey, 86% of respondents said they are prevented from pursuing new digital services or transformation projects.

Here’s why:
- That 86% said factors including reliance on a legacy technology, complexity of implementing technologies and lack of resources and skills had prevented them from pursuing a new digital service or other transformation project their organization wanted.
- A total of 55% said their reliance on relational databases somewhat limited their ability to implement digital transformation projects, while 17% said “severely.”
- An 81% have had a digital transformation project fail, suffer a significant delay or be scaled back in the last 12 months.
- A total of 42% said they were behind schedule, or at risk of falling behind, on their most significant digital transformation project.
- A 73% said, while the huge potential of digital projects is often talked about, most of the time they fall short of being truly transformational or revolutionary–albeit a fall compared to the previous two years.
Perhaps part of the challenge enterprises face is they are reacting to market pressure and not being led by business needs or visions for the business’s future created inhouse. According to the Couchbase survey, 52% have their digital transformation strategies set by IT and not business leadership. The drivers for these transformations tend to be, the survey found, reacting to business pressure such as customer demands for new services or responding to regulatory mandates. Surprisingly, the survey found, only 8% of transformative ideas executed are brainstormed from within the organization.

Perhaps part of the reason why organizations are so reactive is because the rate of disruption is accelerating. According to the survey, 91% of respondents believe the speed of disruption in their industry has accelerated. In fact, 40% of respondents categorized that acceleration as rapid. Moreover, survey respondents plan on spending more on digital disruption this year over last, $30 million compared to $27 million.

— George V. Hulme
September 18, 2019
Does Agile Methodologies Work for IT Infra?

Recently, I had a fascinating conversation with my old friend Joe–the name has been changed but the conversation is real and unedited–an IT Infrastructure program manager from a large IT service provider.

After a bit of chit-chat on weather and football, we started talking about infrastructure in a typical large enterprise.

Joe: I am not sure how agile applies to infra space, I can understand it works well for software development, but not in IT Infra space.

Me: OK, why do you think so?

Joe: Well, we have lots of coupling between components, infra projects tend to be big migration or upgrade moves and highly risky. I know many large enterprises in Europe that we work for and many of them haven’t looked at agile in the infra space.

Forty minutes and couple of beers later…

Joe: I have to send four reports every Friday afternoon on status, risks, issues.

Me: Why do you need so many reports?

Joe: I can’t share all the same info to the client that I share internally. There are some risks that we manage internally. My client wants the info on Jira and confluence. My regional manager wants it on an internal tool (some five-word acronym that I forget!). My account manager wants it as a PPT, as this is an important account and he says this needs to be tracked closely. My global manager wants it on a different tool, inherited from a previous merger as my employer was recently acquired by another service provider.

Basically, most of my Friday is spent copy pasting information across these reports. Then I spend most of Mondays reconciling the numbers and responding to follow-on queries, and Tuesdays justifying the cost of resources, change in resource costs because many projects are usually in Amber or Red. I think it usually works well if I have a good PMO.

Me: How many people do you have on the current programme?

Joe: We have four PMs and seven infra resources on this infra upgrade and migration programme. There is a different provider who looks at the application changes and they have a team of managers and developers etc. The seven infra resources I referred to are also busy with BAU work.

Me: So, you have seven part time infra people who are the actual doers, one programme manager, four PMs and PMO, client side vendor manager(s) and you are running half a dozen projects.

Joe: Yes, as you can see, we can’t be agile in infra.

Me: Why do you need projects? And, why do you need so many of them?

Joe: What? No projects? Surely, that’s how it is always done! How do we ensure control, governance?

— Aditya Vadaganadam

September 18, 2019
Why DevOps is Like Football
When it’s done right, DevOps is an impressive feat. You have groups of professionals all melding together as one large team working toward a common goal. The groups–and the individuals themselves–perform their roles and hand off tasks to each other at just the right times. They follow pre-designed plans, but they constantly tweak those plans, in real time, to deal with variables that threaten to undermine the whole mission.

This is DevOps in action. But take a closer look at some of the descriptors above and ask yourself if it sounds like something else that’s starting to creep into a lot of conversations this time of year. If you’re a sports fan like me, this sounds just like football.

Let’s face it, football parallels have been attached to all kinds of institutions and practices–everything from the military to business to marriage. The emphasis on discipline, teamwork and execution can apply to most initiatives in everyday life.

But I’d argue that football’s very structure aligns especially well with what we do in DevOps.

In football, the goal is to move the ball from the line of scrimmage to the end zone. In DevOps, we have our own starting and ending points. Think of our line of scrimmage as the point where features get added to the application and our end zone as the point where an application is successfully delivered to a customer. Football has its individual units (offense, defense, special teams) and we have ours (development, quality assurance and operations).

In football, the offense players have different specialized skills and all work together to move the ball forward to reach the end zone. The players start with a set of instructions (plays), which they all understand. This is like automation in a DevOps environment. A DevOps team execute its own plays, collaborating and improvising as needed to achieve their shared goal.

Picture what this would look like in a pro football game:
- The quarterback passes the ball: Think of a software developer handing off the application to a test engineer.
- Linemen block for the quarterback: This is like quality assurance continuously protecting the application from bugs that threaten to sack the whole project.
- A receiver runs a route and catches the ball: This is similar to an operations engineer preparing a server and receiving the application.
- A running back runs the ball into the end zone for a touchdown: The application is deployed. Goal!
The team has worked together and achieved the goal–in this case, a touchdown. They will be able to repeat this success time and time again because they have repeatable plays and they collaborate.

Without a DevOps approach imagine what a football game would look like:

No Automation: The team has no plays. Nobody on the field has any idea what his assignment is on each snap, so players basically do their own thing. All the speedy, skilled players yell “Throw it to me!” This kind of freelancing might work for youngsters playing pick-up football, but it won’t work in a more advanced game.

No Collaboration: The positions don’t work together. Even if team members have plays and know them, if the linemen don’t block as a unit, a play will fail. If a running back zigs when linemen are opening up a hole for him to zag, the defense will stop him.

No Shared Objectives: If everybody knows the play but they’re only concerned about executing their parts of it, the offense won’t get far. Imagine:
- The quarterback being concerned with just getting the ball in the air, not where it goes or even if it is caught. The receiver can worry about that. End result: Lots of failed passes–and probably a few interceptions.
- Linemen will block, but they’re not worried about who to block or when. End result: Lots of quarterback sacks.
- Receivers will run routes, but they’ll choose where they want to go. They might run beautifully, and even get open periodically. But if the quarterback doesn’t know where the receivers are, the team won’t complete many passes.
Without a DevOps approach, the chances of getting a ball to the end zone would be very low. And when it did get there, the process would be messy and unrepeatable.

When football is done right, teams defend well, score a lot of points and rack up victories. When DevOps is done right, organizations deliver high-quality software to customers in a rapid, repeatable and reliable way. DevOps, like football, codifies a winning process that serves a vast group of stakeholders both inside and outside of the organizations playing the game.

— Brian Dawson
September 17, 2019
How App Stores Are Addressing Fragmentation in the Linux Ecosystem

According to DistroWatch, 273 Linux distributions are currently active, with another 56 dormant and 521 discontinued. While some of these have shared underpinnings, it still makes for an extremely varied landscape for companies and developers.

It means developers must create multiple versions of their applications to be able to provide their software to all Linux users or just address a fraction of the market. Also, developers require multiple versions of build tools, which inevitably results in significant resource overhead.

Desktop application distribution is complex across all operating systems in general; in Linux, this is further compounded by such fragmentation and inter-dependencies both in the packaging and distribution of software.

For example, Fedora uses the RPM packaging format, while Debian uses the .deb format. Moreover, packages built for one version of a Linux distribution are often incompatible with other versions of the same distribution and need to be built for each version separately.

Developers need to satisfy all these requirements at the same time to be able to successfully offer their products to their users, which can be difficult with limited resources.

In a way, the choice of distribution and subsequent promotion of Linux software is a chicken-and-egg problem. Initially, companies and developers must decide what parts of the Linux desktop system they want to target. Then, once they have their products available to end users through one or more of the distribution channels, they still have to invest in separate efforts to promote their applications and capture usage.

Unfortunately, there is insufficient data to assist in the selection and optimization of either the distribution or the promotion process. Linux software developers generally lack sufficient immediate visibility to the usage and popularity of their software.

Traditionally, there have been multiple dimensions to how creators distribute their software: official websites, where companies and developers offer software downloads on their own domains; code repositories such as GitHub, GitLab and SourceForge; and application repositories in which Linux-based operating systems typically provide software through self-hosted storage locations.

To achieve the highest level of penetration, software developers will often distribute their applications using some or all of the above methods. Likewise, most users will utilize multiple methods to obtain software they need. While this gives developers more freedom in how they deliver applications to the end-users, it creates additional fragmentation.

In recent years, there has been significant effort to address the fragmentation in packaging and simplify the distribution of software through the establishment of software centers, in which some Linux distributions offer store-like destinations with a graphical frontend, software ratings, screenshots, descriptions and other user-focused data that allows better search and discovery.

Currently, there are three of these in the Linux space—AppImage, Flatpak and snaps. They each approach and handle the various aspects of usability in different ways.

AppImage and Flatpak technologies are architected to primarily support GUI desktop applications. Snaps are designed to work with servers, cloud infrastructure and IoT devices, in addition to desktop applications.

Spotify, a snaps publisher, is an example of a company benefiting from this approach. On one hand, it requires the Spotify engineering team to maintain both the snap and the traditional Deb package. On the other, the snap version is significantly easier to release, which has also helped with the simplification of the Deb release, as well.

The primary reason is the ability to use easy rollbacks, which have already been used. This feature removes uncertainty that a possibly broken version of the Linux desktop client application might be pushed out to the users. Spotify engineers first release the snap version, and after ascertaining that it works and behaves correctly, they build the Debian package, too. Snaps allows the Spotify team to more closely follow the weekly cadence of releases they have for the Windows and Mac versions.

The emergence of self-contained packaging and distribution mechanisms addresses a major problem in the Linux application ecosystem–the difficulty in establishing a successful business model because of the packaging and distribution challenges. The new formats reduce the fragmentation, as they minimize the number of permutations developers need to support for their applications.

The shift toward centralized application stores that are not specific to any particular Linux distribution offers higher discoverability and better consistency in the user experience.

Developers need fewer resources to support multiple target platforms, which allows them to focus their efforts on the core product, with the understanding their software will be available to the wider desktop audience, without having to depend on distribution-specific dependencies.

Since the aspects of new, refined models of promotion and distribution in Linux are still relatively young, we see improvements but there is still opportunity to develop and adopt better, data-driven practices.

As this field continues to evolve, the store model will help drive growth and usage of the Linux desktop, offering software developers a refined, more tangible experience than ever before.

— Martin Wimpress

September 17, 2019
How to Overcome Resistance to DevOps Implementation
The number of businesses adopting DevOps to make software releases happen faster without compromising on software quality continues to grow. Within the next five years, the DevOps market is expected to reach almost $13 billion. If you are among those who’ve decided to make a shift to DevOps, you are likely to experience resistance coming from the departments involved in a software development life cycle (SDLC), and even from end users, at the start of your DevOps journey.

In this article, we will outline the reasons why your employees may resist the DevOps approach and show how to overcome their resistance.

About DevOps Approach Implementation

Among the benefits DevOps brings are the fast provision of new infrastructure, quick and reliable delivery of software updates, fewer post-release errors due to the implementation of continuous testing and the alignment of testing and production environments, etc. DevOps implementation requires efforts in establishing tight collaboration between the development, QA and operations teams, as well as adopting new tools and methodologies, such as containerization, IT infrastructure automation, continuous integration and continuous delivery (CI/CD) tools.

A new approach calls for project managers, developers, infrastructure management, QA and operations teams to leave their comfort zones and get used to new working conditions. However, each stakeholder has reasons to resist DevOps implementation and requires a tailored approach to overcome them.

The Resistance of Project Managers

Why They May Resist

Project managers (PMs) are the first to face the changes when the DevOps approach comes into force. Their resistance originates from a range of new duties, new responsibility and knowledge they need to add to their playbook, such as:
- Persuading development, QA and operations teams involved in SDLC in the DevOps necessity.
- Establishing constant collaboration between developers, the QA team and the operations team.
- Getting familiar with DevOps-specific tools and techniques (CI/CD, continuous monitoring, version control tools, etc.) as well as cloud computing technologies like AWS, Azure, Google Cloud Provider (GCP).
How to Overcome Resistance

PMs should become DevOps evangelists as they will be the ones who will contribute the most to overcome the resistance of other parties involved. A CIO should provide a PM with sufficient training on all DevOps-related aspects. Such training should include the management of the development, the QA team and the operations team in the context of close collaboration between them. It should also provide the PM with the understanding of DevOps methodologies and tools–containerization, infrastructure as code (IaC), continuous integration and continuous delivery (CI/CD), cloud technologies, etc.

The Resistance of Developers

Why They May Resist

Developers may resist DevOps adoption due to the following reasons:
- They are not excited by the necessity to work with infrastructure engineers closer and deal with operational issues if the operations team is not competent enough to address them on their own.
- They are comfortable with the technology stack they’ve chosen for a particular project and not accustomed to caring about how familiar the other project team members are with these tools and technologies.
How to Overcome Resistance

PMs should embrace their soft skills to establish a mutual understanding between developers and infrastructure engineers. They should also make sure infrastructure engineers can work with the tools and technologies developers may use within the DevOps approach, e.g., Docker, Jenkins, by providing them with relevant training.

The Resistance of a QA Team

Why They May Resist

A QA team may feel concerned about a faster SDLC DevOps implementation, since it entails less time for testing. Test engineers now need to acquire new skills prescribed by automation across the entire SDLC. In particular, they face the need to learn new tools for automated testing (Selenium, Appium, fMBT) and API testing (JMeter, Postman).

How to Overcome Resistance

To make QA engineers less risk-averse, QA managers in cooperation with PMs, should provide them with appropriate training (tools for automated testing and API testing) and show the advantages DevOps may bring. An opportunity to increase test engineers’ qualification level, less effort on testing activities than required with manual testing are among such benefits.

The Resistance of an Infrastructure Department Manager

Why They May Resist

In the pre-DevOps times, an infrastructure department manager was responsible for controlling the design, configuration and delivery of infrastructure for development, testing and production. With DevOps on board and the IaC approach implemented, the infrastructure manager may feel as if they lose control over the IT infrastructure provisioning and management. What’s more, the scope of the infrastructure manager’s responsibilities extends greatly since they’ll need to run the infrastructure department as a competency center.

How to Overcome Resistance

A company’s CIO should explain the infrastructure manager their new responsibilities upfront. The infrastructure manager should stay assured that they change the area of control, not lose it. Within DevOps, the infrastructure department manager has to transform their approach to work. They must be able to help a PM correctly arrange the development, QA and operations team for each particular project taking into account project complexity and size. The infrastructure manager must be able to proactively deal with the cases when the team members involved in software development lack qualification for a project, and not wait until the lack in teams’ competencies results in the project prolongation, for example.

The Resistance of Operations Engineers

Why They May Resist

With DevOps in place, the operations team will have to coordinate their efforts with developers and the QA team from the very beginning of the project, as well as regularly report to the PM on the tasks completed, which is not the usual work approach. To be able to work in the DevOps environment, operations engineers will also have to get familiar with a range of IT automation, version control and other tools. What’s more, the operations staff’s responsibilities will now include the provision of the information on the IT infrastructure functioning to business and technical stakeholders.

How to Overcome Resistance

To help an operations team adapt to the DevOps culture, the infrastructure department manager should deliver training to make operations engineers familiar with new tools and technologies they will need to use, e.g., version control systems (Git, Team Foundation Server), software deployment automation tools (Chef, Puppet), etc. A PM, in their turn, will need their soft skills to increase the operations team’s willingness to work in close alignment with development and QA teams.

The Resistance of End Users

Why They May Resist

It may sound surprising, but even the primary beneficiaries of the DevOps approach implementation may resist the changes DevOps brings. There are two reasons for it:
- The time available for software acceptance testing gets reduced due to a faster SDLC.
- Acceptance testing becomes automated, and end users hardly trust automated tests.
How to Overcome Resistance

To dispel end users’ DevOps-related doubts, PMs must do their best to earn their trust by establishing strong communication with end users. If, for example, end users know software testing covers critical software functionality and the probability of post-release errors is minimized, their confidence in software reliability and the overall trust in the DevOps approach rises.

In a Nutshell

Each party involved in software development has its reasons to be suspicious when the DevOps approach is coming on board. New duties, close collaboration, the necessity to apply new tools and other DevOps-related changes may make project managers, developers, QA engineers, IT infrastructure managers, an operations team and even end users resist the new approach.

To overcome the resistance, a company’s CIO should maintain open communication with each party involved in SDLC, explain their new duties and listen to their concerns. It’s essential to make sure each team member realizes what changes and advantages DevOps brings into SDLC. To ensure they have the necessary knowledge and skills, appropriate training should be provided to project managers, developers, QA team, IT infrastructure manager and the operations team.

— Andrei Lipnitski
September 16, 2019
An Inside Look at GitOps

At a time when variations on DevOps abound, get ready for the latest: GitOps. This new term promises to bring some order to what can be operational chaos and empower operational teams to leverage best practices learned from their developer counterparts. It’s early days yet for GitOps, but here’s what we think you need to understand about this new facet of DevOps.

What is GitOps?

Everyone we spoke with has a slightly different perspective on GitOps, but this is our take: In its most basic form, GitOps is infrastructure and operational procedures defined as code, and living in a source control system, such as Git. Git becomes the single source of truth for what is running in production (or staging, or development).

Declarative systems, such as Terraform, AWS CloudFormation and Kubernetes help enable this practice; but operations is much more than Infrastructure as Code. True operations requires a way to plan, review and approve moves, adds and changes to the infrastructure. It also requires well documented procedures, or runbooks, to be successful. This is where GitOps differentiates itself. We believe GitOps is the full operations lifecycle of infrastructure management, living as code in a git repository.

GitOps may be used for managing both application deployments and base infrastructure made up of systems needed regardless of the applications deployed (i.e. networking, security, etc.). Both will follow the same procedures to plan, review, approve and release code into production. Done right, GitOps will let engineers constantly apply updates to infrastructure meet all the needs of the applications in a continuous delivery format. The version control system ensures everything is recorded and visible and an audit trail keeps teams compliant. GitOps will make it easy to revert problematic changes, becoming a single source of truth about what is happening in the system from both the software development and infrastructure perspective.

What Technology is Required?

In order to kick off GitOps, you’ll use a Git-based version control system. Then you’ll need the ability to create a declarative cloud infrastructure, using something such as Terraform and Kubernetes, with the help of Helm or Flux. Because both Terraform and Kubernetes are declarative systems, they make it simpler to create and deploy infrastructure as code in a manner that is both replicable and transparent. And you’ll also want to implement a solution such as GitLab that wraps operational procedures around these tools. GitLab, being an end-to-end DevOps solution can manage the backlog of requests, assign those requests to the appropriate subject matter expert to create the change in Git, facilitate the review and approval process, and kick off the execution of the changes using its flexible CI/CD system.

While it is possible to achieve this sort of workflow by using different solutions to merge and ship code to Kubernetes, GitLab is certainly a straightforward path to GitOps. It allows developers and operations administrators to work from the same source of truth and with a common set of procedures and tools.

What Are the Benefits?

In the DevOps world, GitOps is perhaps unique in that it can potentially benefit both operations team members and developers. For starters, GitOps aims to address the sometimes cave-like nature of operations (a single person working largely in isolation) by taking all of the best practices from software development and letting them loose on the deployment side.

Full oversight, architectural review, project management and all the other pieces of the process are brought together in GitOps in an effort to end ops versus everyone else and get all the stakeholders involved in the process. The basics don’t change–an ops engineer will still create an infrastructure–but that will just be one part of a much larger process that everyone can contribute to and collaborate on. No more worrying about an engineer creating a cloud infrastructure that can’t be replicated; with GitOps everything is tracked making repeatability easy. Suddenly it will also be simple to answer the “what changed?” question because no one will have to hunt for an answer. The bottom line is a successful GitOps program will benefit the actual bottom line because operational processes will be dramatically streamlined.

Possible Challenges

With any collaborative effort, change can be tricky and GitOps is no exception to that. It is usually easier to create one-offs and not follow a process, so the first thing to consider about GitOps is it will require discipline from all participants. It will be vital for teams to write everything down.

At GitLab we live and die by this thanks to our culture of collaboration, but we know it’s not necessarily something that comes naturally to any individual or organization. It will be vital for everyone on the team to record what’s going on in merge requests and issues. The temptation to edit something directly in production or change something manually is going to be difficult to suppress, but the less “cowboy engineering” the better GitOps will work.

If GitOps sounds like something your organization might benefit from, our best advice is to start small. Pick a tiny area to begin with, get a win and then build on that. At GitLab we call that iteration and it’s worked well for us.

This article was co-authored by Priyanka Sharma, director of technical evangelism at GitLab.

— Brad Downey

September 16, 2019
Before We Burn the Martech Silo
There’s an ongoing debate about where marketing technology (martech) should sit in an organization. Does it live with the CMO, the CIO, the CXO–a hybrid strategist with both IT and marketing background–or some collaboration? As technology use grows, it’s tempting to move martech under another role. Before we burn the martech silo, like how McDonald’s, Uber, Mastercard and others are doing, let’s look at the implications for DevOps teams.

A recent article in PR Week cited a University of Texas study claiming “only 40% of a corporation’s marketing-related activities today report to the CMO.” Martech is being siphoned off into other areas–sometimes slowly, sometimes abruptly. It’s because the tech outweighs the marketing, at least in the eyes of the C-suite.

In March 2019, McDonald’s announced they were acquiring Dynamic Yield for $300 million. Dynamic Yield rose to fame as one of the leaders in omnichannel personalization. Translation: they watch visitors searching, viewing and ordering. Then, they show those related results to other visitors. With the deal closed, McDonald’s announced they were eliminating their CMO position and creating an SVP of martech reporting to the CIO. Automating the customer experience with in-store kiosks and mobile app ordering is their top priority.

Another case where mobile app ordering dominates, Uber clobbered both its CMO and COO in June 2019, right after going public. Their rockstar CEO said he wants to be more involved in day-to-day operations. Uber rolled non-martech functions into a VP of marketing communications. For Uber, martech is the product–an automated experience connecting riders and eaters with drivers. Everything else is branding and communications.

There’s a third org chart option: the CXO, or chief experience officer. More digital customer interaction is forcing a partnership between technology and marketing insight. Mastercard hired its first CXO earlier this year, bringing in a veteran from the digital marketing industry.

What do these three moves have in common, besides moving martech out of traditional marketing?
- All have powerful, ubiquitous brands with huge recognition.
- Their core product shouldn’t change much moving forward.
- Personalization shapes their offers using global and local trends.
Here’s the thing: Marketing automation automates what marketing knows. When the current and future requirements are clear, evaluating martech tools is easier. Moving martech out of branding, advertising and communication to IT sounds reasonable until you realize what you don’t know.

The martech silo often houses hunters, not farmers. Marketers have different metrics. They discuss impressions, and click-thru, and time viewed, and calls-to-action, and conversions. Strategy and content often have more impact on results than the delivery technology. In a fast-moving environment with many product or service offerings, what works once may not work elsewhere. Creativity comes into play, with cycles of learning.

In today’s SaaS model, marketing can pick a cloud-based tool and explore possibilities with minimal IT support. That’s good, until data and process accumulate, needing a bigger silo. Be careful. Burning the martech silo can be far worse for the organization than leaving it alone, even with all the shadow IT downside. No matter who does it, picking tools for a martech suite before understanding requirements is a formula for disaster. Without integrated marketing experience, IT is set up for failure. It can cascade into marketing failure and finger-pointing.

For a fluid marketing environment, a CXO is best suited for the job. They speak the language of both sides, along with representing external customers and their experience. If your organization doesn’t have a CXO, modeling a DevOps team responsible for martech in that image is the next best option.

DevOps solves the problem of speed marketing needs in making changes. Teams also have deep experience in maintaining applications and overseeing security IT wants. Many teams already have a product marketing person managing their software requirements. What a martech DevOps team needs is people with customer and digital marketing expertise. This include creative writers, graphics artists and advertising and social media specialists.

If that sounds like a problem without a CXO running air cover, you’re right. Budget and headcount come from somewhere. Both the traditional marketing team and the IT team want input, perhaps even control. The key for a martech DevOps team is behaving as if the silo had disappeared, no matter what the org chart says. The DevOps leader should keep everyone informed, managing up and across. It usually means creating different presentation views, one for marketing, one for IT and maybe one for the CEO. Make sure everyone gets answers to their questions in their language.

We’re going to have more martech moving forward. Creating effective martech DevOps teams is a priority, even without sweeping C-level reorganization. Don’t just burn the martech silo–reclaim the pieces that work for both marketing and IT and build the right team around the customer experience. If you need help getting started on a bigger martech mission, let’s talk marketing strategy.

— Don Dingee
September 13, 2019
There’s No Silver Bullet for 5G Deployment
When it comes to 5G, a common mistake is to think there is a silver bullet for successful deployment. Often operators assume there is one magic remedy that if introduced to their network, will see them succeed. For example, automation is a key pillar of the 5G era; end-to-end automation maximizes the efficiencies and responsiveness of 5G. However, it would be a mistake to assume there is one method of automation that leads operators to successful 5G deployment. There are a vast range of automation areas and use cases such as network slicing and DevOps that can play a key role.

But first, the key point about there being no silver bullets was part of the reason Nokia commissioned Analysys Mason, a global research organization, to assess and index operators’ maturity and readiness for 5G transformation. With 5G deployment, there are a number of available automation options that need careful consideration.

While the Index made a number of significant findings on operators’ readiness for 5G and provided guidance for how to align technology investments, one of the most intriguing sections was on DevOps. In short, the Index found DevOps, as a key part of network automation, will enhance the impact of 5G on customer experience and service agility, but on the whole is not being adopted.

Key Findings from the 5G Maturity Index

The 5G Maturity Index showed it is paramount operators invest in a variety of digital technologies that can enhance the impact of 5G, including artificial intelligence learning as well as new processes such as DevOps. The following are the most vital findings of the research:

On Automation
- The 5G world will be characterized by the co-existence of physical and virtual infrastructure for a foreseeable future–without automation, the cost of managing functions across the two worlds will be crippling for operators.
- Even when there is a vision for end-to-end network automation, business process automation is not clearly tied to network automation and is often in its infancy.
On DevOps
- Only 32% of network operators are using or are planning to use DevOps by 2020 as means to improve consumers’ experiences.
- A significant 35% of network operators have no plans to use DevOps at any point.
- The level of confidence in DevOps is low.
- DevOps is becoming the norm in cloud and data center environments, but they are only slowly extending into network services.
Why Deploy in a DevOps Mode?

It’s clear that network operators are not yet deploying in a DevOps mode on a major scale. But what are they missing out on? DevOps has been proven, in the cloud environment, to support rapid and flexible service creation–an important goal for operators. At the heart of DevOps is continuous integration, delivery and deployment. The process enables collaboration throughout the entire development pipeline from concept and building to deployment and testing.

Furthermore, the continuous testing and feedback loops of DevOps enable teams to consistently provide up-to-date and reliable software to customers. This enhances the consumer experience and creates new revenue streams for operators. While operators should consider adopting DevOps, it’s equally important they view it as a part of the automation process–one of the key supporting structures to the building, but not the only one. Together with other vehicles for automation, DevOps is a building block for faster time-to-value and paves the way to an agile 5G environment.

Conclusion

DevOps has a critical role to play in meeting 5G networks’ requirements for faster time to customer and improving the customer’s experiences. What is crucial is that operators remember:
1. There is no silver bullet to overall automation, a menu of complementary options is needed.
2. DevOps is a critical part of this menu, because of its ability to create efficient processes which ultimately leads to a greater user experience for businesses and their customers.
It’s critical to align digital technologies carefully to an overall transformation strategy with clear objectives–DevOps will be a critical element of this, but do not be tempted to fall for the tale of the silver bullet.

This article was co-authored by Stefan Kindt, technology marketing at Nokia Software. Kindt heads the operations automation and DevOps marketing. Prior to his current role, Stefan was leading the cloud market development at Nokia where he was responsible for the market positioning of Nokia cloud to enterprises and communication service providers. With more than 20 years of experience in the ICT and telecommunications industry, Stefan has extensive knowledge in the data center as well as the telco space, with global market insight and first-hand experience from Asia and Europe.

— Volker Held
September 13, 2019