devopsy.kinsta.cloud

Category: Enterprise DevOps

Three Ways Developers Can Worry Less About Security

Over the past 10 years, developers have risen in organizational importance. They’ve acquired more tools, power, responsibility and autonomy. As part of that evolution, many developers today want to touch all of the layers of code and have their hands in every tool that’s being built. Known as “mechanical sympathy,” developers are taking increased initiative in areas they simply haven’t before–like installing security systems or being in charge of their team’s application security lifecycle.

While pride of ownership is never a bad thing, in today’s complex IT environments, where organizations are adopting hybrid cloud solutions, microservices and containers–and are required to push applications and software into production as quickly as possible–this simply isn’t scalable. Pursuing mechanical sympathy in this case can be too costly and ultimately become counterproductive–especially when it comes to security, where an attacker only needs to find one vulnerability to gain a foothold in the system.

Rather than relying on developers to become security experts, organizations should empower them to dig deep on the tasks that matter to them (development) and leverage the expertise of other specialists (security). That way, everyone can focus on their own productivity and deliver the most value to the business.

Here are three ways developers can let go of their need for mechanical sympathy.

Consider the Implications of Being Responsible for Security

Taking on security means abiding by security best practices, like adopting multi-factor authentication and implementing privileged access security for secrets management. None of this is in the typical wheelhouse of the developer, and in fact, developers could actually be increasing risk if they intend on being responsible for security.

Developers should think of it this way: Imagine something goes wrong with the application you put into production, such as leaking sensitive data based on a mistake in the source code or a design flaw. Do you really want to be the team who’s called? The answer is probably no.

By focusing on core developer strengths and allowing security teams to handle an organization’s security strategy, developers are able to collectively ship applications that are functionally sound, as well as secure.

Don’t Solve for Already-Solved Problems

When it comes to DevOps methodology, there are an increasing number of tools now available that automate time-consuming security tasks–such as automated secrets management, encryption and vaulting solutions–which remove those burdens from developers and allow them to work at true velocity.

With these tools available, there’s no need to reinvent the wheel. Developers should leverage these assets to their advantage and abstract away complexity to the security team, who should be governing and provisioning security tools so developers can get back to delivering applications to market as efficiently as possible.

Prioritize the Value Stream

The aforementioned tools provide a great benefit to developers–freeing them up to focus on more impactful work–but it can be easy for the opposite effect to take hold if developers get mired down in the weeds with them. Instead of trying to specialize in tools such as Kubernetes or OpenShift, which have manufactured obsolescence to begin with, developers should be encouraged to think about the big picture–creating a high-performance, automated pipeline that accelerates software delivery from code to production.

By focusing on the health of the pipeline, rather than the specific tools within, developers establish a real-time feedback loop that enables them to anticipate the future and see the amazing projects that are truly possible. It positions them as drivers of innovation and increases their sphere of influence across the organization.

Developers play a critical role in bringing applications and software to market with speed and stability, but when organizations default to letting them own everything from the nuts to bolts, they may end up introducing security issues that could otherwise be avoided.

The best way to ensure security in the development process, however, is to start with a strong security foundation. Organizations that understand developers and the development process are savvier when it comes to the vulnerabilities that could be inadvertently introduced and are in a better position to secure them with minimal impact on velocity.

In such a complex and powerful world of cloud, CI/CD and more, when developers are empowered to let go, organizations can better manage and mitigate risk, and deliver higher quality applications that drive true business impact.

— Brian Kelly

November 12, 2019
Crossing the Ecosystem Boundaries: How ServiceNow, Salesforce and Atlassian Work Together
Although these platforms were originally designed to serve different functional roles—Atlassian’s help desk, ServiceNow’s IT service management and Salesforces’ sales and CRM—from the outset, they all addressed the evolving mega-management and leadership issue: collaboration across teams and business units.

Independent action, siloed organizations and disconnected thinking was increasingly giving way to more agile, interdependent, collaborative problem-solving practices and methodologies. Whether the differences were political, cultural, intellectual or institutional, survival became progressively dependent on living, working and communicating more effectively across all boarders. It was in our individual and collective, both internally and externally, best interest to do so.

Effective companies and the platforms that served them needed to promote a culture that collaborated with less friction across five boundary types (“CCL’s” Center for Creative Leadership’s Boundary-Spanning Leadership)
- Horizontal: Expertise, function, peers.
- Vertical: Rank, class, seniority, authority, power.
- Stakeholder: Partners, constituencies, value chain, communities.
- Demographic: Gender, generation, nationality, culture, personality, ideology.
- Geographic: Location, region, markets, distance.
Collaboration initiatives generally begin by focusing on existing boundaries within a specific business unit or team. Then they expand to forge common ground on specific projects across teams, and finally, moved across all five boundary types with multiple teams and interdependent projects which some refer to as complex adaptive systems. These platforms were launched, quickly developed and continue to evolve to meet this collaboration challenge head on.

Making Sense of the Blurring Boundaries Across Convergent Platform Roadmaps

Boundaries are difficult to identify when they are continuously moving and being redefined. Instead of taking a more traditional route of seizing mind and market share by differentiating and establishing/defending clear boundaries, these platforms are doing the opposite, they are converging. The ecosystems that support each platform are also beginning to look increasingly similar—but more on this in a future post.

The convergence of frameworks such as ITSM and Agile represents a tangential and interconnected set of moving boundaries intersecting somewhere along the spectrum at ITIL—ITIL4 if you’re counting. As all three platforms seek broader market penetration reflecting this full spectrum of frameworks, consider using the ITIL4 Edition’s 17 service management practices as a cheat-sheet for their boundary spanning enterprise roadmaps:
- Business analysis.
- Service catalog management.
- Service design.
- Service level management.
- Availability management.
- Capacity and performance management.
- Service continuity management.
- Monitoring and event management.
- Service desk.
- Incident management.
- Service request management–includes both request fulfillment and access management of ITIL V3.
- Problem management.
- Release management–includes an explanation about releases in waterfall driven environments and DevOps driven organizations.
- Change control–formerly known as change management.
- Service validation and testing.
- Service configuration management.
- IT asset management.
As organizations prioritize and re-prioritize the list reflecting their Agile/ITSM journey, the boundaries between platforms will move, and in many cases, simply disappear. As this journey plays out in your company, the platforms and ecosystems that support each will need be rationalized. Said differently, you may be asked or told to choose one, but as our experience suggests, you won’t need to do that today.

Instead of diving into the technical features and benefits of each platform and ecosystem, take some time to reflect the benefits of expanding your boundaries to address the business challenges you are facing now and expect over the next 18 months.

Where does this leave you and your team today?

While everything else gets worked out….
- Review the ITIL4 Edition’s 17 service management practices and rank prioritize for your team and company.
- Do a self-assessment on your personal commitment to, and comfort with, the benefits of expanding your boundaries.
- Meet with your team and have each of the individuals do the same.
- Encourage conversations about where the platforms appear to be focusing (e.g., Atlassian focusing more on ITSM and ServiceNow paying more lip service to Agile).
- When the time comes to make recommendations on app consolidation or choosing one platform over another, your team will be well prepared to do so with open, informed and agile minds.
One last note on the benefits of expanding boundaries: Leaders who effectively reach beyond their present boundaries have the advantage when faced with solving problems, creating innovative solutions and evolving to thrive in today’s interdependent, complex world. The benefits of boundary spanning leadership include:
- Increased organizational agility to respond to a dynamic marketplace.
- Advanced cross-organizational innovation processes.
- Achievement of mission-critical, bottom-line results:
  - An engaged and empowered workplace at all levels.
  - Flexible, cross-functional learning capabilities to solve problems and adapt to change.
  - Better-managed risks and rewards through enduring cross-sector partnerships.
  - Higher-performing virtual teams.
  - Global mindsets and cross-regional collaboration.
Use these bullets to get your team spanning conversation started.

— Joshua Platt
November 11, 2019
CIOs See Dramatic Change Ahead

Driven by heightened competition and a rapidly changing business environment, many enterprises now find themselves faced with decisions that could make or break their future.

Consider this: Within three years, 44% of respondents to the 2019 Harvey Nash / KPMG CIO Survey expect to fundamentally change their product, service offerings or business model. It’s a finding that spans enterprises both large and small, the survey said.

This could explain why IT leaders reported their technology budget to be increasing the most in 15 years. The report found 45% of respondents favor budget increases for money-saving technology efforts, such as task automation. Interestingly, almost two-thirds, or 63% of organizations, report that considerable amounts of their enterprise technology efforts are managed outside of the control of their IT department.

However, the survey showed those organizations where the IT team is formally involved in business technology decision-making have better business outcomes. These enhanced outcomes include improving new product time-to-market and employee experience to be better than their competitors. Still, four in ten companies are not formally involving IT in those business-led IT decisions.

Increased investments in automation and artificial intelligence (AI) are expected to place considerable churn on the workforce. CIOs are expecting to witness one in five jobs being replaced by AI/automation within the next five years. However, 69% of CIOs also expect enough new jobs to replace those positions lost to automation.

“Boards are asking their CIO and technology team to prioritize automation of jobs. How organizations adapt to automation will increasingly become a priority, and many are not at all ready,” said Albert Ellis, CEO of Harvey Nash.

Despite CIOs enjoying a rising influence within their organizations, fewer CIOs have seats on the board of directors at their organization—having fallen significantly from 71% who held board seats just two years ago to 58% today.

The 2019 Harvey Nash/KPMG CIO Survey, conducted between December 2018 and April 2019 across 108 countries, is based on the responses of 3,645 CIOs and technology leaders.

While automation and AI may, or may not, displace workers in the future enterprises are counting on them today to help close their perceived skills shortage. According to respondent’s, enterprises continue to find it a challenge to obtain the talent they need. In fact, respondents reported skills shortages at their highest level since 2008.

The employees most difficult to find are those in data analytics (44%), cybersecurity (39%) and artificial intelligence (39%).

Finally, respondents surveyed who consider their organizations to be digital leaders (those who see themselves as very effective or extremely effective at using digital technologies to advance their business strategies, performed better than their competitors on every aspect surveyed.

According to the survey, digital leaders outperformed in time to market (53% versus 34%), customer experience (65% versus 49%), revenue growth (55% versus 43%) and profitability in the previous year (50% versus 37%). Digital leaders, the survey said, are also more likely to introduce ‘major new changes to products and services’ in the next three years (55% versus 39%) and focus on profitability.

— George V. Hulme

November 8, 2019
Becoming a Data Company: A Tale of Two Trends

Data is back, baby! Well, that seems like an odd statement. It never quite left. Nevertheless, Google’s recent acquisition of Looker and Salesforce’s acquisition of Tableau seems to highlight this mindset in magnified ways. With eyes set on new leaps in data science and artificial intelligence, it’s easy to understand how our views on data now feel like a shifting pendulum.

The theme of the last decade was “every company is a software company,” driven by ready compute availability from cloud platforms. In the coming decade, it will be time for our data practices to catch up with the ongoing revolution in software development practices–or in other words–for “every company to become a data company.”

Over the past few months, as I have conferred with customers, attended conferences and connected with colleagues, there’s a perception of growing vigor in data conversations. Developing a data competency is rapidly growing in importance to business now, just as establishing a software competency became a priority 10 years ago.

However, my most curious observation has been the discussions around data and software are insular. Conversations either focus on how to move, manage, cleanse and leverage data for analytics, or how to build and release high-quality software quickly. The people who hold these conversations are either data people or software people who live in isolated worlds, seem to have divergent concerns and connect only at the lowest common denominator–the database storage–when the software guy or gal needs to store the data somewhere. There is little talk of the deeper connection and synergy between the two worlds and how it relates to larger trends.

The Next Wave of Digital Transformation

If you zoom out a bit, there’s more than meets the eye. There is an elegant symmetry between the software and data worlds, and it alludes to how the modern enterprise will morph in the age of digital transformation. Just as functions and organizations were reshaped to support new cultural and technology trends such as DevOps, organizations shall be reshaped in a data-first world, too. As software competency grows and matures, the need to acquire data competency becomes more urgent. What seem like separate trends are a maturing continuation of the uber trend of digital transformation, an attempt to acquire competencies in software and data (and necessarily so in that order).

As the diagram below shows, enterprises have been pursuing a journey of developing a software competency and becoming software companies. Those that make sufficient progress in software competency quickly realize that the significantly increased data they now have acquired requires new competencies to harvest and extract value from it. A data competency becomes necessary to completing digital transformation and so begins the journey to becoming a data company. Software and data are two sides of the same coin of digital transformation with the echoing resonance of reassuringly familiar themes. The diagram below highlights some of these.

Mirror, Mirror on the Wall

Speed and safety are central drivers in both competencies. The DevOps movement was driven by these two (often at odds) requirements to deliver software quickly from development to production without breaking the production deployment by ensuring that what is traditionally known as “non-production” accurately mimicked production. The goal is to build and test code so when it is deployed to production it doesn’t break what is already working.

In the data context, as more organizations understand the value of their data, they realize the importance of speed to get data quickly from where it is produced (production) to those that need it (outside of production, i.e. non-production). Safety in the data context is not just about not disrupting production while extracting that data, but also about meeting regulations and remaining compliant with data privacy policies in the process.

The elegant symmetry here is further accentuated by the fact that while the primary concern in the software context has been around getting software from internal development environments to customer-facing production environments, in the data context, the concern is getting data from customer-facing production environments to internal non-production environments. This completes the virtuous cycle of experimentation and learning that modern enterprises must master (see below) in their journey of digital transformation. It requires that organizations rapidly iterate to product-market fit through trial (software release) and learning (data science, analytics, machine learning) and back again.

In this way, digital transformation becomes a virtuous cycle driven by the complementary movements of DevOps and DataOps–the cultural movements, philosophies, technologies and collaborations that underpin the software and data journeys respectively, and the primary drivers of organizational metamorphosis that enterprises are undergoing during the digital transformation journey.

— Karun Bakshi

November 6, 2019
Using Open Source for Better DevOps Outcomes

Many insurers have jumped on the DevOps bandwagon in order to improve the efficiency and effectiveness of their systems and systems implementations. Very few insurers, however, spend much time thinking about the open source software community as a way to improve their DevOps approach–and the skills of their developers–and in that regard they’re missing a golden opportunity. In fact, insurers often restrict their developers from updating or contributing to open source libraries because they don’t realize the potential benefits available to them by participating more actively in the open source community.

A 2016 survey by 451 Group indicated that the more companies used open source solutions in their DevOps portfolios, the more effective the DevOps function was. More specifically, the study reported that among companies with at least 60% of their DevOps portfolio as open source, a full 54% had reached the “fully deployed” DevOps state, with all of their development occurring via DevOps models.

By comparison, only 30% of the surveyed companies that used less than 60% of open source tools in their DevOps portfolios had reached this state of DevOps adoption. Clearly there’s a link between open source use and DevOps success. Given these results, insurers should focus on the benefits of supporting and encouraging their DevOps groups to embrace the open source community.

It’s a dated but still persistent view: Participating and contributing to the open source community means developers are giving away their time by creating and contributing intellectual property (IP) that may benefit others outside the organization. However, the truth of the matter is participating DevOps engineers, developers, analysts, etc. are often fixing bugs and making improvements to code libraries other than their own but that can have value for their own DevOps efforts.

The very definition of open source –software for which the original source code is made freely available and may be redistributed or modified–makes it fertile ground for skills and solutions improvement. Additionally, that process of analyzing and repairing code represents some serious skills development. It’s the kind of development skills that can be difficult to obtain by just working on in-house applications and platforms–which are often proprietary and dated–lacking modern coding techniques.

Since open source literally means a developer and/or organization has the ability to make the code fit their needs however they see fit–they control their own version of the product that may diverge from the original open source project–the opportunity for DevOps solutions improvement is substantial. Moreover, any DevOps related IP a developer creates and contributes is not lost and is certainly not stolen–the IP can be used directly by the developer and that developer can continue to benefit from the IP of other developers on a gratis basis.

Utilizing DevOps solutions from the open source community can be both time and cost effective and can be a positive boon to organizations. While some insurers may not knowingly allow open source code in core systems or applications, its use in a DevOps ecosystem is another matter. That’s especially true since DevOps is a newer and less mature IT function that requires new tool, process and solutions development. Leveraging open source solutions can expedite that process. In fact, many of the key DevOps tools used today either are or started as open source solutions for DevOps problems. One would be hard pressed to find a DevOps development stack–from building/developing to monitoring–that didn’t use at least one open source tool as part of the stack.

Often the fear or apprehension to using open source libraries (for DevOps or anything else) is that anyone could have written it, thus its quality is or could be sub-par. While in some cases this can be true, the same holds true for closed source proprietary software, as many insurers could attest. Simply put, open source tools and libraries are worked on by developers of the exact same caliber–if not the exact same people–who work on closed source proprietary tools and libraries. The only difference is there are often less people involved with proprietary solutions development. And since there are relatively few proprietary DevOps solutions in the marketplace to choose from, open source should be considered a viable channel for DevOps solutions.

Even if there were more choices for proprietary DevOps solutions, the financial savings involved in combining open source solutions with DevOps can be significant. Importantly, open source solutions for DevOps are particularly useful when the right solution for a problem has yet to emerge, almost a given in a maturing DevOps ecosystem. Rather than rushing out to buy something that promises to fix the problem–or writing something themselves–DevOps developers can find open source solutions and use those to experiment, test and otherwise isolate and identify the true nature of and solution to a DevOps challenge.

Another reason DevOps groups should avail themselves of open source techniques and tools is the fact that the tools and techniques available for DevOps–proprietary off the shelf and built inhouse–are still not quite there yet. The tools available are almost always missing something, resulting in developers at many organizations spending most of their time–and the organization’s money–trying to develop fully functional solutions from the ground up.

More often than not, the resulting tool or solution is never as complete or well-built as the open source tool or solution that already exists. Most organizations would be better served tasking their developers with scouring open source libraries, identifying the “best fit” DevOps tools and then spending the time required creating the updates and improvements to the tools that have already been mostly built, saving time and money.

Finally, there are lots of DevOps staff out there who participate in the open source community–in their spare time anyway, or have done so while working on previous projects. That means hiring DevOps staff with open source experience is a shortcut to potential open source techniques and tools. These are people that can hit the ground running by bringing their open source experiences to bear while they’re getting up to speed on the proprietary work they may be assigned. Considering these benefits, it’s past time for DevOps groups at insurers to dive into the open source waters.

— Tanner Gibson

November 6, 2019
IoT, AIOps and OMG!

The task of keeping networking straight while expanding the definition of network to include both physical and virtual networking has fallen heavily upon the op-centric side of DevOps–or what I like to call devOPS. What is defined as networking has grown, sometimes at a ridiculous rate. While the terms are familiar, how you manage them in truly virtualized environments is different than traditional hardware management. More agile, but different.

And that means those who deal with networking in DevOps have to know more, manage more and debug more–all while working on other parts of the DevOps toolchain. It’s been a slog through increasing capability with increasing workload.

There is a confluence of new and different things that promise to set this lopsided progress right. We’ll look at them individually, then talk about the sum of the parts.

Internet of Things (IoT)

For most of us, when we hear IoT, we think of millions of devices slamming data into a centralized service. iPhones, for example, returning information to Apple. Or highly specialized devices such as utility meters that report far more frequently than humans could go out and read them. But IoT is many things, and increasingly IoT is networking devices. They hold a wealth of information about the state of the network at any given moment, and that information is increasingly being funneled into APIs that IT or management apps can take advantage of. Considering that some of those management apps are also getting data from application tooling and server health status, there is a wealth of information coming together in one place.

AI Operations (AIOps)

A networking device can be doing a lot in any given second, and generating data as it works. Servers can be on their way offline when they feed the information out, or maybe sitting idle and wanting more work shipped their way. Application bottlenecks might be identified in the wealth of application tooling information, but buried under the wealth of other response data being shipped back. AIOps aims to help the devOPS side deal with these issues by limiting what makes it to human attention and drawing correlations between issues. Finding that an over-burdened network connection is associated with an apps performance degradation is an easy one to talk about, even though it is pretty simplistic. AIOps can make the connection before a human gets involved, and then the staff member can decide what to do about the actual root cause.

It’s not at all perfect yet–we’re not generally talking about AIOps making a ton of decisions, rerouting traffic or spinning up new instances to resolve issues at this point–but it is definitely in the “useful tool” category. The cost in person-hours to analyze and reduce problems is definitely cut by the use of AIOps as it exists today, and that benefit will only grow moving forward.

The biggest problem is the one that bedevils all data interchange, and in fact is the biggest issue in the world of big data–data standardization.

Enter OMG

No, my OMG in the headline wasn’t what you thought. It was the Object Management Group (OMG). They’ve been around forever, and have done a pretty good job of developing useful standards, particularly in the area of data interchange. I’ve followed them for a long time, and used their standards since … well, anyone remember CORBA? That was my first set of OMG standards.

Now they’ve started an Artificial Intelligence Platform Task Force (AIPTF). This group will be looking at all areas of AI to set standards, but the interesting bits for this blog will be in data standardization and communication. Standardizing data and how it is communicated is the next big step in this process, and hopefully the OMG will set us down that path. They’re not specifically looking to address infrastructure big data needs, but their work will likely guide AIOps moving forward. And there is huge potential in that movement. If AIOps gets some standards that help it to be more proactive, we may actually end up with auto-fixing algorithms for obvious and easy issues. We already have the systemic ability in container managers to increase the number of instances serving an application if the existing ones get over-loaded, if we could expand that type of auto-growth to include networking issues and consider cores used by instance to find the best solution. Members of the team that are more devOPS will have more time to focus on new development. And that’s a huge plus.

The Future is Bright

Whether the work at OMG benefits us will depend upon whom is involved in the various subgroups they have planned to implement, and cooperation with other ongoing efforts at standardization. Yet I’m hopeful that, like with that original CORBA spec, they will provide a roadmap that others can follow to put us on the path to the much-maligned self-healing network. Giving devOPS a chance to keep up, while leaving them ultimately in control of what happens. Current iterations of AIOps do this to a lesser extent, but more inputs and more automated analysis of those inputs will only make AIOps more powerful. Standardization will make AIOps more able to make use of data and better focused on their specific problem at hand. I see the confluence of all three trends to be a huge positive to the industry if we manage to take advantage of it. And some vendor out there will see the potential, taking advantage of it for us, we just have to find them and use their solutions.

Meanwhile, paw through the data available to you by hand again, and keep knocking those walls down.

— Don Macvittie

November 6, 2019
The Challenge of Scaling Metrics in Distributed Systems
Whether migrating from a monolithic architecture or building a distributed system from ground zero, there are many benefits to leverage from a microservices architecture–faster software deployment cycles, enhanced scalability, improved isolation of risk. However, distributed systems are complex and as such, accompanied by many challenges, particularly when it comes to the scalability of metrics.

The Importance of Observability in Distributed Systems

In order to realize the benefits of a distributed system, a strategy of data-driven optimization is essential. The paradox here is visibility simplifies complexity, while the complexity of visibility is dramatically increased in distributed systems.

As New Relic points out in their eBook Cloud Native Is the New Normal, traditional monitoring approaches need to be adapted to meet the demands of decentralized architectures, including:
- Overcoming the loss of end-to-end monitoring capabilities by learning how to connect the dots across a complex network of environments.
- Developing tactics to quickly drill down into issues across interconnected parts to reduce mean time to resolution (MTTR).
- Introducing proactive reliability strategies that focus on measurement first, followed by improvement.
Finding the Right Balance Between Monitoring Nothing and Monitoring Everything

Most application engineers leverage some sort of metrics tooling such as Grafana, Prometheus or Graphite to track vitals for their system health. While monitoring is essential, there is such a thing as bloat when it comes to metrics ingestion; and this overload translates to scalability challenges that affect performance, cost and reliability.

So, how do you know what to monitor? In order to derive the most valuable insights from your system, you must first understand and identify the components, and more specifically which attributes within those components drive key metrics. Knowing this requires an intimate familiarity with how all the pieces work together in your distributed system.

Google’s Site Reliability Engineering book provides a framework for determining what to monitor in distributed systems. They break signals down into four different categories:
- Latency: Understand resource requirements and identify bottlenecks.
- Traffic: Monitor system response to varying levels of load; most useful when paired with other indicators.
- Errors: Track frequency and nature of interruptions in request/response flow.
- Saturation: Surface capacity-related issues.
Even when you’ve identified the most important metrics to track, scalability is still a challenge that must be addressed, particularly when dealing with applications that handle massive volumes of data across a complex application architecture.

M3: Uber’s Open Source Large-Scale Metrics Platform

A solution directly addressing the challenges of distributed metrics at scale is M3, an open source project developed by Uber’s engineering team.

Before using M3, the Uber team sent metrics to a Graphite stack with storage handled using the Whisper file format in a shared Carbon cluster. From there, they created Grafana dashboards to visualize the data and used Nagios for alerting based on Graphite threshold checks via source-controlled scripts. As the volume of metrics continued to expand, manual processes to scale the solution became unsustainable and repeat failures resulting in permanent data loss led them to build a more fault-tolerant, scalable system.

M3 is focused on providing a reliable storage solution for large-scale metrics over long retention periods, solving inherent single node-associated limitations with Prometheus by offering a multi-tenant store for Prometheus metrics. A key component of this solution is M3DB, the scalable storage backend for M3.

M3DB, built in Go, is a distributed time series store and reverse index with configurable out-of-order writes. By enabling the storage of real-time metrics over long periods of retention, M3DB can replace a Prometheus cluster. In developing the M3DB solution, the Uber team essentially deconstructed Prometheus into microservices to enable the scaling of each component individually. By taking this approach, M3DB is able to offer better reliability, efficiency and performance.

At a minimum, M3DB can be leveraged as remote storage for Prometheus (or used to replace any other remote storage solution). There are, however, other features the solution offers, including an external aggregation engine and a separate query engine.

The M3DB project is still under development with a growing community of contributors.

To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon NA, November 18-21 in San Diego.

— Molly Wojcik
November 5, 2019
AIOps: The Path to Greater Resilience and Uptime

Large organizations with complex IT infrastructures are facing a significant challenge when it comes to data. The complexity is such that many IT operations teams are struggling to detect, or predict, the complex failure conditions and circumstances that can adversely impact resilience and uptime.

Monitoring a multitude of systems is a complex task, as the data generated by any system can be vast and far too overwhelming for manual IT operations processes, particularly in relation to analyzing, correlating and prioritizing alerts and remediation.

Companies have much to gain from a better understanding of the data being created from monitoring tools and practices: increasing the resilience of systems, and the speed of remediation cycles delivers significant cost and time savings. Harnessing advanced technologies to deliver more intelligence to teams in charge of performance monitoring is becoming increasingly important, which is where AIOps can deliver significant value on top of existing monitoring and automation tools.

Cutting Through the Noise

Telemetry tools deliver staggering amounts of data. The Microsoft Azure telemetry platform, for example, records 10 petabytes of data per day. When dealing with this volume, the ability to intelligently decipher the severity of alerts becomes vital, especially as many of them may be duplicates or not meaningful in any way. Yet it is IT operations teams that are tasked with understanding how systems are performing, and where resources should be diverted to ensure services are always up and running. To do this effectively, the root causes of alerts must be identified quickly and correctly.

AIOps is increasingly becoming the go-to solution to the problems inherent in dealing with big data. By applying data collection, data modeling and data analytics techniques and using machine learning algorithms to establish patterns, the noise of the data is reduced as operations teams gain more intelligent insights into alerts. So, providing more context to issues flagged within systems is the goal. Eventually, somebody has to take action when an alert is created, so the more intelligence engineers have, the easier it will be to remediate.

The less context delivered with alerts, the more time engineers will need to resolve the issue, as the manual remediation process will likely involve interaction with other teams. Take application performance monitoring as an example. In my experience, operations teams do not have an end-to-end understanding of the applications that are live in the production environment. If an alert is created, the ops team will typically seek assistance from a member of the developer team who built it, or another colleague with a deeper understanding of the application, which of course extends the lead time of the remediation process.

Key Considerations for Deploying AIOps

By establishing data patterns through machine learning and other advanced analytics technologies, the following key drivers for alerts assessment using AIOps can be established:

Frequency: Discovering how regularly alerts are created and whether it warrants automation for analysis, correlation, prioritization and remediation.

Coverage: Telemetry is of course vital to AIOps, as the machine learning algorithms will only be as good as the data they ingest. Putting in place the right telemetry tools is therefore vital to ensure the right data is being gathered. Failing to do so increases the risk of missing critical issues.

Impact: Not all problems require equal energy and resources. Certain issues might only occur every few months and may not be critical. In such situations, it is not prudent to invest significant amounts of time and money into developing the machine learning algorithms that will identify where and when an alert will be created. Again, cutting through the noise of the telemetry will help to identify high-priority alerts.

Probability: What is the likelihood of certain issues recurring? Can automation tools, informed by the machine learning algorithms, be implemented to deal with these, i.e. learn where and when they are likely to occur so they can be dealt with by the AIOps platform autonomously?

Take, for example, the issue of CPU or memory going beyond a certain threshold. Rather than manually contacting customers when an issue is flagged to inform them that services are about to go down before being spun up again, an AIOps platform can automate this process, so remediation happens seamlessly.

Latency: Automation of the remediation process for an alert deemed critical requires increased investment in AIOps. Getting good data from the early stages of AIOps processes will reduce the amount of time taken to both flag and deal with a problem. Containerized microservices, for example, take very little time to recover, so very little investment is needed when applying AIOps tools that automate remediation. Database recovery, however, is a more complex process, the automation of which would require significant investment.

What’s the Customer Value?

For service providers, AIOps is essential if they are to attain the highest levels of resilience, the measure of which all comes down to the Service Level Agreement (SLA). The SLA is measured by a percentage that corresponds to the time systems and are up and running over a year. So, 99.99% uptime is good, 99.999% is great, but 99.9999% is where service providers want to be. To put this into context, downtime for an entire year in the latter scenario amounts to just 32 seconds.

Increased automation is the only way to achieve optimal SLA. To do this, service providers are increasingly turning to AIOps solutions.

Operations teams are in charge of the IT strategies that align with business objectives. For service providers, a key part of this is, of course, ensuring maximum availability and optimal performance of the customer-facing environment. Understanding when and where faults appear, and improving resilience against them, is the best way to ensure this. That is where AIOps truly adds value.

— Prashant Jain

November 4, 2019
Privileged Access Management at the Speed of DevOps

Privileged access management (PAM) is clearly a very important element of effective cybersecurity. PAM has been ranked number one on Gartner’s Top 10 Security Projects for two consecutive years, and it is number four on the CIS Top 20 Critical Security Controls. The struggle for many organizations, however, is they have also embraced DevOps culture, and the traditional approach to PAM clashes with the principles of a DevOps environment. Thankfully, it is possible to effectively manage privileged access in a DevOps environment—it just takes a new approach.

Challenges of Privileged Access Management

Despite advances in identity and access management and the existence of things like two factor or multi-factor authentication and biometric authentication, most organizations still rely on a simple username and password to authenticate an individual and grant access to network resources, applications and data.

The concept of least privileged access—ensuring that users only have access to things they really need access to—has been preached since the dawn of cybersecurity and, when done right, can be an effective strategy. The problem is it’s tedious to do right and it’s very easy to lose track of who has privileged access to what.

That’s where a PAM solution comes in. The whole point of privileged access management is to provide a way to track and manage who has access to what—to grant access when it is needed and revoke access when the need no longer exists. Unfortunately, there is a lot of confusion about PAM and the effort to manage privileged access is challenging for many organizations.

Enterprise Management Associates (EMA) conducted a survey on the state of privileged access management. The survey included responses from companies representing different company sizes and industries around the world and focused on IT and cybersecurity professionals—most of whom are familiar with PAM and are directly responsible for managing or granting privileged access to users.

There were some concerning findings from the survey regarding the perception of PAM. Twenty percent of organizations that reported using a dedicated PAM solution believe it somewhat or greatly reduces user productivity. Many respondents also expressed frustration about the amount of time required to manage privileged access. Half of the survey participants cited manually granting temporary privileges as either very (29%) or extremely (21%) time consuming, and half also said manually revoking temporary privileged access as very (33%) or extremely (17%) time consuming.

PAM and DevOps

As challenging as traditional PAM solutions might be for businesses around the world, they pose an even greater hurdle for organizations that have embraced DevOps. A recent episode of the Defense in Depth podcast, hosted by David Spark and Allan Alford, focused on some of the challenges of PAM. A listener had written to complain that “the traditional approach to PAM breaks everything that is DevOps. Regardless of what your PAM provider says about their ability to scale with your dev cycles, they’re lying about the real level of effort.”

That seems like a fair assessment. Pretty much every company is somewhere on the cloud journey—and working within a DevOps framework on some level. Companies often face backlash from the DevOps team—with some going so far as to boycott efforts to introduce security tools that can’t keep up with the pace of DevOps in the cloud.

DevOps developers express frustration about trying to build an automatic scaling, horizontal infrastructure and being dragged down by demands from the security team to install agents and change authentication modules. They complain it often means rewriting all of the scripts, resulting in significant and unnecessary effort for a solution that ultimately can’t scale properly in the DevOps environment.

That is, unfortunately, true when it comes to traditional PAM solutions. Next generation PAM solutions, however, are built with the cloud and DevOps in mind. They are lightweight, automated, scalable and capable of integrating seamlessly with common tools and platforms.

Rethinking Privileged Access Management

Based on the survey, organizations understand the importance of privileged access management, but they struggle with finding a way to implement it effectively. The question from the podcast illustrates how PAM is also a significant issue for DevOps environments because traditional PAM solutions do not integrate or scale effectively.

At the same time, we all know that PAM is a crucial element of effective cybersecurity. It’s not acceptable to just not manage privileged access, but it also doesn’t make sense to suffer through the headaches and frustration introduced by traditional solutions. Organizations need to find new privileged access management tools that work in the cloud and can scale automatically to provide PAM at the speed of DevOps.

— Tony Bradley

November 1, 2019
Extending Automation to Your First Code Commit

I am a big advocate for software reuse, and an even bigger advocate if that reuse comes in the form of software automation. My team and I rely on reusable software processes that accept simple inputs to produce predictable results. We have tight deadlines with requirements that often seem impossible. Automation makes us more efficient and our work more predictable.

Once upon a time, we used glue and bubble gum to cobble together means by which to build, then test and then eventually deploy application versions. It worked until it didn’t, and industry opportunities arose for mechanization. Enter continuous integration (CI), continuous deployment (CD) and DevOps, in part.

You likely know this and have already taken advantage of CI/CD platforms such as Jenkins, CircleCI and others. The simplicity of the as-code model allows us to turn easy-to-create declarations into the otherwise complex provisioning of servers, databases, caches and file systems, along with the execution of code to leverage what was provisioned.

Life as a developer has been made better using these as-code tools in the tool chain. Spinning up a Kubernetes cluster on Google to (re-)deploy an image built, tested and pushed to DockerHub from a CloudBees/CodeShip pipeline is something I rely upon each time I commit to my Git host.

In this article, I am going to talk about Project-as-Code to extend automation to the first code commit with a working app running through a functional CI/CD pipeline and deployed to a production-like environment.

Barriers Keeping Enterprise IT Departments from Coming Along for the DevOps Ride

Unfortunately, basic usage of all these great tools comes with the added complexity and cognitive load cost. Understanding Docker image creation, HashiCorp Terraform syntax for infrastructure-as-code, Kubernetes cluster deployment and more are just a few of the competencies developers and operators need to have.

Learning a new set of tools every few years comes with the territory, but there’s no reason to ignore new potential avenues for automation. I’ve wondered if there is a better way to abstract and automate the setup of these common pieces in the modern software production pipeline.

Another common issue I’ve noticed—these pipeline components often get set up sequentially, with the first meaningful code commit being the blocker to setting up all the other pieces of the CI/CD pipeline. Without this commit, these tools sit relatively idle while waiting to provide value.

For those of us who work on new projects regularly, getting things started has always had its issues, especially for consulting scenarios where requirements and team dynamics differ between clients. We are disciplined enough to first reach in our toolbox to reuse as much as possible. Still, DevOps has yet to be adopted wholesale in a standard way by many enterprises.

For these situations, I find the benefits of DevOps are easy to explain and understand. Developers like to reuse and fewer barriers, the business likes savings and an enterprise likes productivity and innovation. However, to inject those benefits is still a very manual, time consuming process, making the overall sell of DevOps harder.

The purpose of DevOps automation is to simplify tedious error prone manual processes. I see another opportunity for automation: entire DevOps projects using the same as-code model we have become accustomed to.

Generating Your First Meaningful Commit “As-Code”

Why not use a straightforward declarative model-driven means of not only generating tech-stack specific application source code, but all the config files that lead to DevOps being automated. With this, the toolchain learning curve is minimized, and idle tools now are immediately called to action. The result is a self-sufficient operational DevOps environment working through an application code base that comprises a fully deployable executable first code commit. Developers then get their reusable assets and environment(s), the business sees cost and time savings, and the enterprise gets the productivity and innovation necessary to stay competitive.

In an article written by Pete Hodge titled “Hello production,” he goes into great detail about the value of quickly standing up a flowing CI/CD pipeline leading to a functioning app deployed to production. He contends deploying even a useless application into production, as soon as you can, is the right way to start a new project because it pulls unknown risk forward, opens up parallel streams of work and establishes good habits. Project-as-Code standardizes this thinking with model-driven automation.

The next generation of DevOps is unfolding with automation remaining a key component to expand scale, velocity and productivity. As automation has worked its way up the toolchain, automating the automation behind DevOps will shift the paradigm to improve how projects are created, how DevOps is implemented and the quality and features of the first commit into a Git repository. What now takes weeks to months could instead take hours to days, falling in line with our expectations now met by other successfully mechanized aspects of DevOps.

Hopefully this article will serve as a starting point for your next DevOps project and how Project-as-Code automation can be a part of getting you started easier and faster.

— Steven Randolph

November 1, 2019