devopsy.kinsta.cloud

Category: Blogs

GitLab Adds More AI and Cybersecurity Capabilities to CI/CD Platform

GitLab this week delivered an update to its continuous integration/continuous delivery (CI/CD) platform that adds additional generative artificial intelligence (AI) and cybersecurity capabilities.

The GitLab 16 release included cybersecurity capabilities such as centralized policy management, expanded compliance reports and controls, compliance dashboards and default Level 3 attestation for the supply-chain levels for software artifacts (SLSA) framework.

At the same time, GitLab this week also made available an emergency patch for two modules that have a path traversal flaw, tracked as CVE-2023-2825, that could allow unauthenticated users to read arbitrary files on the server under certain circumstances.

In terms of generative AI, GitLab 16 added a value stream forecasting capability that automatically generates workflow summaries for senior managers. GitLab is also adding a code refactoring capability and the ability to automatically resolve vulnerabilities to an update to GitLab 16. Previously, GitLab made available a ‘suggested reviewers’ features, an ‘explain this code’ and an ‘explain this vulnerability’ capability that are all enabled by large language models (LLMs) developed by Google. GitLab has also partnered with Oracle to provide additional AI capabilities enabled by machine learning algorithms. In 2021, GitLab acquired UnReview, a provider of a tool that uses AI to identify which expert code reviewers to assign to a project based on the quality of their previous efforts and current workloads.

David DeSanto, chief product officer at GitLab, said with 10 AI capabilities GitLab is providing more of these types of capabilities than any other DevOps platform provider. In addition, those capabilities are distributed across the entire software development life cycle rather than being focused solely on making developers more productive, he noted.

That’s critical, because if AI capabilities are only focused on developer productivity, an imbalance is created; DevOps workflows will not be able to absorb the increased amount of code that will soon be moving simultaneously through pipelines, DeSanto added.

AI will also further fuel adoption of platform engineering as a means for centralizing DevOps management. It’s becoming more apparent that infusing these types of capabilities into DevOps platforms built and maintained by an internal IT team is going to be a major challenge, he added.

There is no doubt that generative AI capabilities will soon become widely integrated across DevOps workflows. That will enable developers to build more secure code faster while at the same time making it simpler for software engineers to maintain large codebases. Just about every job function imaginable will be impacted to varying degrees. In the case of DevOps teams, the ultimate impact should involve less drudgery as many of the manual tasks that conspire to make managing DevOps workflows tedious are eliminated. In fact, AI may ultimately make DevOps, as a methodology for managing IT, more accessible to a much wider range of organizations.

As AI advances, there should still be plenty of demand for human expertise. For better or worse, the nature of the expertise required, however, will undoubtedly be at a much higher level than it has previously been as more low-level tasks are automated. The challenge is determining how and where to refocus the efforts of the DevOps teams to add higher value.

May 26, 2023
The Metrics Disconnect Between Developers and IT Leaders

A survey of 350 software developers found a majority (91%) of respondents are unhappy with the actual metrics leadership teams are measuring.

The survey, conducted by Atomik Research on behalf of Uplevel, an enterprise intelligence platform provider, found the top metrics developers say should be tracked include hours worked (52%), work allocation (49%) and the amount of time they have to focus on writing code (46%).

Unfortunately, the survey also revealed more than half of respondents (56%) reported that the CTOs that lead software development efforts make significant strategic decisions without realizing the negative impact on their team. They also often move people around on projects or tasks without knowing the full implications (51%) and overwork developers (44%).

Just under a third (30%) noted their engineering leaders rely solely on gut feelings to measure the effectiveness of their teams, while a third of respondents observed most engineering roadblocks go unnoticed by leadership.

Additionally, 96% noted that not knowing what their own leadership team is working on is detrimental to the larger team.

Christina Forney, vice president of product management for Uplevel, said many CTOs clearly need to be more engaged. Developers don’t mind being measured, but they clearly want to be evaluated based on metrics that matter, she added.

Managing software development teams has become a lot more challenging in an era where many developers are working remotely. IT leaders can no longer manage by walking around as easily as they once did, said Forney. Aligning workflows across a highly distributed team of developers is a challenge no matter how many communications platforms are used.

Surprisingly, the Uplevel survey found more than half of respondents (54%) said they feel more productive in the office. More than a third (35%) also said they preferred some form of synchronous communication to engage with other members of their team versus 27% that preferred some form of asynchronous communication. A total of 38% said they preferred a mix.

It’s no secret that despite long-standing commitments to automation, there are still a lot of manual processes in DevOps workflows that create bottlenecks. In an uncertain economy, however, there’s a lot more focus on developer productivity, so eliminating DevOps bottlenecks has become a major priority. The challenge is those bottlenecks are not as apparent to senior IT leaders, many of whom may have simply become inured to them over time.

Regardless of how bottlenecks occurred, DevOps teams would be well-advised to reevaluate the metrics they track. Mean-time-to-recovery (MTTR), for example, is not as important a metric to the business as the amount of quality code that finds its way into a production environment.

In theory, all the data required to track more meaningful metrics that could improve productivity already exists in the tools and platforms DevOps teams use to build and deploy an application. The challenge has been finding a way to automatically surface that data without adding yet another bottleneck to a DevOps workflow.

May 25, 2023
US DoJ Makes PyPI Give Up User Data ¦ Tape Storage: Not Dead

Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

This week: PyPI complies with a “string of subpoenas,” and LTO continues to grow, despite predictions of its demise. (more…)

May 25, 2023
AWS Identity and Access Management (IAM) Roles and How to use Them

Amazon Web Services relies on the AWS IAM service to govern who is authenticated and authorized to use AWS resources. It plays a hugely significant role in AWS security–and so do its various identities. Let’s look in more detail at IAM roles and how they work.

What is the Purpose of IAM?

When you create an AWS account, it has a single sign-on (SSO) identity called the AWS account root user, which has complete access to AWS services and resources. To avoid potentially disastrous issues arising from this kind of unfettered access, IAM allows shared access to the AWS account via identities.

Based on the security-first model, AWS restricts all actions for all identities by default–apart from the root user. This restriction enables us to manage granular access for identities following the principle of least privilege, so permissions are granted only as required to execute daily tasks.

What are IAM Roles?

AWS users, federated identities and AWS roles are the three major categories of AWS Identities. IAM roles are conceptually similar to AWS users, but whereas a user is uniquely associated with a principal(users/apps/etc.), a role can be assumed by anyone who needs it.

Rather than passwords or access keys, temporary security credentials are provided to whoever assumes the role. This eliminates the overhead of managing users and their long-lived credentials.

Any of the following entities can assume a role to use its permissions:
● AWS user from the same account
● AWS user from a different account
● AWS service
● Federated identity

Structure of an IAM Role

The two essential aspects of an IAM role are the trust policy (who can assume the IAM role) and the permission policy (what the role permits the user to do).

Trust policies use specified conditions to define and control which principals can assume the role. They prevent the misuse of IAM roles by unauthorized or unintended entities.

Permission policies define what the principals assuming the role are allowed to do.

Types of IAM Roles

AWS IAM roles fall into one of the following major categories based on their trust policies:
● Service role
● Service-linked role
● Web Identity role
● SAML 2.0 federation role
● Custom IAM role

Service Roles
AWS services are the trusted entity type for these roles, which are created to allow AWS services to perform actions on the user’s behalf. They do this by inheriting the permissions assigned to the service role.

You might wonder why AWS services need permission to access each other. It’s because, by default, even AWS services have no access to the resources in the AWS account. However, service roles allow AWS services to access resources based on their requirements.

Most AWS services rely on service roles for optimal functioning. For example, they allow Cloudformation to create and delete resources on the user’s behalf based on a YAML or JSON file. The Amazon EC2 IAM role is a special type of service role. EC2 relies on an instance profile as a container for an IAM role, which is then assumed by the applications running within the EC2 instance to perform actions the role allows.

Service-Linked Roles
A service-linked role is a unique kind of IAM role linked to an AWS service. It simplifies the process of setting up a service by automatically adding all the required permissions for a service to perform actions on the user’s behalf. It is predefined by the service. Most service-linked roles do not permit changes to trust or permission policies.

Web Identity Roles
A user assumes a web identity role when they log in to AWS using an identity provider (IdP) such as Amazon and Facebook. Users do not have an identity within AWS itself; in exchange for an authentication token, they get temporary security credentials in AWS that map to an IAM role that is authorized to use the resources in the AWS account.

SAML 2.0 Federation Roles
These roles are assumed by users who are included in an external user directory, typically within organizations. This enables federated single sign-on (SSO), so that organizations can give users access to the AWS console and CLI without having to create a separate IAM user for each person. An organization that manages its employees in Microsoft Azure Active Directory could connect to AWS directly and give its users access to the AWS console and CLI based on the permissions provided to the SAML 2.0 federation role.

Custom IAM role
Custom IAM roles do not fit any of the other categories and support scenarios other than the ones listed above.

What is the Time Limit for Assuming a Role?

An entity can assume a role for as long as the IAM role’s session duration property dictates. For example, if the session duration for a role is set at 12 hours, the temporary credentials will expire 12 hours after they are issued.

If you assume the role using the assume-role* command, you can specify a value for the session duration using the duration-seconds flag, ranging from 15 minutes to the maximum session duration permitted for the role. The default duration is one hour, but it can be extended to up to 12 hours. The session duration can be adjusted by editing the role.

What About Role Chaining?

One role can assume another role in a process known as role chaining. Role chaining cannot extend beyond an hour on the user session; it doesn’t follow the role’s maximum session duration field.

And Cross-Account Access?

IAM roles are often leveraged to enable cross-account access – the process of leveraging a principal in one account for access to resources in a different account. Organizations often maintain multiple AWS accounts to segregate environments such as development, staging, test, UAT, and production, and they grant identities from one account permission to access resources in another.

This could be to process data in the production account and then anonymize and copy it to the UAT account, so that the frequency and attributes of data are the same. This keeps the UAT environment as close to the production environment as possible.

How do IAM Roles Differ From Resource-Based Policies?

Both identity-based policies and resource-based policies are permission policies, but identity-based policies are attached to identities such as users, roles, or groups, and resource-based policies are attached to resources such as S3, Amazon SQS queues, or VPC endpoints.

Identity-based policies specify the actions the identity can take and on which resources, whereas resource-based policies determine who is allowed to access the resource to which they are linked and specify the actions they can perform. Resource-based policies must be inline and not managed. This list provides details of which resources support resource-based policies.

Where both identity and resource-based policies are present, AWS evaluates them together. An action is permitted if it is allowed in either or both policies. If either policy contains an explicit DENY, it overrides the ALLOW.

You can provide cross-account access using just the resources policy rather than using a role as a proxy. This is achieved by attaching a resource-based policy on the resource you want to share. This resource policy comprises all the principals allowed to access this resource, in contrast with an identity-based policy, which specifies the resources a principal has access to.

Using a resource-based policy for cross-account access rather than roles has advantages.

When a user assumes a role, they cede the permissions they have in the trusted account so that they can secure permissions in the trusting account (account with shared resources). With a resource-based policy, the user keeps their permissions in the trusted account, and they also secure access to the shared resources in the trusting account. The principal can access both accounts.

AWS IAM Roles Anywhere

AWS IAM Roles Anywhere is a kind of service role that permits on-prem machines or workloads external to AWS (such as servers, containers, and applications) to access resources on AWS by acquiring temporary security credentials. This completely eliminates the headache of managing long-term credentials.

Workloads are required to have X.509 certificates issued by a certificate authority, which must be registered with IAM Roles Anywhere as a trust anchor. Though a relatively recent use case, it marks a move toward minimizing the usage of long-term credentials.

Benefits of Using IAM Roles

● No management of long-term security credentials
● Support for Single Sign-on (SSO) using SAML 2.0
● Support for web identity federation, allowing users to log in via popular external identity providers (IdPs), such as Amazon, Facebook, and Google
● Enhanced security posture because you don’t have to rotate and replace short-term credentials (An expiry is already associated with them).
● Support for use cases such as cross-account access, Identity federation, AWS IAM Roles Anywhere

Key Takeaways

IAM roles are key pillars supporting, not just the IAM service, but the entire authentication and authorization flow. Their value extends beyond applications such as cross-account access, AWS IAM Roles Anywhere, and identity federation, being important for enhancing any organization’s security posture substantially. When used to their full potential, they can be extremely useful in IAM.

May 25, 2023
Dell Looks to Expand Pool of Available DevOps Expertise

Dell Technologies has quietly launched a training and education initiative to significantly expand the number of IT professionals that have DevOps skills.

Brad Maltz, senior director for DevOps and developer relations ecosystems for Dell, said the next era of IT will require IT professionals that understand how to automate the management of application development and deployment. Given its long infrastructure history, Dell is especially interested in helping IT teams to use a mix of graphical tools and command line interfaces (CLIs) to programmatically deploy software at much faster rates and at scale, he added.

Dell has already hired more than 200 specialists to provide that training as part of a DevOps Transformation Services consulting services engagement, said Maltz. The primary area of focus will be working with IT organizations that are embracing platform engineering to centralize the management of DevOps and GitOps workflows, he noted. Many of those teams are now relying on declarative automation tools that ultimately make DevOps more accessible to a wider range of IT professionals, added Maltz.

The number of IT professionals that employ graphical tools based on IT management frameworks such as ITIL still far outnumber the number of IT professionals that have DevOps skills. The ongoing debate between these two camps essentially comes down to the need to be able to consistently manage IT at scale versus being able to adroitly keep pace with the needs of organizations that are increasingly dependent on software. In the digital transformation era, it’s become clear that more of the organizations that hire IT professionals are placing a higher value on agility, said Maltz.

Bridging the divide between ITIL and DevOps to address that requirement is largely a psychological challenge that requires IT teams to address a range of cultural issues in addition to mastering different types of tooling, he added.

Regardless of whether organizations continue to manage IT infrastructure internally or rely more heavily on managed services provided by Dell, the need to continuously deliver secure software faster has become a business issue. The trouble is the level of DevOps expertise within organizations tends to vary widely. A recent survey published by the Continuous Delivery (CD) Foundation found only 22% of respondents are using both continuous integration (CI) and continuous delivery (CD) to automate all building, testing and deployment of code in a production environment. Most organizations that have adopted DevOps make extensive use of CI to build applications, but CD has proven to be more challenging to master. In fact, the CD Foundation survey found only 7% of respondents described themselves as DevOps engineers or specialists.

The bulk of the managed services that Dell provides are focused on managing infrastructure rather than deploying software, but the company does have a managed DevOps offering based on the application development and deployment platform that an internal Dell DevOps team built to construct and deploy applications. Naturally, Dell has a vested interest in accelerating the rate at which software can be deployed on IT infrastructure it provides and manages.

It may be a while before the overall size of the DevOps community expands, but the training initiatives such as the one launched by Dell will help. However, it may become a lot easier to find and retain that talent in the months and years ahead. In the meantime, organizations would be well-advised to invest in upskilling their own IT teams. After all, it’s less expensive to train someone that already works within an IT organization than it is to compete against thousands of other organizations willing to pay top dollar for the best DevOps expertise money can buy.

May 25, 2023
FinOps Foundation’s FOCUS Aims to Standardize Cloud Billing

Cloud spending is on the rise. As organizations shift more applications to the cloud, there is a growing need to optimize workloads and reduce spending when possible. But a lack of visibility into complex cloud bills often impedes these goals. This is compounded by a lack of data standardization across the multi-cloud landscape, with cloud vendors producing lengthy billing reports in various formats. With things so decentralized, FinOps practitioners often struggle to make sense of it all.

FinOps Open Cost & Usage Specification (FOCUS) is set to change this status quo. FOCUS, an initiative sponsored by the FinOps Foundation, a Linux Foundation technical project, aims to create a standard, vendor-neutral specification for cloud cost, usage and billing data. A common schema for cloud billing should help unify cost data across cloud vendors and greatly aid FinOps objectives in areas like allocation, chargeback, budgeting and forecasting.

I recently met with Udam Dewaraja, FOCUS lead at FinOps Foundation, to understand the context that led to creating the specification. Below, we’ll dive a little deeper into FOCUS, examining the details of the proposed specification and considering what benefits it will bring to cloud consumers and cloud service providers (CSPs).

Why the Cloud Needs a Common Billing Schema

The complexities of today’s cloud services can often create headaches around cloud billing. For instance, AWS alone has hundreds of services and potentially thousands of SKUs. Cloud service providers each have their own data formats and terminologies, which are reflected differently in detailed bills. And with the granularity of cloud metering, enterprises could be staring at hundreds of millions of rows of cost data. Now, extrapolate that across multiple cloud service providers, SaaS and APIs, and you can see the dilemma that the modern CFO faces.

“There are many nuances that FinOps practitioners need to understand,” said Dewaraja. “And a lack of a common definition can be a scalability challenge as you add more things.”

It’s relatively easy to spend money on more and more cloud-based consumption models. As Dewaraja said, “Everyone has the company credit card.” Of course, you can use tools like AWS Cost Explorer, Cloudability or Datadog to garner some insights to reduce spending. But to really get the fuller picture, you need to rationalize cost metrics across many areas. Plus, in 2023, The State of FinOps report found the widest variety of cloud cost management tools in use in the history of the report, indicating tooling sprawl.

Overview of FinOps Foundation

First, for those unfamiliar, the FinOps Foundation is a Linux Foundation-supported group that aims to help practitioners understand the discipline of FinOps. Representing a community of over ten thousand FinOps practitioners, the FinOps Foundation routinely shares best practices and oversees a FinOps framework. “FinOps is more of a cultural practice that should be adopted,” explained Dewaraja. As such, the initiative aims to oversee training, meetups and events and encourage executive buy-in around the FinOps concept.

Understanding the FOCUS Specification

So, what exactly does FOCUS entail? Well, FOCUS is an open source royalty-free specification for cloud billing data. It will provide standard dimensions, attributes, and terminologies for cloud billing data sets. To formalize the specification, the project contributors are actively considering the essential FinOps capabilities and what requirements these cloud bill data sets have in common.

The specification itself has yet to be officially released and is still being formalized at the time of writing. Microsoft and Google have joined the initiative, as well as numerous cloud finance and cost analysis vendors. And while the group waits for additional CSPs to join, the community is building out open source validators and converters, said Dewaraja.

Benefits of a Standard Cloud Billing Spec

Cost management solutions are already available, but what’s not readily available is a method to make business decisions based on broader richer datasets, explained Dewaraja. This is where standardized cloud billing can greatly aid efforts by creating a more holistic picture.

Specifically, this can aid objectives like allocation, chargeback, budgeting, and forecasting. Doing so is often an incremental process, explained Dewaraja—as you gather insights into data, you iteratively use this visibility to make improvements over time.

Some other benefits of adopting FOCUS will likely be:

Putting finance in engineering terms. In the cloud, finance is quite dynamic. And interestingly, the FinOps discipline is becoming engineering’s responsibility as part of the shift left approach, said Dewaraja. Engineers now need to understand this problem and how to architect their solutions cost-effectively. As such, bringing specifications into the mix will help put financial responsibility squarely on the shoulders of engineering terms.

Bringing accountability to cloud consumption. A standard specification should empower essential FinOps capabilities that rely on usage segmentation like internal cost reporting, showback and chargeback. Because if invoices are just one big pie, there is no accountability for individual departments, and you can’t drive the behaviors you need, said Dewaraja.

Win-win for practitioners and cloud vendors. With FOCUS, cloud providers will drive more trust in cost data, providing more visibility. And corporations using the cloud can make their adoption more efficient, likely influencing them to bring on more workloads. “Driving trust and understanding of this cloud data is mutually beneficial,” Dewaraja said.

FOCUS Set to Improve FinOps

Dewaraja, who previously spearheaded FinOps practices at Citigroup, knows all too well the strains of managing disparate cloud billing data. He oversaw a broad, dynamic cloud-native platform of distributed cloud and SaaS and sought to embrace platform engineering principles. In this environment, you need a holistic cost picture that associates various expenses to make the right calls, he said. “You really see success if you can plug this into the workflows of those developers and into business processes themselves,” he said.

Theoretically, FOCUS could deliver on these goals. But although the project is gathering momentum, it’s still in an early stage. For the standard to gain a foothold, it will require greater awareness and more contribution from other CSPs. The specification is community-driven but is being led from the practitioner down, adopting corporate contribution licenses instead of individual developer contributions. Thus, cloud vendors and cost management vendors will likely lead the initial charge.

FOCUS looks great in the long run but will require a little bit of work upfront to get folks on board. That said, it should be on the radar for 2024 roadmaps, said Dewaraja. To learn more about FOCUS and how to contribute, software providers should check out the FinOps Foundation materials.

May 25, 2023
Microsoft Adds Slew of Developer Tools to Azure

At its Build 2023 conference this week, Microsoft announced general availability of an Azure Deployment Environment service that provides developers with a portal through which they can employ infrastructure-as-a-code (IaC) templates based on Terraform or Azure Resource Management files.

In addition, Microsoft has enhanced Microsoft Dev Box, a workstation environment for developers that run Azure, to include a portal through which they can manage multiple environments.

Scheduled to be generally available in July, Microsoft Dev Box is also previewing a capability that makes it possible to customize environments using configuration-as-code files stored in a Git repository. Microsoft is also previewing tighter integration between Dev Box and its Visual Studio integrated development environment (IDE) and Windows to further customize developers’ environments.

Scott Guthrie, executive vice president of the cloud and artificial intelligence (AI) group at Microsoft, told conference attendees these and other tools for building cloud-native applications, along with GitHub Actions, make it simpler than ever to build applications in an Azure cloud service that includes an integrated continuous integration/continuous delivery (CI/CD) platform.

Microsoft is also previewing GitHub Advanced Security for Azure DevOps that natively embeds automated security checks using scanning tools into the Azure DevOps platform.

Other additions to the Azure environment include tools for centralizing the management of application programming interfaces (APIs) using an Azure API Center service that is available in preview.

The Azure Event Grid has also been updated to enable it to use HTTP to pull delivery of discrete events in addition to pushing them.

Finally, Microsoft is also previewing generative artificial intelligence (AI) capabilities to make it simpler to manage cloud costs by, for example, using prompts to summarize invoices or breakdown costs.

Microsoft, along with its GitHub subsidiary, is at the forefront of applying generative AI to application development using tools like GitHub Copilot that developers can use to write better code faster. That copilot capability is now being disseminated across the portfolio of applications and services that Microsoft provides.

It’s not yet clear what the downstream impact of such tools are just yet, but the overall quality of the applications being built should improve as, for example, fewer mistakes are made that can lead to vulnerabilities.

Obviously, the way application software is built, constructed and secured is about to change in the generative AI era. Thanks to its alliance with OpenAI, Microsoft has a significant head start on rivals building generative AI platforms that can similarly be applied to improve developer productivity.

In the meantime, DevOps teams should assume the volume of code simultaneously moving through pipelines is about to dramatically increase as developers become more productive. There will inevitably be more opportunities to apply multiple forms of AI to reduce any DevOps bottlenecks that might arise, but it’s clear most of the immediate benefits are being provided to developers.

One way or another, however, the overall rate at which applications are built and deployed is about to change drastically. The issue is understanding how those advances will impact downstream DevOps workflows that were constructed for a different era of application development.

May 24, 2023
Red Hat Enhances Insights to Simplify RHEL Management

This week at its Red Hat Summit event, Red Hat introduced an expanded set of capabilities to Red Hat Insights that simplify management of Red Hat Enterprise Linux (RHEL).

Managing Linux across the hybrid cloud is complex, and it’s made even more so because of current macroeconomic conditions and the challenge of finding and retaining highly skilled DevOps and IT professionals, said Gunnar Hellekson, vice president and general manager, Red Hat Enterprise Linux, Red Hat.

The enhanced Insights capabilities add a layer of abstraction to help reduce enterprise Linux complexity across the hybrid cloud and make RHEL management easier by lowering skill level requirements for managing Linux server estates, the company said.

Using a single UI, entry-level systems administrators and IT support teams can access Red Hat Insights’ predictive analytics to identify potential bugs, misconfigurations or security vulnerabilities and remediate them. Systems administrators of all skill levels can detect, assess and push fixes for potential problems without a deep understanding of Linux management systems like Red Hat Satellite Server and without having to interact with the command line, the company said.

These enhancements help organizations maintain services and systems even in uncertain macroeconomic times when hiring is constrained. Red Hat research also claimed that these enhanced capabilities enabled IT organizations to find IT issues up to 90% faster and remediate them nearly 66% faster across the hybrid cloud.

A Red Hat Insights image builder service also helps lower the skills level needed to manage Linux estates by enabling IT teams to build standardized, optimized operating system images, including the creation of their own ‘golden images’ that adhere to organization-specific needs for security and compliance. These images can be deployed across public clouds, hybrid clouds, data centers and the edge using the same console.

As the economic downturn continues, organizations are looking to do more with less and identify ways to make the most of the resources, talent and skillsets they already have. Red Hat is making the case for adding a layer of abstraction to allow less-highly-skilled IT and DevOps team members to contribute to managing Linux estates across hybrid, multi-cloud and edge deployments.

“The demands of managing Linux across the hybrid cloud are stretching the ability of resource-constrained IT departments to keep pace,” said Hellekson in a statement. “CIOs need to be able to extend existing skillsets and lower the skill barriers for overseeing Linux estates, which is exactly what the management capabilities of Red Hat Insights are designed to do.”

May 24, 2023
How to Choose an Observability Data Pipeline

A complete full-stack observability platform combines data from multiple sources to gain holistic insights into system behavior. In fact, organizations without a full observability solution could find themselves facing departmental silos since teams with different data sources cannot share insights.

While a key step in observability is the ability to combine logging data with metrics, companies that have developers viewing logging data and their metrics elsewhere are not collaborating on their data in the way observability requires. The solution is an observability data pipeline that best fits your business needs.

What is an Observability Data Pipeline?

An observability data pipeline does two things—centralizes data from a number of different sources and provides teams with the right tools to turn data into business insights. Observability data pipelines are typically user-friendly so that you can easily set up and manage your data.

Observability data pipelines integrate data from a range of sources, including AWS, FluentD and Kubernetes. They have capabilities for searching and viewing log data and notifying engineers through email and even Slack.

An observability pipeline ingests logs, so they can be viewed in a log viewer. Pipelines can also analyze data in real-time, allowing engineers to quickly spot critical security issues before they affect critical systems.

There are a number of full-stack observability platforms, including Coralogix, Splunk, Elastic and even options like Azure Log Analytics and Google Cloud Logging. Coralogix, for example, is a popular observability data pipeline that can analyze data in real-time using Streama and use Kafka Connect to integrate data from a variety of sources and extract trends.

Business Benefits of Observability Data Pipelines

Using observability data pipelines has the potential to transform the way a business does DevOps. Let’s go through a few key benefits of this technology:

● Reducing Cost of Observability Data
Many organizations use systems like Cloudwatch to analyze logging data. However, these often charge per GB of logging data analyzed, raising the question of whether Cloudwatch is really cost-efficient. And it’s not just Cloudwatch. For many systems, logging costs can be increased due to slow query times, noisy logs and the wrong type of storage. Observability data pipelines help organizations manage high volumes of logging data and move away from expensive legacy platforms. This enables enterprises to increase their observability for a modest price.
● Reduced Onboarding, Increased Problem-Solving
Pipelines are straightforward to learn. Unlike options such as ELK, which can be quite specialized and difficult to maintain, observability data pipelines are easy to use and easy to set up. This reduces the time needed to onboard new engineers. Less time onboarding means more time solving problems and serving clients, which increases business revenue.
● Better Data Shaping and Merging
Data from different sources is often stored in multiple formats and structures. Because observability depends so much on well-integrated data, it’s vital to transform this data into standardized formats. Observability data pipelines use log parsing to store key features of logging data for search and analysis.
● Reducing MTTD/MTTR
Having an optimized, user-friendly observability data pipeline reduces downtime by helping engineers solve problems faster.
● Enabling DevSecOps With Observability
We’ve written about the rising prominence of DevSecOps, a paradigm that involves shifting security left in the development life cycle. Key to DevSecOps is the fruitful collaboration between DevOps and Security. This can only be achieved by high levels of observability. Observability data pipelines facilitate this with their ability to integrate data from multiple different sources and present it intuitively. This means security teams can apply their expertise to log analytics that would otherwise have remained siloed.

Wrapping Up

Observability, the ability to use logs, traces and metrics to “picture” the internal state of a system, is vital to any DevOps team. Observability data pipelines can help by integrating and presenting data from different sources. This mitigates the effects of organizational silos and allows DevOps teams to solve problems faster.

May 24, 2023
Is AI the Answer to Devs’ Documentation Problem?

I have written this blog half a dozen times over the years, but every once in a while something comes up that makes me realize I should revisit it once more for those in the back. This time, it was a social call with a group of friends, some of whom are in high-tech, all of whom are smart and often insightful. One member of the group, a tech lead for a company that you do business with, was talking (in vague terms, of course; this person is a professional) about work he is doing, and the group started to make fun of documentation.

Let’s face it—since the very first line of code was written, we have sucked at documentation. It is a rare project that (a) gets just enough documentation and (b) keeps that documentation up-to-date. What we most often get is no documentation or documentation that records the initial design but, by the end, bears little resemblance to the code’s actual workings. The worst of all types of documentation is the “It’s self-documenting!” kind, which has gotten another boost based on the idea that OpenAPI/Swagger will be a tell-all about what is actually happening with the API. Like far too much API testing, it does a good job of telling you about inputs and outputs and, in most cases, absolutely nothing about what is done with the inputs or what magic happens to get you the outputs.

After our call, I got to thinking. What can we do to adequately document the code? I mean, with AI becoming all the rage and the set of lines of source code available for training being so massive, you would think we could do better. Maybe we can—but right now, the language processors that we currently pass off as AI are too busy being conversational or generating the code itself. Both of those spaces are going to take a while to get “good enough” to rely upon without review (not as long as many think, but certainly not tomorrow). Maybe that point will be when some enterprising engineer will turn it on source. Though I think that it will take a documentation specialist and not a coder—after all, we coders created this problem and haven’t fixed it in decades.

Many of us truly believe the “self-documenting” part when that is and absolute lie. It may be self-documenting to someone who wrote it or, in the best cases, someone with deep domain knowledge, but certainly not to the person who has to pick it up. We all have horror stories of working on that application—or porting/rewriting an application—that claimed to be documented or self-documented and was not even close. The saving grace in all of the cases that I worked on was that the code actually worked and performed the job required. Given some extra time to step through it, it could be rewritten even without good comments and docs. (Hat tip to the dev that had to pick up one massive porting project from me because I left to go write about storage—the project was huge and was a blast, but having to hop in halfway through probably made him crazy.)

If you are planning for the future, I recommend a ton of things, but for documentation, only two. First, write design docs, discuss decisions and put them in Git with the source code. Require that they be reviewed and updated after each sprint. Then, make sure that there are comments in the code. Using naming conventions to document code is helpful, to be sure. But more is needed—particularly as code morphs into whatever it is going to be in long-term final form. We have all seen that method or function that has a name … and then doesn’t do what the name implies. Or worse, that does what the name implies but also applies half a dozen side effects because devs needed somewhere to put new functionality. And we haven’t even talked about complexity like feature flags.

So do both, and require that devs spend a bit of time on each during each sprint—not to be a jerk, but to pave the way for dev #327 that inherits this code and has to make sense of it. Help them get productive quickly.

And keep rocking it. We have documentation problems because we solved business problems rapidly. That’s a good thing, we just have to think about the next gen of devs that will be on this project, and give them that little boost into productivity that documentation offers. And maybe you’ll help the compliance team while you’re at it, but that’s a different blog.

May 24, 2023