devopsy.kinsta.cloud

Category: DevOps Practice

Learning ‘The First Way’ of DevOps – Continuous Flow

Many organizations are struggling to realize well-engineered DevOps, even ‘the first way’ of DevOps – continuous flow.

DevOps is an evolving body of knowledge. There is no one-stop shop that covers all roles and the complete range of skills required for DevOps. A DevOps-competent workforce requires an ongoing commitment to DevOps skills development and mastery. Enterprises must approach DevOps training strategically to leverage training resources to their best advantage.

Learning Continuous Flow

The first way of DevOps, continuous flow, is the foundation for DevOps.

Why Is Learning Continuous Flow Important to Engineering DevOps?

A knowledgeable, skilled workforce is needed to meet strategic objectives. Without a solid understanding of DevOps concepts and terminology, communication between people and teams is inefficient and a source of bottlenecks and misunderstandings. Developing hands-on skills for the tools and workflows is needed to understand and implement DevOps pipelines. There is an industry-wide shortage of skilled DevOps candidates. New skills foster innovation. Knowledge workers reduce production costs and reduce mistakes. Having skilled coworkers makes for a satisfying working environment. Employee retention is better. Competitive advantages are derived from knowledge.

How Is Learning Continuous Flow Accomplished?

The following are recommended engineering practices for learning continuous flow.

People are encouraged to be continuous learners with an ongoing commitment to DevOps skills development. DevOps training is approached strategically to ensure DevOps training is well aligned to business goals. DevOps skills gap assessments are conducted and used to define and refine the training strategy. Leaders and cross-functional teams are encouraged to take DevOps fundamentals training. Leaders define a training strategy that best fits the needs and priorities of the organization to meet critical skills gaps.

The training strategy considers which skills and roles can be covered by inexpensive self-study resources. Determine which skills and roles warrant more costly training. DevOps training has a distinct budget. Compliance to DevOps training programs are tracked. DevOps leadership training is offered to management and staff that have a role relevant to DevOps. DevOps tools training includes the recommended engineering practices for using each DevOps tool in the Service Catalog. Staff are encouraged and supported to take DevOps certifications that are relevant to their assigned roles.

A DevOps training program is defined and includes a recommended curriculum for each role including leaders, developers, QA, Ops, and infrastructure and security staff that are operating in a DevOps environment. Recommended DevOps training resources are cataloged, and the catalog is visible for all stakeholders. Merit and recognition programs encourage employees to master DevOps skills.

All staff is allocated time for DevOps training. DevOps mentoring programs supplement formal training. Job swapping and immersive training is encouraged and planned. People are encouraged to attend external conferences and meetups and share what they learn with other team members.

What Is Needed to Implement Learning for Continuous Flow?

The following are recommended engineering practices for implementing a learning program for continuous flow.
Assign roles and responsibilities for DevOps training leadership (e.g., DevOps Sensei, Training PM, evangelist). Conduct a DevOps skills gap assessment. Define a training strategy that best addresses the gaps and fits the needs and priorities of the organization. Define a training budget that includes expenses and time allocations per role and skill.

Determine which skills and roles can be covered by inexpensive self-study resources. Determine which skills and roles warrant more costly commercial training. Identify training goals for each role. Ensure job recruiting profiles match skill requirements for DevOps skills that are most relevant to the organization strategic goals. Put in place the means to make training resources visible and available. Set training goals for teams and individuals. Put in place incentives and recognition programs for completion of training.

Some of the best sources of DevOps training resources are listed below:

1. The DevOps Institute
2. Partners of The DevOps Institute, such as Learning Tree International and Xellentro
3. Udemy
4. Recorded talks and publications available through IT Revolution
5. Books such as my own Engineering DevOps, by Marc Hornbeek.

What this Means

Organizations need a trained workforce to succeed with DevOps—even ‘The First Way’ of DevOps—continuous flow. Organizations need to approach DevOps training strategically to leverage DevOps training resources to their best advantage. To learn more about my blueprint for learning all things DevOps, refer to my book Engineering DevOps.

June 2, 2021
Then and Now: The Evolution of DevOps Culture

There are many available resources dedicated to delivering better business value faster. Based on this advice, many organizations and teams are adopting common software delivery practices — even outside the usual use case of providing software (take, for example, SpaceX teams using agile, or leadership teams that use scrum practices and have daily stand-up calls). This progression into faster, better, cheaper and easier – no matter what you’re delivering – is what makes incorporating DevOps into business practices critical to delivering unmatched value for end users.

Then and Now

A great perspective to consider when approaching the subject of DevOps is how technology has changed just about everything – how we build, connect, share and educate. Up until the 2000s, for example, the standard turnaround time for taking and sharing pictures was one to three weeks, if not longer. This process used to involve film cameras, darkrooms to process the photographs, photo finishing labs and hard copy, physical prints. Today, our mobile devices allow us to snap hundreds of professional-quality images and share them immediately online.

These earth-shattering changes don’t apply to photography alone, but to many other industries including financial services, banking and IT. Consistently, technology is finding new ways to disrupt industries, priming them to be more efficient to fit into our fast-moving lifestyles.

The DevOps Life Cycle

Now, let’s look at software delivery. In the past, the process of traditional software delivery was linear and sequential; waterfall methodologies offered little flexibility and did not succumb to the nature of change and demand for technology services we see today. What we’ve experienced with software delivery now uses DevOps to leverage new insights through people, processes and technology, giving organizations the ability to deliver continuous business value. Yes, that means businesses are adding value to our lives all the time, whether you see it or not. For technology organizations, this process for delivering business value is modeled through what we call the DevOps life cycle — understood through the graphic below.

Getting to a DevOps Culture

Often, an organization’s challenges with DevOps do not lie in the technology or processes themselves, but in the people part of the equation – how we work together to deliver value via cross-functional teams. To understand what building a DevOps culture means, we need to look at what culture encompasses and how our people, process and technology work together to enable a successful DevOps life cycle.

This layered graphic features all the various components related to culture. Each layer encompasses the next.

· Mindset

· Values

· Principles

· Practices

· Tooling

Getting to a DevOps culture involves articulating change both from the bottom up and top down in our organizations and is, therefore, no easy feat. Before we can introduce or discuss tooling, we need to understand how our practices and principles help us achieve our outcomes. Each inner layer requires the outer layer to be successful before we can truly get to defining and strengthening a DevOps culture.

Here are my tips for getting to a DevOps culture:

· Learn from and develop skills from mistakes

· Focus on the important priorities

· Capture collaboration

· Perfect your tooling

The DevOps Principles

DevOps is the collection and collaboration of people, processes and technology to deliver software value continuously. Here are some core DevOps principles you can use to guide you on your DevOps journey.

Create With the End in Mind

High-performing teams work best when leaders and individuals understand their role in the team. Therefore, creating with the end in mind involves reassessing and often asking big picture questions, like “Who is this delivery for?” and “Why are we doing it this way?” Keep the vision in mind and make adjustments along the way.

Even the most famous photographs in history involved many adjustments and numerous attempts to get it right. Creating with the end in mind means knowing we may not have everything right the first time around. That’s okay! Similarly, life isn’t without its own distractions, conflicts and changes. Keeping the target outcomes in mind helps us align to initiatives and actions, and to question when they don’t align.

Cross-Functional Teams

Many products and deliveries involve multiple people. For example, some of the best Marvel cinematic universe (MCU) movies were made thanks to the help of multiple production teams and partner companies coming together to build the powerhouse we know today. Not to mention, in my opinion, some of the best superhero movies of our time. We succeed with software delivery in the same way: by having fully responsible cross-functional delivery teams.

The best engineering teams I’ve been on are made up of independent and autonomous individuals who know how they can contribute to the work in progress. They’re also willing and unafraid to collaborate with others to get this work done.

Commitment to the Journey

“I am not here to be right. I am here to get it right.” – Brené Brown, from the book Dare to Lead.

Even with cross-functional delivery teams and creating with the end in mind, our teams still may not get everything right the first time around. Committing to DevOps is about accepting any failures that come during the journey. Committing to the journey doesn’t mean you can’t define what it involves, even if it’s out of the ordinary. Experimentation is also a part of how we, as developers, adapt to change and adopt novel approaches to delivering faster, cheaper and better software.

The DevOps Principles as Guides

Use these principles as the foundation for how to work together, but also use the right tools, and define and improve processes along the way. In photography, there are three components that serve as building blocks to the final images we capture: aperture, shutter speed and ISO. In DevOps, it’s our people, processes and technologies. Achieving high performance with DevOps involves all of these components working together and adjusting to our environment, situation or team.

The key to getting to the results you desire? Not getting blindsided by any single component of getting to that result.

The DevOps Practices and Tools

I like to consider this section the call to action; the one says to our teams and individuals, “Gear up!” No reasonable photographer would leave for a photoshoot without their camera and camera gear, right? Similarly, DevOps practitioners and teams need the right tools to get the desired results.

Source Code Management

Source code management is about allowing teams to collaborate, have version control, correct and revert software code. This is fundamental for many teams to increase not only visibility and collaboration, but also start automation workflows.

If you are an individual just starting out with writing code, I recommend learning the basics of Git, such as how to add, commit, push, merge, branch and clone code. If you’re a team, I recommend discussing the practices and standards you want to use for collaborating. Always fit your Git structure and workflow to your codebase and team dynamics.

Continuous Integration

What continuous integration (CI) does is take code away from local machines to package it into a deployable artifact in a CI server. What this is designed to do is automate the integration of newly committed code changes. This process is often triggered by code changes pushed into your Git repository. Using checks and other tools, teams can version and release artifacts that have already been vetted for quality and security.

Continuous Delivery

Continuous delivery, on the other hand, takes artifact code to production environments. CD aims to deliver code changes of all kinds to users in a fast, repeatable and safe manner. CI/CD consists of both CI and CD pipelines working together to deliver code to users.

Infrastructure-as-Code (IaC)

Infrastructure-as-code allows users to define hardware through code. We can improve how we provision, configure and maintain hardware systems. Infrastructure provisioning tools like HashiCorp’s Terraform and AWS Cloud Formation, configuration automation tools like Ansible, Puppet and Chef and package managers like Helm allow teams to accomplish much more when software scales and demand increases.

Secrets and Identity Management

Security is everyone’s responsibility. Secrets management and identity management tools make accomplishing DevSecOps easier by securing and managing sensitive information. Teams should consider how they are handling secrets and users within their architecture. HashiCorp Vault, KeyCloak, KMS and other similar tools can help you achieve your goals around security.

There are many other practices and tools DevOps teams and organizations can use to achieve their desired outcomes. What works for one team may not work for another — hence the need to innovate and experiment in a controlled and well-led manner. Other practices that can increase your team’s ability to deliver value include value stream mapping, progressive delivery, GitOps and chaos engineering.

People, processes, and technology belong together in a DevOps culture. While practice makes perfect, what’s more important than perfection is gathering the right answers along the way. Everything doesn’t need to be right the first time around. Just remember that no leader, individual or team in DevOps is alone. We started by defining what it means to build a DevOps culture, change mindsets and set principles and practices, so now I ask you: What does DevOps mean to you?

June 1, 2021
7 Step Transformation Blueprint for SecDevOps

The struggle for continuous software security is obvious, but the solutions are not.

As I indicated in my prior blog, SecDevOps is the Solution to Cybersecurity, a security-first mindset, coupled with SecDevOps-specific practices, provides an opportunity to achieve true continuous security. But, in reality, how can an organization accomplish SecDevOps?

This blog explains how to apply my 7 Step Transformation Blueprint to SecDevOps.

Leadership and engineering solutions require the persistent, methodical application of skills and practices toward leading and designing solutions that achieve business and team goals, including continuous security. While driven by visionary ideals, engineering requires practical, disciplined, progressively refined implementations using carefully chosen dimensions of people, process and technology solutions. At any point in the engineering life cycle, the goal is to achieve a balanced solution while evolving practices towards maturity.

The seven step transformation engineering blueprint prescribes seven steps for achieving and continuously refining digital transformation methodically, no matter what the goals or current level of maturity are. The seven steps are visioning, alignment, assessment, solution, realize, operationalize and expansion. Each step considers the people, process and technology aspects of the transformation.

Step One – Visioning

Top leaders define a strategic vision for the digital transformation for the organization including a motivating vision statement, measurable goals, team values, and major implementation tactics. Identify senior sponsors that will own the transformation at the strategic level. Include key partner organizations that need to be strategically aligned to the transformation. In a SecDevOps transformation, a vision for a security-first mindset and associated SecDevOps practices are called out as the highest priority for implementation tactics supporting the vision.

Step Two – Alignment

Leaders and key team members who are most important to the implementation of the transformation align specific measurable goals and tactics for selected “model” applications. Specific measurable goals around continuous security are set in this step.

Step Three – Assessment

For the current state of selected applications, capabilities are discovered and assessed, deep-dive assessments are conducted for specific topics, and a current state value stream map is created relative to the organization’s goals. My earlier blog, DevSecOps Practices Gap Assessment, explains my recommended approach for conducting an assessment for security.

Step Four – Solution

An expert team performs analysis of assessment data and formulates a future state value stream roadmap including themes, epics and user stories and obtains alignment with key stakeholders. My earlier blog, 9 Pillars of Continuous Security Best Practices, outlines a comprehensive set of practices to consider when building a roadmap for any continuous security solution.

Step Five – Realize

Proof of concept (POC) trials are conducted to validate solution choices. Trials of security tools and integrations of those tools into the SecDevOps platforms also would be conducted during this step. The solution is validated with selected applications and use cases. Training is conducted as the solution is deployed to production. Governance practices for the new solution are activated.

Step Six – Operationalize

Deployed improvements are monitored and controlled with metrics. Retrospectives are conducted to create actionable prioritized lessons learned for continuous improvement. Chris Tozzi’s article 6 DevSecOps Metrics for DevOps and Security Teams to Share suggested metrics that can be developed and leveraged, both for this step and to drive ongoing improvements as the use of SecDevOps practices expand.

Step Seven – Expansion

Once continuous flow (the first way of DevOps) is realized for a select set of applications, the organization can safely expand the solution(s) to other applications across the organization. Further transformation cycles will lead to realization of continuous feedback (the second way of DevOps) and continuous improvement (the third way of DevOps) and apply it to SecDevOps.

What This Means

SecDevOps strategies and solutions are complex. The seven step transformation blueprint described in this blog can help organizations build a strategy and implement SecDevOps as an important part of their digital transformation. To learn more about how to apply the blueprint to continuous security (and other key elements of digital transformation) refer to my book Engineering DevOps.

May 28, 2021
How to Move Fast Without Breaking Security

“Move fast and break things,” is a familiar motto. The phrase, attributed to Facebook CEO Mark Zuckerberg, helps to explain the company’s stellar growth over the past decade, driven by its product innovations. However, while it’s a useful philosophy for software development, moving faster than you’d planned is a risky approach in other areas, as organizations globally realized during the COVID-19 pandemic. While 2020 saw digital transformation program timelines advance by up to seven years, enterprises’ quick moves to the cloud also meant that some things got damaged along the way – including security.

Our recent survey conducted with the Cloud Security Alliance showed that more than half of organizations are now running over 41% of their workloads in public clouds, compared to just one quarter in 2019, and this will increase further by the end of 2021. Enterprises are moving fast to the cloud, but they are also finding that things are getting broken during this process.

The research showed 11% of organizations reported a cloud security incident in the past year, with the three most common causes being cloud provider issues (26%), security misconfigurations (22%) and attacks such as denial of service exploits (20%). In terms of the business impact of these disruptive cloud outages, 24% said it took up to three hours to restore operations, and for 26% it took over half a day.

As a result, it’s no surprise that organizations have significant concerns about enforcing and managing security in the cloud. Their leading concerns were maintaining overall network security, a lack of cloud expertise, problems when migrating workloads to the cloud and insufficient staff to manage their expanded cloud environments. So, what are the root causes of these cloud security concerns and challenges, and how should enterprises address them?

Confusion Over Cloud Control

When asked about which aspects of security worried them most when running applications in public clouds, respondents overwhelmingly cited getting clear visibility of topologies and policies for the entire hybrid network estate, followed by the ability to detect risks and misconfigurations.

A key reason for these concerns is that organizations are using a range of different controls to manage cloud security as part of their application orchestration; 52% use cloud-native tools and 50% reported using orchestration and configuration management tools such as Ansible, Chef and Puppet. However, nearly a third (29%) said they use manual processes to manage cloud security.

In addition, there’s competition for overall control over cloud security: 35% of respondents said their security operations team managed cloud security, followed by the cloud team (18%) and IT operations (16%). Other teams, such as network operations, DevOps and application owners, all figured in, too. Having different teams using multiple different controls for security limits overall visibility across the hybrid cloud environment, and also adds significant complexity and management overhead to security processes. Any time you need to make a change, you need to duplicate the work across each of these different controls and teams. This results in security holes and the types of misconfiguration-based incidents and outages we mentioned earlier.

How to Move Fast and Not Break Things

So how can organizations address these security and management issues, and get consistent control over their cloud and on-premises environments, so they can take full advantage of cloud agility and scalability without compromising on security? Here are the four key steps:

Get holistic visibility across your networks: a lack of visibility across cloud estates was the main security challenge for survey respondents. Organizations need to get holistic visibility across all of their different cloud accounts, including the security controls that are being used both in the cloud and to manage traffic to and from the cloud environments. This enables IT and security teams to quickly identify any potential security issues or risks that could lead to outages.

One management console to rule them all: When using a mix of the cloud providers’ security controls as well as other controls, both in the cloud and on-premises, it’s a huge challenge for organizations to manage policies consistently. If all these diverse security controls are managed from a single console using a single set of commands and syntax, security policies can be applied consistently. This avoids duplication of effort and the error-prone manual processes that lead to misconfigurations and outages. It also provides a full audit trail of every change.

Automate security and compliance processes for speed: Automation is essential to managing cloud environments efficiently while orchestrating changes across a complex array of security controls. It brings speed and accuracy to managing security changes across cloud environments. It also accelerates audit preparation and helps ensure continuous compliance. Automation also helps organizations overcome skills gaps and staffing limitations, which were cited as a challenge by over three-quarters of our survey respondents.

Accelerate incident response, anywhere: Our survey also showed organizations are concerned about overall network security in the cloud. Security management solutions that integrate with SIEM or SOAR tools can help to address these concerns by accelerating incident response. As soon as a threat is detected by the SIEM or SOAR product, the management solution can identify all the applications and servers that the threat could impact. The solution can then mitigate the threat’s risk by automatically isolating any affected servers or devices from the network – whether in the cloud or on-premises – preventing lateral movement.

With a network security automation solution handling these steps, organizations can get holistic, single-console security management across all of their public cloud accounts, as well as their private cloud and on-premises deployments. This helps them to solve the cloud complexity challenge and ensures faster, safer and more compliant cloud management – making it possible for organizations to move fast in response to changing business needs without breaking things.

May 27, 2021
Unifying Partner Ecosystems With a Distributed Ledger
Blockchain has become famous in recent years for powering the cryptocurrency craze. But the fundamentals behind blockchain could prove helpful in other areas of the data economy, too. Because, as the value of data rises, so does the need for accurate, real-time data sharing across multiple partners.

As we saw recently, global trade is surprisingly fickle. If a 1,300 ft long cargo ship can end up blocking the Suez Canal for an entire week, imagine how many data record incongruities the shipping and logistics industries have to sort out each day. Similarly, other partner networks, such as food distribution or airline affiliates, for example, must constantly share data back and forth to remain in business.

Since a distributed ledger is external yet sharable with partners, it opens some interesting data-sharing possibilities. Similar to merges for collaborative software, partners would retrieve and manipulate the distributed ledger, which would act as a source of truth for status on a physical object or a piece of data. Essentially, this process could reduce manual reconciliation efforts and avoid disputes when something goes wrong.

I recently spoke with “father of serverless” and AWS Lambda inventor Tim Wagner. Wagner, now heading Vendia, shared with me why he foresees distributed ledgers as the future for real-time, business-to-business data sharing. “Supply chains are surprisingly brittle,” Wagner said. To solve this problem, we need an “automation of that workflow that isn’t centralized any longer,” he said.

Role of a Distributed Ledger

A distributed ledger could be especially relevant for a multi-party supply chain. Within multi-partner agreements, data tracking is complex, and many partners struggle to come together. “Companies are struggling for a lack of truth,” said Wagner. “Everyone wants their own copy of the data, but they want it to be locked to what everyone is seeing at the same time.”

For example, take supply chain management. Suppose an automotive parts manufacturer produces a chassis. They send it to a shipping logistics provider, who then sends it to an automaker to complete assembly of an automobile. If the chassis breaks at some point during these handoffs, whose fault is it? Without a system of record, you open a big problem of distrust, Wagner explained.

Or, consider an airline ecosystem. Partner airlines must often coordinate multi-carrier flight plans, apply discounts or settle unused travel vouchers. Often, it’s difficult to track the journey of these transactions between organizations. An externalized ledger could help partner airlines follow the route of a single passenger, helping to ease financial settlements.

“It’s not just about sharing, which is hard enough, but it’s really about sharing with control,” said Wagner. These days, tracing the lineage of data is critical, as is having real-time visibility into an object’s state. A distributed ledger could help these partner ecosystems have a record they can all trust. Wagner describes this as a tamper-proof, immutable record — an “unassailable source of truth.”

Would This Replace Partner APIs?

What’s especially interesting to the integration-minded is that the distributed ledger concept could supplant established business-to-business communication models. Over the last couple of decades, most partner integrations have moved to RESTful API connections. One company opens up data access, and another authenticated partner makes HTTP API calls to integrate data into their own application.

Yet, the issue here is that the API is only a layer on top of a private database owned by the provider. Located behind closed doors, there’s no industry-wide real-time data synchronization. API calls are good for data retrieval and manipulation, but stringing a sequence of thousands of transactions together is difficult, described Wagner. Retaining the time-series integrity of a complex database is a little too much for synchronous calls to handle.

Instead of opening access to an internal platform via API, Wagner believes that a distributed ledger is a better fit for certain business-to-business machine automation. “Any place where you see API-level protocols developed to maturity is a good sign; there is a good chance distributed ledgers will find greater speed,” he said. “Trying to pull that information through APIs and pray you’re finding the right data is difficult to manage.”

For example, Wagner referred to how some animal shelters are creating a global repository to avoid siloed records. In doing so, they are making a global animal rescue schema for a unified database that could hopefully help every pet find a home.

Communicating with the ledger would still involve calls, but likely not your typical RESTful requests. Wagner advocates for the use of GraphQL as a better method to involve clients around changes. With a GraphQL API, stakeholders can query, mutate and subscribe against changes in a distributed system. GraphQL isn’t perfect, but its easy data mechanics make it a Goldilocks-style middle ground between REST and SQL, said Wagner.

Getting Buy-In

A synchronized distributed ledger could help resolve disputes and reduce manual checks. But, how do we get buy-in? How can you convince an entire partner ecosystem to align around a distributed ledger as a source of truth?

Wagner says there is usually a leader or single manufacturer leading the charge. This figure is likely a more prominent, well-respected player in the process. But for a digital ledger system to be successful, it must also satisfy other criteria too, Wagner said:
- Data schema forethought: Agreeing upon a shared data schema may be a challenge. Ecosystems will require forethought when designing a data model for the long haul.
- Compliance: There are many new data regulations to consider, such as GDPR, CCPA and PCI. “You can’t blindly copy data the way you used to,” said Wagner.
- Governance: As enterprises deal with data crossing boundaries, they must apply governance measures to satisfy compliance and maintain security. This governance needs to be tightly associated with data and offer fine-grained control.
- High performance: A distributed ledger requires operational flexibility for large objects, such as a 5 TB file, noted Wagner.
- Redaction capabilities: Whereas some partners must see all fields, others may, by law, be limited in what they can view. Thus, a ledger must include the ability to redact fields. Wagner described this as “intentional redaction or visibility control of individual customer records.”
Of course, different industries are at very different points along their digital journey, some more advanced than others. Change may be easier for, say, financial services supporting ACH payment technology, which Wagner described as being “ripe for natural organic expansion into a data-centric methodology.”

The Poor Man’s Blockchain

Tapped into the same feed, an extensive partner ecosystem could use a blockchain-esque arrangement to stay on the same page. This could help reduce manual reconciliation typically handled through email or over the phone.

Yet, as distributed systems experts are few and far between, implementing such technologies will most likely involve a cloud platform solution, said Wagner. He jokingly calls this the “poor man’s blockchain.” Such a solution must be cloud-friendly, enterprise-focused and easy to integrate and deploy.

As more and more critical data begins to reside outside company walls, “organizations have to reconstitute that data to work with partners,” said Wagner. A distributed ledger could, in effect, act as a sort of outsourced centralized IT, enabling a data model for serverless environments.
May 27, 2021
JFrog Adds Private Distribution Network for App Binaries

At its online swampUP conference today, JFrog launched a Private Distribution Network (PDN) that makes it simpler and faster to distribute application binaries across an extended enterprise.

Yoav Landman, JFrog CTO, said the PDN provides the equivalent of a content delivery network (CDN) designed specifically for distributing application binaries using a peer-to-peer network managed by JFrog. Each binary distributed is an immutable signed release that provides visibility and traceability to validate the source destination as well as the integrity of the binary, added Landman. That approach essentially applies concepts from a blockchain network to the distribution of application binaries, noted Landman.

The PDN is based on patent-pending network optimization technology that employs caching, distribution nodes and lightweight accelerators that enable IT teams to distribute binaries through inbound-only firewalls or to environments that have limited connectivity, said Landman.

Finally, the PDN also comes with tools to analyze binaries for open source security vulnerabilities and compliance with license policies in addition to role-based access controls (RBAC).

Landman said the goal is to provide a way to distribute binaries via a single click all the way to edge computing platforms in a way that is tightly integrated with the continuous integration/continuous delivery (CI/CD) pipelines and software development life cycle (SDLC) processes a DevOps team already employs.

Currently available via a private beta, PDN is scheduled to become available in the third quarter 2021 with pricing based on a per-use model that requires no upfront costs.

For years now, JFrog has been making a case for applying DevOps best practices to the deployment of binaries rather than just managing source code. As enterprise IT environments become more distributed, the need to manage the deployment of binaries at scale becomes more acute. IT teams are not going to be able to dispatch software engineers to manually install binaries on every edge computing platform employed. The goal is to provide IT teams with complete transparency into software distribution in a way that makes it simple to address compliance requirements, said Landman.

Most organizations already find deploying software to be a major challenge. CI processes are generally well understood as part of any application development process. CD has proven to be a more elusive goal, because each target platform on which software is destined to run tends to be unique, which makes automating the delivery process challenging. That issue is only going to be more difficult to address as organizations attempt to install software across hundreds, or even thousands, of edge computing platforms that might be based on a wide range of types of platforms. A PDN alleviates many of these issues by providing IT teams with a network to centrally manage the process, noted Landman.

At this point it’s now only a matter of time before most organizations confront this next major DevOps challenge. As organizations look to process and analyze data at the edge to drive any number of digital processes in near real-time, the amount of software deployed outside of a cloud or traditional data center is about to explode. The challenge now, of course, is finding the most efficient way to secure delivery of all that code.

May 26, 2021
Cattle, Not Kittens – Revisited

The early days of DevOps were filled with pronouncements that we needed to treat our servers as cattle, not pets (kittens). For a year or so, the refrain was everywhere. And we did. We detached servers from dedicated hardware, we made spin up and spin down so easy that we just replaced malfunctioning servers (which has caused its own problems), we have even made servers designed to spin up, do one thing well, then spin back down until needed again – something unthinkable even a decade ago.

And we’ve reaped benefits. Servers suck up less attention from the DevOPS side, and standardization has made it easier to code for them, moving things along and allowing members of another team to pitch in when there is a crunch because of familiarity.

But IT is filled with people, and something happened in the transition. In hindsight, we should have seen it as a possibility, but things changed so much, so fast that I don’t think anyone was considering this possibility.

Our DevOps tools/toolchains/processes became our pets (kittens).

Now the special snowflake that requires specific knowledge and has to be treated with kid gloves is the DevOps step, not the servers it uses. Like pet servers, a fragile step that is working is avoided like the plague, when it should be shored up. Like pet servers, a process that is locked into a specific tool, OS, or platform is left that way, even when other options are a better solution. Like pet servers, the person who “knows all about” that is left to manage the step that should, by all rights, be easily understandable by anyone on the DevOps team.

It’s time to start treating DevOps like cattle, not kittens. Some steps (like some servers) will be more difficult to make less pet-like and more cattle-like, but if history is any gauge, the payoff is worth the investment.

We don’t have a cross-toolset language, other than doing things in the various shells and calling the shell scripts from a given toolset (not the best solution, just an example of where we’re at), but that shouldn’t stop us from standardizing and genericizing as much as possible. Less fragility, more reusability and a broader support team is good for everyone.

You all are knocking it out of the park. We – all of us – are spending more time at work and at home in the digital space than ever before, and, generally speaking, the tools “just work.” That’s all on you. Keep kicking it. But standardizing to keep things running smoothly just makes sense. You know your org; it is possible that the easiest route to cattle is to standardize on a single toolchain (perhaps one or two toolchains with reusability documented) possibly to shove a bunch of steps into scripts and share amongst whatever tools each team chooses (I, and history, strongly discourage this approach, but neither I, nor history, has to live in your environment. There might be a good reason for this, so I include it here).

All I’m saying is, don’t re-implement every step over and over if you can avoid it, and don’t make steps so arcane that you need a specialist (not internal, and definitely not external). Keep rocking it, and get those cattle moving!

May 26, 2021
Oracle and Allies Launch ARM-Based Cloud Service

Oracle announced today it will make available for the first time ARM-based processors on the Oracle Cloud Infrastructure (OCI) that will also be supported as targets for deploying applications using GitHub, GitLab or Jenkins platforms. The latest Oracle service is based on processors from Ampere Computing.

The company also revealed today it has joined the Continuous Delivery Foundation (CDF), an arm of the Linux Foundation that oversees the development of open source DevOps platforms and tools such as Jenkins and Spinnaker.

Matt Leonard, vice president of compute for OCI, said the OCI Ampere A1 Compute provides an alternative to existing support for the x86-based processors that Oracle provides. That service will be offered at a cost of one cent per core hour with virtual machine sizes ranging from one to 80 OCPUs configured with one to 64 GB of memory per core or as a bare-metal service with 160 core and 1 TB of memory. Pricing per GB of RAM per hour is $0.0015. At that price point, Leonard said Oracle is now offering the lowest price per core of any cloud service provider.

The growing availability of ARM-based processors is starting to have an impact on cloud economics at a time when organizations are significantly increasing the number of applications being built and deployed on public clouds. Many of those organizations, to reduce overall costs, are now actively evaluating ARM-based services, noted Leonard.

To help jumpstart application development on the platform, Oracle is providing organizations with three classes of OCI Ampere A1 Compute services. An Oracle Cloud Free Tier gives developers $300 in free credits for 30 days. The Always Free ARM option gives developers four A1 cores and 24 GB memory, while an ARM Accelerator option provides open source developers, independent software vendors (ISV) partners, customers and universities with ARM-based development projects that need additional resources beyond what the Oracle Cloud Free Tier provides with credits for a 12-month period.

Oracle Linux, Java, MySQL, GraalVM and the Oracle Container Engine for Kubernetes (OKE) service are also all available on the OCI Ampere A1 Compute along with integrations with DevOps platforms from GitHub, GitLab and/or Jenkins. Other vendors providing support for OCI Ampere A1 Compute include SUSE, Datadog, OnSpecta, NGINX and Genymobile.

Oracle has also made available an Oracle Linux Cloud Developer image that enables DevOps teams to install, configure and launch a development environment via an OCI console that includes OCI client tools, utilities and common programming languages such as Java, GraalVM, Python, PHP, Node.js, Go and C/C++.

Finally, each Ampere core is single threaded by design with its own 64 KB L1 I-cache, 64 KB L1 D-cache and a huge one MB L2 D-cache to eliminate any ‘noisy neighbor’ issues that IT organizations might otherwise encounter.

Oracle is clearly betting that ARM-based processors will present the company with an opportunity to gain share at the expense of larger infrastructure-as-a-service (IaaS) rivals such as Amazon Web Services (AWS) and Microsoft Azure. It’s not clear to what degree IT teams might employ a different cloud service just to access ARM-based processors. However, in a world where IT teams now routinely employ multiple clouds, as far as Oracle is concerned, the playing field has never been more level.

May 25, 2021
Three Ways Machine Learning Can Change Incident Management

Incident management for important applications has many connotations, many of them negative. Most are well-deserved. Stress, panic, hurry, urgent, drop-everything, all-hands-on-deck, emergency, to name a few. Typically, there is a critical problem, ranging from complete failure to loss of certain functionality or performance. The problem or outage potentially means loss of revenue, loss of customers, negative social media commentary or bad reviews and more. Time is of the essence, dictating that the primary objective is fixing the incident as quickly as possible. Often, this means that symptoms or partial causes are mainly addressed rather than the underlying root cause.

Without fully addressing the underlying issue, companies virtually guarantee that the same problem – or a similar one – will reoccur. Not identifying root cause often prevents a durable fix. In addition, companies lose the opportunity to proactively improve application code or infrastructure based on real-world experiences and issues. Postmortems may only result in reviews of monitoring and observability solutions and the inevitable updates to alert rules. Most DevOps professionals not only understand but have lived through these frustrations on an ongoing basis. Management, then, often wonders why their systems are so unstable.

Changing the model for incident management has been limited by a combination of the overriding urgency combined with short-staffed, overworked teams. Although AI and machine learning has been positioned as the panacea for nearly every kind of technical ill, this is a clear case where “machines” could fundamentally enhance human efforts to improve a situation. The best troubleshooters exhibit a combination of instinct, experience and patience to carefully sift through reams of data, spotting unusual events and their correlation with bad outcomes. This turns out to be a perfect application for machine learning.

By ingesting all logs and metrics to find meaningful problems or anomalies, systematized machine learning can quickly and comprehensively sift through all data to gain a thorough understanding of the real problem. Using this approach significantly speeds up the root cause analysis. Such an applied solution introduces three game-changers for incident management.

Three Game-Changers for Incident Management

First, incident remediation times can be reduced dramatically and with less of an all-hands-on deck, drop-everything frenzy. Rather than SREs and engineers spending hours digging through dashboards, traces and millions of log lines, an automated approach can accomplish it more quickly, more thoroughly and with less impact on human professionals, allowing them to focus on other responsibilities. This instantly relieves stress and panic and alleviates pressure because incidents can be remedied better and faster, restoring the application to its expected service level. The value is, of course, two-fold: faster time to resolution and greater efficiency or productivity of the teams who would otherwise be involved. Ending or mitigating the frustration professionals have to deal with is an additional bonus.

Second, besides faster MTTR, an automated, machine-learning-driven approach is more comprehensive and gets to the root problem. This means that future incidents may be avoided altogether. This also means a potential end to the alert rule “hamster wheel.” The knee-jerk tendency to continually adjust alert rules and settings to enable better “early warning systems” is far less necessary. Instead of trying to gain an early indication of a similar failure in the future, the cause of such failures can be found and addressed proactively. Eliminating such problems in the first place is far better than trying to evolve alerts to catch unresolved causes. Alerts are still important, but machine learning can shift the basis for them to more root-level issues, and they can be constructed automatically by the system.

Third, automated, machine-learning-based incident management can help proactively catch silent bugs that may not yet directly be part of an incident. It can also inform developers earlier in the cycle. Not terribly long ago, new releases were tested extensively before deploying to production, allowing for the creation of specific test plans, conducting stress tests and potentially catching bugs that might have potentially terrible consequences in the future. Now with faster, non-stop development cycles, testing mainly occurs during production. Machine learning can surface new or unusual errors, event patterns and patterns in metrics. The approach can proactively surface subtle bugs early, before they impact users or cause widespread failure.

Changing the DevOps Life Cycle

Each of these three results significantly changes not only the incident management life cycle, but also DevOps as a whole. Poring over logs and metrics is a perfect application for machine learning and offers the opportunity to fundamentally change incident management, transforming it from a scourge and necessary burden to a competency. An automated system of machine learning applied to incident management advances DevOps rather than encumbers it.

May 21, 2021
Implementing an On-Demand Feature Environment
Not long ago, developers at LeanTaas implemented an on-demand feature environment. It came about as the result of a collaborative effort to support a faster go-to-market strategy. Moving to an on-demand feature environment resulted in greater agility, effectiveness, quality and a better bottom line for the organization. With careful implementation, any development team can benefit from this exercise.

With an on-demand feature environment, you can easily test specific features without disrupting development activities that already are in process. Setting up the environment can offer several benefits and enhance the overall outcomes of your product release. Here, we’ll walk through the steps we took to get this implemented.

First, we decided to come up with some standards and changes in our development practices:

Naming Conventions

Every Git branch name should start with the associated Jira ticket and include details about what that branch is for. We updated the Jira ticket with environment URLs, automated QA test results and comments from product manager reviews.

Syntax:

<JIRA_TICKET>-<SHORT_TITLE>-<PRODUCT_NAME>.<DOMAIN_NAME>.com

For example, LT-123-demo-product1.xxxx.com

In this case, LT-123 is the Jira ticket, demo is the short title, product1 is the product name and xxxx is the domain name. So, our feature branch URL would be accessible at https://lt-123-demo.product1.xxxx.com, and the internal name would be dev-product1-lt-123-demo, which developers can use to ssh and debug issues (if any) and refer to when they check logs (if necessary).

Also, following a set naming pattern helps us write a regular expression in a front-facing server environment (we use NginX) that helps proxy/route the requests to the respective feature-specific runtime environments deployed in an AWS EC2 instance.

If you are using NginX, you can use following regex to achieve this:

server_name ~^(?<featurebranch>.+)\-product1\.xxxx\.com$;

set $featurebranch dev-product1-$featurebranch

proxy_pass https://$featurebranch;

Data and Microservice Dependencies

For the environment to work in its own isolated state, it needs a data set stored in its own database. So our teams came up with an automation script for generating an anonymized sample dataset that’s required for this feature instance. We packaged this data, along with other required artifacts, into a golden Amazon Machine Image (AMI) for the specific product. For easy reference, we stored this custom AMI ID in the AWS systems manager (SSM) parameter store so that our Terraform scripts can use this reference while spawning an EC2 instance.

Once the feature-specific instance came up, we realized it couldn’t play well with our other microservices such as authentication, authorization or notification. They couldn’t differentiate between a real product instance versus a short-lived, feature-specific instance. So, we had to add an additional header in our internal requests, and enhanced these other microservices to be aware of this additional header so that they could exactly respond back to a feature specific instance. When it comes to usage of message services, every feature branch has its own SQS queue, which followed the same naming conventions of the internal hostname. Platform services relied on this same standard to ensure these feature branches worked seamlessly.

Automation/Manual Quality Checks and PM Reviews

Now that the feature-specific instance is up and running in its own isolated environment and with its own dataset, it enables a fully functional playground for the QA team and PM team to extensively test new features in an isolated fashion. Having this playground is a game changer for our productivity. It helped parallelize the iteration cycles. Each feature started getting its own feedback and progressing towards the finish line; essentially swimming in their own lane.

Cost Considerations

So far, so good. Feature-specific application instances are up and running and development iterations are progressing. But what is the lifetime of these instances? We don’t want to let them run forever; we also want to consider the fact that not every new feature will end up getting released. So, how do we handle such cases? We want to hook up these instance termination actions seamlessly into the developer’s work stream. Here’s how we solved for this:
- Delete the feature branches-related infrastructure as soon as the feature branch itself is deleted.
- Stop the EC2 instances every night and start them every morning by using a combination of AWS Cloud Watch Events and Lambda.
- Automatically delete the feature branches that are active for more than seven days. We later ended up providing an additional feature; skip flag, for some feature branches because sometimes we’d end up making changes that were complicated or revamping things which needed more time.
Notifications

We use Slack, so we decided to send Slack notifications every time a feature environment was created and attach test results, as a thread, to the same Slack message.

We also decided to use AWS tags and maintain branch_owner (developer name), branch_status (synced with Jira to indicate In-Dev, In-QA, PM Review, Code Review etc.) and tag EC2 instances. We also sent a daily report with the number of active branches, which helped in our daily standup and also provided a quick, at-a-glance overview of features that were ready for release.

For implementation of CI/CD pipelines, we created a new Jenkins pipeline that had the ability to perform some prereq checks, code quality checks, build applications into Docker containers, spin up infrastructure using Terraform and deploy the Docker container into newly created infrastructure using Ansible, run automation tests against the feature environment and send notifications in Slack as well as update Jira tickets with the build URL and test results.

Sample screenshot of our pipeline

Once our feature environment was certified, then a pull request is submitted to merge in Master, before merging code review is performed by other developers and engineering leads, and then, finally, merged into Master. Once the feature branch is merged into Master, it gets deleted via a webhook.

Now, code is built against the master branch and deployed into the master environment to ensure integrity. We perform one more round of QA using automated and manual checks, regression testing, internal penetration testing and performance testing. Once these are passed, the Docker image is finally promoted to production.

Following these steps to a seamless implementation of an on-demand environment will help any organization take the same steps towards a more efficient process. If you can apply these steps to your workflows, your metrics could improve exponentially.
May 20, 2021