devopsy.kinsta.cloud

Category: DevOps Practice

  • Stytch Launches New, Flexibility-First SDK

    Stytch Launches New, Flexibility-First SDK

    Passwordless solutions have been a trend for a while now, improving user experience (UX) while reducing exposure to common attack vectors. Now, Stytch’s new JavaScript SDK aims to make password-free authentication a bit easier by increasing customization and flexibility while offering fully baked authentication out of the box.

    Julianna Lamb, Stytch CTO, said the solution provides dual benefits by doing the heavy lifting of implementation while providing the flexibility to create a powerfully branded UX from the customer’s first sign-on experience. The Stytch JavaScript SDK can offer an all-in-one experience that handles the entire user auth journey with a pre-built frontend component. But it can also give teams complete control over designing a sophisticated and tailored authentication journey. In an era where tech companies chase the ultimate customer experience as their holy grail, this should appeal to teams wanting to improve a customer’s first impression of their brand, Lamb said.

    Multiple Options for a Smarter Front Door

    Like many authentication tools, Stytch’s product suite originally offered two options: One was a standard widget to handle frontend components while backend APIs provided logins. The other option provided direct API integration, in which the developers built their frontend, handled tasks like session management and setting cookies on the frontend and used the API to do things like send a one-time passcode or email a magic link.

    “But we found people really want flexibility when it comes to any solution they’re using,” Lamb said. “They also don’t want to do the heavy lifting when it comes to things like session management. Those things can be complex to reason through and having those out of the box is really compelling.”

    The new SDK decouples drop-in UIs from helper functions, giving teams multiple options. They can place Stytch’s UI components anywhere they want in the product, sculpting the authentication flow to their specifications. As a headless SDK, it can be embedded client-side in the application to handle all the calls to the API, do the authentication, create a session and check its validity. The result: Less backend code to write and less logic to implement.

    Customizable UI flows mean that teams can fully own the UX experience and build a UI that is customized exactly how they want it to look—and they can do this without having to build an internal API to route authentication requests from client to server. Teams looking for ways to make a brand impact right out of the gate can capitalize on this ability by creating a unique user experience.

    “Authentication is the front door to your application,” Lamb said. “Sign-up is the first impression the user has of your app—and the ability to customize it with your brand look and feel so the UI fits your aesthetic is super-critical.”

    Frictionless Flexibility in Identity Verification

    Because teams may have different authentication needs within one product, the SDK can be used in tandem with the core Stytch API to meet shifting needs. Teams could generate embedded magic links via the API while providing Google OAuth and Google One-Tap sign-in options via the SDK.

    The solution also reduces friction by making it easier to not only authenticate users but verify them on return visits to continue the same session. By loading the SDK on the frontend, teams can automatically connect Stytch’s sessions product to the browser cookie, feeding user session data directly into the cookie.

    The Stytch team is currently building out more SDK functionality and frontend components, Lamb said. Given the ongoing evolution of simple authentication tools that meet complex requirements, it will be interesting to see where innovative authentication goes next.

     

  • DevSecOps: Realities of Policy Management

    DevSecOps: Realities of Policy Management

    Policy management is essential to scale cloud environments and is key to secure DevOps practices. It enables organizations to manage policies put in place that secure the cloud environment, ensure Kubernetes configurations are secure and enable the continuous monitoring of a company’s overall security posture. 

     As businesses migrate workloads across multi-cloud architectures to achieve the agility and scalability needed to keep up with the pace of digital transformation, the growing demands placed upon developers often leave a gap that exposes potential threats and risks in the configuration settings, making security an even greater focus in DevOps. In fact, IDC’s 2020 survey found that 67% of breaches in the cloud are caused by misconfigured applications or infrastructure, including some of the industry’s largest breaches like Marriott’s second breach.

     To add to this, the first State of Policy Management report by Nirmata and the creators of the CNCF project Kyverno revealed that nearly 50% of users in cloud-native environments have now adopted some type of policy management solution. This tipping point of mainstream adoption across production cloud-native environments is making it critically important for DevOps teams to look at their practices and simplify and operationalize policy management across their Kubernetes stack by eliminating vulnerabilities through built-in, curated policies, without the barrier of learning complex policy languages. But this widespread adoption also acknowledges that organizations are finally realizing the need to put attention, investment and innovation into addressing security and compliance gaps as they adopt Kubernetes through proper DevSecOps practices and tools so applications being built can empower businesses.

    To confidently build cloud applications in Kubernetes, DevSecOps teams need to accept these DevSecOps realities and apply policy management effectively. 

     Customization Creates Security Risks

    Kubernetes can be highly customized, but DevSecOps teams need visibility into what’s happening in each cluster as organizations scale to ensure application reliability and security. Exploits on containers—including malware installation, cryptomining, host access and privilege escalation—offer opportunities for more security vulnerabilities. They can exist in images, production-accessible container registries, failed builds and third-party admission controllers in Kubernetes clusters.

    To address these risks in applications running on Kubernetes, it’s important to protect your environment with three A’s: Authentication, authorization and admission. This should be done at the cluster layer which enables secure access to authenticated entities that are authorized to perform certain actions. One way to accomplish this is through policy management. In fact, according to the State of Policy Management report, the top use case for policy management is for Kubernetes admission control (31%). When a request is authorized, having a policy in place ensures the request goes through another set of filters. For example, an authorized request may be rejected by an admission controller due to quotas or due to other higher-priority requests. In addition to validation, admission webhooks can also mutate incoming requests as a way of processing request objects for use before reaching the Kubernetes API server. 

    Configuration Issues

    Container images, namespaces, runtime privileges, persistent storage and control plane, together with network policies that are not compatible with best practices, are a source of misconfiguration and risk exposures. It’s this potential for greater risk exposures that have led configuration management to be a key driver of policy management adoption. In the Nirmata State of Policy Management report, it ranked third in security tools that are adopted today and fourth for the organization’s plans for future adoption.

    There are several ways to define security policies to protect cloud environments. Policies can be used for auditing purposes, to reject pod creation or to mutate the pod and limit what it can do. By default, pods can receive traffic from any source and send traffic to any destination. Network policies allow you to limit the ingress and egress access for your pods. The network policy typically translates to firewall rules.

    Lack of Adequate Authorization 

    Cloud authentication allows authorized users to securely access information stored in the cloud with authentication provided through cloud-based services. In The State of Policy Management report, key findings revealed that more than 24% of respondents today are using tools like policy management to authenticate users accessing applications. With data theft one of the fastest-growing and most expensive cybercrimes, the ability to process information and have strong authorization measures needs to be part of DevSecOps best practices.  

    In Kubernetes environments, as a user request is authenticated, it goes through an authorization workflow that decides if the request should be granted. It evaluates the request attributes against all policies and allows or denies the request. The main authorization mechanism is role-based access control (RBAC). Each authenticated request has an HTTP verb like GET, POST or DELETE, and authenticated entities have a role that allows or denies the request. Other authorization mechanisms include attribute-based access control (ABAC), node authorization and webhook mode.

    As the trend of moving security further to the left in the application life cycle accelerates, developers are being required to take active responsibility for it, which creates a higher desire for automation. The rapid releases can expose applications to many vulnerabilities, raising the risk of a breach.  With DevSecOps, organizations can transform the development pipeline by shifting security and compliance to the left, enabling developers to check the code before every commit. In doing so, security and compliance vulnerabilities can be identified and fixed in development, saving the organization from high costs and negative publicity from potential breaches. 

    There’s no doubt that Kubernetes is complex and challenging to adopt. On top of this, the development landscape is flooded with tools and more demands are being placed on engineers.  These shifts are changing the traditional role of a developer, leaving a gap for organizations to address to get the most out of DevOps. But by taking a secure, continuous, iterative approach that also includes policy management, companies can reduce the risk and exposure of their cloud environments. The more mature an organization’s DevOps practices are, then generally, the stronger the security and compliance practices are—they grow in relation to each other. The most innovative companies accept policy management in DevSecOps as a necessity of development and adopt some level of policy management best practices. With the right implementation tools like policy management and security and compliance best practices, organizations can remain efficient in agile and DevOps while adhering to critical go-live deadlines—which can’t be missed in this fast-paced business environment. 

  • Twitter/Bluesky ADX Algorithm | Cloud Energy Use ‘Tripled’ | Apple Staff are Revolting

    Twitter/Bluesky ADX Algorithm | Cloud Energy Use ‘Tripled’ | Apple Staff are Revolting

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Bluesky—Twitter’s research spinout—opens its ADX algorithm, Irish data center electricity worries grow, and Apple employees are the latest to rise up against hybrid working. (more…)

  • Pulumi Extends Infrastructure-as-Code Reach

    Pulumi Extends Infrastructure-as-Code Reach

    At its online PulumiUP conference, Pulumi announced it will make its Pulumi CrossCode translation technology available to third parties and DevOps teams. The technology was developed to integrate a wide range of programming languages with its infrastructure-as-code (IaC) platform.

    The company also announced it has added support for YAML—widely used to configure Kubernetes environments—as well as any variant of Java. It already supports .NET, Node.js, Go and Python programming languages.

    Finally, Pulumi also added packages for Oracle Cloud, Databricks and EventStore that simplify deployments on those platforms. The company already provides similar integrations for Amazon Web Services (AWS), Microsoft Azure, Google Cloud, Kubernetes, Auth0, CloudFlare, Confluent Cloud, Datadog, DigitalOcean, Docker, GitHub, Kong, MinIO, MongoDB Atlas, PagerDuty, Snowflake, Spot by NetApp and SumoLogic. There are also now components that provide support for container applications, Kubernetes clusters and serverless applications along with an AWS Cloud Development Kit (CDK).

    Pulumi has been making a case for an IaC approach that enables developers to use programming languages they already know to provision infrastructure as an alternative to having to deploy and master tools such as Terraform.

    The Pulumi CrossCode translation technology the company developed is now being made available to any organization that wants to add support for another programming language. It converts any infrastructure-as-code format, including Terraform, CloudFormation, Azure Resource Manager and Kubernetes configuration, to any programming languages supported by Pulumi.

    Pulumi CEO Joe Duffy said the goal is to extend the overall reach of the translation technology the company created to simplify provisioning any type of infrastructure.

    While many developers use tools such as Terraform, the need to limit misconfigurations is driving many larger enterprises to revisit their approach to IaC to improve software supply chain security, noted Duffy. Today, Pulumi claims more than 2,400 organizations have adopted platforms that enable developers to provision infrastructure and apply guardrail policies to reduce errors. That approach prevents developers from making configuration mistakes that can lead to exfiltration of data via, for example, a port that has been accidentally left open, he added.

    Too many developers assume that their cloud service provider is securing both the underlying platform and how it is configured only to discover later that it was their responsibility to validate those configurations. Security teams, meanwhile, can’t keep pace with the rate at which cloud infrastructure resources are being provisioned without some ability to apply policies that enforce rules for provisioning that infrastructure within the context of a larger DevSecOps workflow.

    Ultimately, the goal is to employ automation to resolve these issues once and for all as more organizations adopt DevSecOps best practices. In the meantime, the challenge is doing so in a way the average developer will accept and embrace. Otherwise, they will continue to find ways to work around security teams in the name of productivity, regardless of how insecure the underlying IT environment might become.

  • Internal Developer Portal: What it Is and Why You Need One

    Internal Developer Portal: What it Is and Why You Need One

    One main advantage of the cloud is that it offers developers flexibility and a self-service solution. Instead of having to contact the central infrastructure team whenever a dev team needs a new server or a cluster, the dev team can do it themselves through a simple web UI. This change drastically improves productivity. But there’s a caveat: Most companies don’t actually want to use the default portals provided by cloud providers. There are a few reasons for this: Security, simplicity or the need for custom, company-specific options, to name a few. That’s why we now see a move toward internal developer portals. In this post, you’ll learn what internal developer portals are and how they can make your developers’ lives easier. 

    What is an Internal Developer Portal?

    An internal developer portal is a self-service application and data store that lets developers and managers track and organize everything their engineering teams build and operate.

    Imagine that you need to create a new microservice. Your developers will first need to provision infrastructure for it; for that, they’ll use a cloud provider portal. Next, they’ll need to create new CI/CD pipelines or modify existing ones. Then, they’ll probably need to implement some changes to existing components like the API gateway and database. Lastly, they’ll probably need to add some documentation or make their new microservice visible to the rest of the company.

    Overall, they’ll need to touch a few different UIs or portals. Now, imagine they can do all of that from one place. That’s what we call an internal developer portal. It’s a self-service portal that lets developers catalog and understand relationships, ownership, resources, external service data and anything else they need to understand and operate their software.

     Why Does an Internal Developer Portal Matter?

    Now that we know what an internal developer portal is, let’s take a step back and discuss why internal developer platforms (IDPs) are on the rise. The main reason is because of the increased complexity of both software and infrastructure. Especially when we’re talking about cloud-based microservices environments, it’s challenging even for senior engineers to keep track of all dependencies and things that must happen every time changes are needed. Microservices architectures are heavily decentralized. Although you’ll still see the typical three-tier software architecture, in most cases, it’s no longer easy to tell by looking at a random component where it fits in the bigger picture just by its name or server location.

    Natural Evolution

    The second reason is that it’s simply a natural next step in the evolutionary process for self-service portals. Old-school, on-premises-based environments barely had any self-service capabilities. Do you remember how difficult that was? To request a new server, you had to email the central infrastructure team. If you were lucky, you’d get what you needed in a matter of hours or days. But more often than not, it took a week or two. And if you made some mistake in the initial request, that would cost you an extra week (or more) of waiting. More advanced companies? Even when they created some custom self-service solutions, these were limited. Often, it wasn’t much more than an online form to be processed by the infrastructure team which often became a bottleneck for all dev teams.

    When most companies started moving to the cloud, they began to see the benefits of self-service platforms. The ability to create servers ad hoc within a matter of minutes led to increased developer productivity, faster time-to-market and better quality software. But again, cloud-provider platforms are suitable for managing cloud infrastructure; you need more than that for a complete solution. Your developers can be more effective at creating and changing cloud infrastructure, but everything else was still done separately. And that’s how we naturally evolved into the demand for internal developer portals. To bring even more flexibility and clarity while increasing your dev teams’ productivity, you need to take the next step in the self-service platform evolution and combine all the things developers need into one internal developer portal.

    Clarity 

    Also, from a business perspective, internal developer portals bring clarity and offer a good overview of all your infrastructure and software development efforts. Even if you’re in the cloud and your dev teams can manage the infrastructure as they choose, you still need some control and oversight—for example, to avoid wasting money on test servers that developers forgot to delete. Sure, you can get simple reporting from your standard cloud-provider portal, but what if you’re using multiple cloud providers or have a hybrid-cloud environment? An internal developer portal can benefit both developers and their managers. Long story short: IDP portals are the future of software development.

    What Can an IDP Actually Do?

    How do you create an IDP for your company? You can build your own from scratch, use some OSS offerings or go for ready-to-use products.

    Build Your Own

    The first option would give you the most flexibility and ability to perfectly tailor the solution to your needs and the needs of your developers. However, building your own IDP from scratch is almost like building your own operating system from scratch. It requires a lot of work which could negate the main benefits that IDP is supposed to bring. You want to implement IDP to increase developer productivity and decrease time-to-market, but it would take you months to build it and years to maintain it to keep it working properly with constantly changing cloud APIs.

    Open Source IDP

    The next option is open source IDPs. Since IDPs are, in general, quite complicated pieces of software, there aren’t many options to choose from. The two most popular are Backstage and Gimlet. Backstage is more of a DIY platform for building developer portals, which means you’ll still need to spend quite a lot of time creating something that actually benefits your company. Gimlet is quite popular, too, but it focuses on Kubernetes-based environments. Both are great tools—don’t get me wrong—but if you’re looking for something ready to use and feature-rich, consider the third option.

    Ready-to-Use Solutions

    The last option is to use a ready-to-use product. Ready-to-use solutions combine the IDP functionality with incident response capabilities as well as metrics, dependencies visualization and filtering across multiple clouds.

    Internal developer portals are gaining popularity for good reason. In fact, many companies have been trying to build IDPs for years without even knowing the solution had a name. This is because an IDP-like solution will naturally evolve in any company that cares about developer experience and moving faster. The problem is that it usually takes years to come up with a single solution that fulfills all their needs. Another problem is that most of these organically created internal developer portals focus only on the tech side and skip the business needs. That’s why a purpose-built tool can be much easier and faster to implement while also doing its job much better.

  • Smoothing the Transition From REST to GraphQL

    Smoothing the Transition From REST to GraphQL

    Since its open source release in 2015, GraphQL has taken the development world by storm. GraphQL, the query language originally developed by Facebook, is a big boon for frontend developers, as it enables you to grab the exact data you need with a single request. This solves the problems of over-fetching and under-fetching, which classical REST APIs are prone to. GraphQL also offers a nice developer experience with a consistent schema and documentation. Some are even treating GraphQL as a convenient meta layer to aggregate multiple underlying services and abstract their interfaces with simple commands.

    For these reasons and more, many development teams are considering switching from REST to GraphQL. Even enterprises are beginning to use GraphQL in large production environments. But, how can these teams make the switch without breaking existing integrations? How can we ensure that things like developer experience, security, and access control all remain stable?

    I recently met with Anant Jhingran, the co-founder of StepZen, to discover tips for transitioning from REST to GraphQL. According to Jhingran, most organizations see the benefit in adopting GraphQL but are hesitant to forgo years of investment into RESTful architectures. But according to Jhingran, it’s rarely a case of “either/or”—more often, he sees projects running both styles simultaneously or layering one on top of the other. Below, we’ll explore some things to keep in mind as you invest in GraphQL.

    The Decade of Developers

    We’ve reached ubiquity of web APIs among companies big and small. “APIs have turned out to be phenomenal for software supply chains,” said Jhingran, the former CTO of Apigee. They help share data between partners and can enable platform ecosystems to thrive. Jhingran describes how within the last decade, APIs and API management have peaked to near-ubiquity in terms of adoption.

    But not all APIs behave the same way. Although representational state transfer (REST) is the most popular style, it’s still quite a general architectural guideline for web APIs—how the data comes out of each service is still highly nuanced. The structure of requests and responses are different from service to service, causing developer experience hangups at scale.

    GraphQL, on the other hand, “is a fantastic way of structuring data,” said Jhingran. With a single call, you can return the explicit information that the client requires in an expected format. For an e-commerce application, for example, this might be customer information, order numbers and delivery status. Displaying this all with one call is an awesome experience for frontend developers. And, GraphQL APIs are easy to explore and navigate since the schema is baked in. This improves upon REST APIs, where documentation may not always match the actual production implementation.

    “This decade is the decade of developers,” said Jhingran. Satisfying developer experience has become more mission-critical as enterprises work to increase their footprint in the cloud marketplace and GraphQL is part of this movement. “For people who build apps, it’s just better,” said Jhingran.

    But just because frontend teams clamor for the ease of use of GraphQL doesn’t mean backend teams can easily restructure their entire framework around it. Doing it right requires many skills, and the journey can be arduous, explained Jhingran.

    It’s Rarely REST Vs. GraphQL

    REST and GraphQL are often pitted against one another to the point that they seem like polar opposites. But in practice, teams often use GraphQL to build on top of APIs already in place. Jhingran describes the most common pattern as “GraphQL comes in as a layer that connects the dots between various REST APIs.” Either GraphQL is consumed directly or as a REST layer above GraphQL. “It could go either way,” he said.

    GraphQL is a fantastic data infrastructure tier as it can mix and match database backends. It could, in essence, become the new plumbing upon which applications are constructed. But while GraphQL may be excellent for internal construction, the name of the game is stability when externalizing services to the outside world. With REST, the big goal is to decrease broken clients and ensure high-grade security for each resource. Therefore, Jhingran foresees certain use cases where using a REST wrapper around GraphQL services to deliver fixed JSON responses would be preferred.

    Another point is that GraphQL is not only relegated to appeasing frontend developers—it has some backend potential, too. “GraphQL is an opportunity to reimagine frontend and GraphQL is an opportunity to reimagine backend infrastructure,” said Jhingran. This concept has weight nowadays, as Netflix engineers, for example, use GraphQL as a backend for microservices architecture.

    Making the Transition From REST to GraphQL

    Not all apps are going to be transformed on day one. Teams may encounter cultural resistance to new technologies. Interface redesigns, especially for those external apps which are exposed to third parties, must also require extra precaution to maintain app compatibility to avoid breaking change.

    Thus, it may take time to transition effectively. “Getting it started versus getting it right are not the same thing,” said Jhingran. He said people who get started with GraphQL may struggle for months due to competing best practices and tools. For example, Jhingran identified two main perspectives when approaching GraphQL: Programming and declarative.

    • Programming: Teams use GraphQL libraries such as Apollo to program their way out of your GraphQL endpoint. In this approach, you just start writing. It’s quick to get started, said Jhingran, but you end up describing everything you do and encountering many contingencies along the way.
    • Declarative: The other option is a more declarative approach—instead of telling the system how, you tell it what to do. For example, this subsystem should do X, this subsystem should do Y and so on. According to Jhingran, adopting tooling with declarative snippets to create GraphQL endpoints is “better in the short and long run.”

    GraphQL Security

    A March 2022 report from Salt Labs found a shocking 681% increase in API-based cyberattacks over the previous 12 months. There are many concerns surrounding API security right now—these valuable endpoints are prone to broken access control, data overexposure and over-permissive states.

    As I’ve covered previously, GraphQL security is a bit nuanced compared to REST. Within the REST paradigm, you can easily assign permissions and rate limit calls per resource. But with GraphQL, it’s not that simple. Since you can batch multiple queries, callers have much more power at their fingertips, and hackers might abuse such power if the proper authorization isn’t in place.

    Companies may rightly worry about opening another can of worms with GraphQL. But according to Jhingran, these worries are all solvable. A comprehensive GraphQL endpoint will likely require another layer to uniquely segment authorization to retain a least-privilege model for all resources and backend endpoints.

    Future of GraphQL

    To date, GraphQL has gained modest prominence. But, Jhingran foresees an incoming inflection point in the next two to three years. “The dominoes are beginning to fall,” he said.

    Now that the market is saturated with APIs, characteristics outside of functionality will be what sets a service apart and gives it a competitive edge. This includes things like performance, security and developer experience, the latter of which makes GraphQL an attractive proposition. This, paired with the explosion in GraphQL tools, will undoubtedly help developers perform their jobs better.

    Looking to the future, Jhingran predicts increased usability and abstraction in how developers work with GraphQL. He compared it to how disjointed pages on the web became interconnected and searchable with links. We see similar thinking happening in the world of APIs, he explained, in which the disjointed APIs require a better connection. Similar to how Google simplified search, he foresees GraphQL and the tools that work with it helping to decrease cognitive load by abstracting complexity with declarative commands.

  • Git for Mainframe: Back to the Future 

    Git for Mainframe: Back to the Future 

    Software delivery modernization should be done with care, especially with mission-critical mainframe apps. But for some, modernization will feel like a trip back to the future.

    Mainframe applications are typically systems of record with high-value transactions, significant compliance risk and an essential need for stability and security. They’ve been running reliably, often for decades, with code changes made by teams of dedicated developers using time-tested interfaces like ISPF (green screens) and Eclipse integrated with mainframe-based source management tools—Endevor is the most widely used example.

    So Why Modernize?

    Three primary drivers of change are present:

    • Business Agility: Digital disruption is generating an increased volume and velocity of business demands often resulting in cross-platform application architectures. It’s increasingly common to have cloud/mobile customer-facing apps calling back-end mainframe services for seamless experiences.
    • Staffing: A talent crunch resulting from career mainframers retiring while traditional mainframe practices and tools lack appeal for the next generation. Backfilling is a challenge as next-gens would rather be using popular, cloud-native tools that offer broader career potential.
    • Performance: DevOps adoption has raised the bar in terms of productivity, delivery velocity, quality, lead times, etc. The opportunity to achieve comparable performance levels with mainframe development is entirely possible but requires modernization.

    The confluence of these drivers leads to an obvious conclusion—the time for modernizing mainframe application development is now. Version control is the foundation of software development, acting as the focal point with IDEs/code editors, Agile tools, DevOps automation tools (like code scanners and CI/CD orchestrators) and service management tools. This article focuses specifically on the adoption of Git as a critical enabler of modernization.

    The Role of Git

    The next generation of talent, typically millenials and Gen Zs, know and love Git. With 94% of professional developers viewing Git as a fundamental development tool per Stack Overflow, it has become the de facto standard for enterprise IT, eclipsing all alternatives. Built initially for open source development of Linux, it offers a decentralized model where devs can work independently in branches yet collaborate closely through pull requests, issue management, etc.

    It has become so dominant that popular editors like VS Code have built-in Git capabilities. With the introduction of the free Code4z extension pack by Broadcom, VS Code and compatible tools (e.g., cloud-hosted GitHub Codespaces) are now viable options for mainframe developers, making Git adoption a natural, ‘parallel’ modernization opportunity. The Developer Cockpit Simulator illustrates the Git-ready experience using VS Code for mainframe application development.

    Back to the Future

    Beyond version control, leading Git platforms like GitHub and GitLab have expanded to become comprehensive DevOps platforms (e.g., GitHub Issues, Actions, Codespaces, Projects, etc.).  Yet the precedent for this evolution was set decades ago when Endevor was envisioned as an environment for development and operations with SCM at its core. Yes, DevOps before ‘DevOps’ ever existed. For many mainframers, seeing the Git version control tools evolve into comprehensive platforms comes with a distinct feeling of déja vu.

    With automation that has stood the test of time, Endevor delivers code changes reliably and, with workflows optimized for the platform, does so very efficiently and securely. And while Endevor is continuously updated with new DevOps capabilities—including the recently introduced Dynamic Environments to facilitate developer self-service—there is no denying that Git is valuable from an enterprise perspective. In addition to attracting next-gens and facilitating onboarding, a Git experience empowers any developer, if authorized, to easily update mainframe apps. All they need is a few mainframe basics and experience with the relevant languages (e.g., COBOL, PL/I).

    Back when the mainframe was the sole enterprise platform, a single, common experience was the norm, but IT evolved along many dimensions resulting in a sometimes disparate and siloed environment. Today, however, most developers are comfortable using multiple languages in a hybrid environment, so normalizing around a modern developer experience makes sense and brings things full circle.

    Git-Native Approach

    One option to take advantage of Git is to move source code to a Git enterprise server, but the inherent costs and risks associated with this approach should be carefully considered. Career mainframers are some of the most productive developers around and moving to a new paradigm will naturally be disruptive. In addition to changing workflows, this approach reestablishes build and deployment automation that has been working reliably for decades, which is both time-consuming and risky. Mainframe applications are often monolithic and complex with career mainframers’ expertise often filling gaps in documentation, so preserving this institutional knowledge should be a priority.

    Career mainframers can, of course, learn Git, but training takes time. And as many are nearing the end of their careers, will they be motivated to embrace the change? Repo migration may have the unintended consequence of accelerating the retirement of a critical knowledge base. As Deloitte forewarns in their Modernization and the Aging Workforce paper, “Don’t underestimate the value of legacy coders.”

    There are also costs associated with the migration—testing, documentation, training, project management—so the opportunity cost of porting code rather than investing in other high-priority business initiatives further qualifies this option.

    With these factors taken into consideration, migration should start with applications that are both stable (e.g., infrequent change requests) and pose a low risk to the business (e.g., lower dependence on career mainframers).

    A Git-native approach would require a specific tool such as Team Build, for example; a lightweight processor-based build engine that enables teams to collaborate natively using enterprise Git repositories while building on the z/OS platform. It is independent of Endevor and can be used for any build-and-ship needs for z/OS. It converts build logic in JCL and makes it maintainable off-platform with JavaScript.

    Hybrid Git Experience

    This ‘risk-managed’ option offers the benefits of Git—easier recruiting and onboarding, increased collaboration, etc.—while eliminating many of the costs and risks outlined above. With it, career mainframers continue working as they always have with existing tools and practices, without disruption, while next-gens use the tool they prefer, Git.

    Endevor Bridge for Git enables the Git experience with Endevor continuing to serve as the main repository. It accomplishes this by synchronizing a ‘source control mirror’ on the Git enterprise server. With this option, those who prefer to use Git can do so while colleagues using native Endevor can work on the same codebase at the same time. They can use Eclipse, VS Code or VS Code-compatible cloud-based IDEs, which offer the added benefit of working in Git without a local copy of the code.

    And with enablers like the Zowe CLI, modernizing-in-place offers the best of both worlds: Mainframe and off-platform tooling. For example, run Jenkins CI/CD pipelines for code scanning, testing and other functions off-host following a code change while allowing developers to continue to use the ISPF interface. To summarize, this option offers the advantages of the Git-native model without repo migration.

    Recommendations

    Due to the nature of mainframe applications, risk management is usually paramount so a thoughtful, pragmatic approach to the adoption of Git is recommended. Provide those career mainframers that are open to adoption with the time they need to embrace the technology by introducing Bridge for Git, for example. As organizations become more comfortable with the hybrid model and are satisfied they have enough Git-based developers with institutional knowledge of their codebase, they can identify apps to port to their Git enterprise servers.

    At the other end of the spectrum, for companies with a Git-only imperative, one option is migrating an initial set of applications quickly using, for example, Team Build. However, a cold cut-over at the portfolio level could be excessively disruptive and it’s not recommended. Accelerated retirements and/or increased retention costs are a real risk, so a phased approach over time is the best approach. The key to success with Git adoption is recognizing it is a journey for the organization that requires a well-thought-out plan.

    For a third-party perspective on the adoption of Git for mainframe apps, IDC has published an Infobrief, Git for Mainframe: Considerations for Enterprises, highlighting the opportunities and the challenges.

    There are many paths to modernization, so take advantage of others who have blazed the trail.  But whatever the approach to Git adoption, younger team members should be prepared for a “We were doing that way back when” story or two.

  • This is Not Just a Test: DevOps and the Need to Automate

    This is Not Just a Test: DevOps and the Need to Automate

    Creating software is as much about the process as it is about the product. In recent years, we’ve seen the speed of innovation reach new heights due to more efficient development and quality control. DevOps has further accelerated development by combining software and operations in unified, self-improving teams. With automation taking care of rote tasks, developers can now tackle more difficult issues, resulting in even greater software quality and faster development. The need for speed to market is vital; a month or two can make or break a product launch. Then again, if a product isn’t tested thoroughly in the rush to get to market, flaws can slip through and an organization’s revenue and reputation can take a serious hit. In the U.S., software failures in businesses’ operations systems cost a total of almost $1.6 trillion in 2019. The problem is that it can be a bottleneck and create major delays in the release cycle. In fact, A recent survey of 4,300 developers revealed that it is the top reason that releases are delayed.

    In the past, the waterfall approach was common. Tests usually took place after code was written; carried out by testers who had no interaction with the original developers. As a result, bugs that could have taken moments to correct during development required days or even weeks to fix afterward. Even worse, the complexity of the finished software and its many dependencies meant that fixing a localized bug could introduce additional problems that needed to be fixed after another round of testing, further delaying the release.

    But even in a modern DevOps environment, testers and developers are still speaking entirely different languages. By and large, testers do not code and software developers are not expert testers. The two teams typically converse through multiple logs, event viewers and other operating system tools. It creates a great deal of friction and slows down the process.

    Shift Left 

    DevOps represented a big improvement over waterfall methods but, even so, it’s far from a set-in-stone process. It, too, has evolved and right now, shift left testing has moved into the spotlight.

    In this approach, the guiding philosophy is to test early and often, beginning very early in the development cycle and continuously throughout every stage whenever there’s an opportunity. As a result, bugs are caught early when the relevant code is still fresh in the developers’ minds, so it’s easier to fix. In the end, businesses are able to release higher quality software much faster and more frequently.

    Environmental Tests

    When it comes to the DevOps process, it’s not always clear where and how to automate the process. The following looks at how that can be incorporated into DevOps based on four different environments in a continuous delivery (CD) pipeline.

    These environments include local development (usually a coder’s personal device), a continuous integration (CI) or development server, test servers for user interface (UI) testing and the production environment itself. Each has stages with requirements that must be met for code to advance.

    Whenever any new software features or capabilities are entertained, though, developers and the QA team should begin by answering the question, “How can this be tested?”

    Moving Through the Stages

    Once code is created, it must be able to be built without error, passing all unit tests and local verification.

    This stage is particularly important because the development environment echoes the requirements of production; if it works here it’ll likely also work for the customer.

    At this stage, running early regression and developer verification tests ensures the build is ready while providing feedback about the reliability of the environment. It’s useful to have individual test cases as part of more than one schedule. This enables developers to test minor parts of the software without undergoing regression tests of all product features. These tests should be automated and triggered as part of the same process.

    Developers should have the ability to perform basic tests on their own code. After all, the sooner a flaw is identified, the easier it is to eradicate. Just as testers are not experts in coding, neither are developers expert testers. To shift left and effectively integrate tests at all stages of the development life cycle, it must be automated. This will allow developers to test larger quantities of code earlier and more frequently.

    Once through this stage, the software can advance to the test environment.

    This is where a variety of product tests take place, including those for function, load and performance. Passing this stage entails completing all automated functional UI tests, verification of visual appearance, an inspection of server logs and performance evaluation.

    This is where most regression tests are conducted and automation is crucial for scheduled tests to be triggered.

    Next, the product moves into the production stage.

    Tests still occur in this stage to pinpoint any remaining bugs. Automated tests are then run against the production environment to closely mirror interactions with real end users. At this point, it’s highly unlikely there will be major issues but, if there are, running tests at earlier stages could very well be the culprit.

    Throughout the process, ideally, testers will use a visual, no-code tool to create these automated tests. In this way, testers are able to more rapidly and intuitively implement test automation without having to tap into programming resources. Additionally, the integrated documentation of the actions taken during automation will be interactively connected to visual documentation of the test results. The upshot is that testers and developers gain a shared language for both building test automation and troubleshooting the issues it identifies.

    This is Not Just a Test

    Effective, scalable tests are crucial to delivering the reliable, high-quality software that today’s businesses rely on for day-to-day operations. This is why DevOps and development teams need to integrate tests at all stages of software development. Shift left speeds up the creation of better quality software, which provides a significant market advantage. After all, it’s not just about software quality; it can show which companies are best prepared to meet market demands.

  • Why We Still Need Specialists

    Why We Still Need Specialists

    A few years back, I wrote here that specialists—particularly infrastructure specialists—were not going anywhere.

    I think it is worth updating that prediction and reminding you all that specialists across the bulk of DevOps subdomains are still necessary—even if you have broken down their silos.

    Spend time in any of the variety of places that DevOps folks hang out and listen to/read what they actually say. When they express their sentiments as a real-time activity, the idea that we made one team responsible for most, if not all, of an application’s life cycle seems rather brutal.

    In most industries, we have increasing specialization as complexity increases. Even if it was allowed, most carpenters are not qualified to do plumbing or electrical work. Oh, most know the basics and could do it in a pinch, but they aren’t the expert that the plumber or electrician is. IT, for the last many years, has tried to do the opposite, with a decent rate of success. But practicality wins out over rigidity to make that success.

    We still have specialists. And you should ignore people that suggest they need to go away. The situation is already bad enough without having a person you can turn to for help with a harsh issue in this language, or on that piece of networking gear. While some of this—language specialists are the easy example—is merged into a DevOps team, some of them—network admins, for instance—only partially are. The networking within/between containers is often handled by the DevOps team (using one set of specialties), while the networking that allows all of the organization to run is still handled by operations. But I can’t think of an org I deal with regularly that doesn’t house these specializations somewhere.

    This is one of the things that has traditionally held back low-code/no-code solutions—they assume these specializations and will require specialists in whatever the underlying technologies are when things inevitably go wrong. The same is true (to a lesser extent) of DevOps, AI and automation. Someone has to know what is going on underneath—they must have specialized knowledge—in order to fix it when things go wrong.

    Remember, you are asking your DevOps teams to be both Jack of all trades and master of some. That is a recipe for failure, but DevOps is designed to withstand or overcome failure in short order, so I only mention this so you understand why new hires seem wild-eyed at times and old hands seem jaded—there is a lot going on that they’re mastering and remastering.

    So, stand up if you’re one of the “sailing against the wind” crowd. Be proud you’ve managed to thrive in an environment with a ton of specialist parts and make sure you know who to ask for help when something goes wrong and you end up out of your depth. Even if the answer to “Who can I ask for help?” ends up being “Learn it for yourself and then be prepared to help others,” you’ll rock it either way and the rest of us who have struggled through it feel your pain and respect your work. So, thanks—for those who’ve never experienced it, but rely upon your work.

  • vFunction Tool Uses AI to Help Convert Monolithic Java Apps

    vFunction Tool Uses AI to Help Convert Monolithic Java Apps

    vFunction today added the ability to employ artificial intelligence (AI) to automatically assess the level of effort required to convert a monolithic Java application into a set of microservices.

    Bob Quillin, chief ecosystem officer for vFunction, said the goal is to make it simpler for IT teams to more effectively employ the company’s vFunction Application Transformation Engine by using a vFunction Assessment Hub that assesses the level of technical debt involved.

    Legacy monolithic Java applications tend to have a lot of technical debt and dependencies that make them a challenge to convert into microservices, he noted. Most simpler Java applications have either already been converted or are in process, Quillin added.

    The vFunction Assessment Hub enables IT teams to better predict the outcome of refactoring a Java application to better prioritize which Java application should be converted first based on, for example, the level of scalability that can be achieved, said Quillin. Current assessment approaches involve a lot of manual effort and which typically requires organizations to hire external contractors to perform, he added.

    An AI-based approach reduces the total cost of creating a business case and constructing a roadmap for modernizing existing Java applications, said Quillin. The vFunction Assessment Hub measures application complexity based on code modularity and dependency entanglements and then assesses the level of risk changes might have on the stability of the application based on the depth and length of the dependency chains. It then surfaces benchmarks for debt, risk and complexity in addition to identifying aging frameworks that could create security risks and licensing issues.

    Thus far most organizations have been content to lift and shift Java applications into the cloud without modernizing them. The application simply now runs on infrastructure that is managed by an external cloud service provider. That approach does little to improve the scalability of a monolithic Java application, noted Quillin. The goal should be to convert that Java application into a set of microservices that run natively in the cloud, he added.

    The vFunction Assessment Hub is already being used within a Java Refactoring Service for the Azure cloud provided by Microsoft that is based on agent software to analyze code and the vFunction Application Transformation Engine.

    In general, organizations are embracing microservices to build applications that are both more scable and resilient. In the event a microservice becomes unavailable, requests for services are rerouted to another microservice to make sure the application degrades gracefully, rather than simply crashing altogether. The challenge, of course, is that the dependencies that can exist between hundreds of microservices can be more challenging to manage over time than those of the monolithic application being replaced.

    It’s not likely that microservices-based applications will entirely replace monolithic applications any time soon, but the bulk of new applications are being built on top of a microservices-based framework. As such, many DevOps teams are now managing a broad mix of monolithic and microservices-based applications across IT environments that become increasingly more complex to manage with each passing day.