devopsy.kinsta.cloud

Category: Enterprise DevOps

  • ServiceNow Extends Reach Into DevOps Ecosystem

    ServiceNow Extends Reach Into DevOps Ecosystem

    ServiceNow this week, as part of a massive Paris update to its core software-as-a-service (SaaS) platform for managing IT operations, has extended its DevOps integration efforts by adding connectors for Microsoft Azure DevOps (ADO) and Atlassian Jira platforms.

    Pablo Stern, senior vice president for IT workflow products at ServiceNow, said those connectors are the latest in a series of moves that ServiceNow is making available to bridge the divide between traditional IT service management (ITSM) and DevOps processes.

    ServiceNow this week also announced it adding a Process Automation Designer tool to its App Engine platform for building applications on the ServiceNow platform. IT teams will be able to use that tool to create workflows spanning ITSM and DevOps processes, noted Stern.

    In addition, ServiceNow announced enhancements to its Flow Designer and IntegrationHub tools to simplify the building of workflows, and unlocked a Source Control function across its application portfolio.

    Finally, Service Now is now bringing third-party data into the ServiceNow Service Graph and Configuration Management Database to make it easier to integrate ServiceNow tools with those from partners such as SolarWinds.

    Stern said the ServiceNow strategy is not to build a DevOps platform, but rather to foster interoperability across what has become a broad DevOps ecosystem. Most IT teams are now trying to create workflows that span traditional ticket-based approaches to manage IT processes and DevOps teams that routinely leverage programming tools to automate those processes, so the goal is to enable each organization to strike a balance between those approaches as they best see fit, said Stern, citing JP Morgan as a customer using ServiceNow connectors to DevOps platforms to achieve that goal.

    Very few IT organizations are relying on DevOps processes end to end to manage IT. Organizations have been managing IT using processes based on the ITIL framework for several decades. The challenge they face is ITIL-based processes are not often agile enough to support digital business initiatives that typically require rapid responses to any event, especially when the application involved generates revenue for the business. Every second of downtime or sub-optimal performance impacts the bottom line. Minimizing disruptions to those applications requires more reliance on both automated DevOps and DevSecOps processes.

    Naturally, the arguments over the merits of DevOps versus ITIL approaches to managing IT will remain unabated for years to come. Proponents of ITIL approaches often note there are applications that are so critical to the business that a more deliberate approach to rolling them out and updating them is an absolute requirement.

    Regardless of IT philosophy, it’s clear IT and DevOps processes will be employed side-by-side for many years come. The number of IT operations professionals who have the programming skills required to automate IT processes is still fairly limited compared to the total population of IT administrators. As such, IT teams might want to focus less on the debate itself in favor of determining where they are on the maturity curve when it comes to managing IT. Once that’s determined, it then becomes a lot easier to discern the next step in their IT journey.

  • The 2020 DevOps Dozen² Awards Nominations Are Now Open

    The 2020 DevOps Dozen² Awards Nominations Are Now Open

    The DevOps Dozen Awards are back for their sixth year and open for nominations. We are excited to celebrate the hard work and continued success of DevOps leaders, whose achievements over the past year deserve recognition. This year, we have expanded our awards program to include two different DevOps Dozen sections; hence, the name DevOps Dozen² (Squared).

    There are a dozen awards for each section this year:

    • DevOps Dozen Tools and Services Awards — 12 awards recognizing the companies that develop and deliver outstanding solutions to empower developers, DevOps and IT operations teams.
    • DevOps Dozen Community Awards — Closer to the original vision that we had with the DevOps Dozen when we instituted the awards five years ago, these 12 awards honor the leaders who have made a positive impact in the DevOps community.

    Nominations are made by the audience. However, it will work a little differently for the Tools and Services Awards categories than it will for the Community Awards. While anyone can nominate a tool or service for the final phase this year, there is a $250 fee for each Tools and Services nomination. This is meant to cut down on the spam and superfluous nominations we have received in the past.

    We are interested in serious tools/services in the DevOps space. Nominate only those you think are truly deserving of an award. Commercial tools/services are not eligible for the Community Awards and will not make it through to the finals.

    Winners will be chosen from among the finalists, with general public voting accounting for 40% of the final total and our judges’ selections for 60% of the total.

    The last day to submit your nominations is Oct. 15. To vote, follow the steps outlined on the DevOps Dozen² website.

    Meet the Judges

    Who are this year’s judges? You may already be familiar with their names. The 2020 panel will be lead by:

    • Alan Shimel, founder and editor in chief, DevOps.com
    • Jayne Groll, CEO and co-founder of the DevOps Institute
    • Mitch Ashley, CEO and founder of Accelerated Strategies Group
    • Charlene O’Hanlon, managing editor, MediaOps Inc.

    The 2020 DevOps Dozen² Categories

    DevOps Dozen Tools and Services Awards

    1. Best End-to-End DevOps Tool/Service
      Consolidation in the DevOps space has led to a greater selection of platforms offering end-to-end solutions. This award honors the platform with the most diverse and robust solution for DevOps teams.
    2. Best DevOps Repo/GitOps Tool/Service
      Git has led to GitOps, which has become an integral part of the DevOps toolbox. This award is presented to the most outstanding Repo/Git/GitOps tool.
    3. Best CI/CD Tool
      This award recognizes the best CI/CD tool that helps teams to deliver quality code better.
    4. Best Value Stream Management Tool
      Value stream management has come into its own as an important tool for measuring the business value of DevOps and application development. This award recognizes the value stream management tool that most effectively helps visualize the value stream, track metrics and measure improvement.
    5. Best Observability Solution
      This award recognizes the best observability solution for observing and monitoring to understand the performance of individual components and systems and entire applications.
    6. Best DevSecOps Solution
      This award honors the best DevSecOps solution, which enables security to be included earlier and continuously throughout the SDLC.
    7. Best Testing Service/Tool
      Testing is a mainstay in the DevOps toolchain, and this award recognizes the best testing tool for ensuring the quality of code.
    8. Best Kubernetes Platform/Service
      The adoption of containers has paved the way for an entire solutions ecosystem supporting and enhancing the container technology. Kubernetes is perhaps the most widely recognized and utilized tool in the space. There are many platforms and services that offer Kubernetes-based solutions. This award honors the best Kubernetes platform/service available.
    9. Best Cloud-Native Security Solution/Service
      The cloud-native movement has taken the IT world by storm due to the depth and breadth of the solutions available and the benefits of building applications and services specifically for cloud-native environments. This award recognizes the best tools, solutions and services considered cloud-native.
    10. Best DevOps for Mainframe Solution
      Mainframes are far from being the dinosaur of the IT world, and a variety of solutions are available for enabling DevOps practices in mainframe environments and hybrid environments. This award recognizes the best of them.
    11. Best DevOps for DataOps/Database Solution
      Databases often have been left out of the DevOps conversation, but that is changing. Recently, the whole concept of DataOps has started gaining traction. This award recognizes the best solution in the database for DevOps and DataOps categories.
    12. Best New DevOps Tool/Service Provider
      This award recognizes the best new service provider or tool that has brought innovation to the DevOps space in the past year (must be less than 24 months old).

    DevOps Dozen Community Awards

    1. Best DevOps Industry Implementation
      This award honors organizations that have successfully adopted and implemented DevOps principles and practices.
    2. Best DevOps Survey/Analysis/Research
      This award recognizes research that has significantly and positively impacted the DevOps community.
    3. Best DevOps-Related Video Series (video)
      This category highlights the most entertaining, educational and inspirational DevOps video series—those with the most engaging and highest-quality content.
    4. Best DevOps-Related Audio Podcast Series
      Which is your favorite DevOps podcast from this year? There are many great DevOps podcasts sparking some interesting discussions on a multitude of topics from CI/CD and testing to DevSecOps, Kubernetes, containers and more.
    5. Best DevOps Book/eBook of the Year
      Each year sees a plethora of well-crafted DevOps-related books and ebooks. This category highlights the most popular and best read of the year.
    6. Top DevOps Evangelist
      This category is one of our favorites. There are so many great evangelists who spread the DevOps gospel and we want to know who does it best. Nominees can work within the DevOps space or outside of it.
    7. Best DevOps Transformation (non-vendor)
      In a year in which digital transformation became a must-have for survival, which organizations stood out in their efforts to transform their business through DevOps?
    8. Best DevOps Virtual Event of the Year
      COVID-19 has changed the game when it comes to hosting events. In 2020, virtual events have become the new norm, reaching a global audience while everyone stays safe at home. Which DevOps virtual event stands out from the others?
    9. Most Innovative DevOps Open Source Project
      Open source projects are meant to promote innovation and increase efficiency, velocity and quality. While there are many great open source projects out there, the best ones are those that create a community of people who work on and with the technology and support each other.
    10. DevOps Executive of the Year (non-vendor) This category celebrates excellence among DevOps leaders at the C-suite (or equivalent) levels.
    11. Best DevOps Presentation of the Year
      This category recognizes the best DevOps-themed presentation from the many (virtual) conferences during the year.
    12. Best DevOps.com Article of the Year
      DevOps.com publishes multiple new articles every day. Which was the best article from 2020? Let us know so we can recognize the author.

    All the solutions nominated to the Tools and Services Awards will automatically be registered into the DevOps Tools Guide authored by MediaOps and Accelerated Strategy Group. This initiative is our effort to create a comprehensive listing of tools and tools vendors to help end users better understand the DevOps market landscape.

    With so many talented and passionate leaders in our community, choosing a winner is no easy task. Though not everyone can take home the award, I truly believe every single person who is a part of this great community and has done some remarkable work to empower and support other members of the DevOps community is a winner. By acknowledging them in our award process, we want to highlight their work and show how important they are in the DevOps space.

    For more information on the awards, please visit the DevOps Dozen² website. If you have any questions regarding entry process, awards, timing or any technical issues regarding the website, please contact us at DD@devops.com.

  • Onboarding Is Science, Not Art

    Onboarding Is Science, Not Art

    Make sure your new team members are getting a well-rounded onboarding

    In Fortnite: Save the World, there is a questline in which a character named The Cloaked Star answers, “Figure it out yourself,” to a variety of questions from a new hero who was a DBA before the end of the world came. It is hilarious because, at that point in the story, the new hero doesn’t have the information to do so.

    Most of us have onboarding of new DevOps employees so simplified it would make us from five years ago cry tears of joy. Give them a container or virtual that is already equipped the way they need and they are ready to be productive, right?

    Welllll … kind of. Let’s stay totally stoked by the idea that pre-made tooling that will run fine on whatever they’re running it on is a huge step, then acknowledge that in the age of DevOps, tribal knowledge is still very much a thing. There is this other repository we hit for X, an alternate API for Y, that system that does not work the way one expects, and so on.

    We need to apply the same science to knowledge transfer that we do to tooling. It is not all self-documenting, it is not all obvious, and that knowledge absolutely needs to be transferred to new employees.

    Some of us have this down pretty well. Far too many of us do not.

    Two solid ways to transfer knowledge are documentation that is both thorough and up to date, and a buddy system. Both have strengths and weaknesses, but both address the parts that are not automated. If you go with documentation, make certain someone is responsible for keeping it up to date in the fast-paced world we find ourselves in. If you adopt the buddy system, make certain your team knows what information is most important to impart. While each buddy will be asked questions, it is the things that might surprise new employees that buddies should be prepared to impart.

    You are building the future “legacy code.” How far in the future that is remains to be seen, but given our rate of change, probably sooner than you think. Even if you use the buddy system, there needs to be some level of documentation included. The number of projects that use something like “Joe’s database designed in his basement” because it had the best caching available but then employee turnover removed everyone with any knowledge of the weak spots in the chosen product, is stunning.

    So make sure your new team members are getting a well-rounded onboarding. Make certain they have the info they need to be productive and avoid pitfalls that your team has already identified. Make their transition easier, and they’ll be productive sooner. And hey, you might avoid that big bug caused by someone’s lack of information.

    Whatever you do, no matter their level, make sure they know the important bits. And keep kicking rear, now with new team members, equipped to help you along.

    And don’t be The Cloaked Star. Help a new employee out.

  • JAMstack Boards the Enterprise

    JAMstack Boards the Enterprise

    Set course for development via JavaScript, APIs and Markup. Engage.

    In recent years, the JAMstack web development trend has garnered interest among front-end web designers. A 2020 study by Netlify found 1 in 3 sites serving a million users use JAMstack.

    JAMstack has been viewed as a development style for blogs and startups, but lately, some proponents believe it’s time for enterprise adoption. So, what benefits could JAMstack and serverless offer enterprise architecture? I recently talked with Ryan Coleman, vice president of engineering and product at Stackery, to find out.

    Defining JAMstack

    First off, what is JAMstack? JAM stands for JavaScript, APIs and Markup. It’s is a new(ish) style for building websites and applications based on modern tools and serverless web architectures. We’ve seen its adoption grow exponentially within the last five years or so.

    As defined at jamstack.org, JAMstack enables fast and secure sites that pre-render files directly from a CDN. This avoids tight-coupling between the client and the server, also known as headless. Applied in the right context, it can bring performance, security, scalability and developer experience benefits.

    Admittedly, there is some variation on implementation of this style—PayPal’s Jamund Ferguson simply calls them static apps—but, “the thing that they all have in common is that they don’t depend on a web server,” noted jamstack.org. Jekyll, Hugo, Nuxt, Next and Gatsby are among many JAMstack frameworks and static site generators, and many adopters are listed here.

    JAMstack Benefits

    No longer are IT teams operating a web farm. Rather, they are shifting the process of interacting with live web servers left. According to Coleman, JAMstack is more aligned with these modern practices across the board. Therefore, he sees many potential benefits for its use within enterprise environments.

    Serverless

    With JAMstack, web pages are built during the build process attached to a CDN pipeline. “Instead of running a server farm, you are externalizing it,” notes Coleman. “It’s the CDN’s problem.”

    JAMstack plays nicely with a serverless mentality. “JAMstack helps rethink what production footprint actually looks like,” Coleman said. By reducing your active server footprint, serverless implementations decrease infrastructure costs.

    Security

    With the proliferation of web vulnerabilities, companies are seeking new ways to limit their exposure and avoid catastrophic breaches. JAMstack can help enterprises keep pace, offering a “much tighter place to apply security controls,” he said.

    It allows teams to think of their security infrastructure in smaller pieces, which affords better control and scalability for things such as authorization. “Serverless architecture helps enable enterprise-grade SAML-based Auth tied into your stack,” Coleman noted.

    Democratization Effect

    Coleman also sees opportunities for small teams to disrupt the dominant players by utilizing it. “Companies need to move quickly, achieve enterprise-grade architecture and secure products with less investment on the onset,” he said. These abilities could, for example, enable a FinTech startup to compete on the same playing field as an established bank.

    “How do you create an architecture that is living, and how do you make sure it is as secure and clean as possible?” he asked. “How do you get a whole team together where actually doesn’t have to use 1,000 lines of YAML, and how do we continue to maintain?”

    Since JAMstack adopts APIs, companies could rapidly stitch together high-performing applications. Furthermore, by pairing low-code tooling and JAMstack, teams could drag-and-drop snippets of AWS-managed services to construct web applications very quickly.

    Bi-Directional Interactivity

    JAMstack has a reputation for creating compelling user-facing apps. Yet, Coleman acknowledged a common misconception: JAMstack is static and, therefore, not for enterprise.

    However, JAMstack can be more than a static app; the structure can support secure interactive capabilities. By taking CDNs and buckets and utilizing API connections to rich features such as AWS managed services, it can essentially reach the same enterprise-grade capabilities as other development styles, he said.

    JAMstack: No Longer Only for Blogs

    “Serverless + JAMstack is where web app architectures are going,” Coleman wrote in a recent blog post. Though many view it as a static app, there is growing potential for using this development approach to enable applications with dynamic interactivity. “With AWS in the backend, you can create an interactive enterprise-grade app.”

    Another benefit of JAMstack is the decoupled nature between client and server. This abstraction layer makes for building web applications across multi-cloud arrangements. And as web APIs become more accessible through low-code, the entry barrier to connecting to a MySQL or Postgres database—or handling identity management, for instance—is lowering.

    Once only viewed as good for blogs, JAMstack is gaining more credibility within enterprise architecture as APIs and serverless become more commonplace, offering the potential to increase performance, increase rapid web application delivery and solve other concerns within enterprise situations.

    Coleman described JAMstack as enabling developers to “build that web application the way you want.” How, then, do you want to build?

  • Why It’s Time to Migrate Away From COBOL

    Why It’s Time to Migrate Away From COBOL

    Every few years an alarm goes up about the shortage of COBOL programmers, and then it dies down. While it’s hard to tell whether this cycle occurs because programmers are found or because other solutions are identified, the latest resurgence of interest is at least in part due to the pandemic and state governments’ use of software for social services that couldn’t stand the increased caseload of tens of millions of unemployment claims.

    As we continue to sit in the midst of the pandemic, organizations are thinking about the many ways businesses will have to change. For instance, will cash still be used in five years or will the business need to support some kind of cryptocurrency? It’s time to ask those big questions now—and determine whether it’s possible to make changes without touching COBOL code.

    For most organizations, I believe that in answering these questions they’ll find that it’s time to start migrating away from COBOL.

    The Rise (and Fall) of COBOL

    COBOL is one of the earliest programming languages. It was invented in 1960 and rose to prominence as a language that required minimal programming skills. Part of COBOL’s initial attraction was that it was a domain-specific language designed for business—not only was it easy for programmers to learn, it also could be understood by executives; hence, the acronym for “Common Business-Oriented Language.”

    COBOL was also one of the few programming languages that understood the requirements of business computing—most simply, the fact that an amount like 10 cents can’t be represented accurately in binary was a huge challenge. If you’re a bank, small rounding errors on each transaction add up. Libraries for business arithmetic could be written in other languages, but they’re scarce and hard to find.

    While COBOL never disappeared, its influence faded in the 1980s but many large institutions failed to transition away from it. Today, there are billions of lines of code in governments, banks, enterprises and elsewhere performing essential business functions with few programmers able to maintain them.

    Part of the reason why COBOL fell out of fashion was its inability to adopt things such as modularity or breaking down problems into smaller, functional pieces. When COBOL was invented, old business software was monolithic—and monolithic in a very deep sense. Internally, the software wasn’t partitioned into separate components that could be updated without affecting each other. It also tended to model forms that humans would fill out and couldn’t be submitted until the form was complete. Putting a web interface in front of those monoliths leads to a predictable result: long, complex forms that can take hours to fill out and are close to unusable on the modern web.

    While the COBOL language itself has evolved over the years (with revisions released in 1961, 1974 and 1985 and object-oriented COBOL released in 2002), you still have to ask: what will it take to bring a COBOL program written in the 1970s or 1980s up to date? Is redesigning a program to take advantage of COBOL 2014 any easier than porting it to Java or Python or Go or Rust? Furthermore, COBOL looks unfamiliar and forbidding to most modern programmers, which likely caused it to lose popularity in the first place.

    Finding an Alternative

    So, where are we now, with our billions of lines of COBOL running the world’s governments and finances? We certainly need more people who know and understand COBOL programs. There’s a lot of old code that needs to be maintained, pure and simple.

    But it goes deeper than that. COBOL is just another programming language; if we’re going to maintain (or replace) that software, we need programmers who understand the engineering decisions that made the software what it is. Programmers used a lot of tricks to work around the limitations of 1960s computers. Those tricks just don’t make sense to modern programmers; they solve problems we no longer have. We also need engineers and managers who are willing to look at our current situation—for example, the huge surge in unemployment applications—and think beyond the short-term solution. What does it take to reinvent current systems rather than replacing them? How can these systems become more human-centric? Can they be redesigned to match the way we live and work now? Putting a web front-end on a monolithic business process from the 1950s is the road to failure.

    The new generation of programmers must understand financial programming and how to design business systems. There’s little value in migrating from COBOL if you don’t get something better in return. What you should get in return is the flexibility to add new features and new user interfaces, the ability to adapt to changes in competitive conditions and implement changes in the regulatory environment, and so on. We don’t yet know what those new features will be (perhaps cryptocurrency or virtual reality user interfaces), but we do know that businesses will need the flexibility to add them.

    Simply put: Scrambling for COBOL programmers whenever there’s a crisis is not a viable business decision. Transitioning from COBOL is a change businesses will have to make eventually. This transition should be made when businesses have time to think it through, time to hire the right people and time to do a good job—rather than waiting for the next crisis.

    In the meantime, while businesses are baking in the time to think through their transition strategically, it’s important to note that this isn’t an all or nothing decision—i.e. let’s scrap our legacy software and replace it by Jan. 1, 2021. This is not Y2K. Using a microservices approach offers numerous advantage. For one, businesses can split their existing infrastructure into different independent services. They can start with pieces such as authentication, then move on to things such as billing and other services. Whatever businesses decide, they need an approach that’s modular so that it’s easy to add new features as needed. That requires careful thinking about design—and that thinking is more important than the programming language itself.

  • The Pros and Cons of Configuration-as-Code

    The Pros and Cons of Configuration-as-Code

    Configuration-as-code, or “GitOps” as many are now calling it, is a simple but astounding idea: Standardize configuration, check it into version control; no need for each Operations/DevOps person to develop their own set of scripts to get the job done. Configuring or reconfiguring an environment is as simple as checkout and run, assuming everything in the configuration is tested. For a project or a team, this is a serious time-saver and limits the amount of human error introduced into our systems. For projects rolling out code frequently, it is pretty much required.

    But like everything, it has its limits. I worked with a product management engineer once who had concocted a script that had tens of thousands of lines of code and was every bit as complex as the product he used it to manage. His scenario was the typical precursor to configuration-as-code: He would go show off the product and whatever functionality he needed could be managed through his script. The problem was maintenance; as the script was central to how he did his job, changes to it were a big deal. And even though he was the only one maintaining it, he did get burned a couple of times by changes he’d made, because it was that complex.

    When our configuration-as-code starts to cross teams, or we take it to the enterprise level, we risk finding its limitations. Our environments are complex, very complex. And we can do a lot with configuration-as-code, but throwing everything plus the kitchen sink into that system will create issues we need to overcome.

    A single project is constrained. We know what we need to do to spin up the environment. We can identify weak points and build checks and balances around them.

    The enterprise environment is not a project. Agile projects and DevOps thrive because they have a stable working environment to run on. The volume of change in things such as storage allocation, routing and VLANs is not the volume of change that an agile project sees. Good documentation is as solid a solution as configuration as code for something that changes rarely and the size/number of scripts and configurations that need to be maintained is smaller.

    There is a good use case for corporatewide configuration-as-code for recovery purposes. It is far easier to restore the environment in a disaster if you have all configuration and the scripts to install that configuration available. Unfortunately, that use case is minimized by the march of technological change. During the one total destruction restoration I’ve been involved in (was in charge of), many of our configurations were no longer usable because the products we were running in production at the time of the disaster were not the products we could purchase for the rebuild. We ended up with a shiny new data center from building to servers, but it still involved a lot of manual configuration. So while the use case is good, it is not a panacea unless you are constantly replacing your infrastructure with the newest products, limiting the usefulness of this use case also.

    Some organizations have already moved to configuration-as-code via methodologies such as chaos engineering, but most of us are just starting to consider moving configuration-as-code beyond the project level. When the (admittedly large) step of truly moving infrastructure management toward configuration-as-code, start with inventorying what your operations team does on a daily basis. There are a ton of things—the famous low-hanging fruit—that can be boiled down to scripts or standardized configurations and checked into version control. This gives a central point of truth for how things are configured, tracks changes for when things go wrong, and shortens startup time for new DevOps/Operations staff by giving them a repository of tools to review and work with. When working in the cloud, everything is scriptable, so make that side of operations is completely configuration-as-code. In the data center, you end up with diminishing returns after daily tasks are automated, but assigning someone to codify all possible changes is a worthy goal, as long as the work is prioritized to make the most frequent or error-prone configuration issues a priority. Once the focus is down to hardware, the rate of return for those invested hours is going to go down a bit, since specialized hardware is full of proprietary or market-focused config info. To draw on a comparison I’m familiar with: An F5 ADC has very specific use cases that must be reconfigured centrally, whereas restoring your Kubernetes cluster will generally require less focused configuration and teams spinning up containers will take it from there. Not that Kubernetes configuration is simple; but each project handles the details they require while corporate-shared resources tend to need all of the details handled by the infrastructure team.

    We can discuss this at length, and I have with others who think differently, but small tools are better. Do one thing, and do it well—the original Linux mantra—then make scripts that call the small ones. This contains changes over time and doesn’t create dependencies or touches on things that might not need it in a given situation.

    And keep rocking it. All configuration-as-code does is reduce manhours and opportunity for human error, so embrace it, but be controlled about it. Don’t throw everything in there and then forget about it. Make a plan, make a maintenance plan. Iterate. Improve. And use that extra time to solve user problems, which is why we’re here in the first place.

  • Z is for Zowe – the Open Path to Mainframe DevOps

    Z is for Zowe – the Open Path to Mainframe DevOps

    The energy and excitement in the IBM Z mainframe community during the two years since the announcement of Zowe have not been seen in a generation.  By enabling mainframers to work like technologists who work on the cloud, mobile and distributed platforms, Zowe unlocks the power of DevOps for mainframers, ensuring the vitality of the platform for the long-term.

    It accomplishes this by opening up the world of enterprise DevOps tools, both familiar and special-purpose ones, for mainframe applications, allowing users to easily create toolchains that include the mainframe. This freedom to leverage any enterprise DevOps tools across the entire deployment pipeline is achieved simply and elegantly.

    The Zowe Backstory

    The Zowe journey began when Broadcom, at the time CA Technologies, along with IBM and Rocket Software realized the challenges they were hearing from customers—IT isolation, need for increased delivery velocity and recruitment and retention problems—were bigger than any single vendor could solve on its own. The challenges of this scale required an ecosystem-wide response.

    So the vendors came together with the Linux Foundation’s Open Mainframe Project to create the open source framework known as Zowe (as in David Bowie, for the uninitiated). The Zowe community operates transparently and welcomes contributions from all in the form of code, enhancement requests, educational content and shared experiences. A virtual meetup held earlier in the year attracted participants from 47 companies from across the mainframe ecosystem.

    Traditionally, mainframe application development and operations have been separate from the rest of the IT organization due to specialized processes, roles and tools designed around the platform. While they have served the mainframe community extraordinarily well over the decades, the world outside has changed with demand for ever-shorter software delivery cycles resulting in the DevOps model.

    In short, the tipping point has arrived and the time to normalize is now.

    What Is Zowe?

    Zowe is a modern interface to the mainframe that extends z/OS-based services and tooling to the greater IT landscape, including DevOps. As a framework, it consists of four main components:

    • Zowe API Mediation Layer (APIML)
    • Zowe Command Line Interface (CLI)
    • Zowe Explorer (Visual Studio Code extension)
    • Zowe App Framework (UI)

    Broadcom is proud to have initially contributed the first three of these components. The first two were created as part of our CA Brightside offering, which won an award for the Most Innovative DevOps solution—yes, a mainframe offering won a DevOps innovation award!

    These components provide a modern way of working that supports core DevOps practices such as empowering developers with shift-left testing responsibilities and DevOps engineers with robust automation options.

    This article describes the framework’s ability to onboard the mainframe to enterprise DevOps, so developers, systems programmers and others who work with the mainframe can now do so the same way their peers do with other IT platforms (i.e., cloud, mobile, distributed). These shared experiences close the gap between mainframers and others while preserving the core advantages of the platform. Common tools fuel a common language that benefits all, especially when deploying hybrid applications (e.g., web front-end with mainframe back-end).

    Through automation and script libraries, Zowe helps reduce the range of mainframe-specific knowledge necessary to be productive, enabling organizations to more easily recruit, onboard and retain the next generation of talent. By using modern DevOps tools and techniques combined with access to “big iron” applications, the next-generation now sees mainframe as an exciting career choice.

    Recent milestones achieved by the Zowe community include the launch of the long-term support (LTS) release ensuring the stability and longevity of the framework and a Zowe Conformance program ensuring the interoperability of a growing ecosystem of conformant tools.

    The framework drives mainframe DevOps in several ways:

    Zowe API Mediation Layer – Discover and Securely Consume

    As RESTful APIs become table stakes, the greater the challenge becomes, making those services easily discoverable and consumable in a secure way. The Zowe API Mediation Layer (APIML) addresses this need by exposing mainframe services, ranging from z/OS system level to database/transaction services to tool-based services, through a centralized gateway and catalog.

    Conformance of these APIs with the Zowe LTS standard eliminates breaking changes while bringing additional value, including the usability and productivity benefits of single sign-on (SSO).

    At Broadcom, we’ve leveraged Zowe to achieve conformance across our product portfolio and have a healthy pipeline of Zowe-conformant products soon to be announced. By spanning the traditional Dev and Ops domains, we’re facilitating true mainframe DevOps adoption—teams can combine their preferred mainframe-native tools with off-platform ones, including ones used by their peers working on other platforms, to optimize speed, quality and productivity.

    Zowe CLI – Orchestrate and Automate

    For DevOps Engineers and DevOps-savvy self-service developers, CLIs are a “go-to” tool. They use CLIs for a wide range of platforms including AWS and Azure, Git, Kubernetes and so on … why not mainframe? The use of CLIs paves the way for robust, script-based automation, which is a cornerstone of DevOps. The Zowe community already offers some basic scripts, free for anyone to use.

    Using a CLI may be new for some, so community members often share tips and tricks to help these users navigate the productivity curve quickly.

    Broadcom currently offers a dozen or so CLI extensions to mainframe-native tools, once again spanning the traditional Dev and Ops domains to improve productivity across roles and processes.

    Zowe Explorer for VS Code – Visualize and Navigate

    Visual Studio Code is exciting for any mainframers editing code, especially developers and sysprogs. While ISPF and Eclipse will remain viable options for the foreseeable future, the extraordinary success of the lightweight VS Code, which is backed by Microsoft, and the rapid emergence of browser-based editors/IDEs expand the range of options. With 11 million users around the globe, VS Code provides an experience users love and a marketplace of 20,000 extensions enabling users to personalize their UX and optimize their workflows.

    Zowe Explorer, which streamlines interaction with mainframe data sets, USS files and jobs, is only one of the many VS Code extensions available in the VS Code marketplace for mainframe users (and a successful one with more than 12,000 downloads). Unlike Eclipse extensions, all of the mainframe extensions from Broadcom are free, and many are open-sourced.

    In the Queue: SSO/MFA, SDK, Mobile App

    The Zowe community has been working diligently on the implementation of SSO and multi-factor authentication (MFA), which eliminates the drudgery of constantly logging on to multiple systems.

    Also, software development kits (SDKs) are in the works, the goal of which is to make Zowe services (and by extension, the mainframe) easier to use by providing reusable code for common mainframe operations in popular languages and runtime libraries. The beginnings of a Zowe Node.js SDK and Zowe Python SDK already exist.

    Finally, a Zowe mobile app to help systems and operations support staff improve system availability is, like the SDK, being incubated by the community.

    Decision Time – Zowe Open Path or Closed, Proprietary

    Mainframe teams face a choice: Continue with a vendor-controlled model, in which vendors determine which tools you use and how you work or adopt the open path. While the prescriptive (aka opinionated) approach of the former may have been acceptable for a siloed mainframe, the open approach is the foundation of enterprise DevOps.

    As an indication of the early success of the open path, we’ve been delighted to learn about many mainframe shops adopting Zowe on their own, with CI/CD integration and code analysis a common entry point.

    For those with reservations about open source, consider the prevalence of Git, Visual Studio Code, Jenkins, SonarQube and so on in the DevOps landscape. Microsoft CEO Nadella recently stated, “We’re all-in on open source” and Zowe co-founder IBM recently acquired Red Hat—more shining examples that open source tools are a trusted part of the technology stack.

    Change is never easy, which is why no-fee Design Thinking workshops are available to help adopters prioritize the DevOps use cases enabled by Zowe. Because so many see “open” as the least costly, least risky and fastest path to mainframe modernization, familiarity with Zowe is quickly becoming an industry standard. Ask your vendors if they’re joining the Zowe community and ecosystem.

    With ongoing commitment from IBM, Rocket and Broadcom and the guidance and support of the Open Mainframe Project, the excitement around will Zowe continue so join the open mainframe movement!

    To learn more about Zowe, visit Zowe.org and the community blog site.

  • The CMDB Is Dead – Long Live the CMDB

    The CMDB Is Dead – Long Live the CMDB

     Many enterprise IT organizations have a configuration management database, or CMDB, in place. These systems are important for overall configuration management, governance and DevOps coordination. Yet, these legacy data repository systems can fall short in our dynamic IT environment. They require manual updating and can’t always support real-time decision-making. Furthermore, we have new requirements since the onset of widespread WFH workplaces. In this article, we talk about the past and current role of the CMDB, how it is evolving and how integrating it with other IT operations systems can modernize infrastructure monitoring and configuration management.

    History Lesson

    Remember the enormous fuss over Y2K? IT execs were concerned that the new century turnover would break their systems, which led to a secondary concern that they weren’t sure what was even on the network. The 2001 release of ITIL v2 best practice led to the notion of a CMDB, which would enable the tracking of every technology asset and its configurations and show the relationship between those assets. But getting all that data in one place proved to be a problem—and one that has persisted to this day.

    Is the CMDB Still Relevant?

    CMDBs have been around for a long time, and they still play a valuable role in the enterprise IT organization. In fact, a CMDB has never been more essential than now. The year 2020 has been chaotic for most IT organizations as employees quickly transitioned to working from home and are using a wide array of personal devices and networks, all of which need to be monitored, maintained and secured. As shadow IT persists and organizations move increasingly toward rapid deployment and release models that incur frequent, ongoing change, governance and control become critical.

    Contrary to what some may think, you can’t safely go faster without a level of control. This is a balance that DevOps practitioners must strike, and it entails a deep understanding of all elements within the IT environment and how they relate to one another. If the enterprise doesn’t have a CMDB (or something akin to it) to provide visibility into infrastructure components, configurations and relationships, the IT operations team is working at a disadvantage. Such lack of knowledge slows down developers—and nobody wants that. Even worse, without always updated, reliable infrastructure data, DevOps teams risk a catastrophic result with each software release.

    Modern CMDB Requirements

    Today’s CMDB is critical for incident management and must be supported by a mature, well-understood change enablement practice in multi-cloud and hybrid environments, yet legacy platforms aren’t designed to monitor systems residing outside of the corporate firewall. More importantly, modern applications are built on containers and ephemeral microservices that appear and disappear frequently. It is nearly impossible for IT staff to manually update the CMDB under such constraints, and the truth is, people weren’t typically doing this regular updating years ago. Today, the only viable way to maintain state and configuration data to support accurate, real-time decision-making is through automation.

    Most CMDBs on the market today lack automation; they still look and act much like a data repository. The appetite for heavy manual work is razor-thin, and IT organizations simply don’t have the people power to do it regardless. Simultaneously, the role of the IT operations professional has evolved dramatically—it is no longer as focused on installing, patching and rebooting systems. Rather, these individuals are now expected to manage applications and platforms in an end-to-end fashion, serving the business with their unique requirements spanning distinct business units and use cases. Therefore, the more we can use technology to reduce the load of traditional incident management tasks and enable operations to be a true partner with development, the better we can serve the business and end customers.

    So, What Now?

    Integrating modern and legacy tools is one practical solution for resolving these issues. Newer IT operations management (ITOM) technologies that incorporate artificial intelligence and machine learning (AIOps) to automate the discovery of assets and topology mapping can bring the CMDB into the modern age. An open ITOM framework for integrations can keep the CMDB always up-to-date, without manual intervention. This enables IT operations staff to focus on optimizing and improving infrastructure for reliability and performance, versus digging into one dashboard after another to make sense of the status quo.

    Understand the requirements and scope of your CMDB. Tracking the configurations of components that are not business-critical only expands on the effort and resources needed to successfully maintain the CMDB, creating waste in the process. Keep it simple and iteratively improve the CMDB over time. Most importantly, get started now. Even seemingly small, simple improvements can have a positive cascading effect down the road.

    IT environments will only get more complex—even though the end goal is simplicity, cost savings and efficiency. To keep pace, enterprise IT organizations will need to rethink the traditional CMDB. By integrating the CMDB with modern IT operations monitoring, management and automation systems, IT operations teams can finally have a reliable repository of data on all of the company’s on-premises assets and multi-cloud resources, while also gaining the ability to automate manual tasks that impair productivity and eventually remediate where appropriate with the right approach to AI and machine learning.

     

    This article was co-authored by Curt Thorin, a solutions consultant at OpsRamp. He has 20+ years’ experience in enterprise IT software and services. Curt can be reached online at LinkedIn.

  • DevSecOps and Passwords: Lessening the Burden

    DevSecOps and Passwords: Lessening the Burden

    The point of DevOps is to make IT more responsive (and hopefully more stable, but we have evidence that is not always the case). The point of DevSecOps (so far) is to make security more responsive and to build security right into the stack. These are both worthy goals. As we all have pointed out repeatedly, an already overworked security team needs DevSecOps to keep up with the increased rate of change that a more responsive IT department represents.

    And yet, while we’re automating and simplifying things, the security end of DevOps is doing little for end users. Automating protection of App X is good for the App X team and good for the company, but it means little to the ultimate users of security. We all know security is a burden that users bear in the hopes of protecting the company, but no one is a fan. So maybe you didn’t have self-service password management, and now you do. Great! That helps end users. But most of us were already implementing self-service in an attempt to lift some of the load off of the security and help desk teams by the time DevOps took hold.

    Maybe you’ve implemented SSO as a result of DevOps. Again, good move! Employees spend less time remembering passwords and more time being productive. There’s less stress, too. But again, that was already well along, DevOps or not. And you know what? Employees are still making passwords they can remember, writing down passwords, re-using the same complex password, etc. To make it more difficult to hack, IT has come up with all sorts of strange rules such as, “Password digits, when summed, must be divisible by the square root of pi …” but that does little if the same password is used for LinkedIn (because it is “work-related”) and LinkedIn gets hacked again. And don’t fool yourselves, I know tech-savvy people who do this. Non-technical people are doing it a lot if techies are doing it.

    Which brings us to my suggestion: Your DevSecOps plan should include a password manager budget. There are a ton of them available, they’re cheap enough no matter which one you choose and they will generate complex passwords and then remember them for the user. Give employees password managers, train them a bit on their use, and reduce the chances that their password is the same as the one they used at pleasehackme.com. The investment is small—a few dollars per user for most managers—and choices include vaults that are local or hosted and the ability to set password-generation parameters.

    The only problem with a service-based password manager is ownership of the account after the employee leaves the company, but a policy can clearly delineate when responsibility transfers to the user and arrange for removing them from systems at the same time.

    More secure, less user hassle (true single sign-on, since they only need to know the password to the password management tool) and low cost—this is the heart of DevSecOps. And the investment is small. A bit of cash, a bit of training. What’s not to love?

  • How To Build a Business Data Map

    How To Build a Business Data Map

    Recently on DevOps.com, Anais Baig provided a brief overview of the privacy process and capabilities that developers need to think about to comply with new global privacy laws such as Europe’s GDPR and California’s CCPA. Top of the list for complying with these laws? The need to build a business data map.

    It’s no exaggeration to say that without a comprehensive, accurate data map—or, in other words, a complete inventory of the personal data the business possesses at any given time—it’s impossible to comply with these modern privacy regulations. Developers will understand better than most the difficulty in creating and then maintaining this record.

    But it can be done, and today we’re going to dive into the requirements and some of the technical processes that go into crafting a data map.

    What’s in a Data Map?

    In simple terms, a data map shows the personally identifiable information (PII) your business possesses, why you possess it, where it lives and how it’s being used. To answer these four questions, you’ll need to audit the databases in your business. Be sure to include those that are part of third-party SaaS applications as well as your own proprietary data systems; figure out the fields of information contained in each. Then you’ll need to coordinate with your team to understand the business purpose for which each of those data points is being used.

    The bulk of the work that developers typically do is that first step: auditing the business data system and documenting the database structures. At the very least, these are the things that you’ll need to be able to say about every field of personal information you possess:

    Source: How was it collected?

    Use Case: What is the business purpose for which it is being used?

    Storage: Where is it being stored for additional processing, transformation or record-keeping?

    Access: What business units have access to the data?

    Retention: What is the schedule for deleting this data or renewing the user’s consent to possess it?

    Consent: Was this data obtained through explicit customer consent?

    If you’re a small business, sometimes a data map can be “visualized” with a simple spreadsheet, but it very quickly uses its utility as data infrastructure scales. Larger businesses often find it very useful to construct a data map like an actual map, with a visual representation of the data relationships that exist in the business. In this form, a data map most often contains nodes and links to show how different systems that contain any personally identifiable information link together in your organization.

    Here’s one built by Ethyca:

     

    As you can imagine, without automated tools it takes a good bit of grunt work to get to this point of elegant visualization. For those just getting started mapping data in their business, we recommend a hybrid mapping approach.

    The Hybrid Approach to Mapping

    The hybrid approach means using an automated baseline process to identify and record the individual personally identifiable information that your organization is processing into one central repository, and then using manual label analysis data analysis to annotate and verify entity relationships across applications in your stack.

    Label analysis involves using a data attribute’s name to decide which data dictionary element the attribute identifies. For example, an attribute named ‘email_id’ can be identified as ‘person.contact.email-address’.

    When the attribute’s name alone is not enough, the name of the entity to which the attribute belongs is included in the analysis. For example, the attribute named ‘name’ in an entity named ‘customer’ can be identified as a name of a person, whereas a similarly named attribute in an entity named ‘city_master’ can be identified as a name of a city.

    Sometimes a group of attributes occurring together can be identified easily. For example, attributes with names ‘lat’ and ‘lng’ can be identified as latitude and longitude of a location. Similarly, attributes with names ‘f_name’ and ‘l_name’ can be identified as parts of a person’s name.

    Data analysis lets you manually confirm the relationships posited in your label analysis by using actual data from your databases to test.

    The presence of certain values or patterns in the actual data can confirm certain mappings. For example, if the data for an attribute follows ‘foo@bar.com’ pattern, it can be confirmed that this attribute captures an email address.

    Range values captured in attributes of different tables can be checked to see if those attributes are referring to the same piece of information, such as a user_id, that can be used for linking the tables with specific relationships. Similarly, the absence of values in an attribute can confirm the opposite.

    If you’re not using an off-the-shelf data mapping solution, the label and data analysis processes can be arduous. But if done correctly, by the time you’ve completed the exercise, you’ll have a comprehensive understanding of how the data in your business links together, and you’ll be ready to start answering the compliance-related questions about each PII field you hold. You can also begin thinking about processes for maintaining the map’s accuracy as your data inventory flexes and evolves.