devopsy.kinsta.cloud

Category: DevOps Practice

  • Dell Launches Project Frontier Initiative to Advance Edge Computing

    Dell Launches Project Frontier Initiative to Advance Edge Computing

    Dell Technologies this week launched Project Frontier initiative that promises to make deploying software at the edge simpler using integrated platforms that will be available in 2023.

    Announced at a Dell Technologies Summit event, Project Frontier will create an integrated edge computing platform for vertical industry segments such as manufacturing and retail.

    Dell CEO Michael Dell told conference attendees these platforms will help customers simplify, optimize and scale their edge applications in a secure way.

    The offerings can be centrally managed either by an internal IT operations team or via managed services that Dell provides to simplify onboarding and ensure zero-trust security. The overall goal is to reduce the number of instances that might require IT personnel in the field to configure and update them.

    According to a 2022 IDC survey, 42% of businesses say the most challenging aspect of edge deployments is putting together an entire edge solution. To address that challenge, Dell is also launching a Dell Edge Design Program through which it will collaborate with customers to further advance edge computing solutions.

    Through its Dell Validated Design program, the company is already building edge computing platforms for specific customers, but the goal now is to make it simpler to deploy edge computing platforms at scale.

    The number of application workloads being deployed on edge computing platforms is rising sharply as organizations seek to process and analyze data closer to the point where it is created and consumed. In fact, that shift is at the core of many digital business transformation initiatives that depend on edge computing platforms and devices running, for example, internet of things (IoT) applications to process data in near-real-time.

    A major issue—and one that remains largely unsolved, in many cases—is that organizations lack the ability to consistently deploy software on edge computing platforms. That results in a hodgepodge of homegrown tools deploying software on a wide range of edge computing platforms using manual processes that don’t scale well.

    It’s too early to say just how much software will ultimately be deployed on the edge, but there may come a day when there is more software at the edge than there is in the cloud. Of course, the real challenge will be unifying the management of DevOps processes, regardless of where any given piece of software might be deployed. In fact, Dell is betting that as edge computing continues to evolve, the line between edge computing and cloud computing will continue to blur until they are little more than natural extensions of each other.

    Dell, of course, is not the only IT vendor that views edge computing as the next trillion-dollar IT opportunity. There is already no shortage of options. The issue will be determining which IT vendor has the most robust edge computing ecosystem. In the meantime, DevOps teams would be well-advised to revisit DevOps workflows now with an eye toward making sure they can truly scale. After all, as edge computing platforms continue to proliferate, their ability to build, deploy, update and secure software is about to be put to the ultimate test.

  • Choosing the Right API Solution

    Choosing the Right API Solution

    There are more choices than ever when it comes to APIs–and that’s a good thing. But it also means you’ll need a strategic plan for choosing the right API solution.

    Two questions I’ve found can help guide that decision: 

    1. What problem are you trying to solve right now? 
    2. What functionality will you need in the next five to 10 years? 

    Once you’re clear on the answers, you can evaluate API solutions based on the following five criteria to choose the one best suited to your needs.

    Performance

    In a highly competitive global market, it’s not enough to have reliable uptime. Speed also matters–a lot. A site that loads in one second has a conversion rate three times higher than one that loads in five seconds.

    To accommodate current customer expectations for page speed, API response time should be no greater than 90 milliseconds. One way to blazing-fast API speeds is to take a client-side approach where data is delivered to an edge node of a CDN network instead of stored on a server somewhere in your data center.

    But let’s return for a minute to the first question I introduced above: What problem are you trying to solve? If you’re an online retailer, you want your site to load fast so your customers don’t click away and you want everything on it to be accurate–for example, if an item sells out while a customer is browsing, they shouldn’t be able to put it in their cart.

    Not every function requires the same level of speed. Product description content, for example, is mostly static, while inventory is not. So content delivered by your CMS’s API may not require the same speed as data from a transaction engine. 

    While high performance is important for every API you use, the actual performance benchmarks each one needs to meet vary based on function and on the end-user problem you’re solving.

    Documentation

    Thorough documentation is a must-have for any API solution you adopt. Without thorough documentation, your team will be forced to figure out how to use the API via trial and error supplemented with time-consuming support interactions. Strong documentation lets developers use APIs more effectively and, ultimately, ship products to market faster.

    But even among providers that offer thorough documentation, there are more and less helpful varieties. I’ve read plenty of documentation, for example, that is written with the assumption that the person reading it already understands how the API works. This makes it harder for newer developers to learn and understand the capabilities of an API.

    As you consider providers, look for those with documentation written for someone who is totally new to the API–which you and your team will almost certainly be. As a stylistic guide, look to documentation from Mozilla, MDM and W3 schools, which I think do an excellent job of communicating for non-experts.

    To gauge the helpfulness of a potential provider’s documentation, I recommend having your team read some of it and asking the partner team how often they update it. A best-case scenario is that they consider documentation an ongoing project that will never be complete.

    Security

    Security is pretty self-explanatory, so I won’t spend a ton of time on it here. Any API provider should offer robust security to ensure that the API cannot be used nefariously by bad actors or hacked to expose sensitive information. REST APIs almost always follow standards like 0Auth and TLS to maintain security. It’s worth looking at how the API you’re using describes its security practice.

    Flexibility and Extensibility

    Your team should have the freedom to create whatever they want with the APIs you use, meaning they shouldn’t be hampered by roadblocks in the API’s design. This freedom to create should exist in all aspects of the API: It should let your team create whatever customer experiences they can dream up, and it should let you manage back-end information in whatever way makes sense for your organization.

    In other words, an API solution should be well-defined but should not define everything you can do with it.

    Pricing Model

    The final piece to take into consideration is the API’s pricing model, by which I mean not only its actual cost but also the way the provider charges to use its service.

    For example, does it charge by order? By bandwidth? By number of records? Again, this comes back to the problem you’re trying to solve: If you have a lot of requests to make, for example, don’t choose a service that restricts requests per minute.

    And even if a pricing model works for your business today, will it work as you scale? This question brings me to my final thought.

    API Solutions Should Make it Easy to Grow and Evolve

    I started this piece with two key questions to guide the process of choosing API providers: What problem are you solving and where is your business going?

    If you’re exploring APIs as part of a replatforming project, there’s a good chance you chose a platform 10 years ago that could do everything–but that can’t keep up with evolving customer expectations or that restricts your team’s creativity today. The APIs you choose should let you meet current needs and make it easy to make changes–including swapping one API provider for another–as your needs evolve.

    In other words, when you choose the right API solutions, you should never need to do another replatform.

  • Keeping the DevOps Pipeline Flowing as Attack Surfaces Grow

    Keeping the DevOps Pipeline Flowing as Attack Surfaces Grow

    The attack surfaces that today’s businesses and public entities must manage have never been more complex and difficult to protect. The introduction of cloud and SaaS offerings over the past decade has caused seemingly insurmountable IT sprawl, with a third of successful attacks now occurring via unmanaged or virtually unknown assets. And while shadow IT is an urgent threat confronting many organizations, the migration of the software development life cycle to the cloud—and the rise of machine identities—is compounding the attack surface in profound ways. Emerging out of the Agile and lean strategies for software development in the 2010s, DevOps predates the mainstreaming of cloud computing.

    But once the value of cloud-based IT resources came to be widely appreciated, development teams began moving their developer environments to the cloud in a big way—to the point where many people today think of DevOps and cloud as inseparable. Recent Gartner research showed that much of enterprise IT spending is now directed toward cloud projects. In fact, for IT categories like application software, infrastructure software, business process services and system infrastructure, more than 51% of IT spending will shift away from traditional solutions to the public cloud by 2025. Nearly two-thirds of spending on application software alone will be directed toward the cloud by 2025.

    The explosion in cloud DevOps has brought with it a huge uptick in the number of privileges, credentials, identities and access rights under management. We’re all familiar with how digital identity and access management (IAM) works in conventional settings through combinations of usernames and passwords. In the DevOps world, where cloud containers can run as distinct microservice processes interacting with other processes (machine-to-machine communication), identities and privileges are usually governed through SSL/TLS certificates. These machine identities have emerged as a critical security priority across multiple industries, not just because they’ve proliferated exponentially, but because of the lack of effective tools and strategies available to IT security teams.

    Machine Identities—Scope of the Issue

    A recent global survey of 1,000 CIOs found that the average organization has 250,000 machine identities under management coming into 2022, with that number expected to double by 2024. The study also found that 83% of organizations suffered a machine identity-related outage in the past year, with 57% of organizations experiencing at least one data breach or security incident related to compromised machine identities over the same time period.

    Because the perimeter defense solutions and strategies that formed the basis of IT security best practices for many years are of little use in cloud environments, security admins have had to rely on existing IAM and privileged access management (PAM) tools to secure data, accounts, and development pipelines. Not surprisingly, identities and access privileges have become the leading attack vector in the cloud, providing attackers with an exponentially higher number of vulnerabilities to target within the enterprise than were available even a few years ago. We’ve seen these points of weakness come into play in recent high-profile breaches and attacks such as SolarWinds, Codecov and PHP.

    Embracing Zero-Trust Principles to Secure the DevOps Pipeline

    Fortunately for today’s DevOps teams, cloud-oriented privilege management technologies have evolved rapidly in recent years, especially in the categories of cloud infrastructure entitlement management (CIEM) and time-based role access management or just-in-time privileging. CIEM solutions target the security weak points and process vulnerabilities that are unique to the cloud environments. These include the lack of deep visibility into user, group, and role privileges, which together make it difficult to conduct oversight and control over the activities of users within cloud infrastructure and applications. Where conventional approaches such as IAM and PAM are designed primarily to secure data, apps, and users within the organization, CIEM is designed to manage permissions risks to prevent cloud breaches and data theft.

    Emerging CIEM solutions are bringing to bear advanced capabilities in discovery, monitoring and auditing to significantly broaden visibility into DevOps processes for IT security professionals. With a CIEM solution, security teams can scan cloud applications for existing users, groups, roles and granted privileges down to the resource level. Each user, group, role, and privilege can be categorized to identify and verify that privileged users have the appropriate levels of access. Access reports can be generated to concisely outline users and groups with the most privilege and the highest risk.

    On the audit front, CIEM solutions provide the capability to track and report on levels of privilege as well as all activity associated with those privileges. In this way, it becomes possible to generate privileged access reports for consumption by internal and external auditors to ensure compliance with regulatory and other requirements such as SOX, GDPR, HIPAA, GLBA, PCI-DSS, ISO 27001, SOC 2, FedRAMP and more. Similarly, with the ability to monitor privileged access across all cloud applications, security teams can apply behavior analysis to identify privilege abuse and abnormal activities from users. As with conventional SIEM tools, CIEM solutions will generate alerts to administrators to suspect activities or behaviors—for instance, if a user attempts to circumvent policies preventing direct privileged access to cloud services.

    Just-In-Time Privileging and Zero Standing Privileges

    Importantly, CIEM solutions are also effectively eliminating standing privileges to cloud resources based on least-privilege access and zero-trust principles. The automated granting and expiring of permissions—just-in-time (JIT) privilege grants—is highly effective at minimizing attack surfaces. Because these JIT/zero standing privilege (ZSP) solutions work on a zero-trust model, no one and nothing is trusted with standing access to cloud accounts and data.

    With JIT permissioning, elevated privileges can extend either for the duration of a session or task, or for a set amount of time. Once the task is complete, those elevated privileges are automatically revoked–all without sys-admin involvement. Where a user previously had standing access privileges potentially extending around the clock for months at a time, converting to JIT granting would compress that attack surface to several hours per month. Further, JIT permissioning largely frees organizations from having to maintain and pay for both privileged and non-privileged accounts. Dynamic secrets generation – where a dynamic secret is generated on demand and is unique to a client, instead of a static secret being defined and shared ahead of time – also provides a better model for securing temporarily deployed services and features.

    DevOps and DevSecOps are still new and fast-evolving concepts within the wider computer science and cybersecurity universe. No doubt, DevOps has been wildly successful in accelerating automation and speeding time to market for innovative applications and business services. To date, however, security solutions providers have struggled with cloud access management. In other words, they have struggled to accelerate privileged access solutions that could secure the devices, data, and resources used by DevOps teams, especially in cross-cloud environments. Dynamic permissioning platforms using JIT privilege grants and employing ZSP principles show great promise in solving these problems.

  • Kill the Password: Google on Board | 4-Day Week Proves Worthy

    Kill the Password: Google on Board | 4-Day Week Proves Worthy

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Passkeys is getting another big-tech supporter, and the four-day workweek train picks up speed. (more…)

  • Microsoft Expands DevOps Portfolio in the Cloud

    Microsoft Expands DevOps Portfolio in the Cloud

    At the Ignite conference this week, Microsoft launched a slew of updates to its DevOps portfolio as part of an effort to make it simpler to build, manage and secure cloud applications.

    Microsoft is adding Azure Deployment Environments, available in preview, that enable developer teams to invoke a set of best practices to quickly spin up application infrastructure using project-based templates.

    Microsoft is also making generally available Visual Studio Image for Dev Box, a self-service environment for accessing workstations on the Microsoft Azure cloud. Those images make it simpler to access Visual Studio functions and complimentary tools. Microsoft has also made generally available the latest version of the Visual Studio 2022 application development platform.

    In addition, Microsoft has signed an agreement with Postman to collaborate on integration that promises to accelerate the development of application programming interfaces (APIs) starting with the ability to both import and export APIs from Azure API management into Postman.

    Microsoft is also making generally available a predictive tool that employs machine learning algorithms to automatically scale virtual machines on Azure. In addition, updates to the Azure Monitor Log service now make it possible to ingest logs at a fifth of current costs and archive them for up to seven years. ​​​​​​​Microsoft is also now providing tools to migrate from legacy log agents to the Azure Monitor Agent (AMA) and has added Java profiling with a Java Flight Recorder (JFR) tool that collects data from a Java virtual machine and Java applications.

    In terms of managing application environments, Microsoft has now extended the reach of Azure Automanage to now include servers managed via the company’s Arc management platform, in addition to virtual machines running on the Azure cloud. Other capabilities added to Azure Automange include enhanced backup settings and different auditing modes for the compute server baselines and the ability to specify custom Log Analytics Workspaces and Azure tags to identify resources based on settings.

    Microsoft is also previewing an enhanced Azure remote support service through which Microsoft personnel will collect logs and commands to help resolve issues.

    Finally, in terms of securing applications, a GitHub Advanced Security for Azure DevOps tool that, now available in private preview, promises to make it simpler to identify and fix exposed secrets and vulnerable open source dependencies across the software life cycle using a CodeQL code scanner.

    At the same time, Microsoft is making available Microsoft Defender for DevOps to better secure code created to configure clouds. In addition, it will also enable DevOps teams to better prioritize their remediation efforts across multi-pipeline DevOps environments spanning multiple clouds. Microsoft Defender for DevOps is available in preview and supports GitHub and Azure DevOps with future support for other DevOps platforms planned.

    Another security addition available in preview is Microsoft Defender Cloud Security Posture Management (CSPM), which promises to provide contextual risk-based information via attack path analysis surfaced using a cloud security graph service to identify the most exploitable cloud resources. Microsoft has also made generally available a multi-cloud security framework for Microsoft Defender for Cloud to help organizations adopt DevSecOps best practices.

    Microsoft is also previewing support for agentless scanning for Microsoft Defender for Servers and an expansion of Microsoft Defender for Containers that adds support for Amazon Web Services Elastic Container Registry.

    Microsoft CEO Satya Nadella told conference attendees that Microsoft is making a concerted effort to natively embed security capabilities within its platforms to reduce the total cost of security. Many of the tools DevOps teams rely on today don’t integrate well, which is challenging at a time when application environments are becoming more dynamic as more cloud-native applications are deployed.

    It’s too early to say how much better security will entice organizations to opt for one cloud platform versus another. But it is almost certain that cloud service providers will be expected to contribute to securing software supply chains.

  • GraphQL: Security by Obscurity Just Isn’t Enough

    GraphQL: Security by Obscurity Just Isn’t Enough

    The debate about how to secure GraphQL rages on. Many organizations are hesitant to adopt GraphQL for public-facing APIs as there is no precise method to handle authorization concerns as of yet. Without a role-based access layer to enable fine-grained permissions for each field (and underlying services that GraphQL might wrap), the query language can be prone to access control issues.

    One method to hamper GraphQL access is to turn off introspection features altogether. Some may assume that hackers can’t perform introspection on an endpoint if it’s not visible. But this isn’t entirely true—even though an “internal” endpoint may not be fully documented, any endpoint on the public web can be searched and introspected quite easily by brute-forcing queries or through field suggestions.

    I recently met with Karim Rustom, cybersecurity engineer, Escape Technologies. According to Rustom, “making something obscure isn’t enough to make it secure.” Leveraging a single slip-up in access control could be used as a doorway into these systems. Although it’s a best practice to limit introspection, it’s likely not enough to protect production GraphQL endpoints, he said. Below, we’ll consider some common GraphQL threats and alternative solutions to retain a safe GraphQL implementation.

    Common GraphQL Threats

    Having run thousands of scans on production GraphQL endpoints, Rustom found common trends emerge around GraphQL threats. First, he noticed the potential for denial-of-service attacks to be quite high. GraphQL, by design, is intended to query whatever data the client requires. But sometimes, these queries can return a mountain of data.

    If GraphQL owners don’t place a limit on what can be requested in a single query, the platform can easily fall victim to denial-of-service attacks and service interruptions. And limits aren’t only necessary for bulk read-only operations—although GraphQL is primarily used for fetching data, some methods might initiate intense server-side processing, which should be limited in some form. According to Rustom, such data manipulation methods are especially pervasive in FinTech, food delivery or health care-related applications.

    Another rampant GraphQL threat lies in access control issues. This comes as little surprise, as access control is ranked as OWASP’s most common web application security risk. Often, when creating new objects or new queries, a developer might forget to add the correct authorization, said Rustom. This can become a huge problem and lead to privilege escalation.

    Finally, too many production GraphQL implementations rely on security by obscurity. “Folks think that disabling introspection is a layer of security—we highly stress that it’s not,” said Rustom. He recalled performing penetration testing for a certain client; even though they had disabled introspection, the team was able to fetch ample open paths and discover an unsecured doorway to divulge secret internal knowledge.

    Solutions and Best Practices

    To avoid some of the vulnerabilities presented above, it’s recommended to first place request limits on overly complex GraphQL calls to prevent denial-of-service attacks. One nuance here that Rustom recommended is to consider request complexity—not in terms of the query size, but the overall processing required. He suggested assigning scores for every query and permutation to help inform the decision about whether or not to process the incoming call.

    Furthermore, it’s recommended to disable field suggestions as the feature could be prone to abuse. Using field suggestions, hackers can leak a lot of information to build a partial or complete introspection. These tactics are similar to the word-busting techniques black hats use to perform introspection on REST endpoints.

    Although it’s a good practice to limit the possibility of surveillance, hackers can still discover systems and leverage other common vulnerabilities like broken access control. Rustom suggested other tactics, such as luring attackers into a honeypot or logging them into a fake account. Or, creating error messages that are intentionally misleading for a attacker.

    To retain a safe GraphQL implementation, organizations can use pre-made free and open source tools to audit their surface area and scan for known vulnerabilities. For example, GraphQL.security quickly scans your GraphQL URL endpoint to discover vulnerabilities. Or, Graphinder can examine a subdomain to discover GraphQL endpoints, which could aid auditing purposes. There’s also middleware that wraps a GraphQL service with additional security, such as GraphQL Armor and GraphQL Shield.

    Future of GraphQL in the Market

    There appear to be many flavors of GraphQL throughout the market. We’ve seen it emerge as a mechanism to power backend microservices. We’ve also witnessed the technology adopted as a layer to centralize multiple REST services with a more usable data retrieval mechanism for frontend engineers. Some GraphQL endpoints also run on the public web to serve end clients and external partners.

    “GraphQL is a beautiful technology, just lacking in security,” said Rustom. And unfortunately, his outlook for API security looks grim—amid political turmoil and economic instabilities in regions like Europe, some companies may seek to cut cybersecurity investments to stay lean. For companies that haven’t yet suffered a breach, security is easy to ditch to slim budgets, but it should remain a priority, said Rustom. “Keep securing GraphQL.”

  • Fear of the Unknown

    Fear of the Unknown

    We IT folks are pretty fearless. Because it is newer technology, and change in IT is easier than in areas like heavy manufacturing, we are regularly thrown new tools, products and methodologies and expected to be able to master them. And we do. And it works. If your career is more than a couple of years old, you are not doing what you started out doing. DevOps and Agile have made this process more pronounced because they increased the rate of change, but this phenomenon has been going on since the PC became readily available. Someone somewhere is currently developing your next toolset. And when they do and it gets thrown at you, you’ll figure it out and run with it.

    The problem is that our architectures are never black and white, and we should fear the unknown more. The unknown residual cruft that is all over our networks. Doesn’t matter if “networks” means data center, cloud or both, there’s residual cruft on it. Some organizations even have a full-time position finding cloud instances and services no longer in use and eliminating them. But it is much more broad than cloud instances; there is garbage all over your network and it is time to try to clean it up.

    Most of us believe that we can’t possibly get a handle on everything that is out there. Oh, zombie containers and VMs are relatively easy, but the modules and libraries we loaded up and then, through the natural evolution of the software, stopped using, or whatever was bundled in with the OS chosen for that particular project, but was never used—it’s all over, it’s hidden and lots of it isn’t easy to find. That doesn’t even touch on the software that one employee from accounting absolutely had to have—but that employee is long goneand no one in accounting is willing to remove the software, either because they’re unaware of its existence or because they are unaware of what purpose it might be serving.

    There are tools to help with some of this, like SBOM generators for excess software on instances, and application maps that work much like network maps to help you understand what is interrelated. But they’re not perfect. They can’t be. You’ve been building this spaghetti application infrastructure for years; it’s going to be tough to work out. Get a list of what’s deployed, tag a person as responsible for it–that app in marketing? The person who requested it is responsible for it (or their manager is, if the original person is gone). IT is responsible for cataloging and, in some orgs, managing the app, but the business unit can explain why it is in use, and if it is still necessary.

    It’s a ton of work and, as a former enterprise architect, I can confidently say that it won’t be easy getting names attached to apps. Some apps you might have to turn off to see if they are still in use … and the work will likely never be complete because the infrastructure is changing non-stop as you work.

    But it’s worth it. It saves the org money; it reduces IT footprint and it reduces security exposure. It may even save IT staff time, which is one of the most precious commodities your organization currently has. So, start wrangling. Start with one business unit and ask them to help you–after all, they are almost guaranteed to have running apps or SaaS that IT is unaware of. The goal is not to take control of those apps and relationships, but to inventory them so that when they’re no longer needed they can be cleaned up. A side benefit is that if four business units are using similar software outside of IT’s purview, it may be possible to merge those solutions for a corporate approach … But that’s a blog for another time.

    Meanwhile, keep rocking it! Just remember there is more on the cloud and in your data center than are dreamt of in your architecture plans (many apologies to Shakespeare). And keep solving the business’ problems.

  • 5 Ways to Transform DataOps With Human-in-the-Loop Automation

    5 Ways to Transform DataOps With Human-in-the-Loop Automation

    We are in the middle of a data renaissance. Today, it’s not just about data instrumentation but also learning how to make DataOps a real business advantage for the entire organization. Data has become an inextricable part of products, used to enhance the quality of user experiences. Reliable access to data and the integrity of that data is imperative to drive innovation and business success.

    But just like any process, problems will arise. There will be pipelines that break or data that isn’t instrumented properly. The question then becomes, how can teams more quickly identify problems and continually improve? How can the journey toward data integrity become one of iteration and improvement?

    DataOps teams are by no means strangers to automation—automated CI/CD pipelines and infrastructure-as-code (IaC) are a huge part of their skill set. But there are some components of automation that are less pervasive throughout DataOps, ones that can help promote data integrity and encourage a culture of improvement. 

    It all starts with a different approach to automation—a human-in-the-loop (HITL) approach. Human-in-the-loop automation enables parts of a process to be fully automated while enabling humans to step in at critical points to take action, make decisions and decide the path forward.

    Here are five ways to use human-in-the-loop automation to drive continuous improvement and achieve optimal data integrity.

    Create Better, Earlier Interruption of Problems

    As is life, inevitably things break, issues arise and teams need to step in to remediate. The goal should be to enable better interruption of these problems when they do occur—identifying issues earlier and bringing the right stakeholders together more easily and swiftly.

    HITL automation can help teams not just identify issues earlier but kickstart the incident response and coordinate across teams and stakeholders. Ideally, automated workflows can be triggered from an incoming signal; for instance, from an observability tool like DataDog or BigPanda. An incident response could be started automatically, triggering further automation—like creating tickets, starting a Zoom meeting, creating a Slack channel and bringing together the right team members. 

    By instantly bringing the right people together, the team can investigate collaboratively and more quickly understand the state of the issue. HITL automation should then enable teams to run further scripts to take action and remediate as needed.

    Bake Improvement Into the Process Through Self-Documenting Workflow

    Many companies today are asking, “How do we make sure that we have that full integrated loop, down to the collection point?” Communication between teams is critical to this endeavor. But even before teams can collaborate on improvements, they need to understand the current state of their data so that if, for instance, a team finds that a report isn’t providing what’s needed, there’s a really clean way to get that communication back to the DataOps team to improve the instrumentation in the first place.

    HITL automation can bake improvement into DataOps workflows by automatically documenting every human and machine action throughout a process. Because it’s connected through APIs to all a team’s tools and services, a self-documenting workflow slurps up every action and creates an audit trail. This audit trail is the basis by which teams can more accurately understand not just the integrity of their data, but also how teams are implementing and running processes.

    Bring Human Data to the Forefront of Improvement

    DataOps teams play a big role in making sure the right data is being collected. However, a lot of people are still using systems that are very manual. They may have a good understanding of their log and monitoring data but still don’t have a good sense of the human data around their organization—what processes and workflows people are running.

    Understanding how humans run processes is just as important as understanding technology problems. The self-documenting workflow described above should include documentation of human data, from manually-run scripts to Slack or Teams conversations and, ideally, even auto-update tickets so even maintaining a system of record becomes less manual. Only by looking at the full picture—of human and machine data together—can teams find areas of improvement in their automation, communication and workflows.

    Take an Incremental Approach

    Don’t let “perfect” be the enemy of “done”. Every great transformation I’ve seen started with a small team, a great mandate, and strong support from the executive team. Adding tons of new technology and hiring more people may not be the silver bullet teams need.

    Instead, teams should look at the expertise they have in-house first—the people, technology and processes. Then, begin by taking an incremental approach. HITL promotes incremental, approachable automation by enabling teams to first codify institutional knowledge, analyze, learn and then begin to automate pieces in small batches.

    Diving head-first into a fully automated approach can leave teams in a worse state than nothing at all. Automation is a journey—we learn by doing, by recording human and machine data and then, using insights as our guide, begin to find real value in small doses of automation. These small steps can make a huge impact and, over time, add up to substantial business value.

    Buy for Industry Standards and Build for the Gaps

    As I mentioned above, teams should first start by evaluating their current resources. What skill sets and technology do they already have and how can they use those resources to achieve their data goals? At the same time, many teams starting their automation journey face challenges—lack of development resources, scripts that live on one person’s machine, difficulty bringing all the human and machine data together into a recorded timeline.

    To help data teams effectively and reliably implement HITL automation, investment into an off-the-shelf platform can help bridge the gaps teams and organizations have. Platforms should enable automation through low-code interfaces while providing code-level customization. This combination enables teams to buy for industry standards while building for the gaps—not managing the platform means development resources can be focused on expanding and improving automation.

    Platforms should enable teams to easily connect to APIs and adapt to custom APIs. The continuous change of DevOps processes and tooling coexist with rapidly changing APIs. Therefore, the most promising automation platforms must manage the complexity of APIs, enabling users to focus on the intent, not the mechanics of custom APIs. Automation technologies should seamlessly handle identity, authentication, pagination and caching. 

    Find Leverage With HITL Automation and Drive Business Impact

    As data becomes increasingly ingrained in product life cycles and user experience, automation holds the key to faster innovation, product quality and enhanced reliability. Creating a culture of continuous improvement and implementing technology like HITL automation that promotes data integrity throughout the product life cycle is where I’ve seen the greatest leverage for organizations. 

    To hear more about cloud-native topics, join the Cloud Native Computing Foundation and the cloud-native community at KubeCon+CloudNativeCon North America 2022 – October 24-28, 2022.

  • Microservices Explained: Not Your Father’s SOA

    Microservices Explained: Not Your Father’s SOA

    Microservices are frequently referred to as a variant or derivative of service-oriented architecture (SOA), if not essentially the same thing. While there are similarities and both are designed around the concept of services, that’s where the similarities end. Each was created around a different set of principles and intended to address different problems.

    Microservices architecture is built around key concepts that, when applied, not only differentiate it from other architectures but deliver advantages that accelerate software delivery and help to scale large, complex applications. To deliver on the promise of cloud-native, microservices must be loosely coupled so that, ultimately, they are independently deployable. Any microservice can be updated and deployed—just that containerized microservice—without requiring changes to other microservice(s). Loosely coupled microservices mean they are not sharing databases or maintaining state, for example.

    How big should a microservice be? Or, in other words, how much should it do? The answers range from ‘a single thing’ to ‘a small set of related code with high cohesion’. Another factor is that getting too granular can overrun a microservices-based application with large numbers of microservices, increasing complexity and making triage through observability and tracing very challenging.

    Scaling microservices-based apps is another important factor. Autonomous microservices can be replicated and scaled for performance and workload using container orchestration tools such as Kubernetes.

    Microservice communications are achieved through well-thought-out APIs created around the business or operational capabilities the microservice provides to anyone or anything engaging through the API. The API, and data provided through the API, are often referred to as ‘technology-independent.’ It’s probably better stated that microservice APIs are built using approaches like RESTful and GraphQL over networks that are not tied to any one specific operating environment, technology or programming language. JSON, YAML and XML similarly represent data that is transferable across technologies.

    The following diagram of a media application demonstrates the principles of microservices architecture. Microservices for scheduling media, digital rights management, tracking impressions and adding new ad insertions are loosely coupled, autonomous and have well-defined API interfaces. Each microservice can be changed, enhanced and deployed rapidly and independently of other microservices.
    Microservices SOA

    Service-oriented Architecture, or SOA, emerged to solve a very different problem; the overwhelming size and complexity of monolithic applications. Monolithic applications have very large codebases that often don’t do a good job of compartmentalizing functional or business logic. This makes changes to the codebase challenging even to developers who know much of the codebase well.

    Services in SOA were often used to gather a collection of similar business logic together and then share them as a service with other parts of the application, as needed. Technologies like message buses are used to make requests of a service and operate as traffic managers for requests across multiple services and other parts of the application. While reusability was a critical design consideration, SOA services were still quite large, and deploying services at high velocity was not an important design consideration. SOA services often take considerable time to code and test the service including interdependencies across other parts of the application before deploying as part of a larger software release.

    One important note: There are benefits and drawbacks to microservices and SOA architectures; neither is ‘better’ than the other. Which architecture to use when mainly depends on the purpose of the application you are building. For larger, more complex enterprise application environments that require integration with many other applications, you might choose SOA. That’s a better fit than for smaller applications that don’t require middleware elements for managing the requests within the app or between other applications. Microservices, on the other hand, are better-suited for smaller and well-partitioned web-based systems. They are becoming the standard for today’s cloud-native applications, and also are a great fit for developing a mobile or web application because they give developers more control.

  • Allstacks Adds Alert Capability to VSM Platform

    Allstacks Adds Alert Capability to VSM Platform

    Allstacks has added an alert capability to its software-as-a-service (SaaS) value stream management platform that automatically notifies DevOps teams when software project goals and deadlines are likely to be missed.

    The core Allstacks platform employs machine learning algorithms and artificial intelligence (AI) models to keep track of pull requests, commits and card changes via a set of dashboards that are designed to be accessible to anyone in an organization.

    The Allstacks Alerts capability makes it possible to notify those stakeholders via email or Slack any time a specific indicator, such as a pull request, has been open for four days, which would suggest risks of delays have increased. Support for Microsoft Teams will be added early next year.

    Adam Dahlgren, senior vice president of product for Allstacks, said the goal is to leverage the advanced analytics built into the platform to surface thoughtful observability insights versus simply firing off a series of generic alerts that lack any context.

    Value stream management (VSM) platforms such as Allstacks automatically take technical metrics collected via DevOps tools and correlate them to key performance indicators (KPIs) defined by business leaders. The concept of value stream management traces its lineage back to lean manufacturing methods, which called for each step of a manufacturing process to be continuously measured. As software development has evolved from being a craft to a highly-automated process using DevOps best practices, an appreciation for the value of monitoring the impact of missed software development deadlines on the business has grown as organizations realize how dependent they are on software.

    In fact, many organizations are discovering there is a clear disconnect between their business goals and software development processes. Business leaders are especially anxious to understand how a software development project delay might impact future revenue projections at a time when economic conditions are the most uncertain they’ve been in recent memory.

    VSM platforms also make it possible to track KPIs spanning multiple software development projects; the hope is that doing so will make it easier to see where resources should be reassigned as various bottlenecks are encountered. In the long term, Allstacks is also working toward adding scenario planning tools to make it easier for IT and business leaders to model “what-if” scenarios, added Dahlgren.

    It’s not clear how much organizations are willing to invest in VSM platforms, but the need for them is becoming more acute. The challenge many organizations still face is making sure they have the tools in place to collect the appropriate technical metrics. A SaaS-based platform should make it simpler to achieve that goal and to apply algorithms to what can quickly become a large corpus of data.

    Of course, it’s still early days as far as the adoption of VSM platforms are concerned. But there’s no doubt the divide between application development teams and the rest of the business is continuing to narrow. Less clear is to what degree that increased visibility will lead to a higher degree of empathy between dev teams and the business.