devopsy.kinsta.cloud

Author: Bill Doerrfeld

  • How Will Low-Code and AI Coexist?

    How Will Low-Code and AI Coexist?

    Over the last few years, low-code solutions have become important to help organizations widen programming accessibility to a broader base of users and help fill development gaps. And these technologies are still gathering steam—Gartner estimated the market for low-code development technologies would grow to a total of $26.9 billion in 2023 and that 65% of applications would be developed using low-code by 2024.

    However, AI-based code generation is set to further level the playing field with advanced auto-completion and AI coding assistants. Take GitHub’s Copilot, a helpful AI pair programmer, or OpenAI’s ChatGPT, which can generate complex functions given natural language prompts. Other AI tools trained on UIs, like Galileo, can derive frontends for any project in a matter of seconds. So, are low-code development platforms still relevant in this new paradigm of AI-driven software development? If so, how will low-code solutions and AI coexist?

    Well, generic algorithms can only go so far in creating truly unique and innovative experiences that solve real-world business cases. And since most programmers don’t have the skills to train machine learning or deep learning models from their own datasets, this is one area where low-code solutions could lower the barrier, enabling users to tag unstructured data, generate models, run simulations and share reusable AI across departments.

    I recently chatted with Johanna Pingel, MathWorks AI product marketing manager, to discuss the role of AI in low-code programming. According to Pingel, low-code solutions can lower the barrier to AI functionality, both in quickly integrating AI-driven functionality into applications and seamlessly generating new AI algorithms based on unique datasets. Below, we’ll consider some additional use cases for AI and how low-code platforms may work alongside it.

    Use Cases For AI

    First off, what are application developers using AI for these days? Well, AI has an increasingly wide range of uses. It began with computer image and processing, such as identifying cats and dogs or postal codes. Now, voice recognition, voice-to-text and text-to-voice are commonplace, and AI is being used to create ever more sophisticated applications. One area that Pingel is excited about is applying AI to help with battery life management, predicting the lifespan of batteries for electric vehicles and consumer electronics.

    “Anywhere there’s data, there’s an opportunity to create models and make predictions based on that data,” said Pingel. This especially rings true for scenarios involving data and visualization. For example, consider visual inspection in a pharmaceutical production line to ensure there are no defects with any pills. Another use case is visual inspections of parts in automotive assembly lines. Applied to these scenarios, AI can remove grunt work and make production safer and more efficient.

    ChatGPT is another example of an AI application quickly becoming mainstream, not only for content creators but for scientific research. For example, ChatGPT can spin up MATLAB code and create functions and unit tests, which can be copied and pasted into IDEs. However, code written by ChatGPT should still be checked and verified by engineers and scientists to ensure its accuracy, said Pingel.

    How Low-Code Lowers the Barrier to Bespoke AI

    Low-code solutions can bring many benefits—these platforms help to speed up tedious tasks and level the playing field for those with different coding abilities and desires, said Pingel. Low-code platforms can also be a great tool for engineers looking to incorporate AI into their business systems. For example, having a low-code method could be helpful in generating code for machine learning algorithms and new models, which can help companies create their own algorithms and incorporate them into their own platforms.

    More people are interested in incorporating AI into their applications, but the use cases are growing increasingly more sophisticated. Businesses may start with simple needs, such as identifying faces or identifying text from handwritten letters. These rely on simple datasets, and algorithms already exist to accomplish most of these tasks. Off-the-shelf AI is helpful at the start, but eventually, companies tend to want to use their own data and settings far more specific to their applications, said Pingel.

    For example, a more niche case could be as specific as identifying cracks in a windshield. Or, in a financial industry, it might be analyzing mass amounts of loan applications to determine approval rates. In these situations, using your own internal data becomes necessary to create AI that understands the situation at hand. Yet, the hard part is assembling this data in a way that is accessible for a model to comprehend. Analysts estimate that upwards of 90% of data held within enterprises is unstructured, meaning it requires a heavy amount of tagging to become usable for AI/ML.

    Low-code is a perfect solution to avoid the point-and-click aspects of cleaning, cropping and structuring this data, said Pingel. This can introduce automation into the process of labeling data while still involving some human intervention to fix mistakes. Machine learning models can take a while to train and simulate, underlining the need for a fast way of iterating on models. A common platform could help companies develop custom machine learning algorithms and models and keep track of experiments. This helps to ensure that the most effective model is used for the final application.

    End Benefits of Democratized AI With Low-Code

    Overall, low-code platforms can provide a number of benefits that lower the barrier to unique AI creations. Low-code platforms like MATLAB and Simulink are useful for structuring and labeling data and quickly generating ML models. Low-code also makes it easier to bridge different programming languages and quickly generate code for a variety of different languages, such as C, C++ and Java.

    Additionally, low-code platforms enable users to collaborate easily with other users. But low-code doesn’t mean no-code, Pingel explained. In fact, “you can learn to be a better programming through low-code tools, too,” she said. For example, it could enhance understanding of the structure and syntax of code.

    In general, democratized AI is an important capability for businesses to take advantage of. Although AI-based solutions are coming to market to displace old tools and processes, there is still an argument to be made for low-code—namely in empowering non-technical users and lowering the barrier to creating custom AI-driven functionality.

  • Tips For Standardizing API Life Cycle Management

    Tips For Standardizing API Life Cycle Management

    Software life cycle management is becoming more critical for today’s technology-fueled organizations—because you can’t just release software into the world and expect it to work perfectly for years to come. More often than not, development teams must constantly adapt their software, progressively add features and cater to new use cases as they arise. This is nothing new for web APIs, which are prone to continual evolution and whose integrations require regular upkeep to avoid breaking changes.

    As such, the discipline of API life cycle management has gathered steam. This concept focuses not only on the design and development of APIs, but on the entire life cycle, including testing, monitoring, documentation, maintenance, versioning, CI/CD and even deprecation. Localizing a strategy for API life cycle management from the get-go can increase the longevity and usability of a platform. It also helps software providers treat their APIs as an iterable product rather than a one-and-done offshoot of a digital platform.

    I recently met with Rakshith Rao, co-founder and CEO of APIwiz, to gather insights on standardizing API management across an organization. According to Rao, there are many ways to implement governance to make these interfaces more consistent. For example, automated security analysis, contract-driven development and implementing style guides are just a few ways to wrangle an increasingly extensive API portfolio. Below, we’ll review these tips and others to consider how to best manage API life cycles, hopefully, before sprawl sets in.

    Four Tips to Increase API Governance

    Medium-to-large organizations might be handling thousands of APIs, said Rao. This new paradigm brings many opportunities for digital transformation, but it can also be a nightmare to manage. Too often, APIs aren’t built with tangible business requirements taking center stage.

    Many of these initiatives didn’t begin with consistent policies from the get-go. For example, it’s also easy to skip out on documenting internal services. But, fractured practices harm discovery and reusability. To rectify this issue, Rao shared many ways in which technical leads can promote better API hygiene.

    Enable Visibility of API Usage With API Monitoring

    The API life cycle spans stages of design, deployment, deprecation and eventual retirement. And throughout this life cycle, developers will likely release new versions of the same service. But although you want consumers to adopt the latest version, some large organizations must support legacy versions for years to come, explained Rao. As a result, different API versions are often developed and run simultaneously by various internal or partner teams. Sometimes, “no one turned the tap off and the water’s still running,” he said.

    To enhance visibility for all active (and non-active) sessions, it’s important to conduct ongoing monitoring. This will help discover individual client usage patterns and help avoid shadow or zombie APIs from forming. These forgotten endpoints could pose security vulnerabilities, said Rao, as you might not enforce best practices on older versions if you don’t know they exist.

    Automate Security Analysis to Capture Vulnerabilities at the Design Level

    Not having consistency between APIs could pose data compliance issues. One method to avoid errors is to conduct design reviews on an ongoing basis, and to automate this security analysis at the design level, said Rao. Perhaps authentication tokens need to be encoded, logging requests aren’t coming in, or payloads are structured incorrectly. Whatever the case, having a complete checklist to scan and performing ongoing security analysis early on to catch these inconsistencies can be a big boon to overall governance and long-term cybersecurity.

    Adopt Style Guides but Allow for Flexibility

    Very simple data modeling errors could severely impact a system, said Rao. Thus, it’s in an organization’s best interest to use standard taxonomies and ensure data governance is an overall aspect of life cycle management. One way to ensure consistency is by using specification-driven development—OpenAPI is a common standard for this need.

    Rao also suggested developing API design style guidelines but admitted that these guides become challenging to enforce and limit as an organization scales. This is because various teams in different departments will have different API needs and often design their interfaces for various front-end requirements. His takeaway is to centralize practices but take a GUI approach to configuring and sharing them. This way, API style can be “minimally intrusive but uniformly enforced.”

    Use an Abstraction Layer for Multi-Cloud

    A full 98% percent of enterprises are using at least two cloud infrastructure providers, according to a recent Oracle report. Nearly everyone is adopting a multi-cloud strategy, but this new reality brings novel pressures for DevOps, especially when overseeing the deployment of APIs across different cloud platforms. An API life cycle management platform decoupled from these CSPs could help wrangle polyglot deployments.

    Sticky Notes Aren’t Enough for API Management

    API life cycle management goes well beyond the development and runtime phase. And as more API-first development occurs within organizations, maintaining standard interfaces will naturally require more effort. Rao also advocates for increased infrastructure automation and low-code solutions to meet these new challenges. Automating various parts of the API life cycle so there is less manual work “helps to bring down the guesswork of ensuring reliability when you release an API,” said Rao.

    We’re seeing some enterprises move from supporting hundreds to thousands of APIs. In this new paradigm, “API sprawl is imminent,” said Rao. As such, it will become virtually impossible to manually keep these interfaces in line, understating the need for more automation. Especially during times of economic uncertainty and leanness in the tech market, the democratization of such API management automation will be vital in helping organizations move faster.

    Having broken pieces of information and scattered API contracts will lead to a messy and potentially insecure adoption. As such, API management can no longer be done using sticky notes on the edge of a computer screen, said Rao. Taking the guesswork out, centralizing common policies and shifting more legwork behind the scenes will be required to avoid a splintered portfolio of services.

  • Understanding the Infrastructure Automation Maturity Model

    Understanding the Infrastructure Automation Maturity Model

    Nearly every company wants to evolve its digital transformation and increase the pace of software delivery and operations. And infrastructure automation is a new means to achieve these ends. It supports the emerging discipline of platform engineering and can increase scalability and reactivity to unforeseen events, enabling software ecosystems to be more anti-fragile. Increased infrastructure automation also helps free up engineers from focusing too much on managing DevOps when they really should be building out new features.

    But introducing more infrastructure abstraction can be met with certain challenges. This is often because departments have a wide range of cultures, processes and tooling already in place. With varying maturity levels across teams, it can be challenging to gauge where a company is before attempting to introduce new automation across the board. And as further evolution may require significant changes, having a baseline is key to planning for the future.

    I recently chatted with David Williams, senior vice president of market strategy at Quali, about an internal technique they’ve been using at Quali to gauge the maturity of their clients’ infrastructure automation. Their infrastructure automation maturity model breaks down the journey into four unique stages, moving from the least-mature ad-hoc automation to the most-mature autonomous solutions. Below, we’ll review this model and consider what questions need to be asked to ascertain where your team sits on the spectrum. We’ll see how enterprises can leverage such a model to hone in on improving their software operations.

    Four Questions to Determine Infrastructure Maturity

    DevOps is still a wild west in terms of how people approach it, said Williams. So before attempting to mandate sweeping cultural changes, it helps to first determine the base level of DevOps maturity within an organization. Here are four questions that can quickly assess the state of infrastructure automation within an organization.

    1. What Culture Already Exists?

    First, it’s good to evaluate the culture that already exists. How do teams deliver value and how is this value measured? Perhaps throughput or code quality is measured with specific metrics or SLAs. Understanding the DevOps attitude (and aptitude) within the organization will highlight pre-existing capabilities and show how teams collaborate.

    2. How Are People Organized?

    The genetic makeup of teams can dictate much about development and operations. How are teams organized? Are they large, siloed teams? Or are they two-pizza-sized teams? Understanding team organization and communication patterns are necessary to identify bottlenecks.

    3. What Processes Are Already in Place?

    Next, consider the development timelines and procedures already established and in use. Are teams applying SAFe, Lean, Agile or Scrum methodologies? What are these exact processes and how mature are they? Answers to these questions will demonstrate general release cadences and how teams drive productization.

    4. What Infrastructure Automation is Already Being Used?

    Lastly, you’ll need a window into what level of infrastructure automation, if any, is already in use today. How is it delivered? How is it managed? How is the success of this automation measured, and what is the overall outcome? You will likely find that automation is typically quite siloed and teams have differing approaches. This auditing is vital to identify what works well and what doesn’t.

    The Four Levels of the Infrastructure Automation Maturity Model

    Asking the above questions will quickly determine an organization’s “readiness state,” said Williams. But how do these answers map to the overall maturity of the organization’s infrastructure automation? Well, he shared four basic levels of infrastructure automation maturity that are commonly found in practice throughout today’s software development environments, ranging from bespoke scripts to more robust self-defining automation.

    Level 1: Ad-hoc

    The most basic level of infrastructure automation is ad-hoc integrations involving a mix of old and new technology. This level is chaotic and fragmented, with varying processes used throughout the organization. These bespoke integrations and scripts also may not have excellent visibility, limiting the potential for reusability within an organization.

    Level 2: Automation

    Next is having automation that is specifically tied to a certain technology stack. Nowadays, organizations are commonly hybrid multi-cloud, having adopted automation piecemeal with each new stack. This often results in multiple silos of automation stacks that are blind to each other. At this maturity level, one automation might fire off another automated procedure and so on, but engineers aren’t 100% aware of how they influence each other.

    Level 3: Frictionless

    The next level of infrastructure maturity begins to take the reins with more intentional automation adoption. At this stage, organizations embrace an internal developer platform, similar to Spotify’s Backstage, that helps engineers deliver and manage infrastructure more seamlessly. This tooling is more advanced than other levels integrating key commands into the developer’s IDE. Doing so eliminates the need to switch between environments and reduces friction.

    Level 4: Self-Defining

    Finally, the most advanced form of automation involves deeper intelligence within application operations, including the ability to dynamically optimize infrastructure. At this stage, infrastructure is self-defining, meaning production applications can automatically adapt, be self-governed and enable bursting and non-bursting capabilities. Such intelligence is typically proprietary to cloud providers, said Williams, yet companies often desire this intelligence inherent at the application level. Williams describes this stage as the “intelligent allocation of resources in line with application life cycle requirements.”

    Benefits of Self-Defining Infrastructure Automation

    Self-defining infrastructure automation has many benefits for companies, including having the infrastructure in line with application needs, scalability and measuring usage to optimize cloud spending. Many points within the software development life cycle could warrant increased automation, such as infrastructure-as-code (IaC), deployment release engines, configuration, traffic scalability, cloud management, containerization and documentation. But it’s not until you open conversations around infrastructure automation maturity and understand who is responsible for managing it that you can hone in on what areas can be improved.

  • How To Build Anti-Fragile Software Ecosystems

    How To Build Anti-Fragile Software Ecosystems

    Just as organisms are susceptible to diseases and viruses, software systems are susceptible to hacks and errors. And within many complex interconnected systems, a minor bug could have a cascading effect across an entire organism, causing it to break. As such, increasing the resiliency of a software ecosystem is something engineers continually strive for. But how do we get there?

    One concept to consider is anti-fragile. Being anti-fragile refers to the ability of a system to thrive in adversity and adapt to change. Specifically, with regard to software, anti-fragile can equate to responsive coding practices, such as auto-scaling or default backups, to mitigate unforeseen events. Chaos engineering, for example, is one method to expose edge cases and build with adaptability in mind. And with the introduction of intelligent automation, often fueled by machine learning (ML) and artificial intelligence (AI), it’s becoming more and more feasible to build anti-fragile software systems.

    I recently met with Asim Razzaq, co-founder and CEO of Yotascale and former head of platform engineering at PayPal, to discover what it takes to build anti-fragile software. According to Razzaq, anti-fragile represents the next evolution of reactive, resilient software systems. And introducing anti-fragile concepts into your application networks can be done with just a few (relatively) simple steps. Below, we’ll define what anti-fragile is and see how this concept can be applied to software development and DevOps.

    Understanding Anti-Fragile

    As the adage goes, “What doesn’t kill you makes you stronger.” And it’s true—many natural systems tend to thrive under inhospitable conditions. As described by Nassim Nicholas Taleb in the book ‘Antifragile: Things That Gain from Disorder,’ anti-fragile systems thrive in adversity. For example, consider how human muscle grows under stress and load. Immune systems similarly adapt and respond to new viruses and improve the body’s resilience over time. So, how does this relate to software development?

    Well, organizations need to develop highly-tolerant software systems to meet rising digital consumer expectations. Especially when under extreme loads, systems must be elastic to adapt and survive. To meet high user demands, as well as specific service-level objectives (SLOs) and agreements (SLAs), most software companies are already investing in cloud-native tools that improve reliability. For example, a hallmark of site reliability engineering (SRE) is introducing automation to enhance the stability of applications.

    One example of gauging anti-fragile in software environments is chaos engineering, says Razzaq. Chaos engineering is the act of throwing a wrench into the machine. You randomly introduce bugs and malformed requests and attempt to create as much havoc as possible to see what happens. This could mean pulling out cables and restarting servers to gauge how software systems respond.

    “Software that can respond the best adapts the best,” said Razzaq. For example, if you manually remove a few nodes from the cluster but the system knows it needs seven to function, it could automatically spin up additional nodes. Or if the whole cluster goes down, ideally, the system would automatically restart and scale to the necessary requirements. In essence, anti-fragile software systems can understand and adapt to new environments without much, or any, human input.

    Tips to Construct Anti-Fragile Software Systems

    So, how can we construct anti-fragile software systems — especially when real-world challenges are often unpredictable? Razzaq shared helpful advice to consider when crafting anti-fragile software architecture.

    1. Start With Your Most Critical Systems First

    First, introduce chaos engineering to your most absolute critical systems. Define these vital components at the heart of your product which must remain functional. For a payments processor SaaS, for example, this would be the infrastructure supporting its transactional engine, such as the login and payment microservices.

    2. Determine How to Test Fragility

    Once you’ve identified this critical infrastructure, figure out how to test for fragility in your environment. Introduce chaos engineering to the component to see how it behaves. For example, returning to our payment platform example, this may include switching off a payment transaction type to see how the system functions in an extreme scenario.

    3. Decouple Components

    If systems are too closely coupled, errors and outages can ripple across multiple software services. To avoid that, it’s a good practice to prevent tight coupling between components—especially between critical systems and non-critical ones.

    4. Introduce Alternate Paths

    One method to decouple is by introducing alternate paths. For example, if a credit card verification component goes offline, the system should naturally accept other forms of payment and automatically suggest another form, such as a bank transfer. If all systems are down, the system could still accept unverified payments in a queue and let this continue until the threshold reaches a certain risk level.

    5. Learn From Past Events

    The next frontier of designing anti-fragile architecture is developing self-learning, autonomous systems. Of course, there is a limit to how automation can respond to unforeseen issues. But by feeding a machine learning model with more and more experiences, it could deal with untested new scenarios more gracefully. Of course, AI has the potential to make harmful decisions and even introduce unintentional bias, so organizations should have accountability for the algorithms they oversee, cautioned Razzaq.

    Benefits of Building With Anti-Fragile in Mind

    The goal is to able to thrive and function even under duress. By identifying the elements of core infrastructure to keep alive and putting it through the wringer, you can discover frailties before they cause significant consequences. And by decoupling components and introducing alternate paths, software systems can be better equipped for whatever may come. Uplevelling system operations with increased automation and AI is likely the future of constructing more fault-tolerant applications.

    “Software is everything in this day and age,” said Razzaq. “It increasingly plays a critical role in life-enhancing capabilities.” As such, the end benefits of developing an anti-fragile style could have hugely positive effects. In health care situations, it could make a difference in saving lives. For other less dire scenarios, increased fault tolerance leads to less lost revenue during peak traffic times and more consistent end-user experiences.

  • Software Supply Chain Security Debt is Increasing: Here’s How To Pay It Off

    Software Supply Chain Security Debt is Increasing: Here’s How To Pay It Off

    Last year, the world woke up to the software supply chain dilemma. We saw a spike in attacks as hackers sought to exploit known and unknown vulnerabilities within dependencies. There is also the chance of typosquatting, and malicious code commits to consider. Such supply chain attacks have increased by a shocking 742% over the past few years.

    It’s easy to let software security debt accumulate, as it’s cumbersome to stay on top of frequent updates — especially when new releases cause breaking change. Furthermore, companies might unknowingly collect security debt by hastily looping in projects whose origin is unknown. Plugging these gaps can bring a high burden down the road, which too often falls on the shoulders of developers to manage.

    I recently reconnected with Dan Lorenc, founder and CEO of Chainguard, to discover why risks in the software supply chain contribute to increased security debt and how organizations can proverbially pay it off. Lorenc is a key figure behind Sigstore, the open source utility built to verify the provenance of open source packages. Below, we’ll consider the state of software supply chain security and examine some modern solutions that lessen its severity.

    Why is Supply Chain Security Debt Rising?

    Modern software ecosystems are an intricate mosaic of dependencies, and supply chain security concerns often arise at the cracks between them. “It’s about the gaps and hand-off points between individuals in the software life cycle,” said Lorenc. Since these hand-off points can threaten systems at large, they can accumulate considerable security debt if left unguarded.

    Another reason why security debt is increasing, said Lorenc, is that fixing issues is rarely incentivized—it’s always thought of as “someone else’s problem.” Minimizing this debt will require organizations to change this mindset and follow modern security best practices, like implementing multifactor authentication infrastructure and more frequent code reviews. It will also take a coordinated effort from package managers to integrate tooling that helps verify the legitimacy of packages, said Lorenc.

    Solutions to Reduce Software Supply Chain Security Debt

    Automated Vulnerability Scanning

    In addition to advancing internal culture toward greater security awareness, organizations will benefit from automation that enhances the discovery of vulnerabilities. Because when a severe incident like Log4j hits the presses, it’s a mad scramble to respond. Even a year later, one in four downloads still possesses the infamous vulnerability.

    Integrating automated security scanning is the first step to knowing how these incidents impact your security posture. A tool like Snyk can do a great job of surfacing known vulnerabilities and finding where they lie within your software stack and provide remediation advice. Continually checking for updates can also help pay off security debt “as you go” instead of upgrading everything all at once, says Lorenc. But although scanning tools are essential for increasing visibility, they don’t address the root cause of why they are vulnerabilities in these packages in the first place, says Lorenc.

    Checking Transitive Dependencies

    Reports find that 95% of vulnerabilities stem from transitive dependencies. These are dependencies that are buried deep within the dependency tree of another package. And depending on the programming language, they’re not always that visible. The combinatory effect of transitive dependencies can lead to exponentially more vulnerabilities and issues if not carefully tracked. And since they’re running locally with the same privileges as your other packages, you shouldn’t treat these vulnerabilities any differently, said Lorenc. “You need to realize they can affect your application in the same way.”

    Validating The Source

    The link between the package manager and the source code is a frail area that warrants additional verification. Social engineering tricks can effectively trick users into downloading insecure packages that appear safe at the outset yet hide corrupt code. Or, account takeovers can result in attackers inserting malicious code into commonly used open libraries. Package managers often host built versions and forked versions of software produced by the community on GitHub, yet do not validate a clear link to the original repository. This gap heightens the chance for a malicious package to slip through the cracks, says Lorenc.

    This is something that Sigstore, the open standard for signing, verifying and protecting software, aims to accomplish. In mid-2022, Sigstore was implemented within Kubernetes, signaling the adoption of Sigstore as a standard throughout the open source ecosystem. It’s also been adopted by Npm and PyPI and Maven Central will soon support Sigstore as well, said Lorenc. Although some of these moves have received pushback from developers, Lorenc argued that such verification is necessary to help link packages to their source and increase credibility for open source dependencies.

    Chipping Away At Security Debt

    Like technical debt piles up over time, security debt can quickly rise if left unchecked. Chipping away at increasing software supply chain security debt will likely require a “pay as you go” approach wherein developers continuously scan and update their dependencies. One good sign is that people are paying more attention to risks latent within their code, and the industry is discovering and sharing more potential vulnerabilities before they are exploited.

    “Security has advanced, and there are common best practices across the industry,” said Lorenc. But if these practices aren’t followed, it could bring dire consequences for the organizations and the industry as a whole. And although Sigstore can help bridge the disconnect between the package managers and the source code, there are plenty of other techniques to help improve a security footprint for open source software and cloud-native infrastructure. For example, SLSA and greater use of SBOMs can help address security debt more holistically.

    The software supply chain security dilemma is an advanced topic that requires a coordinated effort from various stakeholders to remedy. The important takeaway is to stay positive, see the debt, and pay it off incrementally, says Lorenc.

  • 6 Ways To Empower Developers and Increase Productivity

    6 Ways To Empower Developers and Increase Productivity

    A big goal of DevOps is to increase productivity. Yet many organizations still struggle with the efficiency of their software release processes. In fact, 54% of engineers stated they have too many manual steps within their deployment process, according to an Armory survey. And with the pervasiveness of multi-cloud, developers now face an escalating number of deployment environments, each carrying their own nuances.

    Development teams want to move as quickly as possible without barriers. Yet, there are often hurdles to seamless deployments. For example, the code might be 100% ready to deploy, yet someone still must validate it before it reaches production. Manual bottlenecks like this in the CI/CD process can make developers feel like their code isn’t worthy of deploying and may even amplify feelings of burnout.

    I recently met with Anna Daugherty, senior product marketing manager at Armory, to see what technology leaders can do to remove barriers within their software release cycles and increase productivity. According to Daugherty, many of these strategies boil down to having empathy for developers and their needs. “Developers aren’t machines; they’re people, and people deserve to have a quality of life when they’re doing their jobs.” Below, we’ll highlight tips and tricks to keep in mind to improve release fluidity.

    1. Identify Where Your Manual Checks Are

    The first step to speeding up CI/CD is identifying where manual steps exist within the development workflow. The DevOps life cycle, often visually depicted as an infinity loop, includes many states, such as design, build, test, deploy, assess and monitor. But if each step has one or more manual steps, you accrue a lot of manual debt, said Daugherty.

    Therefore, engineering leads should seek to remove manual checks whenever possible. Of course, this may not be realistic within certain situations, such as financial institutions that require manual approvals at the last mile. However, everything should be smoothed out and automated up to that point, said Daughtery.

    2. Shift From Continuous Delivery to Deployment

    Depending on who you ask, the ‘CD’ in ‘CI/CD’ may stand for continuous delivery or continuous deployment. As part of reducing friction in the software release process, Daughtery encourages teams to begin to embrace the latter whenever possible. If deployment remains inhibited by unnecessary manual steps, it can decrease scalability and interoperability.

    3. Increase Confidence With Testing

    Of course, if we introduce more automation into the CI/CD pipeline, we must ensure the software is rigorously tested for errors and vulnerabilities. Daugherty encourages development to embrace automated, continuous testing approaches at every juncture, whether pre-production, inter-production, or throughout the application lifecycle. There are also advanced deployment strategies to consider, such as canary testing, which rolls out new changes only to a small subset of users or blue-green deployment can deploy to different places at different times.

    These deployment strategies can help build confidence while reducing the risk of disrupting a majority of your user base with untested production updates. “The more you can integrate testing into your process, the more confident your process will become,” said Daughtery.

    4. Chose The Right Toolsets

    So, how do we implement more automation into the DevOps cycle? And what sort of tools should we look for? According to Daugherty, when scouting for DevOps tools, technology leaders should prioritize open solutions that aren’t closed-off environments. Instead, they should offer flexibility and the ability to integrate with the tools you already use every day. “You can have all the tools in the world, but if they don’t fit together, that can drag down the process.”

    5. Consider Open Source Software (OSS)

    On that note, consider open source software (OSS) where possible. OSS like Argo or Spinnaker has the potential to empower developers and bridge gaps in the CI/CD process. Other projects from the Cloud-Native Cloud Native Computing Foundation (CNCF) and Continuous Delivery Foundation (CDF) are mature with proven stable adoptions.

    And although there is some weariness around OSS, particularly around unknown vulnerabilities and project maintenance concerns, “you can still be successful, secure, and scalable with open source,” said Daugherty. Open source also delivers community benefits and helps engineers feel like they are part of a bigger picture. “[OSS] is the opposite of burnout. It brings people together. It’s a community builder.”

    6. Become Developer-First

    Transitioning into a developer-first organization is another way to increase developer satisfaction and efficiency. But to become a developer-first organization, developers will need more say in the processes—this will involve balancing top-down requirements with total developer control. For example, leaders should suggest tools to enhance productivity but accept feedback on an ongoing basis to see what works best.

    To support these efforts, the concept of developer experience (DX) has entered the lexicon in recent years. Building with developer empathy can result in more usable, frictionless software that increases productivity. DX is crucial within the burgeoning area of platform engineering, as the goal is to create reusable, internal self-service platforms for developers to leverage.

    Interestingly, Netflix extolls what it calls full-cycle development, in which a developer oversees the building and operational aspects of a single component throughout its entire life cycle. “It’s an interesting way of addressing burnout because you feel more connected to what you create,” said Daugherty.

    Final Thoughts: DevOps Isn’t Dead

    Above, we’ve highlighted some potential strategies to increase developer productivity. By replacing manual bottlenecks with more automation, you can increase deployment fluidity. Considering how tools are acquired and iterating upon feedback from engineers can also help improve consistency while retaining quality talent.

    Looking to the future, Daughtery hopes that similar roles such as DevOps, site reliability engineering and platform engineering can come together and join forces. Because, at the end of the day, they share many goals. And contrary to some grabby headlines, DevOps is far from becoming obsolete. DevOps is not dead. It’s very much alive and increasingly important.

  • State of DevOps Report Finds a Rise in Platform Engineering

    State of DevOps Report Finds a Rise in Platform Engineering

    The speed of software development velocity is increasing, and to enable this, platform engineering is emerging to evolve the DevOps practice. Interest in platform engineering is rising—so much so that Puppet emphasized platform engineering in its 2023 State of DevOps Report.

    A whopping 94% of respondents agreed that platform engineering is helping their organizations better realize the benefits of DevOps. The move toward platform engineering has increased over time, bringing benefits such as increased release velocity, reliability and efficiency.

    Below, we’ll highlight some of the key takeaways from the report. We’ll define what platform engineering is and highlight its benefits. We’ll also consider obstacles on the path to embracing a platform-based development culture.

    Platform Engineering Supplants DevOps

    DevOps has been in use for a decade or more at most institutions, maturing steadily with new automation and infrastructure abstraction. The latest evolution of DevOps is a practice known as platform engineering—51% of organizations have already adopted this practice within the last three years and 93% said it’s a step in the right direction.

    Platform engineering is the discipline of designing and building self-service capabilities to minimize cognitive load for developers and to enable fast-flow software delivery.

    In essence, platform engineering is the process of creating standardized, self-service infrastructure that helps developers get up and running quickly. These self-service platforms typically handle capabilities like deployment (43%), operation of infrastructure (43%), monitoring production applications (42%) and security and compliance (42%).

    These shared internal platforms are maintained by a central platform team that embraces a product mindset to iterate the technology based on user feedback.

    State of Platform Engineering

    Platform engineering is relatively new— 27% of platform teams have formed within the last two to three years. Notably, big tech companies hopped on the bandwagon years ago, and non-tech companies are now following. Regardless, 70% believed their timing was “just right.”

    The goals of a platform engineering team are numerous. According to the Puppet report, 54% said problem-solving is the top goal. This is followed by educating and empowering developer teams (47%), setting best practices (46%) and optimizing iteration (44%). A platform engineering team’s scope of work is primarily focused on automating workflows and building and managing infrastructure.

    The ‘correct’ number of internal platforms is a moving target. The majority of organizations (57%) have between two and four platforms, and 30% have five or more internal self-service platforms. This demonstrates that although platform engineering is somewhat centralized, it’s also flexible to the needs of an organization.

    Benefits and Hurdles

    So, what are some benefits as opposed to traditional DevOps? Well, the most common benefit is improved system reliability, at 60%. This tracks, as a standardized platform could reduce human error and missteps in the development workflow. Following this benefit is improving efficiency (59%), speeding up delivery time (58%) and improving workflow and process standards (57%).

    The most common driver of adoption is increasing the speed of delivery. Slight more than half of respondents (51%) said development speed has somewhat increased since the inception of the platform team, and 42% say it’s increased by “a great deal.” Other top drivers include the need to scale, free up engineering time, control infrastructure spending and avoid duplication across teams. Those with more mature adoption tend to be more satisfied with the result, indicating that results increased over time.

    Of course, no new technological introduction comes without its hurdles, and platform engineering is not immune from certain pain points. For example, 34% said that cycle time is slower than expected. This might be due to nascent industry adoption and a lack of best practices.

    Just under one-third (32%) also noticed resistance to platform team adoption, and 32% reported a lack of communication around changes. Organizations will likely have to increase transparency and reduce siloing to encourage productive platform adoption that satisfies the needs of various stakeholders.

    According to the report, product management skills can help realize these goals. Some of the most fundamental product management skills here include strong communication skills (61%), problem-solving expertise (60%) and the ability to foster collaboration (54%). Many companies will also have to convince senior management to invest in platform engineering—nearly half agreed that senior management doesn’t yet understand its value.

    Future Outlook

    Confidence around platform engineering is increasing, especially for teams with mature DevOps approaches. As a result, the practice will likely gain more traction in the coming year—71% of respondents reported their firm plans to hire people with platform engineering experience over the next 12 months.

    Over the next year, organizations that adopt the practice will want to increase awareness of platform capabilities and set expectations across the organization. This will also include setting governance policies and satisfying the needs of specific developers, departments and the organization as a whole.

    Above, we’ve scratched the surface of the current state of platform engineering, its benefits and its pain points. For plenty of other insights, you can pick up a copy of The State of DevOps Report 2023: Platform Engineering Edition from Puppet by Perforce here.

  • The API Economy: Increasing Potential to Monetize Public APIs

    The API Economy: Increasing Potential to Monetize Public APIs

    API-first strategies are becoming more commonplace throughout today’s digital strata. APIs connect microservices architectures, power new digital transformations and help development teams avoid reinventing the wheel for common functions. And although most are built to connect internal components, studies indicate a movement toward more externalization and monetization of public APIs.

    Over three-fourths of developers now view participating in the API economy as a priority for their organization or plan to prioritize it soon, found Rapid’s newly released 2022 State of APIs report. The survey, which surveyed hundreds of professional developers, also uncovered a modest increase in companies seeking to generate direct revenue from APIs.

    Below, we’ll review some intriguing statistics from the Rapid report and consider the major takeaways technology professionals should consider in the year ahead.

    API Reliance Increases

    Reliance on APIs is rising across the board. Over 62.6% of developers reported relying on them more in 2022 than in 2021. Internal APIs help avoid development duplication and third-party APIs can quickly integrate standard functionality related to payments, messaging, geolocation, data sharing and many more areas. For example, ChatGPT has demonstrated the power of standard APIs to deliver advanced AI/ML to the masses in a programmatic way.

    As a result of this interest, the majority of organizations now consider participating in the API economy as a top priority. And as an organization grows larger, its reliance on APIs is likely to increase. A full 40% of the largest companies surveyed reported having more than 250 internal APIs. Nowhere is this more prevalent than in financial services. This finding makes sense, given how APIs can automate payments and underpin the open banking movement.

    In terms of who is using APIs, it’s most commonly full-stack or backend developers, the report found. One intriguing factoid is just how much of this work is now dominating developers’ time. Impressively, 65.5% of developers reported spending ten hours or more working with APIs in their job; 14.5% of them said that work takes up all of their time.

    In terms of design styles, various standards are used throughout the web development sphere. Yet REST is still king—69.3% of engineers used REST in production. REST is followed by webhooks, WebSockets, AsyncAPI, serverless and FaaS, SOAP, GraphQL, Kafka and gRPC, in descending order of popularity. Across the board, these statistics showed a steady increase from last year.

    Of note is that most APIs are built for internal purposes. These often wrap a database or provide open access to an internal microservice. Private use cases are by far the most common approach; according to the report, 75.1% of respondents worked on internal APIs. These findings aligned with other industry reports, such as the Postman State of the API Report, which found that, in 2022, 58% of API initiatives were private.

    Third-party APIs (53.9%) and partner-facing services (48.9%) trailed behind internal APIs in popularity. Interestingly, while internal use cases have remained steady, both partner and public-facing strategies have increased since last year. While the growth was a modest 5% for most sectors, partner-facing APIs in some industries grew by nearly 10%.

    Spotlight on Monetization

    Many still view APIs as operational glue for internal data or connecting partner services. Yet, the possibility of monetization through self-service developer portals shouldn’t be overlooked—countless API-first startups are actively gaining traction and funding. And monetizing this power is seen as a holy grail to some unicorn-chasing VC funds.

    Most APIs are purely for in-house internal integrations and are typically not externalized to the outside world for profit. However, this is slowly changing. In fact, 42.6% of organizations are now monetizing APIs, representing an increase of about 5% from 2021.

    As one may have guessed, financial services are leaders in monetizing their external APIs. The telecommunications industry is also noticeably active in monetization, too. Furthermore, with size comes increased monetization potential—about half of the companies with a large footprint of ten or more internal or external APIs also monetized them.

    API Economy in 2023

    Looking to the future, most anticipate more reliance on APIs across sectors — 69.2% expect to rely on APIs even more in 2023. Yet, organizations will have to grapple with particular hurdles in this newfound landscape. For example, a cloud skills gap might bar some groups from realizing the benefits of an API strategy. Other concerns include ungoverned sprawl, authorization concerns, shadow APIs and escalating attacks on public-facing endpoints. Also, some organizations are still determining how to best secure and scale their use of new styles, like GraphQL.

    A standards-based approach that applies platform-wide governance will likely be necessary to maintain an efficient yet stable API footprint. To meet these high quality standards, testing is nearly universally applied—92.7% of those surveyed test their APIs or plan to do so. The most common tests ranged from functional testing to integration testing, acceptance testing and performance testing.

    The Rapid 2022 State of APIs report received 850 responses from over 100 countries. For more insights, readers can view the complete results here.

  • 5 GraphQL Trends to Watch in 2023

    5 GraphQL Trends to Watch in 2023

    The GraphQL query language had a big year in 2022. We witnessed its increase in production use cases, solving overfetching and underfetching concerns that plagued traditional API integrations. GraphQL can significantly improve usability, aggregate multiple services and optimize how data is ported from system to system.

    Yet, GraphQL is also prone to new security concerns. And as attacks on APIs continue to increase and grow in sophistication, an increased security focus will undoubtedly be required to avoid vulnerabilities. The coming year will likely also see architects grappling with their GraphQL maturity as they consider how best to scale its use across an organization and its various stakeholders.

    Below, we’ll consider where the tides are flowing for GraphQL in 2023. These forecasts are informed by my coverage in 2022 and reports from within the web development community.

    1. Increased Production Use

    A full 62.6% of developers reported relying on APIs more in 2022 than they did in 2021, according to RapidAPI’s State of APIs report. As the number of APIs in use expands, we will undoubtedly see more GraphQL examples. GraphQL is popular due to its usability and flexibility which outweighs traditional integration points. Studies showed it is primarily used for exposing internal data, although partner and public use cases are not uncommon.

    Representational state transfer (REST) remains a dominant style for modern APIs. According to Postman’s 2022 State of the API report, 89% of developers use REST. GraphQL is in fourth place at 38%, superseded by webhooks (35%) and SOAP (34%). Although usage remains small in comparison, it’s steadily gaining traction and will continue to expand.

    2. Unifying Disparate APIs

    GraphQL helps expose fields in an easily consumable manner. But it’s also good for aggregating disparate microservices into a unified schema. As such, it is often used as a meta layer that combines multiple REST services, databases and even GraphQL schemas. For the time being, it appears that REST has staying power. And it’s unnecessary to migrate all REST services to GraphQL. Looking to the future, it is more likely to be adopted as a layer on top of existing REST APIs.

    Combining multiple backend services under the hood and exposing them in a unified schema creates a more usable, localized interface that frontend developers can work from. In the future, GraphQL might become a meta layer for more organizations, increasing the discovery of internal services and helping teams reuse internal microservices.

    3. Supergraphs and Subgraphs

    As mentioned above, GraphQL is used not only to combine legacy REST services but also to combine multiple GraphQL schemas. Some call this a graph of graphs—or a supergraph. A supergraph that combines multiple GraphQL schemas would offer unparalleled developer experience in that engineers can access many services from the same endpoint.

    However, there’s a downside to centralizing with a unified graph. Different internal teams or partners might only need access to a particular set of functions. Giving the entire schema to every person who walks through the door would be TMI, not to mention a potential security risk that breaks the rule of least privilege. So, the idea of a subgraph has gained traction. A subgraph is an Apollo concept, but the theory could certainly be applied in other implementations. In the future, we’ll likely witness more use cases that use subgraphs to help federate a supergraph across an organization, helping segment a graph to specific stakeholders’ needs.

    4. Turning on GraphQL Security Controls

    As of 2022, less than half (43.2%) of GraphQL implementations have disabled query introspection. Disabling field suggestions and introspection are means to reduce the risks of public GraphQL exposures. As GraphQL security best practices become more apparent, these numbers will likely become more commonplace. Other security controls, such as implementing query timeouts, query rate limiting and query cost analysis, can also help curb potential malicious behaviors.

    As we’ve covered before, security by obscurity is not enough to protect GraphQL. Yet, many native GraphQL security and performance directives go unused. Besides turning on these advanced features, new tooling is coming to market that could harden GraphQL. Security vendors continue to add support for GraphQL, making its protection more accessible for non-security engineers.

    5. Managing GraphQL at Scale

    GraphQL has arisen in some large production use cases in recent years, helping developer usability at GitHub, boosting productivity at Netflix and aiding Shopify’s API strategy. These case studies proved its usefulness in large enterprises. But how do we effectively scale GraphQL across a massive organization?

    The concepts of supergraphs and subgraphs are ways to help scale it across a company. But retaining high performance will also be vital to maintaining an effective, scalable footprint. As such, we will likely continue to see companies optimize how GraphQL is used as it matures. This includes optimizing queries and query depth and applying filtering and rate limiting to fit application demands. Taking measurements of GraphQL response times will be necessary to establish benchmarks to improve over time.

    We’re Just Querying the Surface

    Above, we’ve only scratched—or queried—the surface of the exciting developments on the horizon for GraphQL in the coming months. There is much more to be said regarding updates to the query language and improvements within the surrounding tooling landscape. Where do you think GraphQL will go in 2023? Let us know in the comments below!

  • Survey Sheds Light On Cloud, API Expertise Skills Gap

    Survey Sheds Light On Cloud, API Expertise Skills Gap

    Enterprise IT faces a three-pronged onslaught: Digital transformation, open APIs and security threats. First, most organizations are still navigating ongoing digital transformations and cloud migrations. Second, development teams continue to increase their use of open APIs, leading to sprawl concerns. And third, we’ve witnessed a slew of cybersecurity threats that appear to be growing in sophistication. Tech leaders are attempting to govern this influx of new tools and risks, yet it can be challenging to innovate quickly while ensuring that sensitive data is always protected.

    The Axway 2022 Open Everything Survey shed light on some of the above challenges and attitudes of those invested in API development and API-enabled managed file transfer (MFT). Most notably, the survey found that 86% of respondents feared an expertise gap in the coming years, in which their organization won’t have the skilled software and IT infrastructure talent they needed to innovate. Below, we’ll review other findings from the survey, focusing on the responses from those who indicated they’re invested in API development.

    Cloud Migrations Are Still Underway

    Digital transformations are still underway in the vast majority of organizations. Only 12% reported being wholly digital and leading innovation. The rest were either just beginning transformations, were midway through or were still largely brick-and-mortar. More than one-third (38%) also said their company’s digital makeup is a hybrid of old and new technology.

    The most-adopted computing environment is the public AWS cloud, at 47%. Yet other options trailed close behind, including hybrid (37%), managed cloud services (32%), Azure (28%), private (25%) and Google Cloud Platform (18%). Other reports substantiated how modern IT is becoming increasingly hybrid and multi-cloud. As digital journeys continue to shift and evolve, some organizations are embracing multiple fit-for-purpose clouds.

    Although multi-cloud offers unparalleled flexibility, the primary drawback is security. Just over half (55%) of respondents said that tracking and controlling the flow of personal data into and through the cloud is a key hurdle to a multi-cloud strategy. And 53% reported that dealing with different security requirements for regional instances of cloud infrastructure was a top challenge. This concern is warranted, as companies doing business internationally must carefully navigate many specific regional data regulations.

    Open Data Strategies Abound

    As the Open Everything name indicates, many different sectors are becoming more “open.” The report highlighted the adoption habits of APIs within supply chains, where they are most often used to configure applications, operate and monitor applications and integrate internal data sources. Popular use cases here included payment and tracking systems. APIs also have a significant role to play in the financial world—41% of those working in finance said open banking is an opportunity to discover new business models. But nowhere is the need for more open data more crucial than in health care, which was found to be the area most in need of digital transformation.

    We’d like to think of the API revolution as being driven by API-first startups and SaaS services. While these trends get attention, it’s not the most common use case for APIs. In fact, 44% of the time APIs are used to support IT consulting and system integration requirements. This is followed by providing cloud applications for customers (35%), developing a partner network (35%) and offering cloud services for customers or partners (34%). Although there are certainly business use cases for generating revenue through APIs, they are most often adopted to supplement existing platforms or to glue together internal services and data.

    Most Technology Leaders Fear API Sprawl

    Security is an encompassing concern for all walks of IT, but is especially top-of-mind when opening up data. Respondents said security is the most important concern when creating APIs (30%). This concern is warranted, as attacks on APIs have increased exponentially in recent years. Other top concerns are API adoption or consumption (15%), customer experience (14%), ease of API management (14%) and design and development (11%).

    As APIs and microservices become more abundant within an organization, it’s easier for sprawl to set in. Unbridled technology sprawl can have a deteriorating effect on design standards and lead to unknown shadow APIs that may pose a security vulnerability. As a result, over two-thirds of respondents said the prospect of API sprawl worries them.

    To mitigate sprawl and address other concerns, most companies are investing in tools to help manage the growing number of APIs. The majority (83%) said their budget for API management has either increased or stayed the same over the last two years. For 18%, this budget has risen by over 25%. Operationalizing API life cycle management capabilities can have benefits such as standardizing design style and increasing the discoverability of internal services.

    API Security Must Still Mature

    In a perfect world, we could share data without fear of misuse. But in reality, APIs are a common target for hackers. Publicly exposed endpoints with broken access control or missing object-level authorization can be easily abused for things like privilege escalation and data exfiltration. And although these risks persist, many APIs don’t adopt industry best practices regarding API security.

    Although most APIs use authentication and token schemes, under half (43%) implemented OAuth. Furthermore, only 30% implemented policies with an API gateway. OAuth 2.0 and gateways have become best practices in serving APIs, as there are too many vulnerabilities to rely solely on HTTP Basic Authentication and API keys. As such, we will likely see these adoption rates increase as API security matures.

    As with other cybersecurity endeavors, it’s really about taking a holistic perspective and applying defense-in-depth. Thankfully, most technology professionals understand this point—developing a more holistic approach to security was ranked as the top strategy for protecting personal data. Other strategies, like data encryption, advanced multifactor authentication, data loss prevention tools and highly-secured data transactions will also be necessary to protect cloud-native assets.

    Final Thoughts

    The Axway 2022 Open Everything Survey asked 1,000 IT leaders, architects and developers about their top concerns. Above, we focused on results from technology leaders engrossed in cloud and API development. For more insights and the findings from those using managed file transfer (MFT), readers can pick up a copy behind an email gate here.