devopsy.kinsta.cloud

Category: Continuous Delivery

Continuous delivery is a software engineering discipline that allows organizations to consistently develop and release quality software at a rapid pace. It is achieved by automating the software development lifecycle, enabling tasks such as development, testing and release to occur concurrently and continuously.

  • Decidability: SLOs to Measure Quality Attributes

    Decidability: SLOs to Measure Quality Attributes

    You may be familiar with the “-ilities”–the quality attributes that define a system’s so-called “non-functional” parts. One that isn’t on the list is decidability, which is the ultimate quality attribute because it focuses on what truly matters to your team and your customers.

    You can’t do it all. Fast, cheap and good? Here in the real world, we have tradeoffs. Software is the mouth that keeps on eating, but it isn’t a digital free lunch. When we decide what not to do, we bring greater clarity and productivity to the team.

    Of course, the world is full of “shoulds.” We should make our product accessible, we know. It should be secure; it should be reliable. Of course. But how do you decide the limit of your shoulds? To enable decidability, you need context. To gain a contextual understanding of your most critical quality attributes, you must talk to real people and reason about your situation. There’s no other way.

    New Design, Wrong Decision

    I had an eye-opening experience that demonstrated firsthand how quality attributes don’t necessarily deliver business results. Six years ago, I was working on e-learning software for public schools. One of the main problems was user management. We were selling to massive institutions that often had only one administrator responsible for onboarding all the users across hundreds of schools. While school was out, they had to enter data for the schools, create classes and assign teachers.

    Once we had a design for improving the workflow (luckily, before implementing it), we took the mockups to a customer to show the fancy ideas we had built. They surprised us by showing us “what it looks like for them”–a spreadsheet! They were copying/pasting everything by hand.

    So we asked them, “What if we gave you a way to upload the spreadsheet? Would that work?” And, of course, the answer was yes! So we scrapped our plans to build a fancy form and made a drag-and-drop input field to validate and upload the sheet. The decision was clear.

    Measuring Decidability

    How do we measure the critical business drivers and turn them into decidability? The closest answer I’ve found is service level objectives (SLOs). These take the fuzzy -ilities and turn them into metrics and goals that can quickly inform decisions.

    SLOs are a universal tool for measuring quality attributes from an indicator expressed as a proportion of “good” events for a given quality like reliability or scalability. Decidability comes from aggregating the SLO data across your services, geographies, user groups and quality attributes over time.

    Comparing an indicator to a particular goal (tuned over time as the business and engineering context changes), we can derive an allotment of allowable “bad” time or events–an error budget. Setting an error budget for reliability, performance, scalability, etc., allows you to decide how poorly your service can operate and still keep customers happy. Further, you can find the breaking point of user behavior and defend your service proactively from violating the quality attributes you expect.

    Technical debt inevitably stems from all the -ilities you should focus on but couldn’t quite get to, given your situation. Some technical debt is deliberate and even prudent. To tackle technical debt, you must decide which new features to put on hold to make space in the roadmap. SLOs provide a data-driven baseline to make this decision quickly and accurately by telling you when your -ilities are within error budget or out of whack.

    Scaling Decidedly

    You might start on a small, single-purpose project that’s rarely used. But if that changes, you must decide how to invest, redesign and maintain it. Suddenly, you might care about performance, scalability, security and the rest. First, you need decidability so you can prioritize your work. So you gather the people together who care about the service, and you start to think and talk and stack rank. You use your instrumentation to build metrics, set clear goals and do what it takes to keep the system running within expectations.

    Automating action based on error budgets is the magic of setting SLOs. They aren’t just for sophisticated, highly available services. Error budgets are extremely valuable in systems when you aren’t sure what to do. SLOs create decidability in all the trade-offs of all the quality attributes and the functional needs of the app or service you’ve carefully built.

    If you put decidability first, the rest will fall into place.

  • New Relic Extends Observability Reach to More DevOps Platforms

    New Relic Extends Observability Reach to More DevOps Platforms

    New Relic today expanded its ecosystem by adding integrations between its observability platform and a wide variety of DevOps and cybersecurity tools. The New Relic observability platform now integrates with offerings from Atlassian, Amazon Web Services (AWS), CircleCI, Confluent, GitHub, JFrog, Lacework and Snyk and the open source Jenkins continuous integration/continuous delivery (CI/CD) platform.

    Peter Pezaris, senior vice president for strategy and experience at New Relic, said the goal is to make it simple to share analytics based on telemetry data collected from IT operations environments with the tools organizations rely on to build and deploy applications.

    In total, there are now more than 500 integrations with the New Relic observability platform, he noted.

    Only 27% of respondents to a recent New Relic survey reported that they have achieved full-stack observability and only 5% claimed they have a mature observability practice in place. A third (33%) of respondents also said they still primarily detect outages manually or based on complaints from end users. More than half (52%) of respondents said they experienced high-business-impact outages once per week or more and 29% said they take more than an hour to resolve those outages.

    On the plus side, nearly three-quarters (72%) of respondents expected to maintain or increase their observability budgets next year, with more than half (52%) expecting observability budgets to increase over the next year.

    Observability, in one form or another, has always been a tenet of any DevOps practice. For the most part, the best that could be achieved until recently was continuous monitoring of a set of pre-defined metrics. With the arrival of observability platforms, it’s becoming much simpler to not only monitor metrics but also launch queries that surface issues long before an IT service can be disrupted.

    An observability platform achieves that goal by making it possible to query logs, metrics and distributed traces in a way that surfaces patterns that have not been pre-defined. In effect, an observability platform makes it simpler to debug application environments where the number of dependencies that exist between services are becoming too complex to track or decipher manually.

    Of course, those applications will need to be instrumented using agent software capable of generating the data observability platforms require. Fortunately, open source projects such as OpenTelemetry are helping to drive down the total cost of instrumentation by making agent software more accessible to a wider range of application developers.

    It’s not clear how quickly IT teams will be embracing observability platforms, but there is now a clear opportunity to narrow the divide between IT operations teams and application developers, noted Pezaris. As each DevOps platform makes it easier to collect data via application programming interfaces (APIs), richer telemetry data can be collected, he added.

    In the meantime, as application environments continue to become more complex in the cloud-native era, most DevOps teams will need additional tools. Existing legacy monitoring tools don’t make it easy to discover, for example, which microservice might be adversely impacting the overall performance of an application because calls are being rerouted. The frustrating thing, as always, is that it can take days or weeks to pinpoint an issue that might take only a few minutes to fix.

  • Why Backups and Compliance Go Hand-in-Hand

    Why Backups and Compliance Go Hand-in-Hand

    It’s hard to run a business. Besides the numerous challenges you have to deal with daily, ensuring the security of your business and customer information is essential for operating smoothly. In order to mitigate information security risks that would affect companies and their users, a number of security regulations have been created, including a set of best practices that companies must follow.

    Compliance with these regulations will allow you to:

    ● Reduce security and privacy risks
    ● Strengthen control of key processes
    ● Align cybersecurity requirements with business processes
    ● Better understand cybersecurity goals and how to handle business security challenges
    ● Maintain effective cybersecurity programs
    ● Build customer trust and improve company reputation

    One critical requirement to achieve compliance is backups. Organizations back up information that’s vulnerable to threats like software or hardware failures, cyberattacks or human error. Any one of these problems could potentially lead to data loss, system malfunctions, financial losses and reputational damages. In this post, you’ll learn more about why data backups are an important part of compliance and about the benefits of having a robust backup strategy.

    Why are Backups Important for Compliance?

    Regulations like GDPR, PCI-DSS and HIPAA, along with others, were created to ensure that organizations implement the most reliable methods to secure all types of sensitive data from infrastructure failures and malicious attacks.
    Compliance can’t be achieved unless an organization handles data in a way that adheres to the outlined regulations.

    Certifications like ISO 27001 and SOC 2 are considered the most thorough audits an organization can go through to prove they have taken all the necessary steps to protect business and user data. Unlike the regulations mentioned above, ISO 27001 and SOC 2 aren’t legal requirements, but they’re especially beneficial to SaaS companies, data centers and organizations that deal with sensitive information. They make it very clear that the company takes data security very seriously and considers it a top priority. By properly implementing the data and privacy controls required, you can increase reliability and user trust as well as gain a competitive advantage over other companies.

    To be able to obtain such certifications and comply with regulations and different standards in general, you’ll have to maintain robust, up-to-date backup processes that ensure data is always protected but also available to users—even in cases of unexpected events like natural disasters, cyberattacks, data losses and human error. In other words, to comply with regulations, your backup processes must match your business continuity plan and the data backup tools you are using should be able to handle your backup processes.

    Backups, of course, do more than enhance business continuity, as mentioned above. Although they can’t prevent an attack or a data loss from happening, they’re an integral part of your overall security strategy since they allow you to restore lost or corrupted information, mitigating the consequences of such an event.

    There’s no fixed rule for how frequently you should back up. Still, ideally, to secure valuable information, you should implement a combination of full and partial backups as often as possible. Fortunately, great data backup solutions can efficiently handle this complex but crucial task.

    The Difference Between Backups and Compliance

    Backups and data availability are just one of the many requirements for a company to comply with any regulation. It’s an important part, but it’s not enough on its own. Obtaining a security certification isn’t a quick and easy process that you can decide to do and then finish a month later. It’s a time-consuming process that will require buy-in from every member of your team and will ultimately involve in-depth tests and evaluations of how you secure data.

    Although each certification is slightly different, all of them require a company to implement a wide range of security procedures. For example:

    ● Structure your organization in a way that allows optimal management of your services.
    ● Have a strong disaster recovery plan.
    ● Communicate in detail with users about how your system works, your security policies and your processes.
    ● Perform risk assessments regularly to identify risks, measure their impact and create a plan to address them quickly and effectively.
    ● Continuously monitor your security controls to verify they’re operating as expected.
    ● Make adjustments as needed to stay ahead of the latest security threats.
    ● Control physical access to your devices and prevent unauthorized access to your systems.
    ● Integrate best practice security procedures into your day-to-day workflows.
    ● Optimize system operations to detect vulnerabilities early and deal with security incidents.
    ● Clearly describe how you handle changes in your infrastructure to ensure that they won’t affect your business operations.

    The above is far from an exhaustive list, nor will it guarantee regulatory compliance. There are numerous criteria you need to fulfill. Although compliance might seem a huge undertaking, the benefits surpass the difficulties.

    As the security risks in the technology world are constantly changing and becoming more sophisticated, organizations must be able to deal with them or face the consequences. For businesses that want to protect their customers and themselves, following security standards has become necessary. But security compliance isn’t an easy task, and it’s made more complicated by the fact that all of the vendors you use must also be compliant.

    Having a reliable data backup strategy is a crucial step for compliance with regulations and achieving business continuity. Fortunately, there are top-notch data backup tools that can make this complex process easier. These tools allow you to back up regularly to secure your valuable data, and will often automatically run backups on a schedule of your choosing. If you wish to remain compliant and ensure your data is protected, I highly recommend installing a backup service.

  • Get Comfortable with Shifting Right to Improve Resiliency

    Get Comfortable with Shifting Right to Improve Resiliency

    The operations team receives a page at 2:00 a.m. Something isn’t working correctly. Do they know how to fix it? Perhaps more importantly, do they know whether it matters to customers or how it impacts the business? There are different techniques that companies implement to maximize the quality of their software. We talk about shifting left or right, chaos engineering and others. While they each have their own goals to improve quality, there’s an underlying common goal as well: All three techniques increase your team’s knowledge of the system and their ability to resolve problems quickly.

    Here, we’ll walk through the software life cycle and discuss the common challenges to software quality, running complex systems and managing dependencies between teams. The common thread is a proper observability strategy which can help software quality, hand-offs between different teams and resolve problems quickly by ensuring whole-system resiliency from design to maintenance.

    Techniques for Improving Quality

    Shift left is a concept that has attracted much attention recently. The practice revolves around the idea that detecting and correcting problems sooner minimizes their impact. As the thinking goes, they’re easier to fix, they wind up costing less and there are fewer ongoing issues with bad code.

    The rise of DevOps and DevSecOps has propelled the popularity of shift left. Yet, there’s a fact that development teams often miss: no matter how early an organization spots errors, problems will inevitably remain. Early testing can’t catch everything. As a result, some organizations have begun to rebel and move to a shift-right framework. It normalizes errors and aims to better resolve issues across the development process, including post-production.

    Shift right is the perhaps less-extreme cousin to its rebellious relative, chaos engineering. With chaos engineering, team members cause problems on purpose to keep everyone on their toes and test the limits of the system. They shut down servers, disconnect wires, delete files, etc. The goal is to confirm system resilience–from a user perspective, there should be no impact even with all of this chaos. A resilient system means that teams can fix failed components at their leisure during the day. With a resilient system, critical failures are less likely to happen and it becomes second nature to fix them when the dreaded 2:00 a.m. page does occur. Unlike chaos engineering, shift right doesn’t explicitly break things–it just makes the assumption that broken things will exist so teams need to be ready to handle them.

    Both concepts (and yes, even chaos engineering) are valid—but each company needs to carefully analyze which is right for them. One variable to consider is the complexity of the system you’re building. Working on a rocket launch or a surgery robot? Probably should shift left as much as possible. Changing the color of a button? Maybe it’s not the end of the world to verify the fix in production. For most of us, projects fall somewhere between these two extremes–so sometimes shifting left is necessary, and other times shifting right is pragmatic. A careful analysis of the risks should be performed to determine what kind of testing is needed.

    Although teams tend to be pretty good at analyzing the complexity of their software and making development process decisions, they often forget the complexity of other companies’ code which they depend on. Even a simple web application deployed today has tons of dependencies: The server it’s running on, the local ISPs, transit ISPs, load balancer, DNS provider, TLS certificate, CDN and browser, just to name a few! After analyzing the complexities of all of the dependencies across the internet, it becomes clear that finding every possible issue that might impact their application is a daunting task.

    The goal, again, is to find any issues before they reach production. Or if they reach production, fix them before they impact customers. Or if they impact customers, fix them as soon as possible after that. To do any of these things quickly, the level of observability of the overall system is paramount. With simple systems—or even complex systems that are entirely under one company’s control—tools such as application performance monitoring (APM) are common. But when the scope of the system is as big as the internet, internet performance monitoring (IPM) tools are necessary instead.

    With both APM and IPM tools, the goal is to know as much about the system as possible – enabling teams to identify problems quickly, pinpoint the root cause, and fix it. When an organization is equipped to monitor development and operational processes effectively from application to delivery, there’s a shift toward higher quality code, fewer errors, improved security and lower costs for the enterprise. There are also fewer disruptions and distractions for engineers, DevOps teams and others.

    Dealing With Disconnects

    Whether shift left or shift right is used, there will be problems that enter production. There is an invisible wall in many organizations–where engineering teams “throw code over the wall” to the team which is now responsible for running it. In some companies, this is the operations team; in others, it’s site reliability engineering or production engineering–the common trend is that in most companies, it’s not the same team who created the software.

    These operations teams often lack the nuanced information they need to resolve production issues. They might spend hours or even days attempting to track down the source of the problem. There’s often finger-pointing between different teams who created or own different portions of the system. Things can quickly devolve into chaos—and tempers can fray. After all, these kinds of problems always happen at 2:00 a.m., right?

    That’s where end-to-end observability enters the picture. Imagine a framework with proof that the development team has performed sufficient knowledge transfer to the operations team. That the operations team has asked all of the relevant questions about running and maintaining the application. Using the right tools, engineering and operations teams need to monitor every aspect and every component of the system. Creating this monitoring strategy means that everyone involved also builds an understanding of the system–everyone is on the same page. All of a sudden, there is a common language to discuss problems, and teams operating the software in production can take debugging a lot further before they need to ask for help.

    Ultimately, shifting left or shifting right is a personal choice for every organization. Even if every possible test case is executed before deployment to production though, issues will happen–so take the opportunity to implement a robust system-level monitoring strategy that can be used to solve technical questions during that 2:00 a.m. call.

  • ActiveState Makes All Tiers of Curated Artifact Repository Service Free

    ActiveState Makes All Tiers of Curated Artifact Repository Service Free

    ActiveState today announced it is making all tiers of its ActiveState Artifact Repository service available for free for a limited time. The move aims to enable organizations to better secure open source software components incorporated within applications.

    Loreli Cadapan, vice president of product for ActiveState, said the ActiveState Artifact Repository exposes a set of curated instances of open source software for creating binaries that are validated to be secure. That capability reduces the likelihood vulnerabilities will be inadvertently introduced into application environments by developers that have downloaded open source modules from repositories that may contain compromised code, she added.

    In addition, the ActiveState Artifact Repository also makes it possible to automatically generate software bills of materials (SBOMs) that identify which software components were used to build an application after all the code used to construct an application is stored in the repository, Cadapan noted.

    ActiveState’s repository is delivered as a cloud service to organizations that would otherwise have to build their own. Cadapan noted that the goal is to make managing and securing software supply chains simpler when more attacks are being launched against them.

    Most organizations have limited visibility into how they are consuming open source software. It’s become apparent many organizations have no idea what software components have been incorporated across their software supply chain. As a result, when a vulnerability is discovered, there’s no easy way for them to know whether they are impacted.

    However, awareness of this issue has risen, thanks to an executive order issued by the Biden administration which requires federal agencies to have an SBOM for every application they employ by next summer. The executive order was issued in the wake of a zero-day vulnerability in the Log4j tool discovered late last year that is widely used to create logs within Java applications. Many organizations are still looking for all the vulnerable instances of Log4j in their application environments. SBOMs must provide a list of the “ingredients” used to create an application to make it easier to find any component that might one day be similarly affected by a zero-day vulnerability.

    Organizations that want to implement a similar SBOM policy will spend massive amounts of time cataloging software on an ongoing basis. Every time an application is updated, there will be a need to once again verify what components are being employed within the context of larger DevSecOps workflows.

    SBOMs will undoubtedly play a critical role in improving overall application security. However, having an SBOM is not the same thing as operationalizing it. Organizations are ultimately hoping to be able to use an SBOM to decide whether to green-light the deployment of an application, so DevOps teams should expect to be required to not only provide an SBOM but also continuously update it. The challenge, of course, will be determining how many internal resources to devote to that effort versus relying on external platforms to manage that process.

  • Prioritizing Product Security With DevSecOps

    Prioritizing Product Security With DevSecOps

    Building software with strong security can no longer be an afterthought for organizations. The need for a reliable cybersecurity posture has proven vital amid the constant attacks we’re seeing across industries, all over the world. Product-specific security is top of mind for most companies, and a successful product security practice requires integrating with the ethos of the engineering team—DevSecOps. Consider a three-fold strategy, which has proven successful again and again in the industry.

    It consists of first creating a culture of security, with all stakeholders understanding the priorities, strategies, and business value. Second, teams must integrate security with the software development life cycle (SDLC). It’s not enough to shift left–organizations must shift their thinking from shift left to shift everywhere. And lastly, monitoring product systems at all times is non-negotiable. In a 24/7/365 world, there must be teams to watch over security 24/7/365. Part of the engineering deliverable becomes providing data to these teams so that they can have all eyes on the products, always.

    A Successful Practice Starts With the Culture

    In building a strong culture, the goal is to make the entire engineering team a part of the mission. This includes putting a product security team in place to support engineering needs as well as providing continuous education with resources available for addressing any eventuality.

    While every organization has its own unique needs, considerations and approaches to creating a product security team, these five roles are highly recommended:

    1. Technical Product Manager (TPM): Responsible for driving security priorities in planning
    2. Security Architect: Sits on the Architecture Review Board and ensures architecture and design have security processes incorporated
    3. AppSec Engineer: Responsible for implementing the security development life cycle
    4. Adversarial Engineer: Hammers on the attack surfaces to identify weaknesses
    5. Security Champions: Engineers from each Scrum team with an interest in supporting their teams with security fundamentals; they are instrumental in scaling out the overall product security effort

    Creating and executing an engaging security education program is also pivotal to winning over the hearts and minds of employees. Consider initiatives such as in-depth security boot camps, vulnerability-of-the-month presentations, capture-the-flag (CTF) contests and security consultation programs.

    Integrating Security Into the SDLC

    From a high level, all SDLCs consist of planning, development and release, and each phase should have its own set of security controls.

    Planning: The product security TPM is responsible for working with the product management team to determine security priorities–derived from sources like risk reduction, executive priorities, customer requests, security findings and regulatory requirements. Determining what is most important is key.

    Development: Much of the heavy product security lifting comes during the development phase, which includes implementing threat modeling, security baselines and automated scanning.

    Threat modeling: The process of identifying potential threats in a system. Once identified, the threat modeling team can create mitigations, and results are integrated into penetration testing to verify that the threats did not make it into implementation. It’s most efficient to test models during the architecture phase before coding begins to avoid rework and Band-Aids. For example, a company may have a policy disallowing access to port 22 from the Internet. Threat modeling automation can detect violations in the design and notify the owning team asking them to close the port. Tools that support this use case include IriusRisk and ThreatModeler, as well as Microsoft Threat Modeling Tool, which is a free option but does not include automation.

    Security Baselines: Teams should create secured images that are fully patched, meet security policies and have mandatory security agents preinstalled. Doing this for both OS images and containers allows developers to get a level of security for “free” and avoid ongoing security reviews. Baselines should automatically update and flow into CI/CD systems.

    Automated scanning: To keep up with Agile processes, scanning should be automated and have a low number of false positives. In fact, it is better to report on fewer issues than kill credibility with a lot of noise. Continuous tuning will increase the fidelity rule sets and helpful tools include static application security testing (SAST), software composition analysis (SCA), and dynamic application security testing (DAST). Automation is critical to securing applications that use a CI/CD development process and providing security right at the code push is invaluable–developers get immediate feedback, not a ticket weeks later when they likely have forgotten important context. Integrating these tools into build pipelines is a solved problem.

    For example, the below YAML demonstrates automating source scans for embedded secrets (e.g., passwords and keys).

    sast:secrets-detection:
    extends: .sast:secrets-detection
    stage: deployment:development

    Release: When teams reach the release phase, adversarial testing becomes an important component. Penetration testing can be in-house, outsourced or both. An internal red team will have a better understanding of the technology and, therefore, have a broader scope than external security partners. For example, they test across product boundaries, test processes and people and test live production systems. For a mature product, consider a bug bounty program. This will invite white hat hackers to find weaknesses in exchange for cash. But “mature products” is essential; otherwise, the number of payouts will become overwhelming.

    24/7 Monitoring

    As mentioned earlier, 24/7 monitoring is key, especially for any cloud-based technology. A dedicated security operations center (SOC) can help, scrutinizing for threats against the company, employees and customers. The SOC’s responsibilities are much broader than those of a product security team, and they allow product security folks to sleep well at night. Security teams should configure systems, load balancers, public cloud platforms and networks to feed data directly into the security information and event management (SIEM) system. The SOC can then respond to custom automated alerts that identify any data anomalies that are being streamed to them.

    As products–and technology overall–become more advanced, the way organizations secure them must do so as well. The perfect security approach varies from business to business, but empowering IT, engineering and security teams is a beneficial first step to identifying, optimizing and integrating security for both employees and customers.

  • CloudBees Survey Sees VSM Driving Digital Transformation

    CloudBees Survey Sees VSM Driving Digital Transformation

    A global survey of 600 senior executives conducted by CloudBees suggested a high correlation between investments made in value stream management (VSM) and digital business transformation success.

    The survey found 90% of respondents agreed that organizations that committed to VSM are likely to be further ahead in terms of achieving their digital business transformation goals, with 92% noting VSM also creates more time to focus on innovation.

    Hope Lynch, senior director for platform at CloudBees, said the challenge is that too many executives equate VSM with legacy Six Sigma processes rather than application development and deployment processes that advance digital business transformation initiatives. The survey found that while 90% of senior executives claimed they were familiar with the VSM concept, only half could select the correct answer when asked. In fact, two in five respondents confused the concepts of value stream management and value stream mapping. Nearly two-thirds (72%) of respondents admitted there is still confusion about what VSM is.

    VSM traces its lineage back to lean manufacturing methods, which called for each step of a manufacturing process to be continuously measured. As software development has evolved and organizations realize how dependent they are on software, there is a growing appreciation for the value of monitoring things such as the impact of missed software development deadlines on the business. At their core, VSM platforms aggregate DevOps metrics and then match them to a set of key performance indicators (KPIs) defined by business leaders. Once those metrics are gathered and correlated, organizations can apply advanced analytics.

    Eventually, the goal is to apply algorithms and other forms of predictive analytics that will enable organizations to more easily identify the precise impact software development processes have on the business, noted Lynch.

    In the meantime, the CloudBees survey identified manager commitment as the main roadblock to VSM adoption, with 35% of respondents saying getting mid-level employees committed and 31% saying getting senior leadership committed are the biggest roadblocks. That’s a significant issue because more than half of respondents (56%) said C-level executives make VSM investment decisions.

    Survey respondents also reported that scaling VSM efforts throughout the organization is a significant issue for a third (33%) of respondents.

    In general, many VSM initiatives were sidetracked by the COVID-19 pandemic, noted Lynch. However, organizations will now be looking to evaluate the results of those efforts, she added. Digital transformation plans are in place and going well, according to 96% of survey respondents, with only 18% reporting that they are moving, but not fast enough.

    However, all software development processes could be improved by surfacing real-time insights, added Lynch. Most senior executives today don’t have a lot of visibility into the software development processes that are driving multiple digital transformation initiatives, she added.

    In theory, VSM platforms should one day help narrow the divide that has long existed between IT and the rest of the business, as dashboards make it simpler for everyone involved to appreciate the cascading impact a software development decision has across the entire business.

  • Index Tracks Impressive Growth of API-First Startups

    Index Tracks Impressive Growth of API-First Startups

    APIs are the building blocks of modern software, and their use continues to grow around the globe. A full 89% of investment into APIs will increase or stay the same over the next 12 months, Postman’s 2022 State of the API report found. And the evolving API-first approach is ushering in many innovations and startups into the field. This trend is epitomized by the latest research from GGV Capital, which maintains an API-First Index of leading developer-first software companies.

    The API-First Index tracks API-first companies that have raised $50 million in funding or more. And as of Q3 2022, the number of businesses that met this threshold increased to over 60. Just in Q3 alone, $350 million in funding was raised by companies on the index, and, in total, the index represents over $14 billion in funding. Undoubtedly, interest and investment in this space are expanding and will continue to increase in the year ahead. It’s a good area to keep tabs on, as the emphasis on modularity impacts how software is pieced together.

    I recently met with Tiffany Luck, partner at GGV Capitol, who oversees the API-First Index. Below, we’ll explore the index a bit more and get Luck’s viewpoint on why these findings represent a unique moment for API-first technology and business models as a whole.

    The API-First Index

    The API-First Index focuses on companies that serve niche functionally through APIs; also known as API-as-a-product. The list includes an assortment of API-based tools from FinTech, communications, health care, commerce and other sectors.

    For example, Stripe is likely the most well-known API-as-a-product, as it is commonly integrated as a way for businesses to accept payments. There’s also MessageBird, an API for sending notifications across any channel. Other examples include Mapbox, an API for geolocation data, Mux, an API for creating live and on-demand video experiences, Nylas, which offers a popular email API and Alpaca, an API for stock and cryptocurrency trading and many others.

    Currently, the index doesn’t include the entire API platform landscape of tools, which ranges from areas like API management to testing, documentation, monitoring, design, security and observability. There are also middleware integration platforms to consider as well. With these included, the list of well-funded API-related toolsets would surely be even longer.

    Why the Excitement About API-First?

    The number of people deciding to build API-first companies is rapidly increasing. These API-based web components are interchangeable and represent the future of software development. “The world is going toward web and cloud components,” said Luck. The trend represents a global movement with much effort reflected in Europe, Asia and the U.S. Furthermore, API models seemed resilient throughout ongoing economic uncertainty and the patchy VC funding of 2022. “APIs are holding up very well compared to the broader VC landscape,” said Luck.

    So, why is there so much excitement around APIs? Well, people are looking for best-in-class structural blocks that can be used to build common features like geolocation, AI, weather, payments or specific CRM functions. Niche APIs can swiftly deliver particular features without the bulk of a large platform. Another reason is that they’re relatively sticky—API-based delivery models are efficient at building a self-service platform that expands over time, which is a very keen prospect to investors, said Luck.

    Unicorn APIs Solve Hard Problems

    In evaluating API-first companies, Luck looks for those offering B2B solutions to super-hard, shared problems. This equates to reusable functionality that is difficult to build, maintain and scale individually. Furthermore, the solution must solve a need that’s common across companies but that is not core to their products.

    For example, Luck has her eye on a few impressive API-first use cases. Stream, for example, powers in-app user-facing features. Knock offers solutions for handling notifications. But many of the top most-challenging areas include financial services, as this functionality is difficult to build from a regulatory and connectivity perspective. Examples include Pinwheel, which aggregates real-time income and employment data. Method API does something similar but for aggregating consumer debts.

    Luck described APIs as a “modern way of accessing what would otherwise be technically inaccessible.” We’re seeing this action unfold in open banking, as new standards around the globe now mandate that consumer financial data be programmatically accessible, often in the form of an API connection.

    Developer Experience: The New Competitive Advantage

    In a competitive market surging with powerful developer-facing tools, non-functional aspects such as developer experience (DX), security, performance and reliability are becoming more of a focus. According to Luck, developer experience is probably at the top of the pyramid as it represents a catch-all for many of the aforementioned traits. Specifically, developer tools with high-quality DX have excellent documentation and must maintain backward compatibility to avoid breaking changes on the consumer side. “Ultimately, what you’re implementing impacts that customer’s end customer. Customers want zero negative impact and zero disruption,” she said.

    The API-First Index is good for the industry as it helps educate the public on private investing rounds. And as discussion around it increases, it brings awareness and visibility to the benefits of building API-first.

  • 5 Tips to Consider When Introducing Infrastructure Automation

    5 Tips to Consider When Introducing Infrastructure Automation

    Although we discuss infrastructure automation often, the fact is that a lot of manual processes still exist throughout DevOps. Whether it’s rewriting scripts to spin up new servers, updating cloud-native policies or configuring computing environments, manual toil is an all-too-common reality. To make matters worse, engineers commonly create lengthy scripts that work in isolation but are too bulky to be reusable by other developers working with slightly different constraints.

    More infrastructure automation is the key to increasing deployment agility, but setting aside time to automate your daily workflow can easily take a back burner, especially when developers are already behind on their long list of to-dos. So, for the time-pressed, where should they start? What exactly should be automated? And how can we effectively reuse and share automation to help other areas of the business?

    I recently connected with John Laffey, technical product marketing director, Puppet. Laffey has thirty years’ worth of tech support and engineering operations knowledge under his belt and was “doing DevOps before it was called DevOps.” He shared with me five helpful tips to think about when introducing more infrastructure automation into your workflows. Below, we’ll expand on each of those points and consider other best practices around automation as the industry moves toward platform engineering.

    1. Automate Things That Interrupt Your Day

    First, Laffey recommended starting with the things that are easy to fix. These are the tedious actions that interrupt your day. Perhaps a disk runs out of space and troubleshooting takes 15 or 20 minutes. Minor interruptions like this might seem like part of the job, but they definitely add up over time.

    2. Automate Things You Can Reuse

    Secondly, try not to automate things for an ultra-specific purpose, said Laffey. Instead, focus on building automation that is more generic and can have more value to others. This follows the DevOps mantra of teamwork and collaboration. One-off automation that is too verbose or specific to a certain condition won’t apply well across groups.

    3. Automate Stuff That Doesn’t Change Much

    Next, automate the infrastructure that isn’t prone to change. These are fairly reliable things that you’re sure will be part of your systems for the foreseeable future. Because if a component does end up having to be updated often, supporting the automation could be more hassle than it’s worth. “Don’t trade fixing problems for fixing automation,” said Laffey.

    4. Start Small

    Here’s some sage advice applicable to most new projects with a big scope—start small. Laffey recommended starting by automating the easy stuff that has proven immediate value. These tasks should be modular and do one specific thing, like rebooting a machine or draining a log file. “Automate things in chunks that you can put into a toolchain.”

    Too often, teams try to automate everything all at once. But by taking big bites, you can quickly get off track and potentially add to technical debt. If you keep things modular, you’ll find it easier to change and reuse this automation.

    5. Share and Be Open

    This last tidbit is more of a cultural change—don’t code so that you’re the only one who knows about it! Sometimes, engineers don’t necessarily want the world to know about the automation they create. They might worry about automating their job away. Yet, Laffey said, these worries are misplaced. There will always be more work to do, and automating toil away can free up time to help you focus on more exciting projects.

    So, Laffey encouraged DevOps and SREs to be open about the automation they create. Ensure they’re well documented and stored in a centralized repository to allow other departments to reuse your code. This can avoid the duplication of having five different versions of the same script written in different languages within the same organization!

    Next Step: Move From Scripts to Industry-Standard Tooling

    It can be challenging to argue for investing time and energy into infrastructure automation, especially when engineers are hard-pressed to keep the lights on. Yet, small gains in automation can make huge waves in the long term.

    Eventually, said Laffey, you’ll want to move away from using scripts within your toolchain, which he describes as a low-end DevOps maturity. But as you update your process, you’ll want to replace these scripts with more modern industry-standard tools like Terraform or Puppet’s DSL. He recommended slowly revamping scripts in a toolchain as you update them one by one. “Start where you are and avoid a monolithic approach.”

    In essence, platform engineering, Laffey explained, is wrapping DevOps principles at a higher level to make them more efficient. And we will likely see an increased emphasis on infrastructure automation as platform engineering gains more mindshare. “It’s going to become a necessity.”

  • Codenotary Extends Dynamic SBOM Reach to Serverless Computing Platforms

    Codenotary Extends Dynamic SBOM Reach to Serverless Computing Platforms

    Codenotary has extended the reach of its platform for automatically generating software bills of materials (SBOMs) to serverless computing platforms running software constructed using functions.

    Codenotary CTO Dennis Zimmer said because serverless apps are dynamically created, it’s not possible to generate SBOMs using traditional approaches. The TrueSBOM platform makes it possible to create an SBOM in real-time by adding one line to the source code of any application. The TrueSBOM for Serverless edition of the platform now extends that capability to the serverless computing frameworks made available in the cloud by Amazon Web Services (AWS), Microsoft and Google.

    That approach makes the SBOM a part of the application itself rather than a text file that is stored separately, he noted. Each time an application is updated or a scanner discovers a new vulnerability, a new SBOM is automatically generated, added Zimmer.

    Awareness of the need for SBOMs has skyrocketed since an executive order issued by the Biden administration made it clear that federal agencies would require them from software providers starting next year. Many enterprise IT organizations are likely to follow suit as part of a larger effort to better secure software supply chains in the wake of a series of high-profile cybersecurity breaches.

    Most internal DevOps teams already have a good handle on what software components are used within the applications they deploy. The issue is that the organizations that use that software can’t easily verify what components are being used, noted Zimmer. That’s problematic because an organization may prohibit deployment of a specific software component because of a known vulnerability. TrueSBOM allows organizations that use an application to maintain control over their software environment regardless of what software artifacts are used to build closed or open source components, Zimmer added.

    It’s unclear how most organizations will operationalize SBOMs now that more of them are being created. Ideally, organizations should be able to approve only software with components that have been verified as secure.

    As SBOM mandates become more stringent, the number of application providers that are anxious to comply with rules for securing software supply chains should increase. The issue now is finding a way for both the developer and consumer of that software to streamline a verification process that, in its current form, is too cumbersome to manage effectively. As a result, many organizations are now putting together dedicated teams made up of cybersecurity professionals and software developers to manage SBOM processes, said Zimmer.

    Of course, there is a massive amount of code that has already been deployed that today doesn’t show up in an SBOM. While many emerging applications—created using artifacts such as functions and containers—are likely to be increasingly instrumented to dynamically generate an SBOM, the number of legacy applications that might need to be similarly instrumented is several orders of magnitude greater. As a consequence, the amount of effort required to generate SBOMs for every application in use might be greater than teams first anticipated.