Category: Blogs

  • Secure Coding Adventure Park Virtual Summit: Wilder, Faster and More Visionary Than Ever

    Secure Coding Adventure Park Virtual Summit: Wilder, Faster and More Visionary Than Ever

    DevSecOps helps secure code from the start of production, rather than patching and debugging it at a later stage. It brings development, operations and security teams together so they can release secure software faster. At Secure Coding Adventure Park Virtual Summit, taking place on September 29, industry-leading AppSec and DevSecOps practitioners, analysts and visionaries will come together to share their wildest rides on the security roller coaster and discuss everything you need to know about build secure code from the ground up.

    The agenda for Secure Coding Adventure Park Virtual Summit features a mix of topics, including AppSec, DevSecOps, open-source security, CI/CD and more.

    Security luminaries from AWS, Secure Code Warrior, GitLab, Sysdig, WhiteSource and many other well-known IT companies, will discuss:

    • How to prevent supply chain attacks from code through production
    • Why developers struggle with application security
    • The role of AppSec testing in securing your software supply chain
    • How to protect your open source projects and create a secure CI/CD pipeline
    • How remote work has influenced DevSecOps adoption
    • How to best approach modern software security by automating your CI pipeline for simplicity, visibility and control

    This year’s Secure Coding features outstanding industry leaders, such as:

    • Harry Mower, general manager, AWS DevOps Tools at Amazon Web Services
    • Maciej Mensfeld, senior product manager at WhiteSource
    • Marina Segal, director of product management at Sysdig
    • Stefania Chaplin, EMEA’s solution architect at Secure Code Warrior
    • Cindy Blake, senior product marketing manager and security evangelist at GitLab
    • Rhys Arkins, director of product management at WhiteSource
    • Eric Tice, director global open source technical consulting and COE at Wipro
    • Jayne Groll, co-founder and CEO at DevOps Institute
    • Eilam Milner, co-founder and CTO at Argon Security
    • Jonathan Leitschuh, security software engineer at Gradle Inc.
    • Ricardo Sueiras, technical evangelist at AWS
    • Sonya Moisset, principal security engineer at Photobox, Epic Women in Cyber/Tech founder, GirlCode ambassador
    • Scott Gerlach, co-founder and CSO at StackHawk
    • Dave Walker, principal specialist solution architect for security and compliance at AWS

    Our very own Jackie Whetzel sat down with Stefania Chaplin for a quick interview about her upcoming session at Secure Coding Adventure Park Virtual Summit, women in tech and her journey as a solutions architect. Check out a clip of the interview below as Stefania shares a sneak peek of what to expect from her upcoming session.

    To check out the full agenda and to register, visit the Secure Coding Adventure Park Virtual Summit website.

  • Transform Mobile DevOps into Mobile DevSecOps

    Transform Mobile DevOps into Mobile DevSecOps

    Mobile developers are mostly focused on adding new features and functionality to their apps. Security features—such as RASP protection, obfuscation, encryption, jailbreak/root prevention and MiTM prevention—are usually addressed at the end of the development cycle if they are addressed at all. Unfortunately, most apps ship with significant vulnerabilities. One internal app test showed that most apps can be cracked within 15 minutes, and a recent academic study found that 67% of the malicious app installs came from the Google Play Store.

    An Appdome global survey of more than 10,000 smartphone consumers released in August 2021 showed that this approach—features over security—is an enormous mistake. Nearly two-thirds (63%) of consumers say they value security as much or more than features when it comes to mobile apps. Nearly three-fourths (73%) said that they would stop using a mobile app if it left them unprotected against attacks.

    The survey revealed a big shift in consumer sentiment. For the first time in recent history, consumers are on the security team’s side! They expect strong protection from threats and fraud. But meeting these expectations is extremely challenging for development teams, because DevOps and security are so often at odds. 

    Here’s why: Nearly all the work done in DevOps is automated, from GitHub to Jenkins to Fastlane, while almost all security work is manual. Sure, security software development kit (SDK) vendors position themselves as an alternative, but buyer beware. Implementing a security SDK is never just a couple of lines of code. It’s still a brittle, manual process that does not offer a guaranteed outcome.

    Mobile app security solutions available today don’t fit into the automated delivery model that is DevOps. The only way to automate security and make it fit within the DevOps workflow is to stop trying to squeeze it in at the end of the Dev cycle. As a result, security does not have a seat at the mobile app development table today. And as long as security is manual and cannot be seamlessly plugged into the existing DevOps workflows, it will remain an afterthought, at best.

    The Encryption Example: Why Security is so Hard

    As an illustration, let’s take a look at incorporating mobile encryption, an absolute necessity to protect data against harvesting, theft, interception, deception and trickery. For every type of data inside the app, developers must choose the right combination for each component of the encryption model, including encryption algorithm/protocols, cipher suites, key strength, key derivation technique, key protection and safeguarding. It has to match both the security and performance requirements for a highly variable set of data format characteristics. Even small errors or mismatches could slow app performance or render the app unusable. On the other hand, the app may perform well, but the encryption may be too weak to provide sufficient protection. 

    It’s complex, time-consuming work that developers don’t want to have to do twice. That’s why they tend to wait until after the app is built to add encryption. Unfortunately, this means all the network information is stored in the app unencrypted, and if a bad actor gets their hands on a UAT version of the app, the “keys to the kingdom” are in the clear, ripe for the picking.

    The demands on DevOps to get new releases out of the door quickly are high. Security is often seen as a negative force that holds back innovation and delays release, especially when it’s left until the very end of the software development life cycle (SDLC). The term DevSecOps is used to describe a security-focused, continuous delivery SDLC. But that should involve much more than penetration testing. Identifying issues isn’t enough. Security needs to be built into the development cycle without slowing down release times or creating uncontrollable costs.  

    The only way to incorporate security into DevOps and to achieve DevSecOps is to imbue security with as much automation as development currently enjoys. By using an automated, no-code platform that builds security directly into the app, security can become a natural part of the DevSecOps workflow. Plus, security can be consistently applied with documentation that demonstrates protection. It’s the only way to enable developers to get their secured mobile apps into the hands of their end-users at a rapid pace, align all stakeholders to a common shared set of objectives and outcomes and eliminate the sources of friction commonly found between DevOps and security teams. 

  • Can Web 3.0 Solve the Linkrot Crisis?

    Can Web 3.0 Solve the Linkrot Crisis?

    Data on the web is more ephemeral than you realize. A recent examination of linkrot and content drift by Harvard Law School found that as time progresses, web-based content is becoming more and more lost. By analyzing half a million links within New York Times articles, 6% of deep links from 2018 were found to be inaccessible, and 72% of links from 1998 were completely dead.

    The rotting web could have severe consequences that affect society’s collective understanding. Consider links used to substantiate pivotal arguments in the course of law. Studies now show that 49% of hyperlinks in Supreme Court decisions no longer function. The linkrot situation undermines humanity’s shared intelligence, which the Internet once promised.

    So much of the content on the web is outdated, inaccurate or has changed its URL destination, a problem known as content drift. Therefore, band-aids like the Wayback Machine internet archive have risen as a steward of the historical web archives. However, some proponents in the industry argue we need a more modern, decentralized, immutable reference for this historical data. You may have guessed where this article is heading—blockchain.

    I recently met with Jonathan Victor, Protocol Labs, to see how Web 3.0 and blockchain might be used to solve the linkrot endemic. According to Victor, smart contracts that reference canonical URLs could help create a more resilient web. This would, in effect, require a peer-to-peer model with shared ownership and incentive models to sustain content longevity.

    The State of Linkrot

    The longer data sits, the higher the likelihood of losing track of it. Forgotten databases and rising data storage costs can easily lead to whole networks of websites and user data being accidentally (or intentionally) destroyed. Putting so much faith in private companies to preserve data indefinitely is not viable, explained Victor.

    Public data isn’t really as public as you think it is. And older, user-generated content is often the first domino to fall. For example, a massive amount of media on MySpace was deleted in a botched server migration in 2015. In 2021, Yahoo Answers similarly erased 16 years of user-generated content in its shutdown. Surprisingly, users have little ownership or control of their data on forums and social media. A ban on Twitter, for example, could result in inactive links within posts.

    When an article points to a website, this imbues an implicit dependency on an external organization to keep the domain live, said Victor. The second dependency is on the domain owner to keep content unchanged. “If we can talk about content and how to reference it in a different way, maybe we can create references that are much more resilient,” suggested Victor.

    How Web 3.0 Can Help

    One way to solve the linkrot issue is to create a ‘fingerprint’ for web resources. This could be a canonical reference to specific pieces of content. Organizations would store and retrieve content based on a hash of that contract itself, Victor explained. Such a system would likely require an incentive-based, non-exploitative, decentralized network where web users own their data and control who profits from it.

    Some have coined this concept Web 3.0. Victor describes Web 3.0 as a broad classification of distributed technologies and tools that enable a peer-to-peer internet underpinned by blockchain.

    For example, one open source project spearheading progress on this front is IPFS, the InterPlanetary File System. IPFS is a peer-to-peer hypermedia protocol that uses blockchain to store cryptographic hashes. The project is “designed to preserve and grow humanity’s knowledge by making the web upgradeable, resilient and more open.” With IPFS, a client requests content using the hash and retrieves content. IPFS is based on Filecoin, a permissionless storage network.

    Benefits

    A decentralized nature fits the original goal of the open web, allowing creators to own and share content horizontally. If you can decouple data storage and retrieval from a specific location, it gives you an impressive ability to rearchitect modern communication, said Victor. This could have a number of benefits:

    • Redundancy: First off, a major benefit is the redundancy of data. Distributed nodes assure data is globally available and temporally accessible. Resources are in multiple locations, ensuring better global availability and temporal persistence.
    • Incentives: An incentive-based decentralized network could use block rewards to subsidize costs, helping fund storage initiatives. Building on the blockchain can also ease payment acceptance aspects — great for donation-based data hosting groups.
    • Efficiency: By using smart contracts, you can programmatically handle storage requests. “Blockchain gives you higher-level primitives to run this stuff automatically,” said Victor. This helps tie on-chain activities to real-world storage, Victor added.
    • Low-cost: It can be cheaper to store content on distributed networks than a typical cloud provider. Impressively, Victor estimated it would cost 1/1000 of the cost of Amazon. A distributed environment could also bring speed gains; similar to torrenting, seeding from distributed sources increases speed in orders of magnitude, said Victor.
    • Communal ownership: Such a model could spur more communal ownership for some content. This could benefit public-good assets that don’t have a clear owner. According to Victor, this aspect especially benefits universities and open source data initiatives that rely on public support to function.

    A Compelling Problem

    Constructing a new peer-to-peer Internet obviously has significant barriers. Namely, a decentralized structure requires a large group effort, and this community action may be difficult to encourage, let alone coordinate. It will also require consolidation around open standards and protocols, as many different decentralized projects crowd the scene. Shifting pre-existing architectures to this new style could also introduce roadblocks.

    “A lot of content on the internet becomes a public good,” says Victor. Yet, perpetual access to this content is fickle, even for well-established online communities. Some research has found the average lifespan of a website to only be 100 days. Hosting fees or accidental failures are often root causes for the loss of data.

    The fragility of the web is an issue that will take not only innovative development effort, but a radically new mindset to tackle. “It’s a really compelling problem,” added Victor. Using blockchain-based smart contracts to store canonical references for content is one proposed solution. But Web 3.0 is in a primitive stage, and to progress, it will take significant community action.

    Doing so in a way that is sustainable will be the trick. Open standards and blockchain-based protocols could allow the technical means for such a network to flourish. And a cryptocurrency backing could help incentivize participation and provide a means of funding. This decentralized non-ephemeral storage could help enable perpetual storage for the web’s most precious resources.

  • Highlights from GitOps Evolution Virtual Summit

    Highlights from GitOps Evolution Virtual Summit

    GitOps has become increasingly popular in the DevOps space. DevOps teams have been adopting GitOps in order to better deploy, manage and monitor Kubernetes clusters and applications. On Sept. 23, the DevOps community came together at GitOps Evolution Virtual Summit to explore ways in which organizations can use GitOps to solve real problems.

    The virtual event, hosted by MediaOps, offered expert-led sessions and panel discussions focused on a variety of topics, including:

    • GitOps and the evolution of DevOps
    • A developer’s guide to adopting GitOps
    • Metrics to measure in GitOps environments

    GitOps Evolution Virtual Summit is now available on-demand for up to 30 days after the live event. Register now to get exclusive access to the on-demand virtual environment.

    Meet the GitOps Evolution Leaders

    GitOps Evolution Virtual Summit featured an exceptional lineup of industry thought leaders and doers in the GitOps space, including:

    • Tracy Ragan, DeployHub CEO and CDF board member
    • Viktor Farcic, developer advocate at Upbound, member of the Google Developer Experts and a published author
    • Paul Curtis, principal solution architect at Weaveworks
    • Ravi Lachhman, evangelist at Harness
    • Jayne Groll, DevOps Institute CEO
    • Alan Shimel, MediaOps CEO

    GitOps Evolution Sessions

    GitOps – An Introduction, by Alan Shimel and Paul Curtis

    In this session, Paul and Alan discussed the four principles of GitOps, how an enhanced developer experience increases productivity, how to boost stability and reliability in Kubernetes environments and how to bake in security from the start.

    Automating the Human Side of GitOps, by Tracy Ragan

    GitOps requires many manual steps, from updating .yaml files for each new container image, and managing Pull Requests, Merges and Commits. In this session, Tracy explored how to move GitOps to a fully automated DevOps process and reviewed the basic architecture of GitOps, how it is managed manually and how GitOps needs to evolve to be part of the continuous delivery pipeline.

    How To Apply GitOps On Infrastructure, by Viktor Farcic

    During this talk, Viktor Farcic explored Crossplane combined with Flux as a potential solution for all our infrastructure needs and whether it fits into the broader Kubernetes ecosystem.

    GitOps Blind Spots – Introduction to Pragmatic GitOps, by Ravi Lachhman

    Check out Ravi’s session to learn how GitOps can be a continuum in a journey and introducing Pragmatic GitOps practices and approaches no matter where you are in your software development approaches. Ravi talked about common blind spots with embracing on a pure GitOps approach might leave to be solved for.

    The Evolution of GitOps Maturity for Business Success, by Paul Curtis

    In this session, Paul went on a four-step GitOps journey that started with operating a single cluster and application, to managing large deployments and fleets. He discussed the requirements and business benefits of each step and tied them back to improving Software Delivery and Operational (SDO) performance—including deployment frequency, shorter lead time, mean time to recovery and change failure rate.

    Fireside Chat: The People, Process, and Technology of GitOps, by Jayne Groll, Tracy Ragan and Viktor Farcic

    Join Jayne Groll, Tracy Ragan and Viktor Farcic for a panel discussion probing further into the people, process and technology of GitOps – including what’s next, challenges to culture and theory versus practical implementation.

    For more information and to access the on-demand environment, visit the GitOps Evolution Virtual Summit website.

  • Achieving Business Maturity with Database DevOps

    Achieving Business Maturity with Database DevOps

    Companies have seen unprecedented change over the past year and a half, with IT teams trying to help businesses weather the storm by meeting customer demands for services and products to be delivered in new ways while adjusting to remote work. As more companies introduced hybrid approaches to their workplace, a particular trend began to emerge: DevOps adoption. 

    Many adopted DevOps to navigate the new landscape, boosting software delivery and operational performance with advanced application and database development practices. In fact, research from Redgate Software indicates that DevOps adoption reached its peak earlier this year, with 26% of enterprises leveraging DevOps across all projects—a significant increase from previous adoption rates.

    Although not all companies have been quick to adopt for fear of disrupting existing workflows, a technical skills gap or lack of buy-in from management, there is a strong case to be made for why adopting DevOps practices, including for the database, can help businesses better manage the evolving tech environment.

    Adopting Automated Deployments

    Companies everywhere migrated databases to the cloud over the course of the past year, and that trend shows no sign of slowing. Analysts at IDC predict that by 2022 more than 90% of enterprises will rely on a mixture of on-premises/dedicated private clouds, multiple public clouds and legacy platforms to meet their needs. One of the best ways to help maintain that data is to implement DevOps processes for those databases and automate deployments. This provides a host of benefits, including: 

    • Preventing software errors: Manual database migrations often come with a host of errors that lead to downtime, rollbacks and rework. It’s far better to rely on an automated process that is robust and repeatable and can be implemented across all databases to ensure they’re updated in the same, reliable way. 
    • Staying in the loop: Knowledge of how to release updates and changes is often limited to a handful of people or even just one person. If, instead, it is part of the system and development process, it ensures everyone is in sync with updates and developments and allows anyone on your IT team to handle deployments.
    • Guaranteeing quality and consistency: For companies with security and compliance concerns, the predictable and repeatable processes that automated database deployments introduce ensure nothing is left to chance, while providing an auditable record of exactly what was changed, when and by who.

    Although the research shows 20% of companies cite cost as a roadblock to automation, avoiding automation will leave businesses at a disadvantage as they prepare for the coming year. 

    Creating Communities of Practice 

    When asked how they decide the strategic direction and planning for DevOps, the most popular answer from respondents was communities of practice. This concept was proposed for the first time in the early 1990s, and the practice has seen increased popularity among IT teams recently. It’s described as groups of people who share the same passions or concerns for something they do and often learn how to do that task better based on their regular, frequent interactions. 

    These various communities—regardless of whether in-person or virtual, given the current environment—help foster intrinsic motivation in employees and improve overall decision-making and encourage innovation. According to Redgate’s research, over a third of respondents noted that a community of practice makes decisions, provides recommendations or is consulted before DevOps strategies and planning are put into place. Another interesting point from the research is that the adoption of communities of practice correlates with company size. Enterprise and medium-size companies reported adoption rates of 38% and 36% respectively compared to only 28% adoption in small companies.

    As companies grow and teams mature, these communities not only benefit the organization by bringing more diverse minds to the table when making decisions but benefit individuals by supporting development, enabling networking and improving recognition across the organization. 

    Managing Cross-Database Estates

    Redgate’s research found that 70% of organizations now have more than one database management system in use, and 48% have teams working on three or more different databases. One possible reason for this is the increased adoption of cloud databases. With a large range of cloud databases and platforms in use, each offering different advantages, teams are now making choices based on specific use-cases. There was an increase in the share of respondents using cloud-native databases such as Azure Cosmos DB, MongoDB and Snowflake compared to last year, for example.

    With cloud adoption accelerating due to the pandemic and the need for technologies that can be managed, operated and accessed remotely, the advantages of introducing DevOps processes to maintain database changes and ensure version control have never been clearer. Teams that have automation and advanced development practices in place can continue building and deploying software as smoothly as before the pandemic, even when working remotely.

    Instead of working to build up processes and procedures, mature DevOps teams can focus their efforts on solving problems and ensuring higher quality of code, without the need for laborious manual tasks or code reviews that automated systems perform more reliably. 

    Making the Case for Database DevOps

    When teams are physically separated, working disparate schedules and dealing with their own challenges at home, the more companies can do to reduce workloads and help them focus on value-added work creates a happier workplace and improves the chances for business success. The rise of DevOps for both application and database development helps mitigate many of the challenges organizations face as part of their digital transformation journeys. It’s going to help IT teams remain productive when working remotely and encourage them to explore new database platforms, both on-premises and in the cloud.

    With remote and hybrid work the new normal for the future, companies that set their sights on adopting DevOps across the company will only see these benefits increase. 

  • Harnessing the Value of Log Data Analytics

    Harnessing the Value of Log Data Analytics

    Each day, the average enterprise’s cloud applications, containers, compute nodes and other components throw off millions of tiny logs. Each log is a file whose data describes an event such as a user action, service request, application task or compute error. Logs also capture messages that applications and other components send to one another.

    There’s a wealth of information and value hidden in those logs, which is why businesses are so keen on using them to address business needs like customer engagement, IT security and cloud operations. 

    The need for understanding and managing this data is so high that Markets and Markets valued the market for log analytics data at $1.9 billion in 2020. Analysts predict this to grow to $3.7 billion by 2025. It’s not just the sheer growth of data driving this market—it’s also the demand to gain business insights. 

    That’s why ChaosSearch recently conducted a survey to better understand how businesses are currently using and managing their log data. Insights were collected from practitioners at 50 medium-to-large enterprise organizations. What the survey found is that organizations see a lot of value from tapping into their log data to achieve better outcomes across both business and IT functions. That said, these practitioners face challenges handling the scale requirements and they know there are still more opportunities for them to expand the scope and discipline for log management.

    Top Use Cases

    This study confirmed that businesses are using log data in a number of ways, spanning both business and IT functions. We see that more than 90% of respondents are using log management and analytics for at least two primary use cases. It’s not surprising that the top use cases for log management are security (70%) and IT monitoring (68%), but many are prioritizing BI & analytics (46%) and business operations (28%) as well.  

     

    It’s interesting to look more closely at the security use cases: Investigations (45%) and insider threats (45%) top the list of SecOps uses for log data. Audits and compliance (32%), threat hunting (32%) and anomaly detection (30%) also made the list. 

    Growing Desire to Support Business Operations

    We also saw that these practitioners want to use log data to support business operations—specifically to drive better business outcomes and improve competitiveness. For example:

    1. A global industrial equipment manufacturer attached IoT devices to farm equipment to track weather, temperature, humidity, time of day, depth of planting, soil conditions, seeds planted and geo-routing on farms. This raw data is continually collected and analyzed within the central log analytics solution, thereby allowing the vendor to deliver farm productivity reporting and recommendations back to their clients.
    2. A biopharmaceutical firm tracks cell culture logs with alerts on temperature and humidity. The company uses its log analytics platform to store long-term data to run queries that help in hypothesis testing. They also conduct various long-tail analyses that reveal trends and allow scientists to make informed decisions about future experiments. 
    3. A large online retailer leverages log data from web visits and other customer activities to develop 360-degree profiles of customers and analyze the steps in the “customer journey” to identify opportunities for improved efficiency and increased revenue capture.

    Tackling Challenges of Growth and Complexity

    Even with more interest in using log data for security and business operations, the surveyed practitioners called out a range of challenges. The top three being: Complex infrastructure (48%), managing costs (42%) and managing growth (34%). They also said the lack of centralization, reporting and data quality were problems. These challenges are perhaps exacerbated by the expectation that data will continue to grow and data retention will increase. 

     

    There are a lot of variables that drive the volume of log data generation, like the number of users, devices, applications, IT environments and infrastructure elements. The survey found two things were universal—data growth rates are high and are a source of pain and data retention is an increasing priority. 

    Among the participants, 94% ingest at least one TB or more of data per day, with 18% in the 10TB + range. Consider this—the average daily ingest volume for these companies was 7.9 TB! 

    The problem is, despite the massive amounts of data being ingested daily, only 28% of respondents are capturing 80% of log data or more today. And 78% believe that capturing more than 80% of log data is ideal. That’s a big gap. 

    Practitioners Rate Most Important Log Management Capabilities

    We also learned that these practitioners are generally satisfied with their existing solutions, but many of them still expect to bring in a new solution for log management and analytics within the next three years. That’s because they want better scale and a cloud-native solution. 

    When we asked about important capabilities for a log management platform, scalability (58%), resilience (54%) and advanced analytics (52%) topped the list. 

     

    Budgets Increase to Meet Demand for Log Analytics

    To address these challenges and opportunities before them, survey respondents are growing their budgets; 86% anticipate their budgets will grow by at least 20% and 36% expect an increase of 50% or more. These budgets will not only support more data capture and increase retention periods but also help them go after the growing use cases for supporting the business. In fact, 68% said they expect the number of use cases to grow in the next 12 months.

    There is no doubt: Companies who value log data and invest in capturing data will harness insights that will propel their businesses forward.

  • LogDNA Previews Streaming Module for Observability Platform

    LogDNA Previews Streaming Module for Observability Platform

    LogDNA is making available on a limited basis the ability to process streaming data to its observability platform as part of an effort to give DevOps teams more control over log data.

    Tucker Callaway, LogDNA CEO, said the data ingestion pipeline that LogDNA already provides can ingest, parse and normalize massive amounts of structured and unstructured log data. LogDNA Streaming adds the ability to automatically send log data to any application or analysis tool.

    That capability also helps reduce storage costs by limiting the number of copies of log data that might otherwise be created, he added.

    LogDNA also expects that DevOps teams will also be able to analyze data in flight as part of an effort to make it simpler to surface issues even before log data is stored, noted Callaway.

    While there are multiple streaming data platforms available, Callaway said LogDNA opted to build one on its own to meet the requirements of an observability platform. Ultimately, the goal is to shift observability left at the data level to provide more granular insights in real-time, he said.

    That capability will also lay the foundation for a wide range of autonomous processes that would be driven by the observability data collected via LogDNA, added Callaway.

    LogDNA

    Observability has always been a core tenet of any DevOps best practice. Most DevOps teams typically focus on some form of continuous monitoring to more proactively manage application environments. However, it can still take days, sometimes weeks, to discover the root cause of an issue. Monitoring tools are designed to consume predefined metrics to identify when a specific platform or application is performing within expectations. The metrics tracked generally focus on, for example, resource utilization.

    Observability combines metrics, logs and traces—a specialized form of logging—to instrument applications in a way that makes it simpler to troubleshot issues without having to rely solely on a limited set of metrics that have been pre-defined to monitor a specific process or function. DevOps teams can more easily employ queries to interrogate data in a way that makes it easier to discover the root cause of an issue. An observability platform correlates events in a way that makes it easier for analytics tools to identify anomalous behavior indicative of an IT issue.

    It’s now only a matter of time before providers of the platform widely employ machine learning algorithms to analyze machine data faster. There may even come a day when so-called “war room” meetings—that today need to be convened to identify the cause of an IT issue via a painstaking process of elimination—are no longer required.

    In the meantime, the percentage of applications that are instrumented in an IT environment needs to increase to achieve that goal. Fortunately, thanks to the rise of open source agent software, the cost of instrumenting applications is starting to decline. At the same time, DevOps teams should expect to see a marked increase in the volumes of incoming data, and they will need to find somewhere to store that data as well as determine the length of time it should be retained.

  • How Quantum Computing can Solve Real-World Problems

    How Quantum Computing can Solve Real-World Problems

    The history of technology can seem like it was very predictable in retrospect, but the future typically feels more uncertain. Nowhere is this uncertainty more evident than in the domain of quantum computing. When the spectrum of possible outcomes spans from “quantum computers will be one of the most important technology developments of all time” to “quantum computing may never really become practical enough to justify using over a classical computer alternative,” trying to make forecasts can seem futile. However, as a technologist, investor, consumer or just a curious observer, the boundaries and texture of the uncertainty itself can be very interesting, valuable and informative. 

    Quantum computing isn’t likely a replacement for today’s classical computing—at least not initially—but rather a potential complement to it for specific applications. A 2021 report by Deloitte lists dozens of applications that could be transformed by quantum computing approaches: Protein folding, fluid simulation, credit underwriting, financial risk analysis, supply chain optimization and forecasting, vehicle routing, fraud detection, fault analysis, weather forecasting, semiconductor chip design, product-portfolio optimization, consumer product recommendations—and this is only half the list. 

    Quantum computing could make previously intractable simulation, search and optimization calculations relatively easy and quick. For some kinds of calculations, a quantum computer could be mind-bendingly faster than a powerful classical computer could ever be. While today’s classical computers running machine learning systems can make incredibly accurate predictions, quantum-based systems can, in theory, far outstrip their speed at certain prediction-related tasks.

    The blend of physics, math and computer science behind quantum computers is fascinating, but complex. Computer scientist Scott Aaronson, one of the best science communicators, wrote a recent essay for Quanta magazine called, “Why Is Quantum Computing So Hard To Explain?” 

    To oversimplify, quantum computers are built around “qubits,” which are subatomic-scale components, often kept at incredibly cold temperatures (think: Absolute zero) to minimize interference from the rest of the universe. These components can be put into a quantum physics state called “superposition” that allows them to, in a sense, take on many potential values at once. Even though their value is uncertain, reliable calculations and transformations can be performed on these qubits while in a superposition before being reduced to a more classical state, effectively parallelizing computations. Qubits in a quantum computer can also be entangled with one another (a phenomenon that Einstein referred to as “spooky action at a distance”), harnessing them as a team to increase the variety of logical calculations that it’s possible to compute. For certain types of math—for example, searching for an optimal outcome from myriad options—quantum algorithms can seem close to instantaneous when compared to those that run on classical computing hardware. 

    Quantum Computing Challenges

    Publicly available quantum computing has been on the horizon for decades, yet there’s still no reliable forecast for when products capable of widespread adoption will get released, exactly what form the systems will take or how broad and deep quantum computing’s impact will be. There are many technical challenges when it comes to getting a broadly applicable quantum computer to be useful at scale. And quantum computers are racing against classical computers, which have also been improving very rapidly in power, versatility and ease-of-use. 

    One important bottleneck for quantum computers is that correctly reading the results of a quantum calculation is prone to a very high error rate—the delicate superposition state can deteriorate before the correct result is presented as an output.

    Aaronson explains it:

    The problem, in a word, is decoherence, which means unwanted interaction between a quantum computer and its environment — nearby electric fields, warm objects, and other things that can record information about the qubits … The only known solution to this problem is quantum error correction… But researchers are only now starting to make such error correction work in the real world. When you read about the latest experiment with 50 or 60 physical qubits, it’s important to understand that the qubits aren’t error-corrected. Until they are, we don’t expect to be able to scale beyond a few hundred qubits.

    Some theorists even believe a practical quantum computer is impossible to build because of the inherently high amount of “noise in the system.” Mathematician Gil Kalai predicts that doing error correction to reduce that noise to a usable level would, by basic computing theorems, restrict the computer’s number-crunching capacity so low as to be not worth using.

    Nevertheless, researchers continue to chisel away at the challenge of error correction and to pass key technical milestones. For example, in July, Google researchers published a paper in Nature explaining how they have found a promising error-reduction method that scales as more qubits are added to a system. 

    Hardware, Software and App Landscape

    There are several different approaches to quantum hardware. Rigetti and Oxford Quantum Circuits use superconducting quantum circuits, a more widespread approach favored also by Google, Microsoft, and IBM. IonQ (which is planning to go public via a SPAC transaction) and Honeywell (which recently announced a combination of its quantum computing unit with UK-based Cambridge Quantum Computing) use trapped ions. D-Wave uses quantum annealing, ColdQuanta is using cold atoms and Xanadu is using photonics. Each approach has its pros and cons, and each hardware technology faces a fluid set of shifting bottlenecks in reaching affordable scale. Work at the materials layer continues to advance as well, for example, Raicol Crystals is growing quantum quasi-phase matching crystals that can be used in applications as varied as quantum computing, sensing, encryption and communications. And it’s not certain that there won’t be another approach that leapfrogs them in solving the challenges of error correction and price.

    Other companies, such as Quantum Machines, Q-CTRL and Seeqc, are building control hardware that can operate in the space between classical and quantum hardware. Control systems can exist at temperatures warmer than absolute zero, but colder than room temperature. Some of these systems involve both software and hardware and can perform tasks like monitoring the quantum processor’s performance, performing error correction or serving as an abstraction layer connecting classical and quantum hardware. 

    There are also companies creating quantum computing application layer software, like Zapata Computing, Riverlane or QC Ware. This can include application frameworks, components, algorithms or even turnkey quantum computing applications. Classiq has software for quantum circuit synthesis. There’s also a lot of interesting fundamental research happening at the software intersection between quantum computing and artificial intelligence – e.g. Zapata published work recently using IonQ’s ion trap quantum system to train a machine learning system to generate high-quality handwritten digits with low error rates. 

    One of the first theoretical uses for quantum computing, proposed in 1994 by mathematician Peter Shor, would be to crack the RSA encryption widely used today to protect data. The threat spawned research into post-quantum cryptography, with researchers proposing multiple new approaches to encryption that would need to replace today’s algorithms to preserve the privacy of data. Companies such as ISARA (quantum-resistant encryption) and Quantum Xchange (an out-of-band, symmetric key delivery platform) emerged to help prepare governments and companies now for a world where quantum-based, encryption-cracking algorithms start to work on breaking today’s encryption.  

    There are several widely used software development kits (SDKs) that let developers write software for a quantum computer without actually having one to run it on. In fact, a kind of format war has already broken out between quantum computing development environments.

    One widely-used quantum SDK is IBM QISKit, an open-source Python-based development kit for prototyping software and devices on simulators or IBM Quantum Experience, a cloud-based quantum service. ExxonMobil, Goldman Sachs, Boeing, JP Morgan, Samsung and PayPal are just a few of IBM Quantum’s marquee customers. Google Cirq, another Python-based development kit, touts Volkswagen as a user, and Volkswagen is also a contributor to Google’s TensorFlow Quantum, an open source library that extends Google’s TensorFlow machine learning algorithms to quantum processors for rapid prototyping. Microsoft’s Quantum Development Kit (QDK) is based on Microsoft’s Q# programming language and brings quantum compatibility to Microsoft Azure cloud services. Early customers of Azure Quantum include Dow for chemical simulations and Toyota Tsusho, Toyota’s trading arm, developing traffic light optimizations to reduce urban traffic congestion.

    In a recent piece, The Information noted that “Amazon is looking to hire more than 100 new quantum computing scientists, hardware developers and engineers within its Amazon Web Services cloud business,” following the enormous amount of funding flowing to startups in the space, such as Quantum Motion, Atom Computing and PsiQuantum. Amazon also launched Amazon Braket, a fully managed quantum computing service that gives customers access to hardware from a diverse set of vendors such as IonQ, Rigetti and D-Wave. 

    It’s also possible that the most valuable quantum platforms won’t be versatile, horizontal platforms at all; rather, the space could evolve instead to include special-purpose machines, or quantum application-specific integrated circuits (ASICs). Unlike the way things have played out with classical computing, this could mean that quantum computers would be highly specialized integrated hardware and software machines tailored to specific applications, e.g. fluid simulation or fraud detection. 

    Early use cases may involve quantum systems taking specialized roles as part of a larger classical system, similar to how NVIDIA’s graphics processing units (GPUs), which often run alongside an advanced CPU, can be used to offload specific computations for which the GPU is well-suited. In this model, where quantum components are an “accelerator” to classical systems, a credit underwriter might use a quantum processor to handle one of the tasks involved in scoring applicants and might use a different special-purpose quantum processor to perform a piece of a calculation to optimize its borrower portfolio. 

    For many use cases, the error rates of calculations are still too high for developers to do more than hone their coding chops on quantum simulation platforms, waiting until quantum computers start to really work at scale. Quantum simulators, such as QMware, make quantum software algorithms and applications believe they are running on actual quantum hardware, but they are really running on classical computers pretending to be quantum computers. 

    In the meantime, to manage some of the timing uncertainty, quantum-related startups can pursue profitable projects with government, corporate and university buyers today as a source of funding and requirements. Like some biotech startups, they can do this while assembling a strong team and seeking an early adopter clientele, progressing through scientific milestones while keeping an eye on future commercial markets.  

    The hurdles to generally available quantum computers are much like those faced by alternative energy sources: Unlocking the potential will require breakthroughs in hardware design, lower manufacturing costs and a driving force to spur large-scale adoption. And many approaches won’t work. 

    And as with any technology, it is, of course, possible that software developers will find valuable applications for quantum processors that people haven’t thought of yet today—just as developers found that GPUs meant to handle graphics processing also excel at training highly parallelized machine learning models.

    That said, a broadly useful and effective quantum computer’s arrival has the potential to be as disruptive an opportunity as the creation of the Internet or the use of machine learning to make predictions. 

  • Relyance AI Shifts Compliance Left Using ML Algorithms

    Relyance AI Shifts Compliance Left Using ML Algorithms

    Relyance AI emerged from stealth this week to unveil a namesake platform for managing privacy and data governance in real-time within the context of a larger DevOps workflow.

    Fresh off raising $30 million in seed capital, Relyance AI co-CEO Leila Golchehreh said organizations can now quickly and accurately answer essential privacy questions and ensure compliance and governance by mapping requirements directly to their codebase. The platform provides transparency and visibility all the way down to the source code in real-time using machine learning algorithms, she added.

    Relyance AI co-CEO Abhi Sharma also noted most legal and data protection teams today have limited visibility into digital operations. Developers and data scientists are constantly creating and updating applications. By the time compliance teams gather the data required, it’s obsolete, said Sharma.

    The Relyance AI platform analyzes contractual requirements against the operational reality as determined by data processing in real-time. Compliance teams can then address privacy issues as they arise, added Sharma.

    At the core of the Relyance AI platform are five integrated components that first inventory assets by connecting to application programming interfaces (APIs); generate universal records of processing activities (ROPAs) by scanning code and understanding how data is being processed; share alerts when issues are detected; provide views into data life cycles to highlight data subject access request (DSAR) exceptions and extensions and address vendor life cycle management to ensure third-parties are complying with policies.

    Relyance AI

    Sharma said compliance, much like security, is shifting further left toward application development as organizations look to eliminate issues before they arise in a production environment. The Relyance AI platform is designed to be integrated with continuous integration/continuous delivery (CI/CD) platforms that are the core of modern software development life cycle processes, he added.

    Undoubtedly, as the worlds of compliance and application development collide, there will be cultural challenges similar to what is now occurring with the rest of DevSecOps best practices that require more collaboration between developers and security teams. The challenge is compliance officers tend to be even further removed from application development than their IT security counterparts.

    However, it’s now only a matter of time before compliance becomes code. Modern applications already make use of the Open Policy Agent, for example, to embed compliance rules in applications. AI platforms should make it simpler to identify compliance issues as they arise so that they can be addressed before any infraction ever occurs.

    DevOps teams, of course, have a vested interest in managing compliance as code. It’s not uncommon for DevOps teams to race to build and deploy an application only to encounter a wave of compliance issues that need to be addressed by recoding elements of the application. Not surprisingly, those compliance issues often result in application delivery deadlines being missed. Machine learning algorithms provide an alternative to the current raft of documents that compliance officers create, but not enough other members of the IT team read those, much less understand them.