devopsy.kinsta.cloud

Tag: ibm

  • New Jupyter notebooks enable data scientists and policy makers to analyze real-time COVID-19 data

    New open source COVID-19 notebooks offer a toolkit that helps developers and data scientists answer important questions about the pandemic.

    By Frederick Reiss, Chief Architect, Center for Open Source Data and AI Technologies, IBM

    For data scientists and policy makers who are analyzing the effects of COVID-19 and trying to come up with actionable plans based on data, the information landscape is overwhelming. A near-constant flow of data from research studies, news outlets, social media, and health organizations make the task of analyzing data into useful action nearly impossible. Developers and data scientists need answers to their questions about data sources, tools, and how to draw meaningful and statistically valid conclusions from the ever-changing data.

    Policy makers face similar challenges. The United States has over 3,000 counties, each with a unique story of how COVID-19 is impacting its community. Policy makers are asking questions including: What stories can we tell in the aggregate? Are there patterns we see across the country? What regions or demographics are getting affected the most by the pandemic?

    Open source Jupyter notebooks offer actionable data

    To help, IBM’s Center for Open-Source Data and AI Technologies (CODAIT) created COVID notebooks, a toolkit that helps developers and data scientists answer important questions about the pandemic. We’ve taken care of the more mundane tasks, including:

    • Obtaining authoritative data on the current status of the outbreak
    • Cleaning up the most serious data-quality problems
    • Collating the data into a format amenable to easy analysis with tools like Pandas and Scikit-Learn
    • Building an initial set of example reports and graphs

    Taking care of these tasks frees developers and data scientists to focus on advanced analysis and modeling tasks instead of worrying about things like data formats and data cleaning. Our repository uses developer-friendly Jupyter notebooksto cover each of these initial data analysis steps.

    It’s important to note that the underlying data for COVID-19 changes on a daily basis. As you build your own analysis, you’ll want to update the results of your own notebooks frequently. But rerunning a collection of interconnected notebooks can be challenging. There are multiple stages of analysis, and the output of one step often feeds into multiple other steps. To simplify the process of updating your results with the latest data, we’ve created data processing pipelines using the Elyra Notebook Pipelines Visual Editorand KubeFlow Pipelines. For example, here’s the pipeline for county-level time series data for the United States:

    These pipelines let you rerun the entire extract-transform-analyze workflow with a single click of a button.

    Authoritative data sources

    The tools in our repository use authoritative sources to arrive at aggregate insights policy makers can use to make real-time, critical decisions. For county-level data about the United States, our data extraction notebook downloads the latest data from the COVID-19 Data Repositoryby the Center for Systems Science and Engineering (CSSE) at Johns Hopkins University. This data set is the primary source for many of the predictive models used by organizations working with the Centers for Disease Control.

    Our notebook fills in known gaps in this primary source with additional data from the New York Times Coronavirus (Covid-19) Data in the United Statesrepository (for more-complete data on Rhode Island and Utah) and New York newspaper THE CITY’s digestof the daily reports from the New York City Department of Health and Mental Hygiene (for borough-level data on New York City). Our notebooks use the European Centre for Disease Prevention and Control’s data on the geographic distribution of COVID-19 cases worldwideas their data source for worldwide data at the granularity of individual countries.

    The notebooks download all of these data sets as they run, for two reasons. First of all, these data sets change on a daily basis. Second, the license terms of the data sets do not allow commercial entities to redistribute the data. If you use our open source code in a commercial application, be sure to verify that you are staying within the bounds of the license terms for the underlying data!

    You can use our code as a jumping off point for more in-depth, interesting analysis. For example, you might want to analyze the data to find correlations between poverty level and infection rates. Open source developers and data scientists can easily build on these tools to extend the analysis to their individual use cases.

    Explanation of the technology used for developers and data scientists

    The notebooks in our repository are Jupyternotebooks. We use common Python data science libraries, including Pandas, Numpy, Matplotlib, seabornand scipy.optimize.

    Most of the cleaning and data analysis is done using Pandas dataframes, a framework that underlies a majority of analysis done by data scientists. IBM is also working to extend Pandas for natural language processing use cases. Parts of our COVID-19 time series analysis notebooks use the TensorArray extension typefrom our Text Extensions for Pandaslibrary to store time series tensors in the cells of Pandas dataframes. Our team also leveraged the graphical workflow editorthat we have built as part of the Elyra project to tie our notebooks into workflows that you can run each day as new data becomes available.

    Leveraging the open source tool and data set

    IBM and our team believe in the importance of democratizing technology, activating developers with the most up-to-date datasets and tools, which can help policy makers make the most informed decisions for citizens’ well-being.

    You can start contributing by using the repoto build analysis in your community, county and country. Developers and data scientists can also contribute directly to the tools that were used to do this analysis by making pull requests to our GitHub repository.

  • Introducing IBM Cloudant on Transaction Engine: Cloud Native by Design

    By Adam Kocoloski, IBM Fellow, VP and CTO, IBM Cloud Data Services

    The IBM public cloud is built on a foundation of innovation in open source, leadership in security and enterprise-grade infrastructure, all with the goal of enabling our clients to rapidly build new applications while reducing the complexity of managing those applications as they scale. Today we’re announcing a substantial new capability on this front with the arrival of Cloudant on Transaction Engine.

    IBM Cloudant is an elastically scalable, serverless datastore built to meet the needs of cloud-native application developers. Since 2014 it has powered a wide variety of internal and external applications in IBM Cloud. Like many original “NoSQL” solutions, Cloudant delivered scalability and availability by sacrificing some of the isolation and consistency guarantees of traditional relational database systems. As we enter Chapter 2 of cloud, we’ve tasked ourselves with restoring those mission-critical properties while preserving and enhancing Cloudant’s cloud-native character. Cloudant on Transaction Engine represents our first major step towards achieving those objectives.

    Scaling to New Heights

    The new Transaction Engine architecture efficiently organizes data for rapid retrieval at scale, and we’re passing those efficiencies on to our clients. Cloudant instances running on the Transaction Engine can support databases up to 100TB, and we’ve reduced the per-gigabyte storage pricing for these new instances by 4x. We’ve worked diligently on an optimized hardware profile to drive down costs and leverage new innovations in storage technology to support this new pricing. Data is still stored in triplicate and replicated across IBM Cloud Availability Zones, and you only pay for the storage that you use.

    Query processing also gets a major boost from the Transaction Engine design. Smarter organization of the data in secondary indexes enables us to efficiently scale query throughput without imposing partitioning restrictions on your application’s data model. Create views and secondary indexes on your entire database and query them with predictable, consistent performance as you scale. At launch we are increasing the cap on provisioned read throughput by 5x — and if you need more throughput. Cloudant on Transaction Engine also eliminates Global Queries as a separate category of provisioned throughput, resulting in cost savings of up to 20xfor certain types of queries.

    Strong Consistency

    API calls issued to Cloudant on Transaction Engine operate against consistent, isolated snapshots of your database. Document operations are fully Linearizable, and secondary indexes are kept transactionally consistent with the main database. The native change capture feed exported for every Cloudant database is also totally-ordered, and the service will never replay previously observed events unless asked to by the client.

    While we’ve become experts at designing applications that make the best use of eventually consistent systems, and happily share that expertise with our clients, a data service that is immune to eventual consistency simply eliminates a whole class of issues that a developer otherwise has to consider. We’re delighted to offer this capability in IBM Cloudant and look forward to expanding the set of mission-critical applications that can leverage IBM Cloud as a result.

    Natively Secure

    Organizations entrusting their data to cloud databases need to know that the data is stored securely and protected from improper use. Cloudant on Transaction Engine implements native database encryption so that user data is represented as ciphertext even before it hits the disk. Each database is encrypted with a unique Data Encryption Key at no additional cost, and the root of trust for those encryption keys always traces back to IBM Cloud’s standard key management services, which leverage some of the most secure commercially-available encryption technology in the industry.

    Open by Design

    Cloudant on Transaction Engine reinforces IBM’s commitment to open innovation: the core database features powering the Transaction Engine instances are being developed as part of the next major release of Apache CouchDB. While we firmly believe that Cloudant provides the best vehicle for clients to benefit from these innovations, we are committed to the principle that you own your data, and we will always support an open API and replication interoperability with the open source community. Moreover, our Extension for Apache CouchDB offering ensures you can get the support you need to deploy and manage CouchDB in your own data center or other hybrid cloud environments. Blending CouchDB and Cloudant gives you a unique ability to bring your data exactly where it’s needed around the globe.

    Clients are recognizing the benefits of the updates being announced today.  “As a long-time IBM Cloudant customer, we are excited to onboard the new Cloudant on Transaction Engine benefits into the PayRange mobile payment architecture,” said Prashant Kanhere, CTO, PayRange. “PayRange provides contact-less payment methods for clients across hundreds of thousands of machines, and this capability is more important than ever these days. We are big users of Cloudant global queries and have high throughput requirements, and always looking for ways to optimize for performance, scalability and cost. Cloudant on Transaction Engine provides capabilities to take our application in the right direction on all those fronts.”

    In addition to announcing new capabilities to IBM’s Cloudant on Transaction Engine, we’re also celebrating the 50 year anniversary of the relational model behind Structured Query Language (SQL). In June 1970, IBM researcher and computing pioneer Edgar Frank Coddpublished the seminal paper “A Relational Model of Data for Large Shared Data Banks,” describing the famous relational model that became the foundation of SQL.

    ##

  • DevOps Deeper Dive: IT Terminology Matters

    DevOps Deeper Dive: IT Terminology Matters

    Much like any other profession, the IT industry does not exist in a vacuum. Social upheaval will have as much impact on IT professionals as it does on any other. As conversations concerning race relations become more heated and, hopefully, elevated, many in the IT industry are reconsidering the use of terminology such as master/slave to describe the relationship between applications and devices.

    GitHub this week announced this week it will be changing the term “master” as it is applied to its repository to a neutral term. Use of the phrase master/slave as a means to describe relationships between software and hardware has been employed since the very early days of IT. In fact, some would note the concept of slavery stretches back to the dawn of time.

    Nevertheless, too many of the terms used in IT evoke a sense of domination that reflects a white, male view of the world, said Dr. Shuli Goodman, executive director of LF Energy, an arm of the Linux Foundation committed to combating climate change by modernizing the power grid using open source software.

    “Language reflects our mental models,” she said.

    The IT industry in general would be better off embracing, for example, biological models rather than terminology that suggests domination and submissiveness, noted Goodman.

    Even IBM, which provides some of the oldest and most venerable platforms in the industry, is open to having this conversation.

    The company is committed to recognizing the experiences of everyone in the IT community, said Sanjay Chandru, director of IBM Z DevOps.

    “If that means we need to have a conversation about terminology, we’re open to that,” he said. “Personally, I think GitHub did the right thing.”

    The time to make a mental shift might never be better, said JFrog CEO Shlomi Ben Haim. The IT industry is transitioning toward a more distributed approach to computing that relies less on centralization. As more workloads shift to the edge, IT environments will need frameworks that span everything from the edge to the cloud, he said.

    As a result, the IT industry as a whole is shifting more toward a peer-to-peer relationship between software and hardware, he added. “We need IT to be more flexible.”

    The history of IT is marked by pendulum swings between centralized and distributed computing. Arguably, the rise of cloud computing represented a shift back toward a more centralized approach to computing that has a lot in common with the way mainframes were deployed and managed more than 50 years ago.

    PCs, on the other hand, were at the forefront of a wave of distributed computing that in many cases simply became too unwieldy to manage. Cloud computing afforded IT organizations an opportunity to centralize the management of workloads. However, with the rise of edge computing, workloads might soon be more distributed than ever.

    As is often the case when it comes to IT trends, it’s likely new terminology will soon emerge to describe the relationship between highly distributed instances of software and hardware. Hopefully, more perspective will be brought to bear not only on what those terms mean from a technical perspective but also how they might be perceived both inside IT and socially.

  • IBM Releases Fully Homomorphic Encryption Toolkit for MacOS and iOS; Linux and Android Coming Soon

    June 4, 2020 | Written by: 

    Often, when I begin explaining fully homomorphic encryption (FHE) to someone for the first time I start by saying that I’ve been working in the field for nearly a decade and yet, I still have to pause to spell it right. So, let’s call it FHE.

    Half-kidding aside, FHE really sounds like magic when you hear about it for the first time, but it’s actually based on very sound mathematics. The main difference is that FHE requires a shift in the programming paradigm that we are used to, which makes it a little more difficult to integrate into applications. That was until today thanks to a new toolkit we are making available for MacOS, iOS and soon for Linux and Android. In fact, developers with basic platform tool familiarity can get up and running by following a few simple instructions rather quickly (see video below). It was no small feat to synthesize 11 years of top-notch cryptography research into a streamlined developer experience that is accessible and freely available to anyone in the time most people would spend to brew a pot of coffee or de-clutter a desk.

    What is FHE?

    The common methods of storing and sharing sensitive data with colleagues and partners have weak links. Today, files are often encrypted in transit and at rest, but decrypted while in use. This provides hackers and insiders with repeated opportunities to exfiltrate unencrypted data. FHE plugs these holes. It allows the manipulation of data by permissioned parties while it’s remains encrypted, minimizing the time it exists in its most vulnerable state.

    In conjunction with other techniques, FHE also makes it possible to selectively restrict decryption capabilities, so people can see only the portions of a file that they are entitled to, and are necessary for them to do their work.

    1970s & Beyond

    FHE was first discussed in the late 1970s, but the real breakthrough came at the 41st ACM Symposium on Theory of Computing  on 31 May 2009 where it was demonstrated for the first time by cryptographer Craig Gentry in his highly cited, seminal paper Fully Homomorphic Encryption Using Ideal Lattices.

    While the paper was exciting news, many in the industry felt that FHE would remain on the cryptographic shelf because it was too slow for everyday usage due to the complexity of the computation and the enormous computing power it required. Thankfully, a small team at IBM Research took that on as a challenge and a decade later FHE performance has improved to a level that is adequate for certain applications, which will only improve with algorithmic advancements and future hardware accelerators.

    Use Cases

    FHE holds significant promise for a number of use cases such as extracting value from private data; data set intersection ; genomics analytics; obvious queries (i.e. querying without revealing intent) and secure outsourcing.

    FHE is particularly suited to industries which are regulated and make use of private, confidential and “crown jewel” data, such as finance and healthcare, since the technology can make it possible to share financial information or patient health records broadly while restricting access to all but the necessary data.

    For example, we recently published a paper with Brazil’s Banco Bradesco SA, where we homomorphically encrypted the data and the model, and showed that it was possible to run predictions with the same accuracy as without encryption and with adequate performance. The result, banks can safely outsource the task of running predictions to an untrusted environment.

    Show me the Tool Kits

    The new FHE toolkits for MacOS, iOS are now available in GitHub with Linux and Android expected in a few weeks. Each toolkit is based on HELib, the world’s most mature and versatile encryption library, includes sample programs making it easier to write FHE based code.

    The iOS toolkit includes an easy to follow and simple demonstration of a privacy preserving search against an encrypted database. The database is a key value store prepopulated with the English names of countries and their capital cities across Europe. Selecting the country will perform a search of the matching capital.

    I should point out that these are not perfect or final. We wanted to quickly put them out to get the technology into the hands of early adopters who want to make these concepts less abstract and more concrete as we look to build up a community of users and use cases.

    For a developer perspective, read this Q&A by my colleague Eli Dow and if you have questions join our Slack community.

    Clients interested in on-going training, knowledge transfer and joint development can sign up for a subscription to the FHE 2020 – Research Engagement Program.

     

  • IBM continues momentum in AI and trust leadership

    The AI Fairness 360 toolkit is becoming even more accessible for a wider range of developers

    By Willie Tejada
    Published June 3, 2020

    The field of artificial intelligence (AI) is making progress as it looks to improve industries and society. But, while the technology continues advancing, the idea of “build for performance” will no longer suffice as an AI design paradigm. We are now in an era where AI must be built, evaluated, and monitored for trust.

    IBM® continues to serve as an industry leader in advancing what we call Trusted AI, focused on developing diverse approaches that implement elements of fairness, explainability, and accountability across the entire lifecycle of an AI application.

    The IBM AI Fairness 360 toolkit

    Under our Trusted AI efforts, IBM released in 2018 the AI Fairness 360 toolkit (AIF360), which is an extensible, open source toolkit that can help you examine, report, and mitigate discrimination and bias in machine learning models throughout the AI application lifecycle. It contains over 70 fairness metrics and 11 state-of-the-art bias mitigation algorithms developed by the research community, and it is designed to translate algorithmic research from the lab into the actual practice of domains as wide-ranging as finance, human capital management, healthcare, and education.

    Adding new functionalities to AI Fairness 360

    Now, IBM is adding two new ways in which AIF360 is becoming even more accessible for a wider range of developers, as well as increased functionality: compatibility with scikit-learn and R.

    R users can now use the AI Fairness 360 toolkit

    AI fairness is an important topic as machine learning models are increasingly used for high-stakes decisions. Machine learning discovers and generalizes patterns in the data and therefore, could replicate systematic advantages of privileged groups. To ensure fairness, we must analyze and address any cognitive bias that might be present in our training data or models.

    We are pleased to announce the release of the AI Fairness 360 R package, an open source library containing techniques to help detect and mitigate bias in data sets and machine learning models throughout the AI application lifecycle. Read “The AIF360 fairness toolkit is now available for R users” for details.

    AI Fairness 360 now has compatibility with scikit-learn

    The scikit-learn data science library is enormously useful for training established machine learning algorithms, computing basic metrics, and building model pipelines. In fact, many example notebooks in AI Fairness 360 already use scikit-learn classifiers with pre-processing or post-processing workflows. However, switching between the AI Fairness 360 toolkit algorithms and scikit-learn algorithms breaks the workflow and forces you to convert data structures back and forth. You’re also unable to use some of the powerful meta-programming tools from scikit-learn like pipelines and cross validation.

    So, the AIF360 team added a new aif360.sklearn module to the latest version release of AIF360, 0.3.0. This module is where you can find all of the currently completed scikit-learn-compatible AIF360 functions. Get all of the information about this update in “The AIF360 team adds compatibility with scikit-learn.”

    Moving trusted AI forward

    Users can use AIF360 to detect fairness issues at training time from Watson Studio and use Watson® OpenScale™ to detect fairness at runtime. Watson OpenScale already supports some of the AIF360 metrics, and longer term we are working on integrating more AIF360 metrics into Watson OpenScale at both design time as well as runtime. AIF360 will also be made available through the IBM Cloud Pak for Data Open Source catalog.

  • IBM Data Asset eXchange launches new data sets and exploratory Watson Studio notebooks

    IBM Data Asset eXchange launches new data sets and exploratory Watson Studio notebooks.

    By Edward Leardi
    Published May 26, 2020

    The IBM® Data Asset eXchange (DAX) is an online hub for developers and data scientists to find free and open data sets under open data licenses. A particular focus of the exchange is data sets under the Community Data License Agreement (CDLA). Since launching the exchange in 2019, the Center for Open-Source Data & AI Technologies (CODAIT) team has been working on steadily adding new data sets to the exchange, as well as resources that help explore these data sets.

    Our latest update to the Data Asset eXchange adds a host of new data-related assets and user experience enhancements. For existing data sets, we’ve added seven new Watson Studio notebooks as well as three Watson Studio projects (a new class of data assets that package multiple notebooks together). Along with these notebooks, we’ve also added eight new data sets to the exchange, featuring domains such as oil extraction, remote sensing, and speech recognition. Finally, we are working on improving the way DAX displays data set previews. We have begun to add data glossaries and detailed metadata sections to provide users with extra context behind a data set’s features and use cases. We have also started working on accommodating text, image, and audio data record preview allowing users to sample data sets without having to download the entire data set archive.

    New Watson Studio projects

    We are now gradually adding Watson Studio projects to data sets, which include sets of notebooks that illustrate how users can extract, clean, analyze, and model the data. To import a project into Watson Studio, visit one of the three data sets discussed below on the IBM® Data Asset eXchange (DAX) and click Run dataset notebooks or preview the code by clicking the link in the Use the Dataset section. To learn more about Watson Studio projects, review this tutorial. A fourth project is already under development and we will be adding more projects where relevant.

    Data Asset eXchange Weather Project

    Our first Watson Studio project release is the DAX Weather Project, which uses the NOAA Weather – JFK Airport data set. This project has three notebooks performing different functions. The first of these is a data cleansing notebook that walks the user through how to impute missing data values and encode certain weather features for better machine learning model performance. The project also includes a data analysis notebook that visualizes the data set’s feature dependencies and trends across time. Finally, the project includes a time series forecasting notebook to build an ARIMA model and evaluate its performance using the RMSE metric.

    Data Asset eXchange Fashion MNIST Project

    The Fashion-MNIST Project builds off its namesake DAX data set by exploring potential uses for an image data set of clothing articles. This project starts with a data exploration notebook that visualizes various clothing article categories and performs dimension reduction. The second notebook uses the scikit-learn library to compare the performance of traditional machine learning methods to classify clothing labels. The third notebook also designs a classifier, but this time using Keras to build a deep learning convolutional neural network.

    Data Asset eXchange Groningen Meaning Bank Project

    Our third Watson Studio project, the Groningen Meaning Bank Project utilizes the DAX GMB data set to explore named-entities within text. The project’s first notebook acquaints the user with the different types of entity and part-of-speech tags found in the data set, as well as visualizes attributes of the corpus such as most common tokens. The second notebook walks the user through how to build a simple named entity recognition model, complete with feature engineering and model result analysis sections.

    New exploratory data analysis notebooks

    Along with the Watson Studio projects, we also released a new batch of exploratory notebooks that accompany our DAX data sets. These notebooks can be accessed by clicking Try the notebook on each of the data set’s DAX landing pages.

    Contracts and Finance Proposition Banks

    The Contracts Proposition Bank and Finance Proposition Bank data sets, which consist of proposition bank-style annotations of legal and finance domain sentences, now have notebooks that load and visualize these annotations. Included, for example, are graphs that visualize feature distributions such as part of speech tags, as well as graphs that visualize the CoNLL node graph format the annotations are stored in.

    IBM Project Debater Wikipedia Oriented Relatedness and Category Stance

    Two more notebooks that were recently released supporting IBM Project Debater® data sets, Wikipedia Oriented Relatedness and Wikipedia Category Stance, explore text data extracted from Wikipedia. Both notebooks walk you through how to load the data into Pandas DataFrames. The Wikipedia Oriented Relatedness notebook visualizes a sample of concept relatedness data (scoring the relatedness between two Wikipedia articles) while the Wikipedia Category Stance notebook visualizes a sample of category stance data (pro or con stance of a Wikipedia article on a certain topic).

    IBM Project Debater Claims Sentence, Mentions Detection, and Sentiment Lexicons

    The final three notebooks added, Claim Sentences Search, Mention Detection Benchmark, and Sentiment Composition Lexicons also all feature data sets originally emanating from IBM Project Debater. The Claim Sentences Search notebook visualizes the data set’s collection of debate topics using topic modeling. The Mention Detection Benchmark notebook tokenizes the included text data and explores the various entity types present in the data. The Sentiment Composition Lexicons notebook counts and visualizes the sentiment of various bigram word pairs included in the data set.

    New data sets

    TF Speech Commands

    The newly added TensorFlow Speech Commands data set contains over 65,000 short audio clips of 30 common spoken English words. The data set is a good fit for training an audio classifier to detect speech commands such “Yes” or “No”. The data set also contains audio files with background noise that can be used to merge with the speech clips to diversify the training data.

    WikiText-103

    The WikiText-103 data set features over 100 million text tokens extracted from the set of verified ‘Good’ and ‘Featured’ articles on Wikipedia. It is available under the CC BY-SA 3.0 license and is a good fit for long-term dependency language modeling.

    Oil Reservoir Simulations

    The Oil Reservoir Simulations data set consists of 60,000 physics-based simulated oil reservoirs generated by IBM researchers. The features and production-rate labels of this data set are sequence-based, making this data set a good fit for testing and validating sequential algorithms. A detailed notebook is included with the data set and provides visual explanations of the simulations run to generate the sequential data. Included in the data set are the input files to the physics-based simulator as well as raw and pre-processed versions of the data, making this data set a good fit for both novice data scientists and advanced researchers alike.

    Wikipedia Entity Graph

    The Wikipedia Entity Graph data set developed by IBM Research consists of a knowledge graph of entities from Wikipedia where each entity is supplemented by a context document that represents all the contexts in which the entity appears on Wikipedia. This data set can be used for problems and techniques that perform joint modeling of graph structure and textual data.

    Mono Lake Surface Water Extent Landsat8 Data

    The Mono Lake Surface Water Extent Landsat8 data set contains Landsat8 satellite imagery data that was post-processed by researchers from IBM Research to measure surface water extent information for the Mono Lake during the time period of 2013-04-18 to 2019-12-31. Surface water extent is important to the studies of land use, water management, and ecosystem health. This data can be used to predict a time-series of lake water extent and condition in order to monitor how a lake is changing over time.

    Taranaki Basin Curated Well Logs

    The Taranaki Basin Curated Well Logs data set consists of a curated set of 407 oil wells located along the western coast of New Zealand. The underlying data was obtained from the New Zealand Petroleum & Minerals Online Exploration Database and processed and cleaned by IBM Research to produce a simple CSV file containing well logs, well coordinates, and well geological features.

    SimpleQuestions and WebQSP Relation Detection

    The SimpleQuestions Relation Detection and WebQSP Relation Detection data sets are sets of entity relation annotations generated by IBM Research from underlying question-and-answering data sets. The relation detection task deals with generating semantic relationships between entities in a text. The SimpleQuestions Relation Detection data set was derived using the SimpleQA data set developed by Facebook Research, while the WebQSP Relation Detection data set was derived from the WebQSP data set created by Microsoft Research.

    If you have any comments or feedback on the Data Asset eXchange, please get in touch with us using GitHub or Slack.

  • Humio accelerates its momentum with extended collaboration with IBM

    Humio  delivers streaming observability at scale to more customers around the globe with integration with IBM Cloud Pak for Multicloud Management

    Humio is pleased to announce that we signed an agreement with IBM to make the Humio log management platform available as part of the IBM Multicloud Management Pak.

    Humio, the only log management platform enabling complete observability for all streaming logs in real time, self-hosted or in the cloud, today announced the signing of an original equipment manufacturing (OEM) agreement with IBM that will allow the companies to collaborate on solutions that can help clients continually ingest data across their infrastructure, and help identify a variety of problems from application service down-time to cyberattack.

    The agreement enables IBM and Humio to create solutions that combine Humio’s log management platform with IBM Cloud Pak for Multicloud Management. Humio minimizes the challenge of sifting through the rapidly, ever-growing volume of event and data logs by enabling users to log all data — structured or unstructured — in real time and at scale. As a result, IBM will offer clients a suite of products to provide complete data observability and management.

    Keeping up with today’s complex environments can overwhelm even well-staffed IT teams, especially since traditional log management tools present significant incremental costs, data storage problems, and search challenges. Customers leveraging Humio’s platform through IBM will have access to its additional support services including but not limited to global IT support, implementation services, predictive analytics, system developers, augmented reality, multi-vendor processing, and a single point of contact interface.

    “Observability is critical for organizations of all sizes, and we are thrilled to join IBM in helping businesses to gain instant visibility of all of their data at scale through our unlimited pricing plan that removes the single biggest barrier to effective log management. The collaboration with IBM further validates Humio’s technology and allows us to quickly expand our global reach to provide even more customers with our modern log management platform.”
    Morten Gram, EVP at Humio

    With this first OEM relationship, Humio can further increase its market share and more quickly deliver its platform to more customers across the globe. IBM plans to go to market with Humio’s platform directly.

  • IBM launches Equal Access Toolkit to help developers build accessible websites and applications

    Increase the accessibility of your application and website from the start with the open source IBM Equal Access Toolkit.

    By Si McAleer, Program Director, IBM Accessibility

    Although nearly 1 in 4 adults in the U.S. face some type of disability, an industry sample has found that in 2020 over 98 percent of home pages had a detectable accessibility error. Accessibility can be forgotten or left until too late in the process when it is difficult to retrofit the site or application. Knowledge, discipline, and tools are all essential to building in accessibility throughout a development process.

    As we near Global Accessibility Awareness Day, I am excited to announce a new open source offering and design toolkit that give designers and developers the tools they need to make their websites and applications accessible.
    The Equal Access Toolkit and Checkerare tools for developers and testers to use to embed accessibility directly into their workflow.

    ·        The IBM Equal Access Toolkitis a public set of guidelines that deliver phase-based guidance about accessibility to all members of a team creating an enterprise offering.
    ·        The Accessibility Checker is part of an open suite of automation tools. It is a browser extension that allows developers or auditors to evaluate a web-based component or solution for accessibility issues.
    Specifically our project offers:
    ·        Clear guidance on what’s needed at each stage of development, from initial research through planning, design, development, verification and launch
    ·        A prioritized breakdown of accessibility requirements at each stage
    ·        Test automation
    ·        Browser-based accessibility checking with powerful navigation and built-in help that supports learning exactly the right information, at exactly the right time

    Ensuring all team members are responsible for accessibility
    All of the tools are built on the same rules engine that is kept up to date to meet the latest accessibility standards and share guidance elements so that all team members can be responsible for accessibility. Each guideline comes in bite-sized snippets to highlight where team members should focus their attention and get the most value at the most efficient pace.
    The toolkit enables product teams to create accessible tools along the way without the need for an accessibility expert or formal education. At the same time, designers and developers can fill in gaps as they go without a loss of continuity.

    See the checker in action
    The tool offers a full accessibility evaluation of the web page which you can use to understand any potential violations by either looking at the page element or the accessibility checklist item.

    Get started
    Access the checker via GitHub or get the browser extension through you’re the Chromeor Firefoxstores.
    If you want to run the checker in a batch configuration, you can get the checker through the NPM repository. Because the toolkit is technology-agnostic, you can use it in a variety of Node.js-based environments for automated testing.

    Conclusion
    IBM’s long and storied history with diversity and accessibility has taught us that incorporating those aspects into our work creates better innovation and outcomes. We understand it is not always easy to find the best practices or guidance to fully understand or test accessibility solutions. This is why we wanted to establish an open source platform that empowers offering teams to conceive, design, develop, ship, and report on their accessible offerings regardless of their accessibility expertise.
    In honor of Global Accessibility Awareness Day, we encourage you to visit https://ibm.com/ableto access the IBM Equal Access Accessibility Toolkit and Checker and make accessibility an inherent part of your development process.

  • Three Promising Call for Code Solutions to Help in the Fight Against COVID-19

    Three Promising Call for Code Solutions to Help in the Fight Against COVID-19

    By Bob Lord, SVP of Cognitive Applications, Blockchain and Ecosystems, IBM
    Imagine you’re driving to the local grocery store to pick up necessities for your family during the COVID-19 pandemic. As soon as you arrive in the parking lot, you notice the line to enter the store is endless. People are huddled closely together waiting to enter, barely adhering to the social distancing guidelines issued by the government. Now, imagine avoiding the line to enter the store, saving your space within a virtual queue, and waiting in the comfort of your own vehicle—all with the click of a button –social distance intact.
    We are facing unprecedented challenges and believe that emerging technology can help solve such problems across the globe. We launched the Call for Code Global Challenge to take on society’s most pressing issues. Since 2018, this movement has grown to over 300,000 developers and problem solvers across 168 countries who have answered the call. While we originally focused this year’s competition on addressing climate change, we quickly recognized the opportunity to dedicate the ingenuity of this developer community to respond to COVID-19 as well. These are two pressing issues that have the power to compromise our health, our planet, and our survival. We recognize the urgency to act, so we also created an accelerated timeline for the Call for Code COVID-19 track. Today, we’re excited to share three outstanding solutions, like Safe Queue which addresses the scenario mentioned above, that have the potential to change the way we react to the pandemic.
    Together with Chelsea Clinton, Vice Chair of the Clinton Foundation, today I announced the top three solutions that have the potential to help mitigate the devastating impact of COVID-19:
    Are you Well?– [video] The “Are You Well?” app, created by a team from Altran in India, is designed to be a comprehensive medical assistance system that could reduce the stress on over-taxed medical systems. A mobile app would help individuals evaluate their symptoms, aided by IBM Watson Assistant. The solution would then leverage a global dashboard that assigns cases a high, medium, or low level of risk based on thresholds set by healthcare providers, and connect them with medical professionals who could use the data to prioritize cases and offer care in a safer manner.
    COVID Impact– [video] COVID Impact was created by a global team that came together at the University of British Columbia, hailing from Canada, Bangladesh, Kazakhstan, and Mexico. The solution aims to reduce the financial impact of COVID-19 on small businesses. A predictive assessment tool, created with IBM Watson Tone Analyzer linguistic analysis and IBM’s ILOG CPLEX rules engine for decision making, could forecast businesses’ financial risk levels. COVID Impact offers a curated list of federal resources for small businesses, and a real-time news feed of sentiment-analysis evaluated articles. The solution also offers an economic impact heatmap that visualizes where need is greatest, which can help officials direct business aid to specific regions or industries.
    Safe Queue – [video] Safe Queue, created by a single developer in Los Angeles, is a community-driven mobile app that is intended to replace physical lines at shopping centers, small businesses, and polling places with on-demand virtual lines, to enable a safer way to manage entry during COVID-19. Safe Queue uses GPS location data to create a virtual queue of those within 1000 feet of a location, allowing employees to control the queue digitally, and validating entry with a randomly generated QR code for each customer. This solution builds on IBM Cloud Foundry for web app hosting, HERE technologies for geolocation, and IBM Cloudant to store data. The developer grabbed the Call for Code Community Cooperation starter kit and ran from there.
    Medical professionals, first responders, and essential workers are rising to the challenges we’re faced with today and are leading the way. IBM and David Clark Cause have led a broad ecosystemincluding the Linux Foundation, UNHR, Verizon, Persistent Systems, Arrow Electronics, Ingram Micro, Cognizant, ICBC, Altran, and celebritiesto do our part. President Bill Clinton announcedour partnership with the Clinton Global Initiative University for the second year, with a dedicated University Edition within Call for Code. Even Lady Gaga recognized the scope and urgency of the issues we’re facing and urged “tech rock stars” to lend their time and talent to the cause. In fact, the developer who created Safe Queue joined Call for Code after seeing her shout-out on a local broadcast.
    The response for the 2020 Call for Code Global Challenge has been tremendous. So, we are honored that some of the most eminent leaders in human rights, disaster risk reduction, business, and technology are serving as judges to decide the winners, including President Clinton, Mark Cuban, Mami Mizutori of United Nations Office for Disaster Risk Reduction, and Laurent Sauveur of United Nations Human Rights.
    We are excited to work quickly in partnership with the Linux Foundation, Anthem, and these teams to get their innovations deployed in communities where they’re needed most to help mitigate the effects of COVID-19. Anthem will work closely as an active partner with Altran’s “Are you Well?” team to explore how their idea could scale on the Anthem digital platform. Through Code and Response, we bring open source solutions to life through real-world deployments to help first responders in their efforts to save lives. In the past 18 months this has included deployments of Project Owl and DroneAidin hurricane-hit Puerto Rico, and Prometeoin areas of Spain prone to wildfires.
    There is still an opportunity to join the 2020 Call for Code Global Challenge. Register and get all the resources you need here, including exclusive access to a new disease tracker APIfrom The Weather Company. We’ll continue to accept submissions for both climate change and COVID-19 through July 31, and plan to announce our global challenge winners in October.
    I can’t wait to see what happens next… and to safely wait for groceries in a virtual queue sometime soon!

  • IBM Looks to Drive Edge Computing

    IBM Looks to Drive Edge Computing

    IBM today at its Think Digital conference launched an edge computing initiative anchored around an IBM Edge Application Manager, through which a single administrator can manage up to 10,000 nodes.

    Hillery Hunter, vice president and CTO for IBM Cloud, said IBM Edge Application Manager is based on the Open Horizon open source software originally developed by IBM that is now under the auspices of LF Edge, an arm of The Linux Foundation focused on networking and edge computing platforms. Open Horizon makes uses of machine learning algorithms to automate the management of fleets of edge computing platforms.

    IBM also launched IBM Telco Network Cloud Manager to orchestrate both virtual and container network functions. IBM Telco Network Cloud Manager is built on top of an instance of Red Hat OpenShift, which is based on Kubernetes, and Red Hat OpenStack Platform.

    Finally, IBM is making available a range of edge-enabled applications, including IBM Visual Insights, IBM Production Optimization, IBM Connected Manufacturing, IBM Asset Optimization, IBM Maximo Worker Insights and IBM Visual Inspector.

    Like most IT vendors IBM sees the rise of application workloads on the edge as an opportunity to drive adoption of hybrid cloud computing. In the case of IBM, all of these offerings run on the Red Hat OpenShift platform.

    IBM also announced the IBM Edge Ecosystem, which brings together independent software vendors (ISVs) and IBM partners to craft edge computing solutions. Additionally, IBM launched the IBM Telco Network Cloud Ecosystem, which is more narrowly focused on edge computing platforms deployed for telco-focused environments such as a wireless 5G network.

    IT vendors participating in those initiatives with IBM include Cisco Systems, Dell Technologies, Juniper, Intel, NVIDIA, Samsung, Packet & Equinix, Hazelcast, Sysdig, Turbonomics, Portworx, Humio, Indra Minsait, EuroTech, Arrow, ADLINK, Acromove, Geniatech, SmartCone, CloudHedge, Altiostar, Metaswitch, F5 Networks and ADVA.

    Edge computing will undoubtedly force organizations to embrace DevOps more aggressively. Managing thousands of edge computing platforms will require not just higher levels of IT automation infused with machine learning algorithms, but IT teams will also need to be able to deploy, secure, update and manage applications distributed across thousands of endpoints.

    IBM, of course, is not the only IT vendor eyeing the edge computing opportunity. The challenge IT organizations will face first is building new classes of latency-sensitive applications running in near real-time that will be built mainly using microservices. Once those applications are built, organizations will then need to update them continuously within the context of a highly scalable IT management framework.

    Naturally, some organizations will prefer to rely on external IT service providers to rise to that challenge while others will continue to build and manage their own applications and infrastructure. Regardless of the approach, the level of scale and complexity at which IT is being applied is about to reach a level that once would have been unimaginable. Not only will more applications run in near real-time, those applications will be integrated and share data with batch-oriented applications running in the cloud and local data centers. The challenge is finding a way to automate the management of IT at scale without requiring organizations to rely on manual processes, for which there are simply not enough IT professionals available to manage.