Cataloging Unstructured Data in IBM Watson Knowledge Catalog with IBM Spectrum Discover

This IBM® Redpaper publication explains how IBM Spectrum® Discover integrates with the IBM Watson® Knowledge Catalog (WKC) component of IBM Cloud® Pak for Data (IBM CP4D) to make the enriched catalog content in IBM Spectrum Discover along with the associated data available in WKC and IBM CP4D. From an end-to-end IBM solution point of view, IBM CP4D and WKC provide state-of-the-art data governance, collaboration, and artificial intelligence (AI) and analytics tools, and IBM Spectrum Discover complements these features by adding support for unstructured data on large-scale file and object storage systems on premises and in the cloud.

Many organizations face challenges to manage unstructured data. Some challenges that companies face include:

• Pinpointing and activating relevant data for large-scale analytics, machine learning (ML) and deep learning (DL) workloads.
  
• Lacking the fine-grained visibility that is needed to map data to business priorities.
  
• Removing redundant, obsolete, and trivial (ROT) data and identifying data that can be moved to a lower-cost storage tier.
  
• Identifying and classifying sensitive data as it relates to various compliance mandates, such as the General Data Privacy Regulation (GDPR), Payment Card Industry Data Security Standards (PCI-DSS), and the Health Information Portability and Accountability Act (HIPAA).
  

• Event-based cataloging and tagging of unstructured data across the enterprise.
  
• Automatically inspecting and classifying over 1000 unstructured data types, including genomics and imaging specific file formats.
  
• Automatically registering assets with WKC based on IBM Spectrum Discover search and filter criteria, and by using assets in IBM CP4D.
  
• Enforcing data governance policies in WKC in IBM CP4D based on insights from IBM Spectrum Discover, and using assets in IBM CP4D.
  

IBM Spectrum Discover integration with WKC enables storage administrators, data stewards, and data scientists to efficiently manage, classify, and gain insights from massive amounts of data. The integration improves storage economics, helps mitigate risk, and accelerates large-scale analytics to create competitive advantage and speed critical research.