Skip to content

Data Discovery

Dashboard

This screen is reached via Structured Data > Analytics in the sidebar and provides an overview of the selected data sources. Users can filter data sources and choose a time range of 30, 60, 90 days, or custom.

It displays the following metrics:

  • Total discovery count

  • Average processing time

  • Total schemas

  • Total tables

  • Total columns

  • Sensitive columns

  • Column pending review

Sensitive Data Distribution: This graph visualizes the distribution of sensitive data across the selected data sources. Users can switch between linear and logarithmic scales to better analyze the data.

Classification Status: This pie chart shows the proportion of data classified as Safe, Sensitive, Suspect, or Review, providing a quick overview of the current classification status.

Discoveries

This area allows users to monitor and manage all ongoing data discovery jobs across multiple data sources from a single interface. Track progress, review results, and maintain full control over every discovery process in your environment.

If a metadata scan is unexpectedly interrupted or restarted, it resumes from where it left off thanks to Metadata Checkpoint, instead of repeating already-completed steps.

Available Actions

The following actions can be performed on discovery jobs:

  • Edit: Modify the selected discovery job.

  • Review: Open detailed scan results, including Scan name, Data source, Database type, Start and end date, and Total duration information. Includes Schemas, Tables, Column Filter, and Reports tabs.

  • Run/Pause: Start or pause the discovery process.

  • Others: Includes the View Details, View Logs, and Delete actions.

    • View Details: Displays detailed information about the discovery job, including Basic Information, Source Configuration, Performance and Sampling, and Rule Configuration settings.

    • Delete: Deletes the selected discovery job from the system.

    • View Logs: Displays the audit logs generated for the discovery job. Each log entry contains the date, log details, user information, related table, and associated columns.

Schemas Tab

The Schemas tab lists all schemas included in the discovery. For each schema, it shows the number of tables and how many of them were scanned. Clicking a schema name redirects to the Tables tab for further details.

Tables Tab

The Tables tab provides detailed information about the discovered tables. For each table, it shows the Status, Schema (the schema the table belongs to), Row count, and Classification status information.

Discovered database objects are grouped and displayed separately by entity type — Tables, Views, and Stored Procedures — and views are clearly marked as views so they can be distinguished from tables at a glance.

Clicking a table name opens its columns and column details. These details can be exported to Excel for further analysis.

Note

The Qualified Name structure is standardized across all data assets (Datasource, Catalog, Schema, Table, and Column): the datasource name is included as the starting point of every identifier, so objects with the same name in different datasources are distinguished safely, and object names containing a dot (.) character are handled correctly.

Column Filter Tab

It provides an overview of column-level classification results. It displays a classification status pie chart along with lists of non-reviewed columns, reviewed columns, and the columns identified as safe, sensitive, suspect, review or failed.

Review with AI: Click the Review with AI button, shown on the right of the tab bar while on the Column Filter tab (only visible when the AI feature is enabled), to open a side panel. The panel has a single Include SUSPECT columns option, unchecked by default. Click Start Analysis (disabled until the option above is set) to begin.

Once the analysis completes, results are shown under two tabs:

  • Suggested: Columns for which AI has high confidence in the classification result.

  • Uncertain: Columns that require additional review due to low confidence or ambiguous classification results.

Select one or more columns and click Apply selected (n) to apply only the selected suggestions, or Apply all to apply every suggestion. Re-analyze reruns the AI-based classification process and updates the results accordingly.

Columns that have been reviewed and approved by AI are marked with an AI Review badge, so it is visible in the interface which columns went through AI-assisted review.

System Activity Tab

The System Activity tab provides access to monitoring and audit logs related to the discovery process.

  • Monitoring: Displays operational logs generated during the discovery process. Each log entry contains information such as the date, message, log details, schema, table, column, and associated error codes. To export monitoring logs, click Export Monitoring Log in the upper-right corner of the page.

  • Audit Logs: Displays user activities and system actions related to the discovery process. Each audit log entry contains the date, message, log details, schema, table, column, and user information. To export audit logs, click Export Audit Log in the upper-right corner of the page.

Creating a New Discovery

To create a new discovery job, click the + New Discovery button and follow these steps:

1. Source Configuration

  • Select a Data Source.

  • Enter a Display Name.

2. Rule Configuration

  • From the Laws & Standards menu, select the required standards and move them to the Selected area using the Add Selected Items button.

  • From the Classification menu, select the required classifications and move them to the Selected area using the Add Selected Items button.

3. Filter Configuration

Define filters for Schema, Table, and Column using include and exclude patterns.

Include: Add schemas/tables/columns using regex patterns. Separate multiple patterns with commas.

Exclude: Exclude schemas/tables/columns using regex patterns. Separate multiple patterns with commas.

Examples:

  • .SNOWFLAKE. → contains SNOWFLAKE
  • ^SNOWFLAKE.* → starts with SNOWFLAKE
  • ^SNOWFLAKE$ → exactly SNOWFLAKE

Note

If metadata has already been scraped for the selected data source, the Include/Exclude fields suggest schemas or tables based on that metadata.

4. Performance & Sampling

  • Choose a Sampling Method: Full Scan, Random Sampling (default), Row Limit, or Percentage Sampling.
    • Percentage Sampling can only be selected for PostgreSQL, MSSQL, Oracle, SAP HANA, and Databricks sources; it is disabled for other source types.
    • Selecting Row Limit or Percentage Sampling shows a Row Limit or Sample Percent field, respectively.
    • Selecting Full Scan hides the minimum/maximum sample count fields.
  • Choose whether to enable the Exclude columns marked as false positive from the next scan option.
  • Enter minimum/maximum counts.
  • Set the Threshold (percentage for classification detection, determining whether a classification is suggested as a candidate).
  • Choose whether to enable the Exclude Null Values option.

  • Enter Number of Threads.

  • Choose whether to enable the Debug Log option.

5. Schedule Inactivity Periods

Define periods when the job should not run for performance optimization.

  • Click Add for Blackout Period.
  • Select the day and time range.

After completing all steps, click Save to store the configuration or Deploy and Run to start the discovery immediately.