Metadata

Metadata is a description of the dataset you want to synthesize. Metadata is required for any SDV synthesizer.

Your data can be present in one or multiple data tables. Metadata includes the names of tables, columns, data types in each of the columns, and relationships. For example, your data may look like this:

An example of multi table data. The tables are connected to each other through primary/foreign keys.

Click below to see the metadata description for this dataset:

Click to see the metadata

{
    "METADATA_SPEC_VERSION": "V1",
    "tables": {
        "hotels": {
            "primary_key": "hotel_id",
            "columns": {
                "hotel_id": { "sdtype": "id", "regex_format": "HID_[0-9]{3}" },
                "city": { "sdtype": "categorical" },
                "rating": { "sdtype": "numerical" }
            },
            "column_relationships": []
        },
        "guests": {
            "primary_key": "guest_email",
            "columns": {
                "guest_email": { "sdtype": "email" },
                "hotel_id": { "sdtype": "id", "regex_format": "HID_[0-9]{3}" },
                "checkin_date": { "sdtype": "datetime", "datetime_format": "%d %b %Y" },
                "checkout_date": { "sdtype": "datetime", "datetime_format": "%d %b %Y" },
                "room_type": { "sdtype": "categorical" }
            },
            "column_relationships": []
        }
    },
    "relationships": [{
        "parent_table_name": "hotels",
        "parent_primary_key": "hotel_id",
        "child_table_name": "guests",
        "child_foreign_key": "hotel_id"
    }]
}

SDV treats metadata as the ground truth

All SDV synthesizers frequently refer to metadata as the ground truth whenever they are creating or evaluating synthetic data. For high quality synthetic data generation, it's vital that your metadata accurately describes the data.

See metadata in action! Your data may have a column containing values such as 94102, 94130 and 94702. What should SDV learn about this data in order to create synthetic data?

If the metadata specifies the data is numerical, SDV learns these values on a sliding number scale. But if the metadata specifies the data actually represents concepts such as postal codes, SDV can better ensure your synthetic data will be valid.

Create and validate your metadata

There are many ways to create metadata. If you're directly connecting your database, SDV creates metadata based on your database schema. Otherwise, you can auto-detect the metadata based on any local data that you've loaded. Yan always view and modify your metadata as needed.

Please spend some time inspecting and updating your metadata to ensure it accurately describes your data.

from sdv.metadata import Metadata

# Create metadata via auto-detection (if you don't already have it)
metadata = Metadata.detect_from_dataframes(data)

# Carefully inspect and update your metadata
metadata.visualize()
metadata.update_column(
    column_name='room_type',
    sdtype='categorical',
    table_name='guests'
)

metadata.validate()

# When you're done, save it to a file for future use
metadata.save_to_json('my_final_metadata.json')

Resources

Click the links below to learn more about metadata.

Sdtypes

Learn SDV's system for keeping track of data types for each column.

Metadata API

Use Python to auto-detect, inspect, and update metadata.

Metadata JSON

This is the final format that your metadata will be saved as.