Metadata Spec

Metadata is a description of the dataset you want to synthesize. This could be one or multiple data tables. Metadata includes the names of tables, columns, data types in each of the columns, and relationships.

For example, your data may look like this:

An example of multi table data. The tables are connected to each other through primary/foreign keys.

Click below to see the metadata description for this dataset:

Click to see the metadata

{
    "METADATA_SPEC_VERSION": "V1",
    "tables": {
        "hotels": {
            "primary_key": "hotel_id",
            "columns": {
                "hotel_id": { "sdtype": "id", "regex_format": "HID_[0-9]{3}" },
                "city": { "sdtype": "categorical" },
                "rating": { "sdtype": "numerical" }
            },
            "column_relationships": []
        },
        "guests": {
            "primary_key": "guest_email",
            "columns": {
                "guest_email": { "sdtype": "email" },
                "hotel_id": { "sdtype": "id", "regex_format": "HID_[0-9]{3}" },
                "checkin_date": { "sdtype": "datetime", "datetime_format": "%d %b %Y" },
                "checkout_date": { "sdtype": "datetime", "datetime_format": "%d %b %Y" },
                "room_type": { "sdtype": "categorical" }
            },
            "column_relationships": []
        }
    },
    "relationships": [{
        "parent_table_name": "hotels",
        "parent_primary_key": "hotel_id",
        "child_table_name": "guests",
        "child_foreign_key": "hotel_id"
    }]
}

SDV treats metadata as the ground truth

All SDV synthesizers frequently refer to metadata as the ground truth whenever they are creating or evaluating synthetic data. For high quality synthetic data generation, it's vital that your metadata accurately describes the data.

See metadata in action! Your data may have a column containing values such as 94102, 94130 and 94702. What should SDV learn about this data in order to create synthetic data?

If the metadata specifies the data is numerical, SDV learns these values on a sliding number scale. But if the metadata specifies the data actually represents concepts such as postal codes, SDV can better ensure your synthetic data will be valid.

Auto-detect and validate your metadata

SDV allows you to auto-detect metadata based on your data. Please spend some time inspecting and updating your metadata to ensure it accurately describes your data.

from sdv.metadata import Metadata

# 1. auto-detect metadata based in your data
metadata = Metadata.detect_from_dataframes(
    data={
        'hotels': hotels_dataframe,
        'guests': guests_dataframe
    })

# 2. carefully inspect and update your metadata
metadata.visualize()
metadata.update_column(
    column_name='room_type',
    sdtype='categorical',
    table_name='guests'
)

metadata.validate()

# 3. when you're done, save it to a file for future use
metadata.save_to_json('my_final_metadata.json')

Resources

Click the links below to get started with metadata.

Sdtypes

Learn SDV's system for keeping track of data types for each column.

Metadata API

Use Python to auto-detect, inspect, and update metadata.

Metadata JSON

This is the final format that your metadata will be saved as.