Multi Table Metadata API

This guide will walk you through creating the metadata using the Python API.

Creation API

Get started by creating a blank MultiTableMetadata object.

from sdv.metadata import MultiTableMetadata

metadata = MultiTableMetadata()

Automatically detect the metadata based on your actual data. Different methods are available based on the format of your data.

detect_from_dataframes: Use this function to automatically detect metadata from multiple tables at a time.

metadata.detect_from_dataframes(
    data={
        'hotels': hotels_table,
        'guests': guests_table
    }
)

Parameters

(required) data: A dictionary mapping the name of the table to the pandas.DataFrame that contains its data

Output (None)

Deprecation Notice: In previous versions of the SDV, you could detect one table at a time using detect_table_from_dataframe. We do not recommend using this older function.

detect_from_csvs: Use this function to automatically detect metadata from muliple CSV files

metadata.detect_from_csvs(
    folder_name='my_data_folder/'
)

Parameters

(required) folder_name: The name of the folder that contains all your CSV files. This method assumes each CSV file is a separate table of your metadata.

Output (None)

Deprecation Notice: In previous versions of the SDV, you could detect one table at a time using detect_table_from_csv. We do not recommend using this older function.

＊detect_from_ddl: Use this function to automatically detect metadata from your data that is available in a DDL file from an SQL schema.

This functionality is currently in Beta testing. Please let us know if you encounter any issues.

metadata.detect_from_ddl(
    filepath='my_schema.ddl',
    db_type='ibm_db2',
    database_name='users'
)

Parameters

(required) filepath: The location of the DDL file that contains the schema
(required) db_type: A string with the type of SQL variant. Currently only 'ibm_db2' is supported
database_name: A string with the name of the database. Only the tables from the database will be included in the metadata. In the DDL file, the database name is usually prepended to the table name, such as users.transaction_log.
- (default) None: Detect data from all database names found in the DDL file.

Output (None)

＊SDV Enterprise Feature. This feature is only available for licensed, enterprise users. To learn more about the SDV Enterprise features and purchasing a license, visit our website.

The detected metadata is not guaranteed to be accurate or complete. Be sure to carefully inspect the metadata and update information.

Primary keys and other identifiers are auto-detected, but may be incorrect or incomplete. See set_primary_key and add_alternate_keys method to add them.
Sensitive information may not be auto-detected. Check for columns with an 'unknown' sdtype and use the update_column method to update them.
Simple relationships between tables are auto-detected, but may be incomplete. Use add_relationship to add them.

Inspection API

At any point, during the metadata creation or updates, you can inspect the current state of the metadata.

to_dict

Use this to get a copy of the Python dictionary that corresponds to the metadata.

Parameters (None)

Output A Python dictionary that corresponds to the metadata

python_dict = metadata.to_dict()

Note that the returned object is a representation of the metadata. Changing it will not modify the original metadata object in any way.

visualize

Use this to this to see a visual representation of the metadata. Use the parameters to control the level of details in the visualization and for saving the image.

Parameters

show_table_details: Toggle the display of column details

(default) 'full'

Show all the different column names, primary keys and foreign keys

'summarized'

Summarize the columns based on the data type

None

Hide the details. Only show the table name.

show_relationship_labels: Toggle the display of the table relationships

(default) True

Label each relationship between 2 tables with the column names

False

Do not label the relationships. Only show an arrow between tables.

output_filepath: If provided, save the image at the given location in the given format

The output_filepath must end with the filetype that you want to save as. Popular examples are png, jpg or pdf.

Output A graphviz.graphs.Digraph

metadata.visualize(
    show_table_details='full',
    show_relationship_labels=True,
    output_filepath='my_metadata.png'
)

get_column_names

Use this function to look up column names based on the metadata properties that they have.

This is particularly useful if you want to list all columns assigned to a specific sdtype, such as unknown in order to update it.

Parameters

(required) table_name: A string describing the name of the table to look into
(required) sdtype: A string describing the statistical data type. Common types are 'boolean', 'categorical', 'datetime', 'numerical' and 'id'. But other types such as 'phone_number' are also available (see SDTypes).
<other properties>: Based on the sdtype, provide other parameters. For more information, see the Metadata Spec.

Output A list of strings, with the column names that match the criteria. If no columns match the criteria, then an empty string will be returned.

metadata.get_column_names(table_name='products', sdtype='unknown')

[ 'product_id', 'product_code_name', 'code_type']

Validation API

validate

Use this to validate that the metadata is written according to the specification. This function will throw descriptive errors if there is anything wrong with the metadata.

Parameters (None)

Output (None)

metadata.validate()

InvalidMetadataError: The metadata is not valid

Error: Invalid values ("pii") for datetime column "start_date".
Error: Invalid regex format string "[A-{6}" for id column "hotel_id"

validate_data

Use this method to validate that the metadata accurately describes a particular dataset. This function will throw descriptive errors if there is any mismatch between the metadata and data.

Parameters:

(required) data: A dictionary containing your multi-table data. Each key should be the name of a table and the value should be a pandas.DataFrame containing its data. The data should have the same tables and columns as described in the metadata.

Output (None)

metadata.validate_data(data={
    'hotels': hotels_dataframe,
    'guests': guests_dataframe
})

Update API

It is important to verify and update any inaccuracies in the metadata

update_column

Use this method to modify the information about a column in your metadata

Parameters

(required) table_name: The name of the table to update
(required) column_name: The name of the column to update
(required) sdtype: A string describing the statistical data type. Common types are 'boolean', 'categorical', 'datetime', 'numerical' and 'id'. But other types such as 'phone_number' are also available. For more information, see SDTypes docs.
<other properties>: Based on the sdtype, provide other parameters. For more information, see SDTypes docs.

Output (None)

metadata.update_column(
    table_name='guests',
    column_name='start_date',
    sdtype='datetime',
    datetime_format='%Y-%m-%d')
    
metadata.update_column(
    table_name='guests'
    column_name='user_cell',
    sdtype='phone_number',
    pii=True)

update_columns

Use this function to make a bulk update to multiple columns at once. This function will allow you to set the same parameters for a group of columns.

Parameters

(required) table_name: A string with the name of the table
(required) column_names: A list of strings representing the column names to update. All columns must be in the table.
(required) sdtype: A string describing the statistical data type. Common types are 'boolean', 'categorical', 'datetime', 'numerical' and 'id'. But other types such as 'phone_number' are also available (see SDTypes).
<other properties>: Based on the sdtype, provide other parameters

Output (None)

metadata.update_columns(
    table_name='users',
    column_names=['age', 'transactions', 'session_length'],
    sdtype='numerical',
    computer_representation='Float'
)

update_columns_metadata

Use this function to make a bulk update to multiple columns at once. This function will allow you to set the different parameters for each column

Parameters

(required) table_name: A string with the name of the table
(required) column_metadata: A dictionary mapping each column name you want to update to the metadata information for that column. All columns must be in the table. For the exact format, see the Metadata Spec.

Output (None)

metadata.update_columns_metadata(
    table_name='users',
    column_metadata={
        'age': { 'sdtype': 'numerical' },
        'ssn': { 'sdtype': 'ssn', 'pii': True },
        'gender': { 'sdtype': 'categorical' },
        'dob': { 'sdtype': 'datetime', 'datetime_format': '%Y-%m-%d' },
        ...
    }
)

add_column

Use this function to add a column to your MultiTableMetadata object.

Parameters

(required) table_name: Name of the table that contains the column
(required) column_name : Name of the column to be added
(required) sdtype: A string describing the statistical data type. Common types are 'boolean', 'categorical', 'datetime', 'numerical' and 'id'. Other types such as 'phone_number' are also available (see SDTypes).
**kwargs: Any other parameters you need that describe metadata for a column.

metadata.add_column(
  table_name='users',
  column_name='cell_phone_numbers',
  sdtype='phone_number',
  pii=True
)

add_column_relationship

Use this function to specify when a group of columns within the same table represent the same concept.

While anyone can add column relationships to their data, SDV Enterprise users will see the highest quality data for the relationships. To learn more about the SDV Enterprise and its extra features, visit our website.

Parameters

(required) table_name: The name of the table involved
(required) relationship_type: A string with the type of relationship. This represents a higher level concept. See the tabs below for options.
(required) column_names: A list of column names that are part of that relationship. Make sure that these columns are compatible with the relationship type. See the tabs below for more information.

An address is defined by 2 or more columns that have the following sdtypes: country_code, administrative_unit, state, state_abbr, city, postcode, street_address and secondary_address.

metadata.add_column_relationship(
    table_name='users',
    relationship_type='address',
    column_names=['addr_line1', 'addr_line2', 'city', 'zipcode', 'state']
)

A GPS coordinate pair is defined by 2 columns:

sdtype latitude &
sdtype longitude

metadata.add_column_relationship(
    table_name='users',
    relationship_type='gps',
    column_names=['location_lat', 'location_lon']
)

Output (None)

set_primary_key

Use this function to set the primary key of the table. Any existing primary keys will be removed.

The primary key uniquely identifies every row in the table. When you set a primary key, the SDV will guarantee that every value in the table is unique. At this time, the SDV does not support composite keys.

Parameters

(required) table_name: The name of the table
(required) column_name: The column name of the primary key. The column name must already be defined in the metadata and it must be an ID or another PII sdtype.

Output (None)

metadata.set_primary_key(
    table_name='hotels',
    column_name='hotel_id'
)

metadata.set_primary_key(
    table_name='guests',
    column_name='guest_email'
)

remove_primary_key

Use this function to remove any existing primary keys in a table.

Parameters

(required) table_name: The string name of the table that you'd like to remove the primary key for

Output (None) The primary key will be removed. Any existing relationships that use the primary key will be removed too.

metadata.remove_primary_key(table_name='guests')

add_alternate_keys

Use this function to set alternate keys of the table. This method will add to any existing alternate keys you may have.

Similar to primary keys, alternate keys are also unique in your table. However, other tables do not reference alternate keys.

Parameters

(required) table_name: The name of the table
(required) column_names: A list of column names that represent the alternate keys in the table. All column names must already be defined in the metadata and they must be IDs or other PII sdtypes.

Output (None)

metadata.add_alternate_keys(
    table_name='guests',
    column_names=['credit_card_number']
)

add_relationship

Use this method to add a relationship between 2 connected tables: A parent and child table. The parent table contains the primary key references while the child table has rows that refer to its parent. Multiple child rows can refer to the same parent row.

Parameters:

(required) parent_table_name: The name of the parent table
(required) child_table_name: The name of the child table that refers to the parent
(required) parent_primary_key: The primary key column in the parent table. This column uniquely identifies each row in the parent table .
(required) child_foreign_key: The foreign key column in the child table. The values in this column contain a reference to a row in the parent table

Output (None)

metadata.add_relationship(
    parent_table_name='hotels',
    child_table_name='guests',
    parent_primary_key='hotel_id',
    child_foreign_key='hotel_id'
)

remove_relationship

Use this method to remove the connection between a parent and child table. In the case where there are multiple connections, this method will remove all the connections. Use this if the metadata has incorrectly detected relationships.

Parameters:

(required) parent_table_name: The name of the parent table
(required) child_table_name: The name of the child table that refers to the parent

Output (None)

metadata.remove_relationship(
    parent_table_name='hotels',
    child_table_name='guests'
)

You can save the metadata object as a JSON file and load it again for future use.

save_to_json

Use this to save the metadata object to a new JSON file that will be compatible with SDV 1.0 and beyond. We recommend you write the metadata to a new file every time you update it.

Parameters

(required) filepath: The location of the file that will be created with the JSON metadata

Output (None)

metadata.save_to_json(filepath='metadata.json')

Load previously saved metadata

If you already have a metadata JSON file, you can load it in as a MultiTableMetadata object. Use the method based on the version of your JSON file.

load_from_json: If you recently wrote your JSON file for SDV, use this class method to load it as a MultiTableMetadata object.

from sdv.metadata import MultiTableMetadata

metadata = MultiTableMetadata.load_from_json(
    filepath='metadata.json')

Parameters

(required) filepath: The name of the file containing the JSON metadata

Output A MultiTableMetadata object

upgrade_metadata: If you wrote a JSON file for any SDV version before 1.0, use this class method to upgrade the metadata.

You have older metadata if you see a key named "fields" instead of "columns".

from sdv.metadata import MultiTableMetadata

metadata = MultiTableMetadata.upgrade_metadata(
    filepath='my_old_metadata.json'
)

Parameters

(required) old_filepath: The filepath to your older metadata JSON file

Output A MultiTableMetadata object with the new metadata

Tip! After upgrading, save your metadata so you can use it again

metadata.save_to_json('my_new_metadata.json')

load_from_dict

Use this class method to load a Python dictionary as a MultiTableMetadata object.

Parameters

(required) metadata_dict: A Python dictionary representation of the metadata. See Metadata Spec for more details.

Output A MultiTableMetadata object

from sdv.metadata import MultiTableMetadata

metadata_obj = MultiTableMetadata.load_from_dict(metadata_dict)

＊ anonymize

Use this method to anonymize the column names of your metadata. This makes it easier to share your metadata, eg. for debugging purposes.

Parameters (None)

Output A new MultiTableMetadata object that represents the anonymized metadata

anonymized_metadata = original_metadata.anonymize()

＊This feature is only available for licensed, enterprise users. To learn more about the SDV Enterprise features and purchasing a license, get in touch with us.

The anonymized metadata contains new table and column names. The original names are obfuscated, but the sdtypes and other formatting information remains the same.

>>> anonymized_metadata.to_dict()
{
    'tables': {
        '3oc2d': {
            'primary_key': 'id_0',
            'columns': {
                'id_0': { 'sdtype': 'id', 'regex_format': 'ID_[0-9]{10}' },
                'num_0': { 'sdtype': 'numerical' },
                'cat_0': { 'sdtype': 'categorical' },
                'dt_0': { 'sdtype': 'datetime', 'datetime_format': '%Y-%m-%d' },
                'pii_0': { 'sdtype': 'ssn' },
                ...
        },
        'dpco1': {
            ...
        }
    },
    'relationships': [{
        'parent_table_name': '3oc2d',
        'child_table_name': 'dpco1',
        'parent_primary_key': 'id_0',
        'child_foreign_key': 'id_2'
    }]   
}

PreviousCleaning Your Data NextModeling

Last updated 27 days ago

Creation API

Inspection API

to_dict

visualize

get_column_names

Validation API

validate

validate_data

Update API

update_column

update_columns

update_columns_metadata

add_column

add_column_relationship

set_primary_key

remove_primary_key

add_alternate_keys

add_relationship

remove_relationship

Saving, Loading & Sharing Metadata

save_to_json

Load previously saved metadata

load_from_dict

Parameters

＊ anonymize