Single Table Metadata API

This guide will walk you through creating the metadata using the Python API.

Creation API

Get started by creating a blank SingleTableMetadata

from sdv.metadata import SingleTableMetadata

metadata = SingleTableMetadata()

Auto detect metadata

Automatically detect the metadata based on your actual data. Different methods are available based on the format of your data.

detect_from_dataframe: Use this function to automatically detect metadata from your data that is available in a pandas.DataFrame object

metadata.detect_from_dataframe(data)

Parameters

(required) data: A pandas.DataFrame containing your real data

Output (None)

detect_from_csv: Use this function to automatically detect metadata from your data that is available in a CSV file

metadata.detect_from_csv(filepath='data/guests.csv')

Parameters

(required) filepath: The location of the CSV file that contains your data

Output (None)

＊detect_from_ddl: Use this function to automatically detect metadata from your data that is available in a DDL file from an SQL schema

This functionality is currently in Beta testing. Please let us know if you encounter any issues.

metadata.detect_from_ddl(
    filepath='my_schema.ddl',
    db_type='ibm_db2',
    table_name='transaction_log',
    database_name='users'
)

Parameters

(required) filepath: The location of the DDL file that contains the schema
(required) db_type: A string with the type of SQL variant. Currently only 'ibm_db2' is supported
(required) table_name: A string with the name of the table to extract
database_name: A string with the name of the database. In the DDL file, the database name is usually prepended to the table, such as users.transaction_log.
- (default) None: Don't consider any specific database

Output (None)

＊SDV Enterprise Feature. This feature is only available for licensed, enterprise users. To learn more about the SDV Enterprise features and purchasing a license, visit our website.

The detected metadata is not guaranteed to be accurate or complete. Be sure to carefully inspect the metadata and update information.

Primary keys and other identifiers are auto-detected, but may be incorrect or incomplete. See set_primary_key and add_alternate_keys method to add them.
Sensitive information may not be auto-detected. Check for columns with an 'unknown' sdtype and use the update_column method to update them.

Extracting from MultiTableMetadata (get_table_metadata)

If you have metadata available for a multi-table dataset, you extract out metadata for a single table by providing the table name.

Parameters

(required) table_name: A string with the name of the table

Output A SingleTableMetadata object with the metadata only for that table. It will include primary keys and columns but it will not have any relationships.

metadata = multi_table_metadata.get_table_metadata(table_name='hotels')

Inspection API

At any point, you can inspect the current state of the metadata.

to_dict

Use this to get a copy of the Python dictionary that corresponds to the metadata.

Parameters (None)

Output A Python dictionary that corresponds to the metadata

python_dict = metadata.to_dict()

Note that the returned object is a representation of the metadata. Changing it will not modify the original metadata object in any way.

visualize

Use this to this to see a visual representation of the metadata. Use the parameters to control the level of details in the visualization and for saving the image.

Parameters

show_table_details: Toggle the display of column details

(default) 'full'

Show all the different column names, primary keys and foreign keys

'summarized'

Summarize the columns based on the data type

output_filepath: If provided, save the image at the given location in the given format

The output_filepath must end with the filetype that you want to save as. Popular examples are png, jpg or pdf.

Output A graphviz.graphs.Digraph

metadata.visualize(
    show_table_details='summarized',
    output_filepath='my_metadata.png'
)

get_column_names

Use this function to look up column names based on the metadata properties that they have.

This is particularly useful if you want to list all columns assigned to a specific sdtype, such as unknown in order to update it.

Parameters

(required) sdtype: A string describing the statistical data type. Common types are 'boolean', 'categorical', 'datetime', 'numerical' and 'id'. But other types such as 'phone_number' are also available (see SDTypes).
<other properties>: Based on the sdtype, provide other parameters. For more information, see the Metadata Spec.

Output A list of strings, with the column names that match the criteria. If no columns match the criteria, then an empty string will be returned.

metadata.get_column_names(sdtype='unknown')

[ 'product_id', 'product_code_name', 'code_type']

Validation API

validate

Use this to validate that the metadata is written according to the specification. This function will throw descriptive errors if there is anything wrong with the metadata.

Parameters (None)

Output (None)

metadata.validate()

InvalidMetadataError: The metadata is not valid

Error: Invalid values ("pii") for datetime column "start_date".
Error: Invalid regex format string "[A-{6}" for id column "user_id"

validate_data

Use this method to validate that the metadata accurately describes a particular dataset. This function will throw descriptive errors if there is any mismatch between the metadata and data.

Parameters:

(required) data: A pandas.DataFrame containing data. The data should have the same columns as described in the metadata.

Output (None)

metadata.validate_data(data=my_dataset)

Update API

It is important to verify and update any inaccuracies in the metadata.

update_column

Use this method to modify the information about a column in your metadata.

Parameters

(required) column_name: The name of the column to update
(required) sdtype: A string describing the statistical data type. Common types are 'boolean', 'categorical', 'datetime', 'numerical' and 'id'. But other types such as 'phone_number' are also available. For more information, see SDTypes docs.
<other properties>: Based on the sdtype, provide other parameters. For more information, see SDTypes docs.

Output (None)

metadata.update_column(
    column_name='start_date',
    sdtype='datetime',
    datetime_format='%Y-%m-%d')
    
metadata.update_column(
    column_name='user_cell',
    sdtype='phone_number',
    pii=True)

update_columns

Use this function to make a bulk update to multiple columns at once. This function will allow you to set the same parameters for a group of columns.

Parameters

(required) column_names: A list of strings representing the column names to update
(required) sdtype: A string describing the statistical data type. Common types are 'boolean', 'categorical', 'datetime', 'numerical' and 'id'. But other types such as 'phone_number' are also available (see SDTypes).
<other properties>: Based on the sdtype, provide other parameters

Output (None)

metadata.update_columns(
    column_names=['age', 'transactions', 'session_length'],
    sdtype='numerical',
    computer_representation='Float'
)

update_columns_metadata

Use this function to make a bulk update to multiple columns at once. This function will allow you to set the different parameters for each column

Parameters

(required) column_metadata: A dictionary mapping each column name you want to update to the metadata information for that column. For the exact format, see the Metadata Spec.

Output (None)

metadata.update_columns_metadata(
    column_metadata={
        'age': { 'sdtype': 'numerical' },
        'ssn': { 'sdtype': 'ssn', 'pii': True },
        'gender': { 'sdtype': 'categorical' },
        'dob': { 'sdtype': 'datetime', 'datetime_format': '%Y-%m-%d' },
        ...
    }
)

add_column

Use this function to add a column to your SingleTableMetadata object.

Parameters

(required) column_name: The name of the column to update
(required) sdtype: A string describing the statistical data type. Common types are 'boolean', 'categorical', 'datetime', 'numerical' and 'id'. Other types such as 'phone_number' are also available (see SDTypes).
**kwargs: Any other parameters you need that describe metadata for a column.

metadata.add_column(
  column_name='cell_numbers',
  sdtype='phone_number',
  pii=True
)

add_column_relationship

Use this function to specify when a group of columns represents the same concept.

While anyone can add column relationships to their data, SDV Enterprise users will see the highest quality data for the relationships. To learn more about the SDV Enterprise and its extra features, visit our website.

Parameters

(required) relationship_type: A string with the type of relationship. This represents a higher level concept. See the tabs below for options.
(required) column_names: A list of column names that are part of that relationship. Make sure that these columns are compatible with the relationship type. See the tabs below for more information.

An address is defined by 2 or more columns that have the following sdtypes: country_code, administrative_unit, state, state_abbr, city, postcode, street_address and secondary_address.

metadata.add_column_relationship(
    relationship_type='address',
    column_names=['addr_line1', 'addr_line2', 'city', 'zipcode', 'state']
)

A GPS coordinate pair is defined by 2 columns:

sdtype latitude &
sdtype longitude

metadata.add_column_relationship(
    relationship_type='gps',
    column_names=['location_lat', 'location_lon']
)

Output (None)

set_primary_key

Use this function to set the primary key of the table. Any existing primary keys will be removed.

The primary key uniquely identifies every row in the table. When you set a primary key, the SDV will guarantee that every value in the table is unique.

At this time, the SDV does not support composite keys.

Parameters

(required) column_name: The column name of the primary key. The column name must already be defined in the metadata and it must be an ID or another PII sdtype.

Output (None)

metadata.set_primary_key(column_name='guest_email')

remove_primary_key

Use this function to remove any existing primary keys in the table.

Parameters (None)

Output (None)

metadata.remove_primary_key()

add_alternate_keys

Use this function to set alternate keys of the table. This method will add to any existing alternate keys you may have.

Similar to primary keys, alternate keys are also unique in your table. However, other tables do not reference alternate keys.

Parameters

(required) column_names: A list of column names that represent the alternate keys in the table. All column names must already be defined in the metadata and they must be IDs or another PII sdtype.

Output (None)

metadata.add_alternate_keys(column_names=['credit_card_number'])

You can save the metadata object as a JSON file and load it again for future use.

save_to_json

Use this to save the metadata object to a new JSON file that will be compatible with SDV 1.0 and beyond. We recommend you write the metadata to a new file every time you update it.

Parameters

(required) filepath: The location of the file that will be created with the JSON metadata

Output (None)

metadata.save_to_json(filepath='my_metadata_v1.json')

Load previously saved metadata

If you already have a metadata JSON file, you can load it in as a SingleTableMetadata object. Use the method based on the version of your JSON file.

load_from_json: If you recently wrote your JSON file for SDV, use this class method to load it as a SingleTableMetadata object.

from sdv.metadata import SingleTableMetadata

metadata = SingleTableMetadata.load_from_json(
    filepath='my_metadata_v1.json')

Parameters

(required) filepath: The name of the file containing the JSON metadata

Output A SingleTableMetadata object

upgrade_metadata: If you wrote a JSON file for any SDV version before 1.0, use this class method to upgrade the metadata.

You have older metadata if you see a key named "fields" instead of "columns".

from sdv.metadata import SingleTableMetadata

metadata = SingleTableMetadata.upgrade_metadata(
    filepath='my_old_metadata.json'
)

Parameters

(required) old_filepath: The filepath to your older metadata JSON file

Output A SingleTableMetadata object with the new metadata

Tip! After upgrading, save your metadata so you can use it again

metadata.save_to_json('my_new_metadata.json')

You can also load the metadata from a Python dictionary with the information.

load_from_dict

Use this class method to load a Python dictionary as a SingleTableMetadata object.

Parameters

(required) metadata_dict: A Python dictionary representation of the metadata. See Metadata Spec for more details.

Output A SingleTableMetadata object

from sdv.metadata import SingleTableMetadata

metadata_obj = SingleTableMetadata.load_from_dict(metadata_dict)

＊ anonymize

Use this method to anonymize the column names of your metadata. This makes it easier to share your metadata, eg. for debugging purposes.

Parameters (None)

Output A new SingleTableMetadata object that represents the anonymized metadata

anonymized_metadata = original_metadata.anonymize()

＊This feature is only available for licensed, enterprise users. To learn more about the SDV Enterprise features and purchasing a license, get in touch with us.

The anonymized metadata contains new column names. The original names are obfuscated, but the sdtypes and other formatting information remains the same.

>>> anonymized_metadata.to_dict()
{
    'primary_key': 'id_0',
    'columns': {
        'id_0': { 'sdtype': 'id', 'regex_format': 'ID_[0-9]{10}' },
        'num_0': { 'sdtype': 'numerical' },
        'num_1': { 'sdtype': 'numerical' },
        'cat_0': { 'sdtype': 'categorical' },
        'dt_0': { 'sdtype': 'datetime', 'datetime_format': '%Y-%m-%d' },
        'pii_0': { 'sdtype': 'ssn' },
        ...
    }
}

PreviousLoading Data NextModeling

Last updated 4 months ago

Creation API

Auto detect metadata

Extracting from MultiTableMetadata (get_table_metadata)

Inspection API

to_dict

visualize

get_column_names

Validation API

validate

validate_data

Update API

update_column

update_columns

update_columns_metadata

add_column

add_column_relationship

set_primary_key

remove_primary_key

add_alternate_keys

Saving, Loading & Sharing Metadata

save_to_json

Load previously saved metadata

load_from_dict

Parameters

＊ anonymize