Cleaning Your Data

If you're importing local data, you may need to clean your data before using it.

Data Cleaning API

drop_unknown_references

Multi-table SDV synthesizers work best when your dataset has referential integrity, meaning that all the references in a foreign key refer to an existing value in the primary key. Use this function to drop rows that contain unknown references for your synthesizer.

Parameters

• (required) metadata: A Metadata object

• (required) data: A dictionary that maps each table name to a pandas DataFrame containing data. This data should match your metadata.

• drop_missing_values: A boolean that describes whether to drop missing values in the foreign key

  • (default) True: If a foreign key has a missing value, treat it as an unknown reference and drop it. We recommend this setting for maximum efficiency with SDV.

  • False: If a foreign key has a missing value, treat it as a valid reference and keep it

• verbose: A boolean that controls whether to print out a summary of the results

  • (default) True: Print a summary of the number of rows that are dropped from each table

Output A dictionary that maps each table name to a pandas DataFrame containing data. The data will contain referential integrity, meaning that there will be no unknown foreign key references.

from sdv.utils import drop_unknown_references

cleaned_data = drop_unknown_references(data, metadata)

Success! All foreign keys have referential integrity. 

Table Name    # Rows (Original)    # Invalid Rows   # Rows (New)
sessions      1200                 50               1150     
transactions  5000                 0                5000

simplify_schema

Use this function to simplify a complex multi-table dataset into only a few small tables. This makes it easier to iterate when performing a proof-of-concept using SDV Community. Keep in mind that SDV Community synthesizers like HMA are not designed to handle large, complex datasets.

After completing your proof-of-concept, purchase SDV Enterprise for access to scalable synthesizers. The synthesizers in SDV Enterprise support unlimited tables with complex schemas, so you will no longer have to simplify the schema.

Parameters

• (required) data: A dictionary that maps each table name to a pandas DataFrame containing data. This data should match your metadata.

• (required) metadata: A Metadata object that describes your data.

Output:

• simplified_data: A dictionary that maps a table name to a pandas DataFrame containing the data. The simplified data schema may have fewer tables and columns than the original.

• simplified_metadata: A Metadata object that describes the simplified data

from sdv.utils import poc

simplified_data, simplified_metadata = poc.simplify_schema(data, metadata)

get_random_subset

Use this function to randomly subsample data from single table data. This function will keep the same overall schema (and columns) but it will reduce the number of rows in each table.

from sdv.utils import poc

subsampled_data = poc.get_random_subset(
    data, 
    metadata,
    main_table_name="users", 
    num_rows=100
)

Parameters

• (required) data: Your full dataset. A dictionary that maps a table name to a pandas DataFrame containing the data

• (required) metadata: A Metadata object that describes the data

• (required) main_table_name: A string with the name of the most important table in your dataset. We'll make sure the subsample is optimized for this main table.

• (required) num_rows: The number of rows to subsample from the main table. All other table's sizes will be algorithmically determined based on this.

• verbose: Whether to print out the results

  • (default) True: Print out how many rows were included from each table

  • False: Do not print anything out

Output A dataset with fewer rows than before. The dataset will continue to have referential integrity meaning that there will be no invalid or missing references between the tables.

get_random_sequence_subset

Use this function to subsample data for a sequential dataset. Given multi-sequence data, this function will randomly select sequences and clip them to the desired length.

from sdv.utils import get_random_sequence_subset

subsampled_data = get_random_sequence_subset(
    data, 
    metadata,
    num_sequences=10,
    max_sequence_length=100,
    long_sequence_subsampling_method='last_rows'
)

Parameters

• (required) data: A pandas.DataFrame containing your multi-sequence data

• (required) metadata: A Metadata object that describes the data. The metadata must describe multi-sequence data, meaning that it must have a sequence key specified.

• (required) num_sequences: An int describing the number of sequences to subsample from the data

• max_sequence_length: The maximum length each sequence is allowed to be

  • (default) None: Do not enforce any max length, meaning that entire sequences will appear in the subsampled data

  • <integer>: An integer describing the max sequence length. Any sequence that is longer than this value will be shortened based on the method below

• long_sequence_subsampling_method: The method for shortening sequences that are too long

  • (default) 'first_rows': Keep the first n rows of each sequence as they appear, where n is the max sequence length

  • 'last_rows': Keep the last n rows of each sequence as they appear, where n is the max sequence length

  • 'random': Randomly choose n rows of each sequence, where n is the max sequence length. Note the randomly chosen rows will still appear in the same order as the original data.

Output A dataset with fewer rows than before. The dataset will continue to represent multiple sequences of potentially varying lengths.