> For the complete documentation index, see [llms.txt](https://docs.sdv.dev/sdv/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.sdv.dev/sdv/integration/utils/cleaning.md). # Cleaning Your Data If you're importing local data, you may need to clean your data before using it. ## Data Cleaning API ### drop\_unknown\_references Multi-table SDV synthesizers work best when your dataset has *referential integrity*, meaning that all the references in a foreign key refer to an existing value in the primary key. Use this function to drop rows that contain unknown references for your synthesizer. **Parameters** * (required) `metadata`: A [Metadata](/sdv/integration/metadata.md) object * (required) `data`: A dictionary that maps each table name to a pandas DataFrame containing data. This data should match your metadata. * `drop_missing_values`: A boolean that describes whether to drop missing values in the foreign key * (default) `True`: If a foreign key has a missing value, treat it as an unknown reference and drop it. *We recommend this setting for maximum efficiency with SDV.* * `False`: If a foreign key has a missing value, treat it as a valid reference and keep it * `verbose`: A boolean that controls whether to print out a summary of the results * (default) `True`: Print a summary of the number of rows that are dropped from each table **Output** A dictionary that maps each table name to a pandas DataFrame containing data. The data will contain referential integrity, meaning that there will be no unknown foreign key references. ```python from sdv.utils import drop_unknown_references cleaned_data = drop_unknown_references(data, metadata) ``` ``` Success! All foreign keys have referential integrity. Table Name # Rows (Original) # Invalid Rows # Rows (New) sessions 1200 50 1150 transactions 5000 0 5000 ``` ### simplify\_schema Use this function to simplify a complex multi-table dataset into only a few small tables. This makes it easier to iterate when performing a proof-of-concept using SDV Community. Keep in mind that SDV Community synthesizers like HMA are not designed to handle large, complex datasets. {% hint style="success" %} After completing your proof-of-concept, purchase [SDV Enterprise](https://datacebo.com/pricing/) for access to scalable synthesizers. The synthesizers in SDV Enterprise support unlimited tables with complex schemas, so you will no longer have to simplify the schema. {% endhint %} **Parameters** * (required) `data`: A dictionary that maps each table name to a pandas DataFrame containing data. This data should match your metadata. * (required) `metadata`: A [Metadata](/sdv/integration/metadata.md) object that describes your data. **Output**: * `simplified_data`: A dictionary that maps a table name to a pandas DataFrame containing the data. The simplified data schema may have fewer tables and columns than the original. * `simplified_metadata`: A Metadata object that describes the simplified data ```python from sdv.utils import poc simplified_data, simplified_metadata = poc.simplify_schema(data, metadata) ``` ### get\_random\_subset Use this function to randomly subsample data from single table data. This function will keep the same overall schema (and columns) but it will reduce the number of rows in each table. ```python from sdv.utils import poc subsampled_data = poc.get_random_subset( data, metadata, main_table_name="users", num_rows=100 ) ``` **Parameters** * (required) `data`: Your full dataset. A dictionary that maps a table name to a pandas DataFrame containing the data * (required) `metadata`: A Metadata object that describes the data * (required) `main_table_name`: A string with the name of the most important table in your dataset. We'll make sure the subsample is optimized for this main table. * (required) `num_rows`: The number of rows to subsample from the main table. All other table's sizes will be algorithmically determined based on this. * `verbose`: Whether to print out the results * (default) `True`: Print out how many rows were included from each table * `False`: Do not print anything out **Output** A dataset with fewer rows than before. The dataset will continue to have *referential integrity* meaning that there will be no invalid or missing references between the tables. ### get\_random\_sequence\_subset Use this function to subsample data for a sequential dataset. Given multi-sequence data, this function will randomly select sequences and clip them to the desired length. ```python from sdv.utils import get_random_sequence_subset subsampled_data = get_random_sequence_subset( data, metadata, num_sequences=10, max_sequence_length=100, long_sequence_subsampling_method='last_rows' ) ``` **Parameters** * (required) `data`: A pandas.DataFrame containing your multi-sequence data * (required) `metadata`: A Metadata object that describes the data. The metadata must describe multi-sequence data, meaning that it must have a sequence key specified. * (required) `num_sequences`: An int describing the number of sequences to subsample from the data * `max_sequence_length`: The maximum length each sequence is allowed to be * (default) `None`: Do not enforce any max length, meaning that entire sequences will appear in the subsampled data * ``: An integer describing the max sequence length. Any sequence that is longer than this value will be shortened based on the method below * `long_sequence_subsampling_method`: The method for shortening sequences that are too long * (default) `'first_rows'`: Keep the first *n* rows of each sequence as they appear, where *n* is the max sequence length * `'last_rows'`: Keep the last *n* rows of each sequence as they appear, where *n* is the max sequence length * `'random'`: Randomly choose *n* rows of each sequence, where *n* is the max sequence length. Note the randomly chosen rows will still appear in the same order as the original data. **Output** A dataset with fewer rows than before. The dataset will continue to represent multiple sequences of potentially varying lengths. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.sdv.dev/sdv/integration/utils/cleaning.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.