Data Lineage

SDV's metadata object includes basic structural elements of a database that are typically found in a schema definition. For example:

• The type of data in each column

• The primary key of each table (or composite keys if you have them)

• Foreign keys that reference primary keys of other tables

However, you may have other rules in your dataset that are the result of data moving and evolving over time (part of data lineage flow). For example, only certain types of primary keys are allowed to be referenced from another table, as illustrated below.

Data lineage rules dictate that only accounts that are type "Premium" are allowed to have perks. That means that the foreign key from Perks can only reference a subset of Accounts.

Add data lineage rules as constraints

Data lineage rules cannot be added to the metadata, but they can be incorporated later into SDV's modeling step. When modeling, you can automatically detect data lineage rules as constraints and add them to your synthesizer. This ensures that the synthetic data you generate always adheres to the data lineage rules that are part of your dataset.

from sdv.multi_table import HSASynthesizer

synthesizer = HSASynthesizer(metadata)

# detect data lineage logic as constraints
detected_constraints = synthesizer.detect_constraints(data)
synthesizer.add_constraints(detected_constraints)

# now the synthetic data includes data lineage rules
synthesizer.fit(data)
synthetic_data = synthesizer.sample()

For more information about constraints, please see the Constraint-Augmented Generation docs.