.png%3Falt%3Dmedia%26token%3Db20101b9-6791-419f-8b4f-04e111b1323b){kind=logo}
.png%3Falt%3Dmedia%26token%3Db20101b9-6791-419f-8b4f-04e111b1323b){kind=logo}
Sdtypes
All SDV models require information about the data type for every column. In the SDV, the data types are specified by sdtype, denoting a semantic or statistical meaning.
An sdtype is a high level concept that does not depend on how a computer stores the data. A single sdtype (such as
"categorical") can be stored by a computer in several ways (string, integer, etc).There are 5 common sdtypes that describe columns in a dataset.
Sdtype
boolean describes columns that contain TRUE or FALSE values and may contain some missing data.Sdtype
categorical describes columns that contain distinct categories. The defining aspect of a categorical column is that only the values that appear in the real data are valid.The categories may be ordered or unordered.
An example of categorical data is tax payer status such as
Single, Married filing jointly, Widowed, etc. Only these distinct categories are allowed. If you want the synthetic data to include new values that were not in the original data, then the column is not categorical. For example, if you have address data and would like the synthetic data to create new, unseen addresses, see other sdtypes below.
Sdtype
datetime describes columns that indicate a point of time. This can be at any granularity: to the nearest day, minute, second or even nanosecond. Typically, the datetime will be represented as a string.Sdtype
numerical describes data with numbers. The defining aspect of numerical data is that there is an order and you can apply a variety of mathematical computations to the values (average, sum, etc.) The actual values may follow a specific format, such as being rounded to 2 decimal digits and remaining between min/max bounds.Some data may appear numerical but actually represents distinct categories. For example, HTTP response codes such as
200, 404, etc. are categorical data. These numbers don't have a specific order, and they cannot be combined or averaged.Sdtype
id describes columns that are used to identify rows (eg. as a primary or foreign key). ID columns do not have any other mathematical or special meanings. Typically, an ID column follows a particular structure, for example being exactly 8 digits long with a - in the middle. You may find that some of the columns in your dataset represent high-level, concepts in your domain. Such data might also contain sensitive, Personal Identifiable Information (PII).
For these types of concepts, the synthetic data can contain entirely new values that don't appear in the original data. In some cases, the SDV can also extract deeper meaning from the concepts to undersatnd the context.
Browse below for some common sdtypes related to different concepts.
Personal Info
Location
Networking
Banking
Automotive
Other
These sdtypes describe the information about a person.
＊ phone_number | A local or international phone number such as '+1(555)123-4567'. Different countries have different formats. |
＊ email | A person's email such as '[email protected]' |
ssn | A social security number such as 000-00-0000 |
first_name | A person's first name |
last_name | A person's last name |
＊ Licensed, enterprise users will see higher quality data. The SDV will extract the deeper meaning and replicate it in the synthetic data. To learn more, get in touch with us.
These sdtypes describe a location around the world.
＊ country_code | A 2-character country code such as 'US' |
＊ administrative_unit | The name of a region inside the country such as 'Massachusetts'. Countries call this concept different names such as state or province. |
＊ state_abbr | For countries that call their regions states, this refers to the 2-character code such as 'MA' |
＊ city | The full name of the city such as 'Boston' |
＊ postcode | The internationally-recognized, 5-digit postcode such as 02116 |
＊ street_address | The street and building number such as '123 Main St'. The exact format of this may vary by country |
＊ secondary_address | Additional information about units in the building, such as 'Apartment #3'. |
latitude | The latitude of a location, expressed as a decimal |
longitude | The longitude of a location, expressed as a decimal |
＊ Licensed, enterprise users will see higher quality data. The SDV will extract the deeper meaning and replicate it in the synthetic data. To learn more, get in touch with us.
These sdtypes describe information about computer networks and the internet.
ipv4_address | An IP address, using the v4 protocol |
ipv6_address | An IP address, using the v6 protool |
mac_address | A media access control address |
user_agent_string | A user agent string sent by the HTTP protocol |
These sdtypes describe information needed for banking functions such as payment transfers.
iban | An international bank account number |
swift11 | A SWIFT bank code that uses 11 digits |
swift8 | A SWIFT bank code that uses 8 digits |
credit_card_number | A credit card number, expressed using digits |
These sdtypes describe concepts from the automotive industry.
vin | A vehicle identification number |
license_plate | A license plate number, expressed using digits, letters or other characters. The format varies by country. |
Many other sdtypes are possible. The SDV models can use the Python Faker library for new data types. You can input any of the function names as sdtypes. For example, inputting the sdtype
passport_number will use this function to generate meaningful numbers. Many concepts vary based on the country and language. For example, phone numbers are represented differently in different countries.
The SDV aims to provide worldwide support. You can specify the locales when you create a synthesizer. This lets the SDV know that any higher-level concepts should conform to the right formatting rules for that country.
For example, assume you provide the following info:
from sdv.single_table import GaussianCopulaSynthesizer
synthesizer = GaussianCopulaSynthesizer(
metadata, locales=['en_US', 'nl_BE'])
Then for every concept described in your metadata, the SDV will generate values only from the US or Belgium in the appropriate language (English or Dutch).
The public SDV randomly creates values corresponding to the concept, without taking additional context into account. Sometimes this may not be enough. For example, you may want to extract geographical areas from
phone_number to ensure that it follows the same patterns.Last modified 11d ago