❖ BigQuery

This page describes how to integrate with your BigQuery database.

❖ SDV Enterprise Bundle. This feature is available as part of the AI Connectors Bundle, an optional add-on to SDV Enterprise. For more information, please visit the AI Connectors Bundle page.

Installation

To use this feature, please make sure you have installed the bundle with the optional db-bigquery dependency. For more information, see the SDV Enterprise Installation docs.

pip install sdv-installer --upgrade
sdv-installer install --package bundle-ai-connectors --options db-bigquery

Connector

Create a connector object that interacts with your BigQuery database.

from sdv.io.database import BigQueryConnector

connector = BigQueryConnector()

Parameters (None)

Output A BigQueryConnector object that you can use to import data and metadata

Config File

Create a YAML file that corresponds to the database to connect to. You will need separate config files for your source dataset (for importing training data) and a destination dataset (for exporting synthetic data).

The config file has the following structure:

connection:
  database_name: <string> 
  schema_name: <string>

auth:
  json_credentials_path: <filepath>

Parameters:

connection: This section defines the database tables you want to connect to. It is common to all Google-native databases.

• (required) database_name: A top-level logical container for data and database objects within a database system.

• (required) schema_name: A named logical grouping of database objects within a database, used for organization and access control.

auth: This section provides information about your identity, which is required to connect to the database. It is common to all Google-native databases.

• (required) json_credentials_filepath: A string with the file that contains the credentials. This can be downloaded for user or service accounts via Google's interface. Follow the instructions in these docs to select your account, add a key, and download the JSON file. (If you need help with this step, please contact your database admin.)

Note that the terminology of BigQuery is slightly different from other databases. In the connection parameters: Set the database_name to null, and the schema_name to your BigQuery's dataset_id.

Setting Your Config File

Use the set_import_config and set_export_config functions to add your config YAML file to your connector.

connector.set_import_config(filepath='source_db_config.yaml')
connector.set_export_config(filepath='destination_db_config.yaml')

Use the connector to create metadata, import a subset of training data, and export synthetic data. For more information, see the Usage API.

FAQ

Which permissions are needed for importing and exporting?

Importing data requires read access. For BigQuery, this includes: bigquery.jobs.create, bigquery.tables.get, bigquery.tables.getData, and bigquery.tables.list, bigquery.datasets.get. If you do not have these permissions, please contact your database admin.

Exporting data requires write access. For BigQuery, this includes: bigquery.datasets.create, bigquery.datasets.get, bigquery.jobs.create, bigquery.tables.create, and bigquery.tables.export. If you do not have these permissions, please contact your database admin.

Is there a way to speed up import and export?

Installing google-cloud-bigquery-storage in your environment enables the BigQuery Storage API for faster data import/export. This dependency is not currently required so be sure to install it yourself using your preferred package manager.

pip install google-cloud-bigquery-storage

To see results, make sure you have the bigquery.readsessions.create permission.