Skip to content

Automatic translation — the Portuguese version is the reference. View in Portuguese

Updated on 2026-09-08

Tutorial — connect the customer's BigQuery as a source

Step-by-step guide to connecting your BigQuery as an extraction source, with a data project and a billing project, a service account, and a connection test.

On this page (13)

The BigQuery connector can be registered and used today, with no case-by-case release. Extraction depends on your account's service account and on the workspace cloud environment being configured — without it the run fails with a message. Synthetic examples.

Read first: How data flows · Connector catalog.


What it is#

Connecting the BigQuery in YOUR project as an extraction source (pull mode). Here Ingestia reads tables from your BigQuery using your service account and brings them into the Bronze layer of your workspace — just like pulling from a relational database.

Do not confuse this with Ingestia's internal BigQuery (the destination datalake, which runs with our service account). This tutorial is about using your BigQuery as a source.

What it is for#

Reusing data that already lives in your BigQuery data warehouse (e.g., system exports, marketing tables) inside the Ingestia datalake.

When to use / when not to use#

  • Use it when: your data is already in BigQuery tables and you can create a read-only service account.
  • Do not use it when: the data is in a relational database (use the database connector) or in files (use import a file).

Plan and permissions#

  • Product core · access check · ingestion run kill switch.
  • Minimum roles for the customer's service account:
    • roles/bigquery.dataViewer on the data project;
    • roles/bigquery.jobUser on the billing project (or on the data project, if it is the only one).

Prerequisites#

  1. Data project (Project ID) — where the datasets/tables live.
  2. Billing project (optional) — where query jobs run and are billed; empty = the data project itself pays.
  3. Service account JSON with the roles above.
  4. Optional: a specific dataset and location (e.g., US, southamerica-east1).

Step by step#

  1. In the console, go to ETL/ELT → Connections and choose BigQuery (category Databases).
  2. Data project (Project ID) — e.g., meu-projeto-example.
  3. Query billing:
    • In the same project as the data (default), or
    • In another project → enter the Billing project (e.g., meu-projeto-billing).
  4. Dataset — e.g., vendas (optional; empty lists all accessible ones).
  5. Location — e.g., southamerica-east1 (optional; empty = automatic).
  6. Service account key (JSON) — paste the customer's service account JSON (it is encrypted and never shown again).
  7. Click Test connection and confirm that the datasets/tables appear.
  8. Select the tables and save the source.

Example (synthetic)#

code
Projeto de dados: meu-projeto-example
Billing: Em outro projeto
Projeto de billing: meu-projeto-billing
Dataset: vendas
Location: southamerica-east1
Service account: { "type": "service_account", "project_id": "meu-projeto-example",... }

Grant access (example with gcloud, fictitious values):

bash
# Read access to the data
gcloud projects add-iam-policy-binding meu-projeto-example \
  --member="serviceAccount:ingestia-ro@meu-projeto-example.iam.gserviceaccount.com" \
  --role="roles/bigquery.dataViewer"

# Run query jobs (on the billing project)
gcloud projects add-iam-policy-binding meu-projeto-billing \
  --member="serviceAccount:ingestia-ro@meu-projeto-example.iam.gserviceaccount.com" \
  --role="roles/bigquery.jobUser"

Expected result#

  • Test connection succeeds and discovery lists the accessible datasets/tables.
  • During the load, Ingestia creates the query job and runs a 1-row probe (discarded) to validate the SQL/partition before sending any chunk — so a retry never duplicates data in the landing zone.
  • Tables with require_partition_filter are handled automatically: Ingestia builds a predicate that covers every row instead of failing.

Limits and costs#

  • Chunked extraction: batchSize default 5,000 (100–50,000); maxRows default 200,000 (up to 5,000,000).
  • Queries against your BigQuery are billed to your billing project (BigQuery's on-demand model). Ingestia shows an estimate; the amount charged is the one on your project's invoice.
  • Loading into Ingestia's Bronze is a billable load operation on Ingestia's side.

Security#

  • Identifiers (project/dataset/table/watermark column) are sanitized before entering the SQL; the incremental filter uses a named parameter (@since), never concatenation.
  • The service account JSON key is encrypted and redacted in any error/log.
  • Workspace scope: the data goes to the Bronze of your workspace.

Common errors#

SymptomLikely causeWhat to do
Access Denied:... dataViewerSA has no read access to the dataGrant roles/bigquery.dataViewer on the data project
User does not have bigquery.jobs.createSA lacks jobUser on billingGrant roles/bigquery.jobUser on the billing project
Cannot query... requires a filtertable with a required partition filterIngestia handles it on its own; if it persists, provide the partition column
Not found: Datasetwrong dataset/locationCheck the dataset name and the location
Invalid JSONkey pasted incompletelyPaste the entire service account JSON

Troubleshooting#

  • Use Test connection to tell permission issues (dataViewer/jobUser) apart from configuration issues (dataset/location).
  • Confirm that the service account has access to the correct billing project when billing and data are different projects.

Related#


Last reviewed: 2026-09-08.

Related links