The BigQuery connector can be registered and used today, with no case-by-case release. Extraction depends on your account's service account and on the workspace cloud environment being configured — without it the run fails with a message. Synthetic examples.
Read first: How data flows · Connector catalog.
What it is#
Connecting the BigQuery in YOUR project as an extraction source (pull mode). Here Ingestia reads tables from your BigQuery using your service account and brings them into the Bronze layer of your workspace — just like pulling from a relational database.
Do not confuse this with Ingestia's internal BigQuery (the destination datalake, which runs with our service account). This tutorial is about using your BigQuery as a source.
What it is for#
Reusing data that already lives in your BigQuery data warehouse (e.g., system exports, marketing tables) inside the Ingestia datalake.
When to use / when not to use#
- Use it when: your data is already in BigQuery tables and you can create a read-only service account.
- Do not use it when: the data is in a relational database (use the database connector) or in files (use import a file).
Plan and permissions#
- Product core · access check · ingestion run kill switch.
- Minimum roles for the customer's service account:
roles/bigquery.dataVieweron the data project;roles/bigquery.jobUseron the billing project (or on the data project, if it is the only one).
Prerequisites#
- Data project (Project ID) — where the datasets/tables live.
- Billing project (optional) — where query jobs run and are billed; empty = the data project itself pays.
- Service account JSON with the roles above.
- Optional: a specific dataset and location (e.g.,
US,southamerica-east1).
Step by step#
- In the console, go to ETL/ELT → Connections and choose BigQuery (category Databases).
- Data project (Project ID) — e.g.,
meu-projeto-example. - Query billing:
In the same project as the data(default), orIn another project→ enter the Billing project (e.g.,meu-projeto-billing).
- Dataset — e.g.,
vendas(optional; empty lists all accessible ones). - Location — e.g.,
southamerica-east1(optional; empty = automatic). - Service account key (JSON) — paste the customer's service account JSON (it is encrypted and never shown again).
- Click Test connection and confirm that the datasets/tables appear.
- Select the tables and save the source.
Example (synthetic)#
Projeto de dados: meu-projeto-example
Billing: Em outro projeto
Projeto de billing: meu-projeto-billing
Dataset: vendas
Location: southamerica-east1
Service account: { "type": "service_account", "project_id": "meu-projeto-example",... }Grant access (example with gcloud, fictitious values):
# Read access to the data
gcloud projects add-iam-policy-binding meu-projeto-example \
--member="serviceAccount:ingestia-ro@meu-projeto-example.iam.gserviceaccount.com" \
--role="roles/bigquery.dataViewer"
# Run query jobs (on the billing project)
gcloud projects add-iam-policy-binding meu-projeto-billing \
--member="serviceAccount:ingestia-ro@meu-projeto-example.iam.gserviceaccount.com" \
--role="roles/bigquery.jobUser"Expected result#
- Test connection succeeds and discovery lists the accessible datasets/tables.
- During the load, Ingestia creates the query job and runs a 1-row probe (discarded) to validate the SQL/partition before sending any chunk — so a retry never duplicates data in the landing zone.
- Tables with
require_partition_filterare handled automatically: Ingestia builds a predicate that covers every row instead of failing.
Limits and costs#
- Chunked extraction:
batchSizedefault 5,000 (100–50,000);maxRowsdefault 200,000 (up to 5,000,000). - Queries against your BigQuery are billed to your billing project (BigQuery's on-demand model). Ingestia shows an estimate; the amount charged is the one on your project's invoice.
- Loading into Ingestia's Bronze is a billable load operation on Ingestia's side.
Security#
- Identifiers (project/dataset/table/watermark column) are sanitized before
entering the SQL; the incremental filter uses a named parameter (
@since), never concatenation. - The service account JSON key is encrypted and redacted in any error/log.
- Workspace scope: the data goes to the Bronze of your workspace.
Common errors#
| Symptom | Likely cause | What to do |
|---|---|---|
Access Denied:... dataViewer | SA has no read access to the data | Grant roles/bigquery.dataViewer on the data project |
User does not have bigquery.jobs.create | SA lacks jobUser on billing | Grant roles/bigquery.jobUser on the billing project |
Cannot query... requires a filter | table with a required partition filter | Ingestia handles it on its own; if it persists, provide the partition column |
Not found: Dataset | wrong dataset/location | Check the dataset name and the location |
| Invalid JSON | key pasted incompletely | Paste the entire service account JSON |
Troubleshooting#
- Use Test connection to tell permission issues (dataViewer/jobUser) apart from configuration issues (dataset/location).
- Confirm that the service account has access to the correct billing project when billing and data are different projects.
Related#
- Connect PostgreSQL
- Create an incremental pipeline by watermark
- Cost, limits, and security per workspace
Last reviewed: 2026-09-08.