Read before:
../conectores.md(state map) andfichas.md(index of cards).
What it is#
Reads CSV, JSON and XLSX files from a bucket in your Cloud Storage (service account, either yours or the platform's) and loads them into the Bronze layer; Parquet is recognized and loaded directly.
Supported ingestion modes:
- We pull — We pull the data from your source.
Purpose#
Bring data from Cloud Storage into your datalake and make it available to the Bronze/Silver/Gold layers, BI models and the natural language question wizard.
When to use / when not to use#
Use when you need to consolidate this source into the datalake on a scheduled basis, with isolation by workspace.
Avoid sending files with unstable schema without first checking profiling and schema drift.
Plan and permissions#
- Plan: core product (connectors are part of all plans; the limit is the number of sources allowed by the plan).
- Access: requires an active workspace — if the subscription is overdue, suspended or canceled, runs are blocked.
- Disablement: ingestion runs can be temporarily disabled (maintenance or incident). When this happens, the UI indicates the resource is currently unavailable.
Prerequisites#
- The source bucket/container, and the prefix (folder) from which objects will be read.
- A credential with permission to list the bucket and read the objects under that prefix — that's all. It is not necessary (nor desirable) to grant write, delete or bucket administration permissions.
- Files under the prefix must be in a format the pipeline can read (CSV, JSON or Parquet). Object storage is a transport, not a format: the schema comes from the file.
Credentials#
Sensitive fields. They are encrypted (AES-256-GCM) and are never kept in the connection config, pipeline definition, export or DAG. By default the encrypted value is stored in the database; moving this connection's credential to an external credential vault is an explicit choice made per connection after saving — see ../../administracao/credenciais-e-rotacao.md.
- Service Account JSON (
serviceAccountJson)
Secrets are never re-displayed and are redacted in every error/log (raw token, URL-encoded and in JSON).
Form fields#
| Campo | Rótulo | Tipo | Obrigatório | Observação |
|---|---|---|---|---|
bucket | Bucket | text | yes | — |
prefix | Prefix / folder | text | no | — |
serviceAccountJson 🔒 | Service Account JSON | password/secret | no | secret (encrypted; external vault optional) |
Step-by-step#
- In the Data → Connections menu, use New connection.
- Choose Cloud Storage in the catalog.
- Fill in the form fields (see table above).
- Use Test connection. The result is returned separately for three capabilities, each with OK, Failed, Not applicable or Not tested. In this card they mean: Network — the storage service endpoint is reachable from the application server; Authentication — the credential in the form (access key or service account, not username and password) was accepted; Read — it was possible to list the objects in the bucket/container under the provided prefix (the displayed count is of visible objects, not tables).
- Use Save connection. Saving DOES NOT start any ingestion.
- On the connection card, use Mount extraction (or go to Orchestration → Pipelines and use Connect source). The pipeline points to the saved connection — the credential is not copied.
- In the review, Publish (schedule off) creates version 1 without running anything. Afterwards, Activate schedule and Run now are separate actions.
Example — connection test#
Example configuration (all values are fictitious; secrets appear masked):
# Exemplo sintético (valores fictícios — não são credenciais reais)
bucket = meu-bucket-exemplo
prefix = dados/2026/
serviceAccountJson = •••••• (cifrado — nunca exibido)Expected result#
The Test connection returns the number of visible objects under the provided prefix. After the first load, the contents of those files appear in the Bronze layer of your workspace, ready for the pipeline.
Limits and costs#
- Pagination: hard cap per run (default 10,000 pages / 5,000,000 rows); anti-loop guard.
- Estimated volume: ~3 GB/month (catalog estimate, not a measurement).
- Cost: the extraction itself is not charged by the platform; the Bronze load consumes datalake processing (billable).
Security#
- Scope by workspace: each customer reads and writes only to their own datalake (
<prefix>_bronze/_silver/_gold); there is never leakage between workspaces. - Encrypted secrets: sensitive material is encrypted with AES-256-GCM before being persisted. The external credential vault is optional and enabled per connection; without it the encrypted value is stored in the database. Changing the value here DOES NOT revoke the credential at the source.
- Redaction: every error/log goes through secret redaction (raw, URL-encoded and in JSON).
- Production doesn't fake: a path that would hit a mock triggers the production guard (explicit failure instead of simulating success).
Common errors#
- Authentication failed — the credential for this connector is NOT username and password: it is an access key (or service account). Check the credential provided in the form and whether it is still active at the provider.
- Listing denied — the credential authenticates but cannot list the bucket. Grant bucket listing and read permission for objects under the prefix; read permission without listing is not enough because the pipeline needs to discover which files exist.
- Bucket/prefix non-existent or empty — verify the exact bucket name and the prefix (folder). A prefix with no objects returns zero visible objects, not an error.
- File in unsupported format — object storage transports files; the pipeline needs to be able to read the content (CSV, JSON or Parquet). An object in another format is ignored or fails during load, depending on the pipeline policy.
Troubleshooting#
- Use Test connection to isolate credential/network issues before scheduling.
- Monitor runs in the observability per connector (error rate, p50/p95, freshness).
- A schema change at the source is signaled by schema drift (new column = info; removed column/type changed = attention).
Related#
../conectores.md— connector catalog by state../ingestao-incremental-backfill.md— full, incremental by watermark, backfill and checkpoint../pipeline-medalhao.md— Bronze/Silver/Gold layers and quality../agendamento-retry-dlq.md— scheduling, retry/backoff and DLQ../custo-limites-seguranca.md— cost, caps and workspace security
Last review: 2026-10-04. Card automatically generated from the product connectors catalog.