Read first:
../conectores.md(state map) andfichas.md(index of datasheets).
What it is#
Reads the tables from a Databricks catalog and schema via the Statement Execution API (paginated SQL queries in the SQL Warehouse) and loads into the Bronze layer.
Supported ingestion modes:
- We pull — We pull the data from your source.
Use cases#
Bring data from Databricks SQL into your datalake and make it available to the Bronze/Silver/Gold layers, BI models and the natural language question wizard.
When to use / when not to use#
Use when you need to consolidate this source into the datalake on a scheduled basis, with isolation by workspace.
Remember that the first run with a real credential is followed by our team before the connector is fully released.
Plan and permissions#
- Plan: core product (connectors are part of all plans; the limit is the number of sources of the plan).
- Access: requires an active workspace — if the subscription is overdue, suspended or canceled, runs are blocked.
- Shutdown: ingestion runs can be temporarily disabled (maintenance or incident). When this happens, the UI indicates the feature is unavailable at the moment.
Prerequisites#
- A Databricks SQL account with access to the data you want to ingest.
- In Databricks, generate a personal access token (User Settings → Developer → Access Tokens) and copy the SQL Warehouse ID (SQL Warehouses → the warehouse → Connection details). Provide the workspace URL, catalog and schema to read. The token is encrypted (AES-256-GCM) and is never shown again.
Credentials#
Authentication method: Access token pasted by you. It is encrypted (AES-256-GCM) and never shown again, nor appears in logs or error messages.
Sensitive fields. They are encrypted (AES-256-GCM) and never remain in the connection config, in the pipeline definition, in exports or in the DAG. By default the encrypted value is stored in the database; moving this connection's credential to an external credential vault is an explicit choice, made per connection after saving — see ../../administracao/credenciais-e-rotacao.md.
- Personal access token (
tokenAcesso) — Personal Access Token (PAT) of a user with SELECT on the schema.
Secrets are never re-displayed and are redacted everywhere in errors/logs (raw token, URL-encoded and in JSON).
Form fields#
| Campo | Rótulo | Tipo | Obrigatório | Observação |
|---|---|---|---|---|
urlWorkspace | URL do workspace | texto | sim | Endereço do workspace Databricks, sem caminho. |
warehouseId | ID do SQL Warehouse | texto | sim | Em SQL Warehouses → o warehouse → Detalhes da conexão. |
catalogo | Catálogo | texto | sim | padrão: main; Catálogo do Unity Catalog (ex.: main). |
schema | Esquema | texto | sim | padrão: default; Só as tabelas deste esquema são listadas. |
tokenAcesso 🔒 | Token de acesso pessoal | senha/segredo | sim | segredo (cifrado; cofre externo opcional); Token de acesso pessoal (PAT) de um usuário com SELECT no esquema. |
Step-by-step#
- No menu Dados → Conexões, use Nova conexão.
- Escolha Databricks SQL no catálogo.
- Preencha os campos do formulário (veja a tabela acima).
- Use Testar conexão. The result is shown broken down into Conectividade, Autenticação and Listagem/leitura — each with OK, Failed, Not applicable or Not tested. A capability with no test of its own for this connector shows as Not applicable, and Not applicable never counts as success.
- Use Salvar conexão. Salvar NÃO inicia ingestão nenhuma.
- No cartão da conexão, use Montar extração (ou vá em Orquestração → Pipelines e use Conectar fonte). O pipeline aponta para a conexão salva — a credencial não é copiada.
- Na revisão, Publicar (agenda desligada) cria a versão 1 sem executar nada. Depois, Ativar agenda e Executar agora são ações separadas.
Example — connection test#
Sample configuration (all values are fictitious; secrets appear masked):
# Exemplo sintético (valores fictícios — não são credenciais reais)
urlWorkspace = <urlWorkspace>
warehouseId = <warehouseId>
catalogo = main
schema = public
tokenAcesso = •••••• (cifrado — nunca exibido)Expected result#
The Testar conexão returns the list of discovered tables/resources. After the first load, the rows appear in the Bronze layer of your workspace, ready for the pipeline.
Limits and costs#
- Pagination: hard cap per run (default 10,000 pages / 5,000,000 rows); anti-loop guard.
- SaaS load: default
batchSize5,000,maxRows200,000 per resource. - Estimated volume: ~1 GB/month (catalog estimate, not measurement).
- Cost: the extraction itself is not charged by the platform; the Bronze load consumes datalake processing (billable).
Security#
- Scope by workspace: each customer reads and writes only to their own datalake (
<prefixo>_bronze/_silver/_gold); there is never leakage between workspaces. - Encrypted secrets: sensitive material is encrypted with AES-256-GCM before being written. The external credential vault is optional and enabled per connection; without it, the encrypted value is stored in the database. Changing the value here DOES NOT revoke the credential at the origin.
- Redaction: all errors/logs undergo secret redaction (raw, URL-encoded and in JSON).
- Production does not mock: a path that would fall into mock triggers the production guard (explicit failure instead of simulating success).
Common errors#
- Credential rejected (401/403) — the provider did not accept the credential: expired, revoked or lacking required permission. Reconnect the source (the source and history are preserved; do not recreate the source).
- Rate limit (429) — the provider asked to wait. The extraction respects
Retry-Afterand retries; if it repeats, wait and reduce the schedule frequency. - Resource not found — check the resource identifier (account, object, report) and the granted permissions.
- Pagination cursor stuck — the extraction stops when the API cursor does not advance (anti-loop guard). If this happens in consecutive runs, open a support ticket indicating the connector and the resource.
- Empty records path — the response arrived without records in the expected place. Check the selected resource and filters (period, account, status) before assuming there is no data.
Troubleshooting#
- Use Testar conexão to isolate credential/network issues before scheduling.
- Monitor runs in the observability per connector (error rate, p50/p95, freshness).
- A schema change in the origin is signaled by schema drift (new column = info; removed column/type changed = attention).
Related#
../conectores.md— catalog of connectors by state../ingestao-incremental-backfill.md— full, incremental by watermark, backfill and checkpoint../pipeline-medalhao.md— Bronze/Silver/Gold layers and quality../agendamento-retry-dlq.md— scheduling, retry/backoff and DLQ../custo-limites-seguranca.md— cost, limits and workspace security
Last revision: 2026-10-04. Datasheet automatically generated from the product connector catalog.