Skip to content

Automatic translation — the Portuguese version is the reference. View in Portuguese

Updated on 2026-10-04

Databricks SQL — connector

Reads the tables from a Databricks catalog and schema via the Statement Execution API (paginated SQL queries in the SQL Warehouse) and loads into the Bronze layer.

On this page (15)

Read first: ../conectores.md (state map) and fichas.md (index of datasheets).


What it is#

Reads the tables from a Databricks catalog and schema via the Statement Execution API (paginated SQL queries in the SQL Warehouse) and loads into the Bronze layer.

Supported ingestion modes:

  • We pull — We pull the data from your source.

Use cases#

Bring data from Databricks SQL into your datalake and make it available to the Bronze/Silver/Gold layers, BI models and the natural language question wizard.

When to use / when not to use#

Use when you need to consolidate this source into the datalake on a scheduled basis, with isolation by workspace.

Remember that the first run with a real credential is followed by our team before the connector is fully released.

Plan and permissions#

  • Plan: core product (connectors are part of all plans; the limit is the number of sources of the plan).
  • Access: requires an active workspace — if the subscription is overdue, suspended or canceled, runs are blocked.
  • Shutdown: ingestion runs can be temporarily disabled (maintenance or incident). When this happens, the UI indicates the feature is unavailable at the moment.

Prerequisites#

  • A Databricks SQL account with access to the data you want to ingest.
  • In Databricks, generate a personal access token (User Settings → Developer → Access Tokens) and copy the SQL Warehouse ID (SQL Warehouses → the warehouse → Connection details). Provide the workspace URL, catalog and schema to read. The token is encrypted (AES-256-GCM) and is never shown again.

Credentials#

Authentication method: Access token pasted by you. It is encrypted (AES-256-GCM) and never shown again, nor appears in logs or error messages.

Sensitive fields. They are encrypted (AES-256-GCM) and never remain in the connection config, in the pipeline definition, in exports or in the DAG. By default the encrypted value is stored in the database; moving this connection's credential to an external credential vault is an explicit choice, made per connection after saving — see ../../administracao/credenciais-e-rotacao.md.

  • Personal access token (tokenAcesso) — Personal Access Token (PAT) of a user with SELECT on the schema.

Secrets are never re-displayed and are redacted everywhere in errors/logs (raw token, URL-encoded and in JSON).

Form fields#

CampoRótuloTipoObrigatórioObservação
urlWorkspaceURL do workspacetextosimEndereço do workspace Databricks, sem caminho.
warehouseIdID do SQL WarehousetextosimEm SQL Warehouses → o warehouse → Detalhes da conexão.
catalogoCatálogotextosimpadrão: main; Catálogo do Unity Catalog (ex.: main).
schemaEsquematextosimpadrão: default; Só as tabelas deste esquema são listadas.
tokenAcesso 🔒Token de acesso pessoalsenha/segredosimsegredo (cifrado; cofre externo opcional); Token de acesso pessoal (PAT) de um usuário com SELECT no esquema.

Step-by-step#

  1. No menu Dados → Conexões, use Nova conexão.
  2. Escolha Databricks SQL no catálogo.
  3. Preencha os campos do formulário (veja a tabela acima).
  4. Use Testar conexão. The result is shown broken down into Conectividade, Autenticação and Listagem/leitura — each with OK, Failed, Not applicable or Not tested. A capability with no test of its own for this connector shows as Not applicable, and Not applicable never counts as success.
  5. Use Salvar conexão. Salvar NÃO inicia ingestão nenhuma.
  6. No cartão da conexão, use Montar extração (ou vá em Orquestração → Pipelines e use Conectar fonte). O pipeline aponta para a conexão salva — a credencial não é copiada.
  7. Na revisão, Publicar (agenda desligada) cria a versão 1 sem executar nada. Depois, Ativar agenda e Executar agora são ações separadas.

Example — connection test#

Sample configuration (all values are fictitious; secrets appear masked):

text
# Exemplo sintético (valores fictícios — não são credenciais reais)
urlWorkspace = <urlWorkspace>
warehouseId = <warehouseId>
catalogo = main
schema = public
tokenAcesso = •••••• (cifrado — nunca exibido)

Expected result#

The Testar conexão returns the list of discovered tables/resources. After the first load, the rows appear in the Bronze layer of your workspace, ready for the pipeline.

Limits and costs#

  • Pagination: hard cap per run (default 10,000 pages / 5,000,000 rows); anti-loop guard.
  • SaaS load: default batchSize 5,000, maxRows 200,000 per resource.
  • Estimated volume: ~1 GB/month (catalog estimate, not measurement).
  • Cost: the extraction itself is not charged by the platform; the Bronze load consumes datalake processing (billable).

Security#

  • Scope by workspace: each customer reads and writes only to their own datalake (<prefixo>_bronze / _silver / _gold); there is never leakage between workspaces.
  • Encrypted secrets: sensitive material is encrypted with AES-256-GCM before being written. The external credential vault is optional and enabled per connection; without it, the encrypted value is stored in the database. Changing the value here DOES NOT revoke the credential at the origin.
  • Redaction: all errors/logs undergo secret redaction (raw, URL-encoded and in JSON).
  • Production does not mock: a path that would fall into mock triggers the production guard (explicit failure instead of simulating success).

Common errors#

  • Credential rejected (401/403) — the provider did not accept the credential: expired, revoked or lacking required permission. Reconnect the source (the source and history are preserved; do not recreate the source).
  • Rate limit (429) — the provider asked to wait. The extraction respects Retry-After and retries; if it repeats, wait and reduce the schedule frequency.
  • Resource not found — check the resource identifier (account, object, report) and the granted permissions.
  • Pagination cursor stuck — the extraction stops when the API cursor does not advance (anti-loop guard). If this happens in consecutive runs, open a support ticket indicating the connector and the resource.
  • Empty records path — the response arrived without records in the expected place. Check the selected resource and filters (period, account, status) before assuming there is no data.

Troubleshooting#

  • Use Testar conexão to isolate credential/network issues before scheduling.
  • Monitor runs in the observability per connector (error rate, p50/p95, freshness).
  • A schema change in the origin is signaled by schema drift (new column = info; removed column/type changed = attention).

Related#


Last revision: 2026-10-04. Datasheet automatically generated from the product connector catalog.

Related links