Leia antes:
../conectores.md(mapa de estados) efichas.md(índice das fichas).
What it is#
Reads the documents from an Elasticsearch index (up to 10,000 per sync) and the list of indices in the cluster, and loads into the Bronze layer.
Supported ingestion modes:
- We pull — We pull the data from your source.
What it's for#
Bring Elasticsearch data into your datalake and make it available to the Bronze/Silver/Gold layers, BI models and the natural language question wizard.
When to use / when not to use#
Use when you need to consolidate this source into the datalake on a scheduled basis, with isolation by workspace.
Remember that the first run with a real credential is followed by our team before the connector is fully released.
Plan and permissions#
- Plan: core product (connectors are part of all plans; the limit is the number of sources in the plan).
- Access: requires an active workspace — with the subscription overdue, suspended or canceled, the run is blocked.
- Shutdown: the ingestion run can be temporarily turned off (maintenance or incident). When that happens, the screen indicates the resource is currently unavailable.
Prerequisites#
- An Elasticsearch account with access to the data you want to ingest.
- Provide the cluster URL (https://host:9200 or the Elastic Cloud endpoint), the name of the index to read and a user with read privilege on the index (and monitor to list indices). The password is encrypted (AES-256-GCM) and never shown again. Each index reads at most 10,000 documents per synchronization in this Preview.
Credentials#
Authentication method: Username and password (or key and secret) in Basic authentication.
Sensitive fields. They are encrypted (AES-256-GCM) and are never stored in the connection config, pipeline definition, export or DAG. By default the encrypted value is stored in the database; moving the credential from this connection to an external credential vault is an explicit choice, made per connection after saving — see ../../administracao/credenciais-e-rotacao.md.
- Password (
senha) — Native user's password (or the secret of an API key in Basic).
Secrets are never re-displayed and are redacted in every error/log (raw token, URL-encoded and in JSON).
Form fields#
| Campo | Rótulo | Tipo | Obrigatório | Observação |
|---|---|---|---|---|
urlCluster | URL do cluster | texto | sim | Endereço HTTP(S) do cluster ou do deployment no Elastic Cloud. |
indice | Índice | texto | sim | Nome do índice (ou alias) cujos documentos viram a tabela Bronze. |
usuario | Usuário | texto | sim | — |
senha 🔒 | Senha | senha/segredo | sim | segredo (cifrado; cofre externo opcional); Senha do usuário nativo (ou o segredo de uma API key em Basic). |
Step by step#
- No menu Dados → Conexões, use Nova conexão.
- Escolha Elasticsearch no catálogo.
- Preencha os campos do formulário (veja a tabela acima).
- Use Testar conexão. The result is shown broken down into Conectividade, Autenticação and Listagem/leitura — each with OK, Failed, Not applicable or Not tested. A capability with no test of its own for this connector shows as Not applicable, and Not applicable never counts as success.
- Use Salvar conexão. Salvar NÃO inicia ingestão nenhuma.
- No cartão da conexão, use Montar extração (ou vá em Orquestração → Pipelines e use Conectar fonte). O pipeline aponta para a conexão salva — a credencial não é copiada.
- Na revisão, Publicar (agenda desligada) cria a versão 1 sem executar nada. Depois, Ativar agenda e Executar agora são ações separadas.
Example — connection test#
Example configuration (all values are fictitious; secrets appear masked):
# Exemplo sintético (valores fictícios — não são credenciais reais)
urlCluster = <urlCluster>
indice = <indice>
usuario = <usuario>
senha = •••••• (cifrado — nunca exibido)Expected result#
The Testar conexão returns the list of discovered tables/resources. After the first load, the rows appear in the Bronze layer of your workspace, ready for the pipeline.
Limits and costs#
- Pagination: hard cap per run (default 10,000 pages / 5,000,000 rows); anti-loop guard.
- SaaS load: default
batchSize5,000,maxRows200,000 per resource. - Estimated volume: ~0.5 GB/month (catalog estimate, not measurement).
- Cost: the extraction itself is not charged by the platform; the Bronze load consumes datalake processing (billable).
Security#
- Scope by workspace: each customer reads and writes only to their own datalake (
<prefixo>_bronze/_silver/_gold); there is never leakage between workspaces. - Encrypted secrets: sensitive material is encrypted with AES-256-GCM before being written. The external credential vault is optional and enabled per connection; without it, the encrypted value stays in the database. Changing the value here DOES NOT revoke the credential at the origin.
- Redaction: every error/log is passed through secret redaction (raw, URL-encoded and in JSON).
- Production does not mock: a path that would fall into mock triggers the production guard (explicit failure instead of simulating success).
Common errors#
- Credential rejected (401/403) — the provider did not accept the credential: expired, revoked or lacking the necessary permission. Reconnect the source (the source and history are preserved; do not recreate the source).
- Rate limit (429) — the provider asked to wait. The extraction respects
Retry-Afterand retries; if repeated, wait and reduce the schedule frequency. - Resource not found — check the resource identifier (account, object, report) and the granted permissions.
- Pagination cursor stuck — the extraction stops when the API cursor does not advance (anti-loop guard). If this happens in consecutive runs, open a support ticket with the connector and the resource.
- Empty records path — the response arrived without records in the expected place. Check the selected resource and filters (period, account, status) before assuming there is no data.
Troubleshooting#
- Use Testar conexão to isolate credential/network issues before scheduling.
- Monitor runs in the connector observability (error rate, p50/p95, freshness).
- A schema change at the source is signaled by schema drift (new column = info; removed column/type changed = attention).
Related#
../conectores.md— catálogo de conectores por estado../ingestao-incremental-backfill.md— full, incremental por watermark, backfill e checkpoint../pipeline-medalhao.md— camadas Bronze/Silver/Gold e qualidade../agendamento-retry-dlq.md— agendamento, retry/backoff e DLQ../custo-limites-seguranca.md— custo, tetos e segurança por workspace
Last revision: 2026-10-04. Sheet automatically generated from the product connector catalog.