The generic REST API connector can be registered and tested today. What is proven by tests is our side — request assembly, pagination, resumption, retry with backoff and token protection. The format of your API is what the connection test checks. Synthetic examples.
Read first: Como os dados fluem · Catálogo de conectores.
What it is#
A connector for systems without a ready connector: you describe the API (base
URL, the path that lists records, where the list is in the JSON and how it
pages) and Ingestia reads the records on schedule and loads them as one
table in the Bronze layer. Columns are discovered on first read
(names normalized to snake_case; objects and lists become JSON text).
What it's for#
Bring data from an internal system, a niche vendor or a partner API into the datalake without waiting for a dedicated connector — and without exporting spreadsheets by hand.
When to use / when not to use#
- Use when: the API responds JSON over HTTPS, lists records at an endpoint (GET or POST) and pages by page number, by offset or by next-page URL in the body (or does not page).
- Do not use when: the system already has a connector in the catalog (prefer it — it knows the API envelope and limits), the API requires OAuth with token refresh, per-request signing or a custom header key (these modes are planned for a future connector evolution), or the response is XML/CSV.
Plan and permissions#
- Product core · access check (workspace active).
- The load goes through the shutdown key of the capacity execução de ingestão.
- Recommended: a dedicated read-only token for Ingestia.
Prerequisites#
- The API base URL (starts with
https://) and the resource path that lists the records (e.g.:/v1/pedidos). - Know where the list is in the response: the response is already an
array (
[...]), or the array is under a key (data,results,data.items). - Know how the API pages and the maximum page size it accepts.
- A Bearer token, if the API requires it (sent as
Authorization: Bearer …; encrypted, never re-displayed).
Step by step#
- In the console, go to Fontes › Conectores and choose API REST genérica (category APIs genéricas).
- Fill the API description:
- URL base — e.g.:
https://api.exemplo.com - Caminho do recurso — e.g.:
/v1/pedidos - Método HTTP —
GET(parameters in the query) orPOST(in the JSON body) - Caminho da lista na resposta — e.g.:
data(empty = the response is already the array)
- URL base — e.g.:
- In Paginação, choose the API style and fill the requested fields:
- Nenhuma — a single response.
- Por número de página — page parameter (e.g.:
page), page size parameter (e.g.:per_page) and the maximum size the API accepts. - Por deslocamento (offset) — offset parameter (e.g.:
offset,skip), page size parameter (e.g.:limit) and the maximum size. - URL da próxima página no corpo — where the response provides the next URL
(e.g.:
next,links.next,paging.next; accepts relative URLs).
- If the API requires a token, paste it into Token Bearer. In Query extra, pin
parameters that every call must include (e.g.:
status=active&sort=desc). - Click Testar conexão. Ingestia requests a small page, checks that the list exists at the provided path and shows the discovered table.
- With a successful test, save the source and add it to a pipeline. The Bronze
table is named
registros.
How reading works (and where it stops)#
- For page-number and offset pagination, reading stops at the first page shorter than the requested size — that's why the field asks for the maximum the API accepts. If the API ignores the size and always returns fewer, provide the size it actually returns.
- For next-page URL in body, reading follows the next URL until it is empty or absent. A URL that repeats is treated as the end (it won't loop).
- With an empty token, no Authorization header is sent (open APIs).
- Responses 429 and 5xx are retried with backoff (and honoring
Retry-Afterwhen present). 401/403 stop the process with the message "credencial recusada" — the token is never shown in the error. - Extraction is full on each run (no incremental window). Very large sources may need a narrower path or Query extra filter.
Common problems#
| Symptom | Likely cause | What to do |
|---|---|---|
"A resposta não tem uma lista em <caminho>" | wrong list path | check the API JSON; use dot notation (data.items) |
| Only the first page arrives | wrong page/size parameter names, or size bigger than accepted | fix the parameter names and use the API's real maximum |
| "Credencial recusada" | token invalid, expired or lacking permission | generate a new read token and update the source |
| Columns with odd names | the API uses keys with spaces/accents | normal — we normalize to snake_case; rename in the Silver layer |
| Objects appear as text | nested field | expected behavior (JSON as text); extract in Silver |
Related#
- Catálogo de conectores — state and condition of each connector.
- Criar pipeline full — from source to publish.
- Status e limitações — what we don't promise today.