configuring-imports

Configuring Imports

An import is the data destination in a Celigo integration. It takes records from an upstream step and writes them to an external system -- REST APIs, databases, ERPs, file servers, or AI models. Every import is bound to exactly one connection and one adaptor type.

Imports handle six concerns:

• Field mapping -- transforming source fields into the destination system's expected format (including value resolution via static maps and lookup tables). Uses Mapper 2.0 (mappings[] array) by default; NetSuite and Salesforce imports only support Mapper 1.0 (mapping.fields[] / mapping.lists[])
• Operation logic -- create, update, upsert, delete, attach/detach
• Hooks -- JavaScript pre/post processing at various pipeline stages (preMap, postMap, postSubmit). File-based imports that generate files from records also support postAggregate
• One-to-many -- fan out child records from a parent. Set oneToMany: true and pathToMany to the child array path (e.g., "lineItems") when one source record should create multiple import operations
• Response mapping -- extract fields from the import's API response back into the record for downstream steps. Configured on the flow's pageProcessors[] entry, but planned when building the import. The response is available via _json (the raw API response) and errors. Use _json.fieldName to extract from the response (e.g., _json.id for a created record's ID, _json.output.1.content.0.text for OpenAI responses). Response mapping uses Transformation 1.0 syntax (extract/generate pairs), not the newer expression-based transforms
• postResponseMap hook -- JavaScript processing after response mapping merges the response back into the record. Configured on the flow's pageProcessors[] entry, but planned when building the import. Use to transform or enrich the merged record before downstream steps

Imports are used across flows, APIs, and tools.

Import Execution Pipeline

When records arrive at an import step, this pipeline executes in strict order:

1. Input filter (optional) -- discards records before any processing (configured as an expression on the flow's pageProcessors[] entry)
2. preMap hook (optional) -- JavaScript processing before field mapping
3. Field mapping -- Mapper 2.0 or 1.0 maps source fields to destination fields, including lookups and hardcoded values
4. postMap hook (optional) -- JavaScript processing after field mapping, before submission
5. Submit to destination -- writes the mapped record to the external system
6. postSubmit hook (optional) -- JavaScript processing after the destination responds (access response data, log results, trigger side effects)
7. Response mapping (optional) -- carries data from the destination response back into the record for downstream steps. Configured on the flow's pageProcessors[] entry, not on the import itself
8. postResponseMap hook (optional) -- JavaScript processing after response mapping merges response data back into the record

Key distinction: Response mapping lives on the flow's pageProcessors[] entry, not on the import resource. When building an import that needs to pass data downstream, plan the response mapping at flow design time.

Categories of Import

Record-Based Imports

Submit structured records to APIs, databases, or ERPs. The vast majority of imports.

• NetSuiteDistributedImport -- high-performance SuiteApp writes (add, update, addupdate, delete, attach, detach)
• HTTPImport -- REST/GraphQL APIs (POST, PUT, PATCH, DELETE). Supports connector-assisted (formType: "assistant") and GraphQL (graph_ql) modes
• SalesforceImport -- Salesforce CRUD via SOAP, REST, Bulk, or Composite Record API
• RDBMSImport -- SQL databases (Snowflake, PostgreSQL, MySQL, SQL Server, Oracle). Uses per_record, bulk_insert, or bulk_load query types
• MongodbImport, DynamodbImport, JDBCImport -- other databases

File-Based Imports

Write or upload files to remote storage. Require the file{} configuration block. Two modes:

• Record-to-file -- aggregates incoming records into a file (CSV, JSON, XML, XLSX). The file{} block defines the output format.
• Blob passthrough -- transfers a binary blob as-is from an upstream export. Set the blobKeyPath field to the path in the record that contains the blob key.

Adaptor types:

• HTTPImport with http.type: "file" -- upload files over HTTP to cloud storage APIs (Google Drive, Box, Dropbox, Azure Blob Storage)
• FTPImport -- CSV, XML, JSON, XLSX, EDI files to FTP/SFTP
• S3Import -- objects to Amazon S3
• AS2Import -- AS2 EDI file transmission
• FileSystemImport -- local/on-premise filesystem writes

AI Imports

Invoke AI models for classification, extraction, or safety checks. No _connectionId required unless using BYOK.

• AiAgentImport -- OpenAI or Gemini model invocations with structured output, tool use, and reasoning
• GuardrailImport -- PII detection, content moderation, or custom AI-based validation

Stack and Tool Imports

• WrapperImport -- custom pre-built stack connectors (Walmart, BigCommerce)
• ToolImport -- invoke a Celigo Tool resource

Import Matching (create / update / upsert)

The core decision on most record-based imports is the operation -- what happens to each record at the destination. Create requires no match key; update and delete require one; upsert checks first and does whichever applies. Users describe the intent in business terms ("look up the customer and update them", "match by email and upsert", "skip the ones that already exist") that resolve into five behaviors:

• Always create -- every record is submitted as new. No matching, no checks.
• Create only if missing -- match-check first; submit as create if not found, skip silently if found.
• Update only if exists -- match-check first; submit as update if found, skip silently if not.
• Update only if exists, fail if missing -- strict variant; no-match records error out instead of skipping.
• Upsert -- submit a create when no match is found, an update when matched. The catch-all, and the right default when the user is vague.

When matching applies, decide four things: the matching behavior, the match key field(s) (email, external_id, customer.id -- required for anything other than always-create), the on-match action (update, skip, fail), and the on-no-match action (create, skip, fail).

How a destination implements matching is adaptor-specific -- there is no single "matching mode" field. A destination might expose: a native upsert keyed off an external ID (Salesforce upsert, NetSuite upsert, RDBMS ON CONFLICT); a distinct addupdate operation that handles both paths in one call; an ignoreExisting flag paired with a lookup that probes before writing; two separate create and update endpoints with no upsert variant (see composite imports below); or a lookup endpoint plus separate create and update endpoints, where the lookup runs pre-write to drive the create-vs-update decision. Some destinations have no matching concept at all -- writing a CSV to FTP, sending an email, posting to a webhook -- so every record goes out as-is.

Prefer import-level matching over a separate lookup step. Imports natively support this pre-write check, so a single import that does the matching and the write together means fewer steps, fewer round-trips, and no glue logic to maintain. A standalone lookup earns its place only when the looked-up data has a consumer beyond the write -- a router branching on something other than "does this exist", an AI agent reasoning over the result, or multiple downstream steps reading different fields. If the only consumer is the destination call itself, the work belongs inside the import.

Composite (two-endpoint) imports

When an HTTP destination has no native upsert but exposes separate create and update endpoints, a composite import pins both endpoints on a single import node, role-tagged create and update. The runtime picks per record via a match-key check -- the same way a native upsert would -- so the flow stays one step. Prefer this over two separate imports driven by an upstream lookup or router; reach for separate imports only when the create and update paths must diverge beyond endpoint selection (different mappings, different downstream consumers, or different hook chains).

Quick Reference

Adaptor Decision Matrix

Raw HTTP is the fallback, not the default. Pick the most specific match, in order:

1. Native adaptor -- if the application has its own row (NetSuite, Salesforce, databases, FTP/S3), use it. Do not build an HTTPImport against that app's REST API.
2. Pre-built HTTP connector -- for any other REST/GraphQL app, check the 550+ connector catalog before writing HTTP config (see Check for a pre-built connector). The step is still an HTTPImport, but it runs on a connector-backed connection and takes its endpoint config from the connector.
3. Manual HTTP -- hand-write the config from public API docs only when no connector exists or it doesn't cover the operation you need.

adaptorType is case-sensitive: NetSuiteDistributedImport, not netsuitedistributedimport.

Minimum Required Fields

Every import needs at minimum: name, adaptorType, _connectionId (except AiAgentImport/GuardrailImport without BYOK), and the adaptor config block (http{}, netsuite_da{}, salesforce{}, etc.).

Which Schemas to Read

1. Always: request.yml (base fields)
2. Plus: the adaptor-specific file from the decision matrix
3. If file-based: also file.yml
4. If cloning: clone-request.yml, clone-response.yml

Schema Index

All schemas are in references/schemas/:

• Base fields (all imports): request.yml
• Response shape: response.yml
• Adaptor-specific config:
  • http.yml -- HTTP/REST/GraphQL (methods, URIs, headers, response parsing, upsert via existingExtract)
  • netsuitedistributed.yml -- NetSuite SuiteApp (operation, recordType, internalIdLookup, mapping, lookups)
  • netsuite.yml -- NetSuite legacy
  • salesforce.yml -- Salesforce (sObjectType, operation, api, idLookup)
  • rdbms.yml -- SQL databases (queryType, query, bulkInsert, bulkLoad)
  • ftp.yml -- FTP/SFTP
  • s3.yml -- Amazon S3
  • mongodb.yml -- MongoDB (method, collection, filter, upsert)
  • dynamodb.yml -- DynamoDB
  • jdbc.yml -- JDBC databases
  • as2.yml -- AS2 EDI
  • wrapper.yml -- custom stack connectors
  • filesystem.yml -- local filesystem
• AI config:
  • aiagent.yml -- AI agent (provider, model, instructions, tools, structured output)
  • guardrail.yml -- guardrails (PII, moderation, AI-agent validation)
• File output: file.yml (CSV, XML, JSON, XLSX config for file-based imports)
• Clone: clone-request.yml, clone-response.yml

Related Skills

• configuring-connections > Quick Reference -- connection types and auth for import destinations
• writing-mappings > Mapper 2.0 Workflow -- field mappings on imports
• writing-scripts > Data Pipeline Hooks -- preMap, postMap, postSubmit, postAggregate hooks
• writing-handlebars > Quick Reference -- dynamic values in URIs, HTTP bodies, SQL queries
• building-flows > How to Build a Flow -- wiring imports into flow pipelines as page processors
• troubleshooting-flows > Diagnostic Workflow -- diagnosing import-related failures

How to Build an Import

1. Identify the target application

What system are you writing data to? This determines adaptor type, connection type, and configuration shape.

2. Check for existing patterns

Before building from scratch, look at what already exists:

# Search your account (fast, uses local index)
celigo account search "<keyword>"
 
# Show what an existing import uses (connection) and what uses it (flows)
celigo account dependencies import <

The account index auto-refreshes when stale (>4 hours). Force a fresh snapshot with celigo account snapshot.

3. Check for a pre-built connector

Always run this check before writing any HTTP config. Celigo maintains 550+ HTTP connectors with pre-configured auth, endpoints, and resources. Hand-write a manual HTTPImport from public API docs only when this search comes up empty or the connector doesn't cover the operation you need.

If a connector exists, create the connection from it (http._httpConnectorId -- see configuring-connections > Check for a pre-built connector and global iClient) and take the import's relativeURI, method, and body/response shapes from the connector's endpoint metadata rather than reconstructing them from public API docs. The connector-reference fields on the import itself (http._httpConnectorEndpointId, http._httpConnectorVersionId, http._httpConnectorResourceId) are read-only -- the platform sets them; what you control is the connection and the endpoint config you copy from the connector.

4. Query metadata for the target system

For NetSuite, Salesforce, and RDBMS connections, discover available record types and fields:

• NetSuite: metadata fields returns field IDs, types, and groups — use the IDs for mapping.fields[].generate and mapping.lists[].fields[].generate. Sublist names (e.g., "item", "addressbook") appear as groups, which map to mapping.lists[].generate. Lookup field IDs here are the searchField/resultField values for netsuite_da.lookups[].
• Salesforce: metadata fields returns field API names, types, and relationship info. Use field API names for salesforce.sObjectType lookups and for discovering which fields are createable/updateable.
• RDBMS: metadata fields returns column names and types for a table — use these to write SQL queries (see writing-sql) and verify column names before building bulkInsert.tableName or bulkLoad.tableName.

5. Determine the category

Is this a record-based import (submit records to an API/database/ERP), a file-based import (write files to storage), or an AI import (invoke a model)?

6. Choose the right adaptor type

Refer to the Adaptor Decision Matrix in the Quick Reference above.

7. Build the import JSON

Reference the Schema Index for the exact fields needed. Use the Which Schemas to Read decision rule to determine which files to consult.

File Uploads over HTTP (multipart/form-data)

Some destination APIs accept files only as multipart/form-data POSTs (Jira attachments, QuickBooks attachables, OpenAI file uploads). This is an HTTPImport in file-transfer mode (http.type: "file") and works nothing like a JSON record write. Four pieces have to line up:

1. Where the bytes come from. The import never carries the file itself. A preceding blob export or blob lookup pulls the file into blob storage, and that step's response mapping puts the reference onto the record -- the idiom is {"extract": "data.0.blobKey", "generate": "blobKey"}. The record then carries a blobKey (a storage pointer, not content).
2. The media type. The connection's media type (or the import's request media type override) is multipart/form-data; success/error response media types usually override to JSON so replies parse normally.
3. The request body. Not raw MIME and not a normal handlebars payload -- a JSON array of parts, each {name, value, type} plus optional filename (include the extension) and mime-headers. The file part is "type": "attachment" with "value": "{{blob}}" -- {{blob}} is the only accepted value for an attachment (anything else 422s). Fields the API wants alongside the file ride as "type": "inline" parts; an inline part whose value is a JSON object must be serialized. One file reference per import.
4. blobKeyPath (Advanced settings) -- the JSON path in the record where the blobKey lives (blobKey, or file.blobKey if nested). At send time the platform follows it into blob storage and streams the real bytes into the attachment part.

The platform assembles the final MIME body itself -- it generates the boundary (never hardcode one), writes each part's Content-Disposition, and substitutes the attachment part with the raw file bytes. The parts array is a build recipe, not the payload.

Not every multipart API is form-data. Some upload endpoints expect multipart/related instead (e.g. Google Drive's /upload/drive/v3/files?uploadType=multipart), and the parts-array machinery does NOT apply. There the request body is the literal MIME document: explicit boundary, a JSON metadata part, and a content part referencing {{blob}} (double braces). Check which flavor the destination API documents before building -- mixing them produces an import that saves cleanly and fails at runtime.

Async Destinations (submit, poll, confirm)

Some destination APIs only acknowledge a write (an HTTP 202, a job ticket) and finish it in the background -- bulk loads, file ingestion, document conversion. Attach an async helper to the import (http._asyncHelperId) so the step submits, polls a status export until the external work completes, and only then resolves. The mechanics and constraints (a status export with done/error value lists and poll intervals; no transform, output filter, or hook on the async-configured step; helpers cannot nest) are identical to the export side -- see configuring-exports > Async APIs (submit, poll, fetch). Only add one when the API genuinely cannot confirm the write synchronously.

CLI Commands

Pre-Submit Checklist

Required (all imports)

• name is set
• adaptorType exact case matches connection type (request.yml > adaptorType)
• _connectionId references a valid, online connection (skip for AI imports without BYOK)
• Adaptor config block name matches adaptorType (http{} for HTTPImport, netsuite_da{} for NetSuiteDistributedImport, etc.)

Adaptor-specific

• HTTP: pre-built connector was checked (celigo http-connectors list) -- hand-written config only because no connector covers the app or operation
• HTTP: http.method and http.relativeURI are set (http.yml)
• NetSuite: netsuite_da.operation and netsuite_da.recordType are set (netsuitedistributed.yml)
• RDBMS: rdbms.queryType is per_record or bulk_insert -- NOT legacy insert/update (rdbms.yml)
• Salesforce: salesforce.sObjectType and salesforce.operation are set (salesforce.yml)

Cross-resource consistency

• Connection type matches the import's adaptorType
• If response mapping needed: configured on the flow's pageProcessors[] entry, not on the import itself
• If one-to-many: oneToMany: true and pathToMany is set to the child array path
• If using Mapper 1.0 (NetSuite/Salesforce): mapping.fields[] / mapping.lists[], not mappings[]

Gotchas

1. PUT erases omitted fields. Always GET first, modify, then PUT. The set command handles this.
2. Including a rest: block creates a legacy RESTImport. Use only http: for new imports.
3. Input filter skips that import, not the record. Filtered records skip the current import step but continue to subsequent steps in the flow. They aren't dropped -- check numIgnore on the job if records seem to bypass a step.
4. Multipart file parts only accept {{blob}}. In a multipart/form-data parts array, the file part must be "type": "attachment" with "value": "{{blob}}" -- any other value 422s. Never hardcode the MIME boundary; the platform generates it. Fields the API wants alongside the file ride as "type": "inline" parts.
5. bodyKey/blobKey in logs is an artifact, not a payload field. Audit and debug logs never show the assembled multipart body -- an internal storage pointer appears where the body would be. But if the destination actually received the literal string bodyKey or blobKey, the file part is misconfigured (an inline part where an attachment belongs, or a blobKeyPath that doesn't resolve).

Common Errors