Import Data

Import Data is the first step in your pipeline, where data is taken in from external source. There are 2 types of data source you can import:

  • Online data source
  • Offline data source

NOTE: You need to create connection prior to importing online data source. Please refer to create connection page .

Next: Data synchronization

Online Data Source

There will be 2 substep, first is creating the connection node and followed by extracting table or file from the data source creating connection node

  1. Go to the Pipeline page from the left side menu.
  2. Click on the Add Data Source button on the top left corner.
  3. Choose the connection type, fill the credentials, and input the connection name to show on the pipeline.
  4. Once successful, the Connection Node will appear in your pipeline.
    Extracting data table (or file) from data source
  5. Click on on the connection node on the pipeline map.
  6. A side wizard will show up and click on + Import Data (Import Table or Import File for S3 bucket connection). Choose Import to DDL.Later you also can Import Data for transfering to another connection.
  7. Fill the details for the table you want to import based on the connection type. For Google Sheets, you have to choose the sheet, choose the sync method, and input the name for the node.
  8. Here is the detailed explanation.
FieldDescription
Import MethodImport table: Preset import from Delman Data Lab to import a single Table
SQL Query: User can custom an SQL query on the Data Source
Sync method (detail)Append Data: New data imported will added on top of previous data
Drop old data before inserting: Previous data will be erased and replaced by new data every synchronization process
  1. Click the Save button and the data node will appear in your pipeline, connected from the connection node.
  2. Repeat step 5-9 to import more data table (or file) from a single data source

Offline Data Source

  1. Go to the Pipeline page from the left side menu.
  2. Click on the Add Data Source button on the top left corner.
  3. Scroll down on the New Connection (External) tab and select Upload File and click the Continue button.
  4. Choose your file to upload.
  5. Input the node name and click Import File.