Importing Data with Dataimport

In this technical reference document, we will cover the standard approach to importing data from external sources. Everything you need for this can be found in the dataimport module in hot-deploy.

Opentaps Data Import Strategy

The goal of the Data Import module is not to build a set of data import tools against a particular "standard," but rather to recognize that each organization has legacy or external data in its own unique format. Therefore, the Data Import module is a set of flexible tools which you can use as a reference point for setting up your own custom import and export. The existing services and entities can be used "as is" or with little modification if your data happens to be similar, or you can add to and extend them if you have additional data.

The Data Import module sets up "bridge entities" which are de-normalized and laid out in a way that is similar to most applications' data definitions. There are no foreign key relationships to any other opentaps entity, so any data could be imported into them. You would use your own database's import tools to import records into the bridge entities. Then, you would run one of the Data Import module's import services to transform the data in the bridge entities into the opentaps system. The Data Import services all follow a common standard:

  1. Each row of data in a bridge entity is wrapped in its own transaction when it is imported and succeeds or fails on its own.
  2. When a row of data in a bridge entity is imported successfully, the importStatusId field will be set to DATAIMP_IMPORTED
  3. If the import failed for the row, the status will be DATAIMP_FAILED and the importError field will contain a description of why.