Batch Import of Data

The Importing Data user guide details how the application can be used to import data from a single file, either an Excel spreadsheet or a .csv file. However, you may also wish to import data from a collection of files in one batch using the same previously created data model. The Data Importer Tool allows the user to save a template detailing how to import data from a file, and this template may then be used to guide the import of data from other files in bulk using the Bulk Data Importer tool.

../_images/bulk-importer-tab.png

Limitations of the tool

The same limitations of the Data Importer Tool apply to the Bulk Data Importer tool. All files must conform to the same data orientation and format defined by the import template used for the batch import operation.

Model and Template selection

To import data from a set of files, the correct data model and import template must be selected. You may either double click on an ImportRecord in the data manager to open the bulk importer tool with the correct ImportRecord already selected, or select the data model and Import Record from drop-down menus in the tool.

../_images/bulk-import-model-selection.png

After selecting an import template from the second drop-down menu, a summary of the template, including whether the entities are arranged column-wise or row-wise, the row/column containing the Entity names, the number format, and a list of importable FactKinds is displayed on the top right hand side of the Bulk Data Importer Tool.

File selection

After selecting an import template, select files to import by clicking the Add Files button. This will open a file browser allowing selection of multiple files. The supported file types for selection are Excel spreadsheets, (.xls and .xlsx) as well as CSV files, (.csv). The paths of all selected files will be displayed in a list below.

The Add Files button may be used multiple times to add further files to the set of files to import.

../_images/bulk-file-list-combined.png

Importing data and verifying results

When the list of files to import has been finalized the data import procedure may be triggered via the Start Import/Start Import Dry-run button.

../_images/bulk-import-start-button.png

If the checkbox for dry-run is enabled, triggering the import will attempt to detect FactKinds and Entities within each file and create Facts where data can be matched to an Entity, however no objects will be saved to the database. This mode is useful to verify the results of the import before saving data to the database. If the checkbox is not enabled, then triggering the import will also attempt to save all successfully imported data to the database.

If the import template does not completely match the file being imported, a warning dialog will pop up asking you to review the proposed matches

../_images/bulk-import-unexpected-fact-kind.png
../_images/bulk-import-review-fact-kinds.png

The results of the import or import dry-run will be displayed in the bottom table in a file by file manner. Each row in the results table details the number of successfully created Entities and Facts and also displays a status symbol indicating if data could be successfully imported from that file.

../_images/bulk-import-results-table.png

Common errors

When importing data from a file, there are a number of errors that may prevent data from being imported. These errors generally fall into two classes:

  • Template errors

  • Data errors

Template errors are caused when the file to import does not conform to the data orientation or format specified by the selected import template and/or when the data in the file can not be matched to the selected data model. This class of errors prevents all data from a file from being imported. For example, if the file to import does not have Entity names in the row or column specified by the import template, then no Entities will be detected in the file and thus no data will be imported. This is an example of a Template error.

Data errors are caused by malformed data, generally due to a data type mismatch with the detected FactKind. This class of errors only prevents specific cells from being imported. For example, if the detected FactKind for a cell is "Tensile Strength (MPa)" which should have a float data type, and the cell containing data instead has a value "N/A" (a string type), then the Fact for that cell can not be created.

If data can not be imported from a given file due to a Template error, then either a new import template should be created from the Data Importer Tool, or a different data model should be used for import.