Data Pipeline Import/Export Processes
- Heather Smith
Overview
The Data Pipeline Import and Data Pipeline Export processes in Regent allow import and export of data via a configured Data Pipeline pathway. The data imported or exported through a configured pipeline is handled via Regent development within Azure Data Factory. Data Pipelines and Data Factory integration allow for a more configuration-driven process than other, hard-coded data import/export processes within Regent.
Example of Data Pipeline Use
An example of the use of Data Pipelines is state grant awarding data. Eligibility files are imported via a configured data pipeline and the data is consumed and transformed by Data Factory. The data populates a smart form document assigned to students, then an awarding ruleset uses the data in the smart form to determine awards. After awarding, a payment request file is generated and exported via a configured data pipeline, to be sent to the state.
Data Pipeline Setup
Within Institution Setup, the Data Pipeline tab is available to Administrators. On the Data Pipeline Detail screen, the following setup fields are presented. Some fields are hidden/shown based on whether the Data Pipeline being configured is for an ‘Import’ Data Flow or an ‘Export’ Data Flow.
Most Data Pipeline Setup will be done by Regent, in conjunction with Azure Data Factory development of import and export processes.
Field Name | Field Type/Options | Definition/Rules |
|---|---|---|
Active | Checkbox | Indicates if the Data Pipeline is active, meaning available for selection on the Import/Export Process screens. Checked by default when a Data Pipeline is being added. |
Name | Text box | Identifier for the Data Pipeline. If the Data Pipeline is active, the name will be listed as an option for selection within the ‘Import Data Pipeline' Import Process if the Data Flow selected is ‘Import’ or within the ‘Export Data Pipeline’ Export Process if the Data Flow selected is 'Export.’ |
Data Pipeline Type | Dropdown ‘Azure Data Factory’ is the only option for this field. | Identifies the transformation platform being used to extract and transform the data being exported or received. |
Data Flow | Radio button Options: Import, Export | Identifies if the Data Pipeline is an import or export.
|
Process Type | Dropdown All active Processes are listed in this field. | Identifies the Process Name which will be associated with the Data Pipeline in the Process Log when the Import Data Pipeline or Export Data Pipeline process is ran. |
File Path | Text box | Indicates the location of the container to place the file to be consumed by Data Factory. Only presented when Data Flow is ‘Import.’ |
Data Pipeline ID | Text box | The identifier entered here must match the pipeline identifier in the transformation platform used to execute the process. Only presented when Data Flow is ‘Export.’ |
Azure Subscription ID | Text box | The identifier entered here must match the Azure Subscription ID of the platform used to execute the process. Only presented when Data Flow is ‘Export.’ |
Azure Data Factory Name | Text box | The identifier entered here must match exactly with the Azure Data Factory subscription. Only presented when Data Flow is ‘Export.’ |
Query Name | Dropdown All active configured queries. | The query selected will be used to identify population of students for which data will be exported. Only presented when Data Flow is ‘Export.’ |
File Name | Text box | Text entered will be used as the file name for the export file created by the export process for the Data Pipeline. Only presented when Data Flow is ‘Export.’ |
Projection | Text area | Defines the data columns, types, and formats for the data export. Only presented when Data Flow is ‘Export.’ |
Data Pipeline Imports
Users with the Import Data Pipeline permission have access to the ‘Import Data Pipeline’ Process Type in Import Processes. To import a file to be processed, select the name of the Data Pipeline and select a file, then click ‘Import.’ Results of the import will be shown in the Process Log.
If a Smart Form has been created to house the imported data, Data Factory will transform the data and the transformed data will populate within the smart form for matched students, per developed mapping logic.
Data Pipeline Exports
Users with the Export Data Pipeline permission have access to the ‘Export Data Pipeline’ process in Export Processes. To export data via this process, select the name of the Data Pipeline. If the query for the data pipeline was configured with variable parameters, the data elements for which values should be selected will be presented. Select the values for each field, then click ‘Export.’ The export file will be available in the Process Log.
Selected parameters can also be saved by clicking ‘Save Selected Parameters,’ to set up the export as a scheduled process.