Data Hooks™

ⓘ Data Hooks are only available on version 2.0

# Data Hooks Guide

Flatfile Data Hooks provide automated normalization of data. Data is automatically formatted to your ideal input using the conditions you provide, without your user needing to perform cleanup and formatting. Data Hooks also provide feedback to users via info, error, and warning messages when additional human interaction is needed.

Using the hooks:

In order to use data hooks, register an event listener using the registerRecordHook and/or registerFieldHook functions. You will need to provide a callback function that executes any logic you'd like and returns the hook response structure with errors, warnings, or modifications to the data.

The three levels of info messages that can be used with your users are info, warning, and error. Please note that while they each have a distinct look and feel, only the error level will flag the data as invalid. This is especially important if your configuration is set to allowInvalidSubmit: false.

Record Hooks (Row Hooks)

The registerRecordHook event listener lets you re-format your data automatically as it first loads and on any changes the user might make to the record. The record hook waits for changes on each row of data imported, and can also help format a particular field in the record based on another field. For example, it can make city and state required fields if a postal code is not provided. In the example below, you'll see that the record (row of data) and its index value get passed into the registerRecordHook method automatically for you. To call a particular field in the record, you will chain the key name for the field you want to work with to record in a format like this: record.fieldName. The last piece you will notice is in the example is the out object we define at the beginning of the function. The specific data you want to change will be appended to that out object with its field name (out.fieldName) as an object of its own. There are two keys you can use within the object, value and info. Please note that you aren't required to use both the value and info keys. This means that you can change the value without passing in an info message, and also you can keep the original value and add an info message.

value:

The value key is where the value of the data point that is being set. As you see in the example, this is where we provide context as to how the data is to be pre-formatted.

info:

The info array is used to provide information in the form of errors and warnings to the user to let them know what is happening with the data that is being pre-formatted.

message:

The message key is used to provide the user a tooltip with information within the specific data in error or warning state.

level:

The level key is used to differentiate between "info", "error", and "warning" in the importer. "info" are subtle underlines with some context about the changes "warning" are used to provide a similar context to "info", but stand out visually "error" are generally things within the data that must be fixed before proceeding. Setting an"error" will also change the state to an error state not allowing the data to be submitted ifallowInvalidSubmit is set to false.

Example

In the below example, we are doing a couple of things. In the first example, we are reformatting US postal/zip codes. US zip codes are 5 digits in length, and some zip codes in the country begin with a zero. Spreadsheet software can often automatically drop any preceding zero, which would leave some of these zip codes in your data be only 4 digits. The first hook example makes sure that if the zip code is less than 5 digits, it adds a preceding zero. In the second example, we are giving an error message if the record doesn't have a postal code or both the city and state.

FlatfileImporter.registerRecordHook((record, index) => {
let out = {};
if (record.zipCode && record.zipCode.length < 5) {
out.zipCode = {
value: record.zipCode.padStart(5, "0"),
info: [
{
message: "Zipcode was padded with zeroes",
level: "warning"
}
]
};
}
if (!record.zipCode && (!record.city || !record.state)) {
out.zipCode = {
info: [
{
message: "You must have either City & State or Postal Code",
level: "error"
}
]
};
out.city = {
info: [
{
message: "You must have either City & State or Postal Code",
level: "error"
}
]
};
out.state = {
info: [
{
message: "You must have either City & State or Postal Code",
level: "error"
}
]
};
}
return out
});

The above configuration will result in this type of user experience for reformatting the Zip Code to 5 digits and also for validating the record has a city and state or zip code.

Field Hooks (Column Hooks)

The registerFieldHook event listener allows for you to receive all the values from a particular column when the data is initially loaded. The field hook event listener only happens once per field at the initial load of the data, and are fired before the record hooks. While this hook does allow for you to provide initial formatting for a particular field, it also opens up the possibility to verify the information against an external data source. For example, you have a client uploading customer information, but you don't want to allow the same customer ID to be uploaded twice. You could check the data in the customer ID field against your data and send feedback to the user about any duplicate records the file may contain. In order to use this hook, you will use the registerFieldHook method and provide the column key name as a parameter and then a callback function that will have the values from the column as an array of arrays as the parameter for the callback function. Each array would have the value for that row and the row number, and would be structured like this:

[
["A12345", 1],
["B98765", 2],
["A34567", 3]
]

Inside the function, you will have access to this array, and can send it to your server, reformat it or both. When you are done validating and formatting the data, you will need to return another array of arrays, similar to the one that was output, but the value will be returned as an object with a value key containing the new value and/ or an info key that is an object with message and object keys. This value object should be formatted like and works the the same as the value object from the record hooks. For more information on each key and what it does, please reference the record hook section above. What is being returned would look something like this:

[
[
{
value: "12345",
info: [
{
message: "Removed the preceding letter",
level: "info"
}
]
},
1
],
[
{
value: "98765",
info: [
{
message: "Removed the preceding letter",
level: "info"
}
]
},
2
],
[
{
value: "34567",
info: [
{
message: "Removed the preceding letter",
level: "info"
}
]
},
3
],
]

Example

The first below example simply takes the first value from the "customer_code" column and pads it with a zero at the beginning. The second example uses a fetch request to go out and grab usernames from an API and returns those to the server for us to compare against and let the user know that a User ID already exists in the database.

// simple example - add a 0 to the front of the first ID
importer.registerFieldHook("customer_code", values => {
return [
[
{
value: "0" + values[0],
info: [
{
message: "padded the start with a 0",
level: "warning"
}
]
},
values[1]
]
]
});
// example with a server request
importer.registerFieldHook("user_id", async values => {
let serverResults;
await fetch("https://my-json-server.typicode.com/flatfilers/csb-api/users")
.then(response => response.json())
.then(json => {
serverResults = json.map(x => {
return x.user_id;
});
});
let changeValues = [];
values.forEach(item => {
if (serverResults.includes(item[0])) {
changeValues.push([
{
info: [
{
message:
"Duplicate record. This user is already in the database.",
level: "error"
}
]
},
item[1]
]);
}
});
return changeValues;
});