Are you looking for exact duplicates (all fields identical) or duplicates on a primary or subset of key fields? If you want all fields identical, does this extend to link to form fields, table fields or photo fields and their contents?
I was working a while back on tooling to do merging of records through the scripting interface to handle when I end up with duplicate shipping records due to key mismatch (sometimes the tracking numbers are changed in transit) but I never got it finished because the UI experience wasn’t something I figured out.
If you’re after a subset of key fields, I did a quick POC where you create a new field in your source form with the key. I already had a composite key field built using a calculation field I use which looks like this (square brackets for field placeholders):
IFEMPTY([Marketplace];"No marketplace";
CONCAT([Marketplace], "/",[Store Name], "/",[Order ID]
))
This creates what should be a unique key for the record based on my own metadata (designed to handle direct sales and hosted marketplaces). I then created a new form called “Orders Dedupe” and put in it three fields: a *string* type field called “key”, *link to form* field called “Order Dedupe” and a script field which counts the entries in the order. The link to form field is configured as a JOIN type on the “key” field of the dedupe form and the calculation field in the original form. The script field looks like this (change your ID’s to match):
function Key_Match_Count() {
var order_dedupe = record.getFieldValue('fld-fde68e7d2b384cb2a4452d3ae66bbab1');
return order_dedupe.length;
}
Key_Match_Count();
In this form also create a new saved search that uses the script field and is set to look for values greater than one as those will be the duplicates.
Last step is to populate this form, go back to your base form and create a new form script. I wrote the script below to scan each record, use an md5sum implementation to create a hash of the key field and then look to see if that record exists in the dedupe form:
document.getFormNamed("Script Manager").runScriptNamed("md5sum");
var purchase_key_id = 'fld-3e49aaa5bc32429c8f0f0f234878356d';
var dedupfield__key_id = 'fld-c52906ee940142a0a54fac1a98346afd';
var dedupForm = document.getFormNamed("Order Dedupe");
function Extract_Purchase_Keys() {
for (let sourceRecord of form.getRecords()) {
let purchaseKey = sourceRecord.getFieldValue(purchase_key_id);
if (!purchaseKey) {
console.log("Missing purchase key for record: " + sourceRecord.getUrl());
continue;
}
let purchaseKeyHash = "rec-" + md5(purchaseKey);
let dedupRecord = dedupForm.getRecordWithId(purchaseKeyHash);
if (!dedupRecord) {
dedupRecord = dedupForm.addNewRecordWithId(purchaseKeyHash);
dedupRecord.setFieldValue(dedupfield__key_id, purchaseKey);
}
}
document.saveAllChanges();
}
Extract_Purchase_Keys();
This actually found me a dupe record that I hadn’t found in my orders form when I went back to look at the saved search. It’s a bit of a journey, might turn it into a video at some point when I get some more time.