Add external import sources, work history, screening fields and Phase 1 docs
- New ExternalImportSource and ExternalWorkHistoryEntry models, controller and migrations - Screening fields on external profile imports - Admin and provider profile views, routes and workflow tests updated - Phase 1 validation, route inventory, gap report template and roadmap docs - Add the Google Sheets work-order parser tool (tools/google-sheets-wo-parser)
This commit is contained in:
Codex
@@ -0,0 +1,4 @@
|
||||
.clasp.json
|
||||
.clasprc.json
|
||||
.DS_Store
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2026 Garrison Tech Services
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
# Privacy and safety
|
||||
|
||||
## Data boundary
|
||||
|
||||
This project is designed so each user has their own Google Sheet and their own bound Apps Script project. There is no shared inbox, central database, hosted API, or telemetry.
|
||||
|
||||
The parser searches recent Inbox messages using Field Nation and WorkMarket sender patterns, then reads only the matching messages. It writes parsed work-order fields and the Gmail message ID to that user's Sheet. It does **not** write full email bodies to the Sheet by default.
|
||||
|
||||
## Google permission scope
|
||||
|
||||
Apps Script's `GmailApp` service requires Gmail authorization. Google presents this as broad Gmail access; the script cannot make Google enforce a single-label permission. That is why this project must stay open source and why users should inspect it before authorizing.
|
||||
|
||||
The code limits its own search to recent Inbox messages matching Field Nation or WorkMarket sender patterns and performs no outbound HTTP requests. It never sends, forwards, deletes, marks read, requests, accepts, or declines any work order.
|
||||
|
||||
## Safe use
|
||||
|
||||
- The default automatic scan does not inspect unrelated sender patterns. A Gmail label or filter is optional if you want additional organization.
|
||||
- Keep the Sheet private because it contains the work-order fields extracted from matching notices.
|
||||
- Treat the Sheet as private operational data and do not share it publicly.
|
||||
- Review the `Needs Review` queue after a provider changes an email template.
|
||||
- Never paste passwords, API keys, OAuth tokens, or full private emails into a bug report.
|
||||
|
||||
## Platform boundary
|
||||
|
||||
This parser acts only on emails the user already receives. It does not automate Field Nation or WorkMarket web sessions and does not circumvent their access controls.
|
||||
@@ -0,0 +1,49 @@
|
||||
# Google Sheets Gmail Work-Order Parser
|
||||
|
||||
A free, open-source Google Sheets + Apps Script tool that automatically scans recent Inbox messages from **Field Nation and WorkMarket** and turns their work-order notifications into a private spreadsheet.
|
||||
|
||||
It is intentionally a personal email parser. It does not scrape either platform, sign in to either platform, submit requests, accept work, or share your data with a central service.
|
||||
|
||||
## What it does
|
||||
|
||||
- Checks recent Inbox messages from Field Nation and WorkMarket every five minutes; no Gmail label or filter is required.
|
||||
- Parses Field Nation and WorkMarket notification emails into a normalized archive.
|
||||
- Shows a compact **New WOs** queue, plus views sorted by work date, post time, and earnings before tax.
|
||||
- Preserves the direct work-order link when an email includes one.
|
||||
- Detects duplicates by Gmail message ID.
|
||||
- Routes changed or unrecognized email formats to **Needs Review** instead of silently guessing.
|
||||
|
||||
## What it does not do
|
||||
|
||||
- It does not make work-order decisions for you.
|
||||
- It does not automatically request, accept, decline, or alter a work order.
|
||||
- It does not replace the Field Nation or WorkMarket apps.
|
||||
- It does not collect email addresses, work orders, analytics, or telemetry for anyone else.
|
||||
|
||||
## Install
|
||||
|
||||
See [SETUP.md](SETUP.md). A normal recipient should only need to make a copy of the template and open `WO Parser > Set up / repair workbook`.
|
||||
|
||||
## Privacy and permissions
|
||||
|
||||
See [PRIVACY.md](PRIVACY.md). Apps Script's Gmail service requires Gmail authorization. The source is public so users can see that the script searches recent Inbox mail only for Field Nation and WorkMarket sender patterns and never sends email contents elsewhere.
|
||||
|
||||
## Development
|
||||
|
||||
The Apps Script files are under `src/`. They use no external libraries.
|
||||
|
||||
Run parser fixture checks with a current Node.js installation:
|
||||
|
||||
```powershell
|
||||
node tests/run-tests.cjs
|
||||
```
|
||||
|
||||
The fixtures are fabricated examples. Before calling a format supported, add a redacted real-world fixture for that email variation.
|
||||
|
||||
## Giving it away
|
||||
|
||||
Publish a viewer-only Google Sheet template with a make-a-copy link and this repository. Each recipient owns their copied Sheet, bound script, Gmail authorization, and trigger.
|
||||
|
||||
## License
|
||||
|
||||
[MIT](LICENSE). Unofficial and not affiliated with Field Nation or WorkMarket.
|
||||
@@ -0,0 +1,28 @@
|
||||
# Setup and giveaway guide
|
||||
|
||||
## User setup
|
||||
|
||||
1. Make a copy of the published Google Sheet template into your own Google Drive.
|
||||
2. Open the copy on a desktop browser.
|
||||
3. Select **WO Parser > Set up / repair workbook**.
|
||||
4. Review and approve the Google authorization screen.
|
||||
5. The parser automatically finds recent Inbox messages from Field Nation and WorkMarket. No Gmail label or filter is needed.
|
||||
6. Return to the Sheet and select **WO Parser > Check Gmail now** to verify it works immediately.
|
||||
7. Open **New WOs** on the Google Sheets mobile app when you want the optional spreadsheet view.
|
||||
|
||||
The installer creates the labels `WO Parser/Processed` and `WO Parser/Needs Review`, plus a five-minute trigger. It scans only recent Inbox messages from Field Nation and WorkMarket senders; it does not delete or mark source mail read.
|
||||
|
||||
## Template owner setup
|
||||
|
||||
1. Create a clean Google Sheet under the project account.
|
||||
2. Open **Extensions > Apps Script** and replace the default file with `src/Config.gs`, `src/Parsers.gs`, and `src/Code.gs` from this repository. Set the manifest from `appsscript.json`.
|
||||
3. Reload the Sheet, run **WO Parser > Set up / repair workbook**, and use fabricated/redacted test messages from Field Nation or WorkMarket sender patterns in a test Inbox.
|
||||
4. Share the finished sheet as **Anyone with the link > Viewer**. Do not give strangers edit access to the template itself.
|
||||
5. Share a make-a-copy URL by replacing the final `/edit` portion of the sheet URL with `/copy`.
|
||||
6. Publish this repository beside the template link so recipients can inspect the code.
|
||||
|
||||
Each recipient must run Setup in their own copy. Triggers belong to the Google account that creates them and do not make the template owner a reader of anyone else's Gmail.
|
||||
|
||||
## Updating a released template
|
||||
|
||||
Do not silently download or execute remote code in a user's Sheet. Publish a version number and release notes. For a breaking update, publish a new template and provide an import/export migration path for the old archive.
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"timeZone": "America/Chicago",
|
||||
"exceptionLogging": "STACKDRIVER",
|
||||
"runtimeVersion": "V8",
|
||||
"oauthScopes": [
|
||||
"https://www.googleapis.com/auth/spreadsheets",
|
||||
"https://mail.google.com/",
|
||||
"https://www.googleapis.com/auth/script.scriptapp"
|
||||
]
|
||||
}
|
||||
|
||||
@@ -0,0 +1,275 @@
|
||||
/** Entry points, Gmail intake, and spreadsheet views. */
|
||||
function onOpen() {
|
||||
SpreadsheetApp.getUi()
|
||||
.createMenu(WO_PARSER.MENU_NAME)
|
||||
.addItem('Set up / repair workbook', 'setupWorkOrderParser')
|
||||
.addItem('Check Gmail now', 'runWorkOrderParser')
|
||||
.addItem('Refresh views', 'refreshWorkOrderViews')
|
||||
.addSeparator()
|
||||
.addItem('Show setup status', 'showWorkOrderParserStatus')
|
||||
.addToUi();
|
||||
}
|
||||
|
||||
function setupWorkOrderParser() {
|
||||
const spreadsheet = SpreadsheetApp.getActive();
|
||||
if (isPublicTemplate_(spreadsheet)) {
|
||||
showPublicTemplateCopyRequired_();
|
||||
return;
|
||||
}
|
||||
ensureWorkbook_(spreadsheet);
|
||||
ensureGmailLabel_(WO_PARSER.PROCESSED_LABEL);
|
||||
ensureGmailLabel_(WO_PARSER.REVIEW_LABEL);
|
||||
installParserTrigger_();
|
||||
showSetupCompleteMessage_();
|
||||
}
|
||||
|
||||
function runWorkOrderParser() {
|
||||
const spreadsheet = SpreadsheetApp.getActive();
|
||||
if (isPublicTemplate_(spreadsheet)) return;
|
||||
const lock = LockService.getDocumentLock();
|
||||
if (!lock.tryLock(5000)) return;
|
||||
|
||||
try {
|
||||
ensureWorkbook_(spreadsheet);
|
||||
const settings = readSettings_(spreadsheet);
|
||||
const processedLabel = ensureGmailLabel_(WO_PARSER.PROCESSED_LABEL);
|
||||
const reviewLabel = ensureGmailLabel_(WO_PARSER.REVIEW_LABEL);
|
||||
const query = buildPlatformInboxQuery_(settings, processedLabel);
|
||||
const threads = GmailApp.search(query, 0, settings.batchSize);
|
||||
const allSheet = spreadsheet.getSheetByName(WO_PARSER.SHEETS.ALL);
|
||||
const reviewSheet = spreadsheet.getSheetByName(WO_PARSER.SHEETS.REVIEW);
|
||||
const existingIds = readExistingMessageIds_(allSheet);
|
||||
const rowsToAppend = [];
|
||||
const reviewRows = [];
|
||||
|
||||
threads.forEach(function(thread) {
|
||||
thread.getMessages().forEach(function(message) {
|
||||
if (message.getLabels().some(function(label) { return label.getName() === processedLabel.getName(); })) return;
|
||||
if (!isSupportedPlatformSender_(message.getFrom())) return;
|
||||
const messageData = {
|
||||
id: message.getId(),
|
||||
from: message.getFrom(),
|
||||
subject: message.getSubject(),
|
||||
date: message.getDate(),
|
||||
plainBody: message.getPlainBody(),
|
||||
htmlBody: message.getBody()
|
||||
};
|
||||
|
||||
if (existingIds[messageData.id]) {
|
||||
message.addLabel(processedLabel);
|
||||
return;
|
||||
}
|
||||
|
||||
const result = parseWorkOrderMessage_(messageData);
|
||||
if (result.ok) {
|
||||
rowsToAppend.push(workOrderRecordToRow_(result.record));
|
||||
existingIds[messageData.id] = true;
|
||||
message.addLabel(processedLabel);
|
||||
} else {
|
||||
reviewRows.push([messageData.date, messageData.from, messageData.subject, messageData.id, result.reason, WO_PARSER.PARSER_VERSION]);
|
||||
message.addLabel(reviewLabel);
|
||||
message.addLabel(processedLabel);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
appendRows_(allSheet, rowsToAppend);
|
||||
appendRows_(reviewSheet, reviewRows);
|
||||
refreshWorkOrderViews();
|
||||
updateStatus_(spreadsheet, rowsToAppend.length, reviewRows.length, 'OK');
|
||||
} catch (error) {
|
||||
updateStatus_(SpreadsheetApp.getActive(), 0, 0, 'ERROR: ' + error.message);
|
||||
throw error;
|
||||
} finally {
|
||||
lock.releaseLock();
|
||||
}
|
||||
}
|
||||
|
||||
function refreshWorkOrderViews() {
|
||||
const spreadsheet = SpreadsheetApp.getActive();
|
||||
ensureWorkbook_(spreadsheet);
|
||||
const allSheet = spreadsheet.getSheetByName(WO_PARSER.SHEETS.ALL);
|
||||
const values = allSheet.getDataRange().getValues();
|
||||
const header = values.shift();
|
||||
const records = values.filter(function(row) { return row[20]; });
|
||||
const unseen = records.filter(function(row) { return row[0] !== true && row[2] !== 'Canceled'; });
|
||||
|
||||
writeView_(spreadsheet.getSheetByName(WO_PARSER.SHEETS.NEW), header, unseen, 3, false);
|
||||
writeView_(spreadsheet.getSheetByName(WO_PARSER.SHEETS.BY_DATE), header, records, 11, false);
|
||||
writeView_(spreadsheet.getSheetByName(WO_PARSER.SHEETS.BY_POSTED), header, records, 3, true);
|
||||
writeView_(spreadsheet.getSheetByName(WO_PARSER.SHEETS.BY_EBT), header, records, 7, true);
|
||||
}
|
||||
|
||||
function showWorkOrderParserStatus() {
|
||||
const spreadsheet = SpreadsheetApp.getActive();
|
||||
const settings = readSettings_(spreadsheet);
|
||||
SpreadsheetApp.getUi().alert(
|
||||
'WO Parser status',
|
||||
'Version: ' + WO_PARSER.VERSION + '\nInbox scan: Field Nation and WorkMarket senders\nLast run: ' + (settings.lastRun || 'Never') + '\nLast result: ' + (settings.lastResult || 'Not run yet'),
|
||||
SpreadsheetApp.getUi().ButtonSet.OK
|
||||
);
|
||||
}
|
||||
|
||||
function showSetupCompleteMessage_() {
|
||||
const message = 'The parser automatically checks recent Inbox messages from Field Nation and WorkMarket every five minutes. You can also choose "Check Gmail now" from the WO Parser menu.';
|
||||
try {
|
||||
SpreadsheetApp.getUi().alert('Setup complete', message, SpreadsheetApp.getUi().ButtonSet.OK);
|
||||
} catch (error) {
|
||||
Logger.log('Setup complete. ' + message);
|
||||
}
|
||||
}
|
||||
|
||||
function isPublicTemplate_(spreadsheet) {
|
||||
return spreadsheet && spreadsheet.getId() === WO_PARSER.TEMPLATE_SPREADSHEET_ID;
|
||||
}
|
||||
|
||||
function showPublicTemplateCopyRequired_() {
|
||||
const message = 'This is the public template. Make your own private copy before running setup; the public template never scans the template owner\'s Gmail.';
|
||||
try {
|
||||
SpreadsheetApp.getUi().alert('Make a private copy first', message, SpreadsheetApp.getUi().ButtonSet.OK);
|
||||
} catch (error) {
|
||||
Logger.log(message);
|
||||
}
|
||||
}
|
||||
|
||||
function ensureWorkbook_(spreadsheet) {
|
||||
ensureSheet_(spreadsheet, WO_PARSER.SHEETS.README, ['Google Sheets Gmail Work-Order Parser']);
|
||||
ensureSheet_(spreadsheet, WO_PARSER.SHEETS.SETTINGS, ['Setting', 'Value', 'Notes']);
|
||||
ensureSheet_(spreadsheet, WO_PARSER.SHEETS.NEW, WO_HEADERS);
|
||||
ensureSheet_(spreadsheet, WO_PARSER.SHEETS.ALL, WO_HEADERS);
|
||||
ensureSheet_(spreadsheet, WO_PARSER.SHEETS.BY_DATE, WO_HEADERS);
|
||||
ensureSheet_(spreadsheet, WO_PARSER.SHEETS.BY_POSTED, WO_HEADERS);
|
||||
ensureSheet_(spreadsheet, WO_PARSER.SHEETS.BY_EBT, WO_HEADERS);
|
||||
ensureSheet_(spreadsheet, WO_PARSER.SHEETS.REVIEW, ['Received At', 'From', 'Subject', 'Gmail Message ID', 'Reason', 'Parser Version']);
|
||||
seedReadMe_(spreadsheet.getSheetByName(WO_PARSER.SHEETS.README));
|
||||
seedSettings_(spreadsheet.getSheetByName(WO_PARSER.SHEETS.SETTINGS));
|
||||
}
|
||||
|
||||
function ensureSheet_(spreadsheet, name, headers) {
|
||||
let sheet = spreadsheet.getSheetByName(name);
|
||||
if (!sheet) sheet = spreadsheet.insertSheet(name);
|
||||
if (sheet.getLastRow() === 0) sheet.appendRow(headers);
|
||||
const range = sheet.getRange(1, 1, 1, headers.length);
|
||||
range.setValues([headers]);
|
||||
range.setFontWeight('bold').setBackground('#d9ead3');
|
||||
sheet.setFrozenRows(1);
|
||||
return sheet;
|
||||
}
|
||||
|
||||
function seedReadMe_(sheet) {
|
||||
const instructions = [
|
||||
['Free, unofficial parser for Field Nation and WorkMarket notification emails.'],
|
||||
['1. Run WO Parser > Set up / repair workbook.'],
|
||||
['2. The script automatically checks recent Inbox messages from Field Nation and WorkMarket.'],
|
||||
['3. It checks every five minutes and ignores other senders.'],
|
||||
['4. Read New WOs for un-seen, non-canceled entries.'],
|
||||
['5. Template changes are safely routed to Needs Review instead of guessed.'],
|
||||
['The script does not request, accept, decline, delete, or send work orders.'],
|
||||
['This Sheet is private to its owner. Share it only with people you trust.'],
|
||||
['See the repository README for setup, privacy, and troubleshooting.']
|
||||
];
|
||||
sheet.getRange(2, 1, instructions.length, 1).setValues(instructions);
|
||||
sheet.setColumnWidth(1, 760);
|
||||
sheet.getRange(1, 1).setFontSize(14);
|
||||
}
|
||||
|
||||
function seedSettings_(sheet) {
|
||||
const existing = sheet.getRange(1, 1, Math.max(sheet.getLastRow(), 1), 2).getValues();
|
||||
const existingKeys = {};
|
||||
existing.forEach(function(row) { existingKeys[row[0]] = true; });
|
||||
const inboxScopeRow = ['inboxScope', 'Recent Inbox messages from Field Nation and WorkMarket', 'Automatic sender-only scan; no inbox label is required.'];
|
||||
const inboxScopeIndex = existing.findIndex(function(row) { return row[0] === 'inboxScope'; });
|
||||
const legacyLabelIndex = existing.findIndex(function(row) { return row[0] === 'gmailLabel'; });
|
||||
|
||||
if (inboxScopeIndex >= 0) {
|
||||
sheet.getRange(inboxScopeIndex + 1, 1, 1, 3).setValues([inboxScopeRow]);
|
||||
} else if (legacyLabelIndex >= 0) {
|
||||
sheet.getRange(legacyLabelIndex + 1, 1, 1, 3).setValues([inboxScopeRow]);
|
||||
} else {
|
||||
sheet.appendRow(inboxScopeRow);
|
||||
}
|
||||
|
||||
const defaults = [
|
||||
['lookbackDays', WO_PARSER.DEFAULT_LOOKBACK_DAYS, 'Safety window for new messages.'],
|
||||
['batchSize', WO_PARSER.DEFAULT_BATCH_SIZE, 'Maximum Gmail threads checked per run.'],
|
||||
['homeLocation', '', 'Optional future use for distance calculations.'],
|
||||
['lastRun', '', 'Updated automatically.'],
|
||||
['lastResult', '', 'Updated automatically.']
|
||||
];
|
||||
const missing = defaults.filter(function(row) { return !existingKeys[row[0]]; });
|
||||
appendRows_(sheet, missing);
|
||||
sheet.autoResizeColumns(1, 3);
|
||||
}
|
||||
|
||||
function readSettings_(spreadsheet) {
|
||||
const sheet = spreadsheet.getSheetByName(WO_PARSER.SHEETS.SETTINGS);
|
||||
const rows = sheet.getDataRange().getValues();
|
||||
const values = {};
|
||||
rows.slice(1).forEach(function(row) { values[row[0]] = row[1]; });
|
||||
return {
|
||||
lookbackDays: Math.max(1, Number(values.lookbackDays || WO_PARSER.DEFAULT_LOOKBACK_DAYS)),
|
||||
batchSize: Math.max(1, Math.min(100, Number(values.batchSize || WO_PARSER.DEFAULT_BATCH_SIZE))),
|
||||
lastRun: values.lastRun || '',
|
||||
lastResult: values.lastResult || ''
|
||||
};
|
||||
}
|
||||
|
||||
function buildPlatformInboxQuery_(settings, processedLabel) {
|
||||
return 'in:inbox ' + WO_PARSER.PLATFORM_INBOX_QUERY + ' -label:"' + processedLabel.getName() + '" newer_than:' + settings.lookbackDays + 'd';
|
||||
}
|
||||
|
||||
function isSupportedPlatformSender_(from) {
|
||||
return /field\s*nation|fieldnation\.com|work\s*market|workmarket\.com/i.test(String(from || ''));
|
||||
}
|
||||
|
||||
function updateStatus_(spreadsheet, parsed, reviewed, result) {
|
||||
if (!spreadsheet) return;
|
||||
const sheet = spreadsheet.getSheetByName(WO_PARSER.SHEETS.SETTINGS);
|
||||
if (!sheet) return;
|
||||
const rows = sheet.getDataRange().getValues();
|
||||
rows.slice(1).forEach(function(row, index) {
|
||||
if (row[0] === 'lastRun') sheet.getRange(index + 2, 2).setValue(new Date());
|
||||
if (row[0] === 'lastResult') sheet.getRange(index + 2, 2).setValue(result + ' - parsed: ' + parsed + ', review: ' + reviewed);
|
||||
});
|
||||
}
|
||||
|
||||
function ensureGmailLabel_(name) {
|
||||
return GmailApp.getUserLabelByName(name) || GmailApp.createLabel(name);
|
||||
}
|
||||
|
||||
function installParserTrigger_() {
|
||||
ScriptApp.getProjectTriggers().forEach(function(trigger) {
|
||||
if (trigger.getHandlerFunction() === 'runWorkOrderParser') ScriptApp.deleteTrigger(trigger);
|
||||
});
|
||||
ScriptApp.newTrigger('runWorkOrderParser').timeBased().everyMinutes(5).create();
|
||||
}
|
||||
|
||||
function readExistingMessageIds_(sheet) {
|
||||
const ids = {};
|
||||
if (sheet.getLastRow() < 2) return ids;
|
||||
sheet.getRange(2, 21, sheet.getLastRow() - 1, 1).getValues().forEach(function(row) {
|
||||
if (row[0]) ids[row[0]] = true;
|
||||
});
|
||||
return ids;
|
||||
}
|
||||
|
||||
function appendRows_(sheet, rows) {
|
||||
if (!rows || !rows.length) return;
|
||||
sheet.getRange(sheet.getLastRow() + 1, 1, rows.length, rows[0].length).setValues(rows);
|
||||
}
|
||||
|
||||
function writeView_(sheet, header, records, sortColumn, descending) {
|
||||
const sorted = records.slice().sort(function(a, b) {
|
||||
const left = a[sortColumn] || '';
|
||||
const right = b[sortColumn] || '';
|
||||
if (left === right) return 0;
|
||||
const comparison = left > right ? 1 : -1;
|
||||
return descending ? -comparison : comparison;
|
||||
});
|
||||
sheet.clearContents();
|
||||
sheet.getRange(1, 1, 1, header.length).setValues([header]);
|
||||
if (sorted.length) sheet.getRange(2, 1, sorted.length, header.length).setValues(sorted);
|
||||
sheet.getRange(1, 1, 1, header.length).setFontWeight('bold').setBackground('#d9ead3');
|
||||
sheet.setFrozenRows(1);
|
||||
sheet.autoResizeColumns(1, header.length);
|
||||
}
|
||||
@@ -0,0 +1,55 @@
|
||||
/**
|
||||
* Google Sheets Gmail Work-Order Parser
|
||||
*
|
||||
* This file contains user-visible defaults only. Keep API keys, passwords,
|
||||
* and full email bodies out of the spreadsheet and this source tree.
|
||||
*/
|
||||
const WO_PARSER = Object.freeze({
|
||||
NAME: 'Google Sheets Gmail Work-Order Parser',
|
||||
VERSION: '0.2.0',
|
||||
TEMPLATE_SPREADSHEET_ID: '1xIJXDhNmp5ZLuoJpGkYkTjh9MIapUdpUpPFQD2rbkj4',
|
||||
PLATFORM_INBOX_QUERY: '{from:fieldnation from:workmarket}',
|
||||
PROCESSED_LABEL: 'WO Parser/Processed',
|
||||
REVIEW_LABEL: 'WO Parser/Needs Review',
|
||||
DEFAULT_LOOKBACK_DAYS: 7,
|
||||
DEFAULT_BATCH_SIZE: 50,
|
||||
MENU_NAME: 'WO Parser',
|
||||
SHEETS: Object.freeze({
|
||||
README: 'Read Me',
|
||||
SETTINGS: 'Settings',
|
||||
NEW: 'New WOs',
|
||||
ALL: 'All Parsed',
|
||||
BY_DATE: 'WO Date',
|
||||
BY_POSTED: 'WO Posted Date',
|
||||
BY_EBT: 'WO EBT',
|
||||
REVIEW: 'Needs Review'
|
||||
}),
|
||||
PARSER_VERSION: '0.2.0'
|
||||
});
|
||||
|
||||
const WO_HEADERS = Object.freeze([
|
||||
'Seen',
|
||||
'Platform',
|
||||
'Notification Type',
|
||||
'Posted At',
|
||||
'Work Order ID',
|
||||
'Buyer',
|
||||
'Title',
|
||||
'Pay Total',
|
||||
'Pay Type',
|
||||
'Estimated Hours',
|
||||
'Effective Rate',
|
||||
'Scheduled Start',
|
||||
'Scheduled End',
|
||||
'Address',
|
||||
'City',
|
||||
'State',
|
||||
'ZIP',
|
||||
'Distance (mi)',
|
||||
'Schedule Conflict',
|
||||
'Direct WO Link',
|
||||
'Source Message ID',
|
||||
'Source Subject',
|
||||
'Parser Version',
|
||||
'Imported At'
|
||||
]);
|
||||
@@ -0,0 +1,212 @@
|
||||
/**
|
||||
* Deterministic parsers for notification emails. They deliberately return a
|
||||
* reviewable failure instead of inventing data when a provider changes a
|
||||
* template. The sample patterns are intentionally broad; add a tested pattern
|
||||
* only after placing a redacted example in tests/fixtures.
|
||||
*/
|
||||
function parseWorkOrderMessage_(messageData) {
|
||||
const text = normalizeEmailText_(messageData.plainBody || messageData.htmlBody || '');
|
||||
const context = [messageData.from || '', messageData.subject || '', text].join('\n');
|
||||
const platform = detectPlatform_(context);
|
||||
|
||||
if (!platform) {
|
||||
return parseFailure_('This message does not look like a Field Nation or WorkMarket notification.');
|
||||
}
|
||||
|
||||
const workOrderId = extractWorkOrderId_(text) || extractWorkOrderId_(messageData.subject || '');
|
||||
const directLink = extractDirectLink_(text, platform);
|
||||
const title = extractTitle_(messageData.subject || '', text, platform);
|
||||
const location = extractLocation_(text);
|
||||
const pay = extractPay_(text);
|
||||
const schedule = extractSchedule_(text);
|
||||
const notificationType = detectNotificationType_(context);
|
||||
|
||||
if (!workOrderId && !directLink) {
|
||||
return parseFailure_(platform + ' notification did not contain a recognizable work-order ID or link.');
|
||||
}
|
||||
|
||||
return {
|
||||
ok: true,
|
||||
record: {
|
||||
seen: false,
|
||||
platform: platform,
|
||||
notificationType: notificationType,
|
||||
postedAt: messageData.date || new Date(),
|
||||
workOrderId: workOrderId || '',
|
||||
buyer: extractLabeledValue_(text, ['buyer', 'client', 'company']) || '',
|
||||
title: title,
|
||||
payTotal: pay.total,
|
||||
payType: pay.type,
|
||||
estimatedHours: pay.hours,
|
||||
effectiveRate: pay.effectiveRate,
|
||||
scheduledStart: schedule.start,
|
||||
scheduledEnd: schedule.end,
|
||||
address: location.address,
|
||||
city: location.city,
|
||||
state: location.state,
|
||||
zip: location.zip,
|
||||
distance: '',
|
||||
scheduleConflict: '',
|
||||
directLink: directLink,
|
||||
sourceMessageId: messageData.id,
|
||||
sourceSubject: messageData.subject || '',
|
||||
parserVersion: WO_PARSER.PARSER_VERSION,
|
||||
importedAt: new Date()
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
function parseFailure_(reason) {
|
||||
return { ok: false, reason: reason };
|
||||
}
|
||||
|
||||
function normalizeEmailText_(input) {
|
||||
return String(input)
|
||||
.replace(/<style[\s\S]*?<\/style>/gi, ' ')
|
||||
.replace(/<script[\s\S]*?<\/script>/gi, ' ')
|
||||
.replace(/<br\s*\/?>/gi, '\n')
|
||||
.replace(/<\/p>/gi, '\n')
|
||||
.replace(/<[^>]+>/g, ' ')
|
||||
.replace(/ /gi, ' ')
|
||||
.replace(/&/gi, '&')
|
||||
.replace(/'/g, "'")
|
||||
.replace(/\r/g, '')
|
||||
.replace(/[ \t]+/g, ' ')
|
||||
.replace(/\n{3,}/g, '\n\n')
|
||||
.trim();
|
||||
}
|
||||
|
||||
function detectPlatform_(text) {
|
||||
if (/field\s*nation|fieldnation\.com/i.test(text)) {
|
||||
return 'Field Nation';
|
||||
}
|
||||
if (/work\s*market|workmarket\.com/i.test(text)) {
|
||||
return 'WorkMarket';
|
||||
}
|
||||
return '';
|
||||
}
|
||||
|
||||
function detectNotificationType_(text) {
|
||||
if (/\bassigned\b|you are assigned/i.test(text)) return 'Assigned';
|
||||
if (/\brouted\b|direct opportunity/i.test(text)) return 'Routed';
|
||||
if (/\bcancel(?:ed|led)\b|\bvoid(?:ed)?\b/i.test(text)) return 'Canceled';
|
||||
if (/\bupdated\b|\bchanged\b|\brescheduled\b/i.test(text)) return 'Updated';
|
||||
return 'New';
|
||||
}
|
||||
|
||||
function extractWorkOrderId_(text) {
|
||||
const patterns = [
|
||||
/(?:work\s*order|workorder|\bWO)[ \t]*(?:number|no\.?|id)?[ \t]*[:#-]?[ \t]*((?=[A-Za-z0-9-]*\d)[A-Za-z0-9-]{5,})/i,
|
||||
/(?:order|assignment)[ \t]*#[ \t]*((?=[A-Za-z0-9-]*\d)[A-Za-z0-9-]{5,})/i
|
||||
];
|
||||
for (let i = 0; i < patterns.length; i += 1) {
|
||||
const match = text.match(patterns[i]);
|
||||
if (match) return match[1];
|
||||
}
|
||||
return '';
|
||||
}
|
||||
|
||||
function extractDirectLink_(text, platform) {
|
||||
const urls = text.match(/https?:\/\/[^\s<>"']+/gi) || [];
|
||||
const platformPattern = platform === 'Field Nation' ? /fieldnation\.com/i : /workmarket\.com/i;
|
||||
for (let i = 0; i < urls.length; i += 1) {
|
||||
const cleaned = urls[i].replace(/[).,;]+$/, '');
|
||||
if (platformPattern.test(cleaned)) return cleaned;
|
||||
}
|
||||
return '';
|
||||
}
|
||||
|
||||
function extractTitle_(subject, text, platform) {
|
||||
const labeled = extractLabeledValue_(text, ['work order title', 'job title', 'title', 'work description']);
|
||||
if (labeled) return labeled;
|
||||
|
||||
const subjectWithoutPlatform = subject
|
||||
.replace(/field\s*nation|work\s*market/gi, '')
|
||||
.replace(/(?:new|routed|assigned|available)\s*(?:work\s*order|work|wo)?\s*:?/gi, '')
|
||||
.replace(/^[-:|\s]+|[-:|\s]+$/g, '')
|
||||
.trim();
|
||||
return subjectWithoutPlatform || platform + ' work order';
|
||||
}
|
||||
|
||||
function extractLabeledValue_(text, labels) {
|
||||
for (let i = 0; i < labels.length; i += 1) {
|
||||
const escaped = labels[i].replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
|
||||
const expression = new RegExp('(?:^|\\n)\\s*' + escaped + '\\s*[:\\-]\\s*([^\\n]{1,180})', 'i');
|
||||
const match = text.match(expression);
|
||||
if (match) return match[1].trim();
|
||||
}
|
||||
return '';
|
||||
}
|
||||
|
||||
function extractPay_(text) {
|
||||
const totalMatch = text.match(/(?:total\s*(?:pay|amount|compensation)|pay\s*total|not\s*to\s*exceed|NTE)\s*[:$\s]*\$?\s*([\d,]+(?:\.\d{1,2})?)/i) ||
|
||||
text.match(/\$\s*([\d,]+(?:\.\d{1,2})?)/);
|
||||
const typeMatch = text.match(/\b(hourly|fixed(?:\s*rate)?|flat(?:\s*rate)?|blended(?:\s*rate)?)\b/i);
|
||||
const hoursMatch = text.match(/(?:estimated\s*)?(\d+(?:\.\d+)?)\s*(?:hours?|hrs?)\b/i);
|
||||
const rateMatch = text.match(/\$\s*([\d,]+(?:\.\d{1,2})?)\s*(?:\/|per\s*)(?:hour|hr)/i);
|
||||
const total = totalMatch ? Number(String(totalMatch[1]).replace(/,/g, '')) : '';
|
||||
const hours = hoursMatch ? Number(hoursMatch[1]) : '';
|
||||
const effectiveRate = rateMatch ? Number(String(rateMatch[1]).replace(/,/g, '')) :
|
||||
(total && hours ? Math.round((total / hours) * 100) / 100 : '');
|
||||
return {
|
||||
total: total,
|
||||
type: typeMatch ? normalizePayType_(typeMatch[1]) : '',
|
||||
hours: hours,
|
||||
effectiveRate: effectiveRate
|
||||
};
|
||||
}
|
||||
|
||||
function normalizePayType_(value) {
|
||||
const normalized = String(value).toLowerCase();
|
||||
if (normalized.indexOf('hour') !== -1) return 'Hourly Rate';
|
||||
if (normalized.indexOf('blend') !== -1) return 'Blended Rate';
|
||||
return 'Fixed Rate';
|
||||
}
|
||||
|
||||
function extractSchedule_(text) {
|
||||
const labeled = extractLabeledValue_(text, ['start date and time', 'scheduled date', 'scheduled time', 'service date', 'date and time']);
|
||||
if (labeled) return { start: labeled, end: '' };
|
||||
const range = text.match(/\b((?:mon|tue|wed|thu|fri|sat|sun)[a-z]*,?\s+[A-Z][a-z]{2,8}\s+\d{1,2}[^\n]{0,90})/i);
|
||||
return { start: range ? range[1].trim() : '', end: '' };
|
||||
}
|
||||
|
||||
function extractLocation_(text) {
|
||||
const zipMatch = text.match(/\b(\d{5}(?:-\d{4})?)\b/);
|
||||
const cityStateMatch = text.match(/\b([A-Za-z][A-Za-z .'-]{1,60}),\s*([A-Z]{2})\s+(\d{5}(?:-\d{4})?)\b/);
|
||||
const address = extractLabeledValue_(text, ['location', 'address', 'service location', 'work location']);
|
||||
return {
|
||||
address: address || (cityStateMatch ? cityStateMatch[0] : ''),
|
||||
city: cityStateMatch ? cityStateMatch[1].trim() : '',
|
||||
state: cityStateMatch ? cityStateMatch[2] : '',
|
||||
zip: cityStateMatch ? cityStateMatch[3] : (zipMatch ? zipMatch[1] : '')
|
||||
};
|
||||
}
|
||||
|
||||
function workOrderRecordToRow_(record) {
|
||||
return [
|
||||
record.seen,
|
||||
record.platform,
|
||||
record.notificationType,
|
||||
record.postedAt,
|
||||
record.workOrderId,
|
||||
record.buyer,
|
||||
record.title,
|
||||
record.payTotal,
|
||||
record.payType,
|
||||
record.estimatedHours,
|
||||
record.effectiveRate,
|
||||
record.scheduledStart,
|
||||
record.scheduledEnd,
|
||||
record.address,
|
||||
record.city,
|
||||
record.state,
|
||||
record.zip,
|
||||
record.distance,
|
||||
record.scheduleConflict,
|
||||
record.directLink,
|
||||
record.sourceMessageId,
|
||||
record.sourceSubject,
|
||||
record.parserVersion,
|
||||
record.importedAt
|
||||
];
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
module.exports = {
|
||||
fieldNationNew: {
|
||||
id: 'field-nation-message-001',
|
||||
from: 'Field Nation <notifications@fieldnation.example>',
|
||||
subject: 'New Work Order: Network troubleshooting',
|
||||
date: new Date('2026-08-02T14:00:00Z'),
|
||||
plainBody: `Field Nation\nNew work order available\nWork Order #19660234\nTitle: Network troubleshooting\nBuyer: Example Field Services\nLocation: Raleigh, NC 27609\nService Date: Monday, Aug 3 @ 8:00 AM\nPay Total: $225.00\nPay Type: Hourly Rate\nEstimated 3 hours\nView work order: https://app.fieldnation.com/workorders/19660234`
|
||||
},
|
||||
workMarketRouted: {
|
||||
id: 'workmarket-message-002',
|
||||
from: 'WorkMarket <no-reply@workmarket.example>',
|
||||
subject: 'You have been routed a work order',
|
||||
date: new Date('2026-08-02T14:05:00Z'),
|
||||
plainBody: `WorkMarket\nYou have been routed a work order.\nWork Order ID: WM-884221\nJob Title: POS terminal swap\nCompany: Example Buyer LLC\nService Location: Durham, NC 27713\nScheduled Time: Tuesday, Aug 4 @ 10:00 AM\nTotal Pay: $180.00\nFixed Rate\nOpen assignment: https://www.workmarket.com/work-orders/WM-884221`
|
||||
},
|
||||
unknown: {
|
||||
id: 'unknown-message-003',
|
||||
from: 'Some Sender <example@example.com>',
|
||||
subject: 'Hello',
|
||||
date: new Date('2026-08-02T14:10:00Z'),
|
||||
plainBody: 'This is not a work-order notification.'
|
||||
}
|
||||
};
|
||||
@@ -0,0 +1,43 @@
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const vm = require('vm');
|
||||
const assert = require('assert/strict');
|
||||
const fixtures = require('./fixtures.cjs');
|
||||
|
||||
const context = { console, Date, Math, Number, String, Object, RegExp };
|
||||
vm.createContext(context);
|
||||
for (const file of ['Config.gs', 'Parsers.gs', 'Code.gs']) {
|
||||
const source = fs.readFileSync(path.join(__dirname, '..', 'src', file), 'utf8')
|
||||
.replace(/^const /gm, 'var ');
|
||||
vm.runInContext(source, context, { filename: file });
|
||||
}
|
||||
|
||||
const fieldNation = context.parseWorkOrderMessage_(fixtures.fieldNationNew);
|
||||
assert.equal(fieldNation.ok, true);
|
||||
assert.equal(fieldNation.record.platform, 'Field Nation');
|
||||
assert.equal(fieldNation.record.workOrderId, '19660234');
|
||||
assert.equal(fieldNation.record.payTotal, 225);
|
||||
assert.equal(fieldNation.record.effectiveRate, 75);
|
||||
assert.equal(fieldNation.record.city, 'Raleigh');
|
||||
|
||||
const workMarket = context.parseWorkOrderMessage_(fixtures.workMarketRouted);
|
||||
assert.equal(workMarket.ok, true);
|
||||
assert.equal(workMarket.record.platform, 'WorkMarket');
|
||||
assert.equal(workMarket.record.notificationType, 'Routed');
|
||||
assert.equal(workMarket.record.workOrderId, 'WM-884221');
|
||||
assert.equal(workMarket.record.payType, 'Fixed Rate');
|
||||
|
||||
const unknown = context.parseWorkOrderMessage_(fixtures.unknown);
|
||||
assert.equal(unknown.ok, false);
|
||||
|
||||
assert.equal(context.isSupportedPlatformSender_('Field Nation <alerts@fieldnation.com>'), true);
|
||||
assert.equal(context.isSupportedPlatformSender_('WorkMarket <no-reply@workmarket.com>'), true);
|
||||
assert.equal(context.isSupportedPlatformSender_('Unrelated sender <alerts@example.com>'), false);
|
||||
assert.equal(
|
||||
context.buildPlatformInboxQuery_({ lookbackDays: 7 }, { getName: () => 'WO Parser/Processed' }),
|
||||
'in:inbox {from:fieldnation from:workmarket} -label:"WO Parser/Processed" newer_than:7d'
|
||||
);
|
||||
assert.equal(context.isPublicTemplate_({ getId: () => context.WO_PARSER.TEMPLATE_SPREADSHEET_ID }), true);
|
||||
assert.equal(context.isPublicTemplate_({ getId: () => 'recipient-copy-id' }), false);
|
||||
|
||||
console.log('Parser fixtures passed.');
|
||||
Reference in New Issue
Block a user