technical-services-exchange/docs/URGENT_IMPORT_ROADMAP.md

Urgent Import Roadmap

This roadmap promotes Field Nation and WorkMarket history import into an urgent lane. It starts from the current codebase, which already supports manual imported-history records, proof attachments, visibility controls, and admin verification.

The goal is to let a provider pull their own marketplace history into Provider Exchange / Technical Services Exchange as quickly as possible without scraping private systems blindly, exposing customer data, or mixing imported history into native TSE reputation.

Current Import Baseline

Implemented today:

• external_profile_imports records.
• Provider-entered platform, external ID, profile URL, rating, review count, completed jobs, work categories, selected reviews, operational metrics, notes, visibility, and verification status.
• Proof attachments on imported-history records.
• Public/private visibility modes.
• Admin verification status.
• Public provider-profile display of imported history according to visibility.
• Tests for saving imports, visibility, and admin verification.

Missing:

• File/export upload intake. First pass complete for private source batches.
• Structured parser for exported profile data.
• Field Nation-specific import mapping.
• WorkMarket-specific import mapping.
• Review/work-history deduplication.
• Privacy redaction workflow for customer names, addresses, work-order IDs, and private buyer details.
• Provider preview/approval before imported data becomes public.
• Admin review queue focused on imported proof.
• Repeat import/update flow.
• Import audit trail and failure reporting.

Import Principles

• Providers may import their own records and proof.
• Do not automate login scraping until terms, privacy, and technical risks are explicitly reviewed.
• Prefer user-provided exports, screenshots, PDFs, CSVs, emails, or copied profile data first.
• Keep raw proof private by default.
• Never publish customer names, site addresses, buyer-private notes, work-order IDs, or private marketplace content by default.
• Keep imported history separate from native TSE reviews and completed work.
• Every transformed import should be previewed and approved by the provider before public display.
• Every admin verification action should be audited.

Phase I1 - Export Intake And Raw Proof Vault

Purpose:

Let a provider upload Field Nation or WorkMarket source material without needing a polished parser first.

Status:

First implementation pass added local/private source-batch storage, provider upload UI, admin visibility, audit logging, and feature tests.

Build:

• Import source upload screen under the provider profile. First pass complete.
• Supported source types. First pass complete:
  • PDF;
  • image;
  • CSV;
  • JSON;
  • HTML copy/export;
  • plain text paste;
  • email/message export pasted as text.
• external_import_sources table for raw import batches. Complete.
• Link source batches to external_profile_imports. Complete.
• Store original filename, MIME type, source platform, source type, uploaded by, uploaded at, and parse status. Complete.
• Keep raw source private and admin-visible only. Complete with private local storage by default.
• Add provider notes describing where the source came from. Complete.
• Add a "do not publish raw proof" default. Complete through private storage and needs_review status.

Acceptance:

• A provider can upload Field Nation proof today even before parsing is perfect.
• Raw proof is retained privately for review.
• The existing manual imported-history record can reference the source batch.

Phase I2 - Manual Review Workspace And Redaction

Purpose:

Make imported proof safe to use by separating private source material from public profile summaries.

Build:

• Provider import workspace showing uploaded source batches.
• Redaction checklist:
  • customer names;
  • addresses;
  • work-order IDs;
  • phone numbers;
  • emails;
  • private buyer instructions;
  • payment account details;
  • marketplace-private messages.
• Provider-selected public excerpts.
• Admin-visible private notes.
• Public summary preview.
• Redaction status:
  • needs review;
  • provider reviewed;
  • admin reviewed;
  • approved for public summary;
  • rejected for public display.

Acceptance:

• A provider can safely turn private proof into a public summary.
• The UI makes it hard to accidentally publish sensitive third-party details.
• Admins can see raw proof without making it public.

Phase I3 - Field Nation Parser First Pass

Purpose:

Extract useful provider-owned history from common Field Nation source material.

Build:

• Parser interface with platform-specific adapters.
• Field Nation adapter for provider-supplied source files.
• Extract when present:
  • provider profile ID or profile URL;
  • rating;
  • review count;
  • completed work count;
  • client count;
  • service categories;
  • review snippets;
  • dates or recency hints;
  • on-time or reliability metrics if present;
  • work summary categories if present.
• Store parsed output separately from approved/public output.
• Confidence flags for parsed fields:
  • high;
  • medium;
  • low;
  • needs manual confirmation.
• Parser error and warning display.
• First-pass text parser for copied/exported profile text. Implemented.
• Provider approval from parsed draft into imported-history summary. Implemented.

Acceptance:

• A provider can upload a Field Nation export/screenshot/PDF/text source and get a structured draft.
• The provider can edit or reject every parsed field before saving.
• Low-confidence fields never publish automatically.

Phase I4 - WorkMarket Parser First Pass

Purpose:

Add equivalent parser support for WorkMarket source material.

Build:

• WorkMarket platform adapter.
• WorkMarket-specific field mapping for ratings, reviews, completed work, categories, and proof excerpts.
• Same draft, confidence, preview, and approval flow as Field Nation.

Acceptance:

• WorkMarket imports follow the same safe workflow as Field Nation imports.
• Platform-specific differences stay inside parser adapters, not scattered through controllers and views.

Phase I5 - Normalization And Deduplication

Purpose:

Turn repeated imports into stable provider history instead of duplicate profile clutter.

Build:

• Canonical imported profile per provider and platform.
• Import run records for each uploaded source batch.
• Deduplication by platform, external ID, profile URL, date, review text hash, and proof source.
• Change summary between imports:
  • rating changed;
  • review count changed;
  • completed jobs changed;
  • new category;
  • new selected review;
  • verification changed.
• Provider approval for updates.
• Admin review for material public changes.

Acceptance:

• Re-importing updated Field Nation data updates the existing imported profile instead of creating confusing duplicates.
• Providers and admins can see what changed between import runs.

Phase I6 - Imported History Profile UX

Purpose:

Make imported history persuasive and clearly separate from native TSE reputation.

Build:

• Imported-history profile section redesign.
• Separate cards for:
  • Field Nation;
  • WorkMarket;
  • other platforms.
• Show:
  • provider-attested summary;
  • admin verification badge;
  • last imported date;
  • selected public review snippets;
  • category history;
  • metrics with source labels;
  • proof availability without exposing private proof.
• Clear copy that imported history is separate from native TSE work.
• Provider-controlled public/private toggles per field group.

Acceptance:

• A buyer can understand a provider's external marketplace history without confusing it for TSE-native reputation.
• A provider controls what is public.
• Verification status is visible and honest.

Phase I7 - Admin Verification And Audit

Purpose:

Give admins enough tools to verify imported proof without creating a hidden reputation score.

Build:

• Dedicated imported-history admin queue.
• Filters by platform, status, provider, source type, and age.
• Verification actions:
  • approve summary;
  • request more proof;
  • mark unverifiable;
  • reject public excerpt;
  • approve public excerpt.
• Admin notes.
• Audit logs for every verification action.
• Provider notification when verification status changes.

Acceptance:

• Admins can verify imported proof consistently.
• Providers can see verification results.
• Verification does not silently alter reputation without visible context.

Phase I8 - Fast Personal Field Nation Import

Purpose:

Get the user's own Field Nation data into the app as soon as the safe intake and parser path exists.

Status:

First logged-in capture and private local test import completed on 2026-07-18 for Field Nation provider ID 172630.

Completed:

• Captured profile text, screenshots, structured page attributes, availability, recommendation state, screening text, and work-summary data.
• Exported the full logged-in work-history table: 961 unique rows across 28 pages.
• Confirmed a no-cookie request to the profile URL redirects to Field Nation login.
• Attached the profile capture and work-history CSV as private local import-source batches.
• Parsed and approved a private imported-history summary with Field Nation ID, rating, rating count, completed jobs, client count, timeliness score, backout score, work categories, and endorsements.
• Added first-class screening fields for imported background-check and drug-test status.
• Added an external work-history table and imported all 961 captured Field Nation job rows into the local database as private imported proof.
• Added provider edit display for private imported screenings and recent external jobs.
• Added public/redacted provider-profile display for imported screenings and recent external work when the imported summary visibility allows it.

Remaining:

• Add filtering/search/pagination for large imported external work-history sets.
• Add provider controls for which external work-history fields can be public beyond the current redacted sample.
• Keep API research open, but do not block the urgent import lane on Field Nation API credentials.

Build:

• Use the provider's own exported/downloaded/provided Field Nation material.
• Import into the local/dev environment first.
• Review parsed fields manually.
• Redact sensitive details.
• Save public summary and private proof.
• Admin-verify locally if appropriate.
• Document exact source format and parser gaps found.

Acceptance:

• The user's Field Nation history appears on their provider profile in a safe imported-history section.
• Sensitive third-party details remain private.
• Any parser limitations are captured as follow-up tasks.

Phase I9 - Tests And Fixture Library

Purpose:

Make imports reliable without depending on live third-party sites.

Build:

• Sanitized Field Nation fixtures.
• Sanitized WorkMarket fixtures.
• Parser unit tests.
• Feature tests for upload, parse, preview, approve, publish, verify, and re-import.
• Privacy regression tests that prevent known sensitive fields from becoming public.

Acceptance:

• Import behavior is covered by repeatable local tests.
• Parser work can continue without using real private exports in the repo.

Immediate Implementation Sequence

1. Add source-batch storage and upload UI. First pass complete.
2. Add provider preview/redaction workflow. Partial first pass complete with parsed draft preview and private approval.
3. Add Field Nation parser adapter against provider-supplied samples. First text-like parser pass complete.
4. Import the user's Field Nation data locally.
5. Add WorkMarket parser after the Field Nation path is proven.
6. Add dedupe/re-import and admin queue improvements.

Data Needed From Provider

To implement the Field Nation path quickly, collect one or more provider-owned samples outside the public repo:

• profile page PDF print;
• profile screenshots;
• exported CSV/PDF if Field Nation provides one;
• copied text from profile/reviews/work summary;
• email summaries or downloaded reports if available.

Store real samples only in local/private storage or KeePass attachments. Do not commit real Field Nation exports containing customer or marketplace-private details.