technical-services-exchange/docs/URGENT_IMPORT_ROADMAP.md

# Urgent Import Roadmap

This roadmap promotes Field Nation and WorkMarket history import into an urgent lane. It starts from the current codebase, which already supports manual imported-history records, proof attachments, visibility controls, and admin verification.

The goal is to let a provider pull their own marketplace history into Provider Exchange / Technical Services Exchange as quickly as possible without scraping private systems blindly, exposing customer data, or mixing imported history into native TSE reputation.

## Current Import Baseline

Implemented today:

- `external_profile_imports` records.
- Provider-entered platform, external ID, profile URL, rating, review count, completed jobs, work categories, selected reviews, operational metrics, notes, visibility, and verification status.
- Proof attachments on imported-history records.
- Public/private visibility modes.
- Admin verification status.
- Public provider-profile display of imported history according to visibility.
- Tests for saving imports, visibility, and admin verification.

Missing:

- File/export upload intake. First pass complete for private source batches.
- Structured parser for exported profile data.
- Field Nation-specific import mapping.
- WorkMarket-specific import mapping.
- Review/work-history deduplication.
- Privacy redaction workflow for customer names, addresses, work-order IDs, and private buyer details.
- Provider preview/approval before imported data becomes public.
- Admin review queue focused on imported proof.
- Repeat import/update flow.
- Import audit trail and failure reporting.

## Import Principles

- Providers may import their own records and proof.
- Do not automate login scraping until terms, privacy, and technical risks are explicitly reviewed.
- Prefer user-provided exports, screenshots, PDFs, CSVs, emails, or copied profile data first.
- Keep raw proof private by default.
- Never publish customer names, site addresses, buyer-private notes, work-order IDs, or private marketplace content by default.
- Keep imported history separate from native TSE reviews and completed work.
- Every transformed import should be previewed and approved by the provider before public display.
- Every admin verification action should be audited.

## Phase I1 - Export Intake And Raw Proof Vault

Purpose:

Let a provider upload Field Nation or WorkMarket source material without needing a polished parser first.

Status:

First implementation pass added local/private source-batch storage, provider upload UI, admin visibility, audit logging, and feature tests.

Build:

- Import source upload screen under the provider profile. First pass complete.
- Supported source types. First pass complete:
  - PDF;
  - image;
  - CSV;
  - JSON;
  - HTML copy/export;
  - plain text paste;
  - email/message export pasted as text.
- `external_import_sources` table for raw import batches. Complete.
- Link source batches to `external_profile_imports`. Complete.
- Store original filename, MIME type, source platform, source type, uploaded by, uploaded at, and parse status. Complete.
- Keep raw source private and admin-visible only. Complete with private `local` storage by default.
- Add provider notes describing where the source came from. Complete.
- Add a "do not publish raw proof" default. Complete through private storage and `needs_review` status.

Acceptance:

- A provider can upload Field Nation proof today even before parsing is perfect.
- Raw proof is retained privately for review.
- The existing manual imported-history record can reference the source batch.

## Phase I2 - Manual Review Workspace And Redaction

Purpose:

Make imported proof safe to use by separating private source material from public profile summaries.

Build:

- Provider import workspace showing uploaded source batches.
- Redaction checklist:
  - customer names;
  - addresses;
  - work-order IDs;
  - phone numbers;
  - emails;
  - private buyer instructions;
  - payment account details;
  - marketplace-private messages.
- Provider-selected public excerpts.
- Admin-visible private notes.
- Public summary preview.
- Redaction status:
  - needs review;
  - provider reviewed;
  - admin reviewed;
  - approved for public summary;
  - rejected for public display.

Acceptance:

- A provider can safely turn private proof into a public summary.
- The UI makes it hard to accidentally publish sensitive third-party details.
- Admins can see raw proof without making it public.

## Phase I3 - Field Nation Parser First Pass

Purpose:

Extract useful provider-owned history from common Field Nation source material.

Build:

- Parser interface with platform-specific adapters.
- Field Nation adapter for provider-supplied source files.
- Extract when present:
  - provider profile ID or profile URL;
  - rating;
  - review count;
  - completed work count;
  - client count;
  - service categories;
  - review snippets;
  - dates or recency hints;
  - on-time or reliability metrics if present;
  - work summary categories if present.
- Store parsed output separately from approved/public output.
- Confidence flags for parsed fields:
  - high;
  - medium;
  - low;
  - needs manual confirmation.
- Parser error and warning display.
- First-pass text parser for copied/exported profile text. Implemented.
- Provider approval from parsed draft into imported-history summary. Implemented.

Acceptance:

- A provider can upload a Field Nation export/screenshot/PDF/text source and get a structured draft.
- The provider can edit or reject every parsed field before saving.
- Low-confidence fields never publish automatically.

## Phase I4 - WorkMarket Parser First Pass

Purpose:

Add equivalent parser support for WorkMarket source material.

Build:

- WorkMarket platform adapter.
- WorkMarket-specific field mapping for ratings, reviews, completed work, categories, and proof excerpts.
- Same draft, confidence, preview, and approval flow as Field Nation.

Acceptance:

- WorkMarket imports follow the same safe workflow as Field Nation imports.
- Platform-specific differences stay inside parser adapters, not scattered through controllers and views.

## Phase I5 - Normalization And Deduplication

Purpose:

Turn repeated imports into stable provider history instead of duplicate profile clutter.

Build:

- Canonical imported profile per provider and platform.
- Import run records for each uploaded source batch.
- Deduplication by platform, external ID, profile URL, date, review text hash, and proof source.
- Change summary between imports:
  - rating changed;
  - review count changed;
  - completed jobs changed;
  - new category;
  - new selected review;
  - verification changed.
- Provider approval for updates.
- Admin review for material public changes.

Acceptance:

- Re-importing updated Field Nation data updates the existing imported profile instead of creating confusing duplicates.
- Providers and admins can see what changed between import runs.

## Phase I6 - Imported History Profile UX

Purpose:

Make imported history persuasive and clearly separate from native TSE reputation.

Build:

- Imported-history profile section redesign.
- Separate cards for:
  - Field Nation;
  - WorkMarket;
  - other platforms.
- Show:
  - provider-attested summary;
  - admin verification badge;
  - last imported date;
  - selected public review snippets;
  - category history;
  - metrics with source labels;
  - proof availability without exposing private proof.
- Clear copy that imported history is separate from native TSE work.
- Provider-controlled public/private toggles per field group.

Acceptance:

- A buyer can understand a provider's external marketplace history without confusing it for TSE-native reputation.
- A provider controls what is public.
- Verification status is visible and honest.

## Phase I7 - Admin Verification And Audit

Purpose:

Give admins enough tools to verify imported proof without creating a hidden reputation score.

Build:

- Dedicated imported-history admin queue.
- Filters by platform, status, provider, source type, and age.
- Verification actions:
  - approve summary;
  - request more proof;
  - mark unverifiable;
  - reject public excerpt;
  - approve public excerpt.
- Admin notes.
- Audit logs for every verification action.
- Provider notification when verification status changes.

Acceptance:

- Admins can verify imported proof consistently.
- Providers can see verification results.
- Verification does not silently alter reputation without visible context.

## Phase I8 - Fast Personal Field Nation Import

Purpose:

Get the user's own Field Nation data into the app as soon as the safe intake and parser path exists.

Status:

First logged-in capture and private local test import completed on 2026-07-18 for Field Nation provider ID `172630`.

Completed:

- Captured profile text, screenshots, structured page attributes, availability, recommendation state, screening text, and work-summary data.
- Exported the full logged-in work-history table: `961` unique rows across `28` pages.
- Confirmed a no-cookie request to the profile URL redirects to Field Nation login.
- Attached the profile capture and work-history CSV as private local import-source batches.
- Parsed and approved a private imported-history summary with Field Nation ID, rating, rating count, completed jobs, client count, timeliness score, backout score, work categories, and endorsements.
- Added first-class screening fields for imported background-check and drug-test status.
- Added an external work-history table and imported all `961` captured Field Nation job rows into the local database as private imported proof.
- Added provider edit display for private imported screenings and recent external jobs.
- Added public/redacted provider-profile display for imported screenings and recent external work when the imported summary visibility allows it.

Remaining:

- Add filtering/search/pagination for large imported external work-history sets.
- Add provider controls for which external work-history fields can be public beyond the current redacted sample.
- Keep API research open, but do not block the urgent import lane on Field Nation API credentials.

Build:

- Use the provider's own exported/downloaded/provided Field Nation material.
- Import into the local/dev environment first.
- Review parsed fields manually.
- Redact sensitive details.
- Save public summary and private proof.
- Admin-verify locally if appropriate.
- Document exact source format and parser gaps found.

Acceptance:

- The user's Field Nation history appears on their provider profile in a safe imported-history section.
- Sensitive third-party details remain private.
- Any parser limitations are captured as follow-up tasks.

## Phase I9 - Tests And Fixture Library

Purpose:

Make imports reliable without depending on live third-party sites.

Build:

- Sanitized Field Nation fixtures.
- Sanitized WorkMarket fixtures.
- Parser unit tests.
- Feature tests for upload, parse, preview, approve, publish, verify, and re-import.
- Privacy regression tests that prevent known sensitive fields from becoming public.

Acceptance:

- Import behavior is covered by repeatable local tests.
- Parser work can continue without using real private exports in the repo.

## Immediate Implementation Sequence

1. Add source-batch storage and upload UI. First pass complete.
2. Add provider preview/redaction workflow. Partial first pass complete with parsed draft preview and private approval.
3. Add Field Nation parser adapter against provider-supplied samples. First text-like parser pass complete.
4. Import the user's Field Nation data locally.
5. Add WorkMarket parser after the Field Nation path is proven.
6. Add dedupe/re-import and admin queue improvements.

## Data Needed From Provider

To implement the Field Nation path quickly, collect one or more provider-owned samples outside the public repo:

- profile page PDF print;
- profile screenshots;
- exported CSV/PDF if Field Nation provides one;
- copied text from profile/reviews/work summary;
- email summaries or downloaded reports if available.

Store real samples only in local/private storage or KeePass attachments. Do not commit real Field Nation exports containing customer or marketplace-private details.