data-model prep for refactor, removing observed layer

pm/tasks.org

@@ -1,3 +1,5 @@
+#+title: Scrape-Giant Task Log
+
 * [X] t1.1: harden giant receipt fetch cli (2-4 commits)
 ** acceptance criteria
 - giant scraper runs from cli with prompts or env-backed defaults for `user_id` and `loyalty`
@@ -470,10 +472,128 @@ refactor canonical generation so product identity is cleaner, duplicate canonica
 ** notes
 - Removed weak exact-name auto-canonical creation so ambiguous products stay in review instead of generating junk canonicals.
 - Canonical display names are now cleaned of obvious punctuation and packaging noise, but I kept the cleanup conservative rather than adding a broad fuzzy merge layer.
-* [ ] 1t.10: add optional llm-assisted suggestion workflow for unresolved products (2-4 commits)
+* [ ] t1.14: refactor retailer collection into the new data model (2-4 commits)
+move Giant and Costco collection into the new collect structure and make both retailers emit the same collected schemas
+
+** Acceptance Criteria
+1. create retailer-specific collect scripts in the target naming pattern, e.g.:
+  - collect_giant_web.py
+  - collect_costco_web.py
+2. collected outputs conform to pm/data-model.org:
+  - data/<retailer-method>/raw/...
+  - data/<retailer-method>/collected_orders.csv
+  - data/<retailer-method>/collected_items.csv
+3. current Giant and Costco raw acquisition behavior is preserved during the move
+4. collected schemas preserve retailer truth and provenance:
+  - no interpretation beyond basic flattening
+  - raw_order_path/raw_history_path remain usable
+  - unknown values remain blank rather than guessed
+5. old paths should be removed or deprecated
+6. collect_* scripts do not depend on any normalize/review files or scripts
+- pm note: this is a path/schema refactor, not a parsing rewrite
+
+** evidence
+- commit:
+- tests:
+- datetime:
+
+** notes
+
+* [ ] t1.14.1: refactor retailer normalization into the new normalized_items schema (3-5 commits)
+make Giant and Costco emit the shared normalized line-item schema without introducing cross-retailer identity logic
+
+** Acceptance Criteria
+1. create retailer-specific normalize scripts in the target naming pattern, e.g.:
+   - normalize_giant_web.py
+   - normalize_costco_web.py
+2. normalized outputs conform to pm/data-model.org:
+   - data/<retailer-method>/normalized_items.csv
+   - one row per collected line item
+   - normalized_row_id is stable and present
+   - normalized_item_id is stable, present, and represents retailer-level identity reused across repeated purchase rows when deterministic retailer evidence is sufficient
+   - normalized_quantity and normalized_quantity_unit
+   - repeated rows for the same retailer product resolve to the same normalized_item_id only when supported by deterministic retailer evidence, e.g. exact upc, exact retailer_item_id, exact cleaned name + same size/pack
+   - normalization_basis is explicit
+3. Giant normalization preserves current useful parsing:
+   - normalized item name
+   - size/unit/pack parsing
+   - fee/store-brand flags
+   - derived price fields
+4. Costco normalization preserves current useful parsing:
+   - normalized item name
+   - size/unit/pack parsing
+   - explicit discount matching using retailer-specific logic
+   - matched_discount_amount and net_line_total
+5. both normalizers preserve raw retailer truth:
+   - line_total is never overwritten
+   - unknown values remain blank rather than guessed
+6. no cross-retailer identity assignment occurs in normalization
+7. normalize never uses fuzzy or semantic matching to assign normalized_item_id
+
+- pm note: prefer explicit retailer-specific code paths over generic normalization helpers unless the duplication is truly mechanical
+- pm note: normalization may resolve retailer-level identity, but not catalog identity
+- pm note: normalized_item_id is the only retailer-level grouping identity; do not introduce observed_products or a second grouping artifact
+** evidence
+- commit:
+- tests:
+- datetime:
+
+** notes
+
+* [ ] t1.15: refactor review/combine pipeline around normalized_item_id and catalog links (4-8 commits)
+replace the old observed/canonical workflow with a review-first pipeline that uses normalized_item_id as the retailer-level review unit and links it to catalog items
+
+** Acceptance Criteria
+1. refactor review outputs to conform to pm/data-model.org:
+   - data/review/review_queue.csv
+   - data/review/product_links.csv
+   - data/catalog.csv
+   - data/purchases.csv
+2. review logic uses normalized_item_id as the upstream retailer-level review identity:
+   - no dependency on observed_product_id
+   - no dependency on products_observed.csv
+   - one review/link decision applies to all purchase rows sharing the same normalized_item_id
+3. product_links.csv stores review-approved links from normalized_item_id to catalog_id
+   - one row per approved retailer-level identity to catalog mapping
+4. catalog.csv entries are review-first and conservative:
+   - no auto-creation from weak normalized names alone
+   - names come from reviewed catalog naming, not raw retailer strings
+   - packaging/count is not embedded in catalog_name unless essential to identity
+   - catalog_name/product_type/category/brand/variant may be blank until reviewed; blank is preferred to guessed
+5. purchases.csv remains pivot-ready and retains:
+   - raw item name
+   - normalized item name
+   - normalized_row_id (not for review)
+   - normalized_item_id
+   - catalog_id
+   - catalog fields
+   - raw line_total
+   - matched_discount_amount and net_line_total when present
+   - derived price fields and their bases
+6. terminal review flow remains simple and usable:
+   - reviewer sees one grouped retailer item identity (normalized_item_id) with count and list of matches, not one prompt per purchase row; use existing pattern as a template
+   - link to existing catalog item
+   - create new catalog item
+   - exclude
+   - skip
+7. pipeline accounting remains valid after the refactor:
+   - unresolved items are visible
+   - missing items are not silently dropped
+8. pm note: prefer a better manual review loop over aggressive automatic grouping. initial manual data entry is expected, and should resolve over time
+9. pm note: keep review/combine auditable; each catalog link should be explainable from normalized rows and review state
+
+** evidence
+- commit:
+- tests:
+- datetime:
+
+** notes
+
+
+* [ ] 1t.10: add optional llm-assisted suggestion workflow for unresolved normalized retailer items (2-4 commits)
 
 ** acceptance criteria
-- llm suggestions are generated only for unresolved observed products
+- llm suggestions are generated only for unresolved normalized retailer items
 - llm outputs are stored as suggestions, not auto-applied truth
 - reviewer can approve/edit/reject suggestions
 - approved decisions are persisted into canonical/link files