diff --git a/pm/data-model.org b/pm/data-model.org index ca7963d..5c5a1d1 100644 --- a/pm/data-model.org +++ b/pm/data-model.org @@ -26,14 +26,15 @@ Goals: *** Review/Combined data: - catalog of reviewed products -- links from normalized line items to catalog +- links from normalized retailer items to catalog - human review state for unresolved cases * Pipeline Each step can be run alone if its dependents exist. -Each retail provider script must produce deterministic line-item outputs and -must not group items into product identities before review. +Each retail provider script must produce deterministic line-item outputs, and +normalization may assign within-retailer product identity only when the +retailer itself provides strong evidence. Key: - (1) input @@ -56,7 +57,8 @@ Strictly dependent on Collect method and output. - Extract quantity, size, pack, pricing, variant - Add discount line items to product line items using upc/retail_item_id and concurrence - Cleanup naming to facilitate later matching - - Do not group line items into retailer-level product identities here + - Assign retailer-level `normalized_item_id` only when evidence is deterministic + - Never use fuzzy or semantic matching here - (1) collected items from each retailer - (2) collected visits from each retailer - [1] normalized items from each retailer @@ -64,8 +66,9 @@ Strictly dependent on Collect method and output. ** 3. Review/Combine (Canonicalization) Decide whether two normalized retailer items are "the same product"; match items across retailers using algo/logic and human review. -Create catalog linked to normalized items. - - Group line items only here, when review/combine logic explicitly decides they belong together +Create catalog linked to normalized retailer items. + - Review operates on distinct `normalized_item_id` values, not individual purchase rows + - Cross-retailer identity decisions happen only here - Asking human to create a canonical/catalog item with: - friendly/catalog_name: "bell pepper"; "milk" - category: "produce"; "dairy" @@ -74,7 +77,7 @@ Create catalog linked to normalized items. - Then link the group of items to that catalog item. - (1) normalized items from each retailer - [1] review queue of items to be reviewed - - [2] catalog (lookup table) of confirmed normalized line items and catalog_id + - [2] catalog (lookup table) of confirmed normalized retailer items and catalog_id - [3] purchase list of normalized items , pivot-ready ** Unresolved Issues @@ -106,7 +109,7 @@ data/ normalized_items.csv review/ review_queue.csv # Human review queue for unresolved matching/parsing cases. - product_links.csv # Links from normalized line items to catalog items. + product_links.csv # Links from normalized retailer items to catalog items. catalog.csv # Cross-retailer product catalog entities used for comparison. purchases.csv #+end_example @@ -173,7 +176,8 @@ One row per order/visit/receipt. ** `data//normalized_items.csv` One row per retailer line item after deterministic parsing. Preserve raw fields from `collected_items.csv` and add parsed fields that make later review -and grouping easier. Normalization does not assign product identity. +and grouping easier. Normalization may assign retailer-level identity when the +evidence is deterministic and retailer-scoped. | key | definition | |----------------------------+------------------------------------------------------------------| @@ -181,6 +185,8 @@ and grouping easier. Normalization does not assign product identity. | `order_id` PK | retailer order id | | `line_no` PK | line number within order | | `normalized_row_id` | stable row key, typically `::` | +| `normalized_item_id` | stable retailer-level item identity when deterministic grouping is supported | +| `normalization_basis` | basis used to assign `normalized_item_id` | | `retailer_item_id` | retailer-native item id | | `item_name` | raw retailer item name | | `item_name_norm` | normalized retailer item name | @@ -213,18 +219,19 @@ and grouping easier. Normalization does not assign product identity. | `parse_notes` | optional non-fatal parser notes | Notes: -- `normalized_row_id` is the only required identity at this stage. -- Many normalized rows may later be grouped together during review/combine, but that grouping is not persisted here. +- `normalized_row_id` identifies the purchase row; `normalized_item_id` identifies a repeated retailer item when strong retailer evidence supports grouping. +- Valid `normalization_basis` values should be explicit, e.g. `exact_upc`, `exact_retailer_item_id`, `exact_name_size_pack`, or `approved_retailer_alias`. +- Do not use fuzzy or semantic matching to assign `normalized_item_id`. - Discount/coupon rows may remain as standalone normalized rows for auditability even when their amounts are attached to a purchased row via `matched_discount_amount`. - Cross-retailer identity is handled later in review/combine via `catalog.csv` and `product_links.csv`. ** `data/review/product_links.csv` -One row per review-approved link from a normalized row to a catalog item. -Many normalized rows may link to the same catalog item. +One row per review-approved link from a normalized retailer item to a catalog item. +Many normalized retailer items may link to the same catalog item. | key | definition | |-------------------------+---------------------------------------------| -| `normalized_row_id` PK | normalized retailer line-item id | +| `normalized_item_id` PK | normalized retailer item id | | `catalog_id` PK | linked catalog product id | | `link_method` | `manual`, `exact_upc`, `exact_name_size`, etc. | | `link_confidence` | optional confidence label | @@ -241,6 +248,7 @@ One row per issue needing human review. | `review_id` PK | stable review row id | | `queue_type` | `link_candidate`, `parse_issue`, `catalog_cleanup` | | `retailer` | retailer slug when applicable | +| `normalized_item_id` | normalized retailer item id when review is item-level | | `normalized_row_id` | normalized row id when review is row-specific | | `catalog_id` | candidate canonical id | | `reason_code` | machine-readable review reason | @@ -281,7 +289,7 @@ Notes: - `catalog_name` should come from review-approved naming, not raw retailer strings. ** `data/purchases.csv` -One row per purchased item (i.e., `row_type=item` from normalized layer), with +One row per purchased item (i.e., `is_item`==true from normalized layer), with catalog attributes denormalized in and discounts already applied. | key | definition | @@ -291,6 +299,7 @@ catalog attributes denormalized in and discounts already applied. | `order_id` | retailer order id | | `line_no` | line number within order | | `normalized_row_id` | `::` | +| `normalized_item_id` | retailer-level normalized item identity | | `catalog_id` | linked catalog product id | | `catalog_name` | catalog product name for analysis | | `catalog_product_type` | broader product family (e.g., `egg`, `milk`) | @@ -332,5 +341,6 @@ Notes: - `line_total` preserves retailer truth; `net_line_total` is what you actually paid. - catalog fields are denormalized in to make pivoting trivial. - no discount/coupon rows exist here; their effects are carried via `matched_discount_amount`. +- review/link decisions should apply at the `normalized_item_id` level, then fan out to all purchase rows sharing that id. * / diff --git a/pm/tasks.org b/pm/tasks.org index 31dbaaf..3ec64f1 100644 --- a/pm/tasks.org +++ b/pm/tasks.org @@ -1,3 +1,5 @@ +#+title: Scrape-Giant Task Log + * [X] t1.1: harden giant receipt fetch cli (2-4 commits) ** acceptance criteria - giant scraper runs from cli with prompts or env-backed defaults for `user_id` and `loyalty` @@ -470,10 +472,128 @@ refactor canonical generation so product identity is cleaner, duplicate canonica ** notes - Removed weak exact-name auto-canonical creation so ambiguous products stay in review instead of generating junk canonicals. - Canonical display names are now cleaned of obvious punctuation and packaging noise, but I kept the cleanup conservative rather than adding a broad fuzzy merge layer. -* [ ] 1t.10: add optional llm-assisted suggestion workflow for unresolved products (2-4 commits) +* [ ] t1.14: refactor retailer collection into the new data model (2-4 commits) +move Giant and Costco collection into the new collect structure and make both retailers emit the same collected schemas + +** Acceptance Criteria +1. create retailer-specific collect scripts in the target naming pattern, e.g.: + - collect_giant_web.py + - collect_costco_web.py +2. collected outputs conform to pm/data-model.org: + - data//raw/... + - data//collected_orders.csv + - data//collected_items.csv +3. current Giant and Costco raw acquisition behavior is preserved during the move +4. collected schemas preserve retailer truth and provenance: + - no interpretation beyond basic flattening + - raw_order_path/raw_history_path remain usable + - unknown values remain blank rather than guessed +5. old paths should be removed or deprecated +6. collect_* scripts do not depend on any normalize/review files or scripts +- pm note: this is a path/schema refactor, not a parsing rewrite + +** evidence +- commit: +- tests: +- datetime: + +** notes + +* [ ] t1.14.1: refactor retailer normalization into the new normalized_items schema (3-5 commits) +make Giant and Costco emit the shared normalized line-item schema without introducing cross-retailer identity logic + +** Acceptance Criteria +1. create retailer-specific normalize scripts in the target naming pattern, e.g.: + - normalize_giant_web.py + - normalize_costco_web.py +2. normalized outputs conform to pm/data-model.org: + - data//normalized_items.csv + - one row per collected line item + - normalized_row_id is stable and present + - normalized_item_id is stable, present, and represents retailer-level identity reused across repeated purchase rows when deterministic retailer evidence is sufficient + - normalized_quantity and normalized_quantity_unit + - repeated rows for the same retailer product resolve to the same normalized_item_id only when supported by deterministic retailer evidence, e.g. exact upc, exact retailer_item_id, exact cleaned name + same size/pack + - normalization_basis is explicit +3. Giant normalization preserves current useful parsing: + - normalized item name + - size/unit/pack parsing + - fee/store-brand flags + - derived price fields +4. Costco normalization preserves current useful parsing: + - normalized item name + - size/unit/pack parsing + - explicit discount matching using retailer-specific logic + - matched_discount_amount and net_line_total +5. both normalizers preserve raw retailer truth: + - line_total is never overwritten + - unknown values remain blank rather than guessed +6. no cross-retailer identity assignment occurs in normalization +7. normalize never uses fuzzy or semantic matching to assign normalized_item_id + +- pm note: prefer explicit retailer-specific code paths over generic normalization helpers unless the duplication is truly mechanical +- pm note: normalization may resolve retailer-level identity, but not catalog identity +- pm note: normalized_item_id is the only retailer-level grouping identity; do not introduce observed_products or a second grouping artifact +** evidence +- commit: +- tests: +- datetime: + +** notes + +* [ ] t1.15: refactor review/combine pipeline around normalized_item_id and catalog links (4-8 commits) +replace the old observed/canonical workflow with a review-first pipeline that uses normalized_item_id as the retailer-level review unit and links it to catalog items + +** Acceptance Criteria +1. refactor review outputs to conform to pm/data-model.org: + - data/review/review_queue.csv + - data/review/product_links.csv + - data/catalog.csv + - data/purchases.csv +2. review logic uses normalized_item_id as the upstream retailer-level review identity: + - no dependency on observed_product_id + - no dependency on products_observed.csv + - one review/link decision applies to all purchase rows sharing the same normalized_item_id +3. product_links.csv stores review-approved links from normalized_item_id to catalog_id + - one row per approved retailer-level identity to catalog mapping +4. catalog.csv entries are review-first and conservative: + - no auto-creation from weak normalized names alone + - names come from reviewed catalog naming, not raw retailer strings + - packaging/count is not embedded in catalog_name unless essential to identity + - catalog_name/product_type/category/brand/variant may be blank until reviewed; blank is preferred to guessed +5. purchases.csv remains pivot-ready and retains: + - raw item name + - normalized item name + - normalized_row_id (not for review) + - normalized_item_id + - catalog_id + - catalog fields + - raw line_total + - matched_discount_amount and net_line_total when present + - derived price fields and their bases +6. terminal review flow remains simple and usable: + - reviewer sees one grouped retailer item identity (normalized_item_id) with count and list of matches, not one prompt per purchase row; use existing pattern as a template + - link to existing catalog item + - create new catalog item + - exclude + - skip +7. pipeline accounting remains valid after the refactor: + - unresolved items are visible + - missing items are not silently dropped +8. pm note: prefer a better manual review loop over aggressive automatic grouping. initial manual data entry is expected, and should resolve over time +9. pm note: keep review/combine auditable; each catalog link should be explainable from normalized rows and review state + +** evidence +- commit: +- tests: +- datetime: + +** notes + + +* [ ] 1t.10: add optional llm-assisted suggestion workflow for unresolved normalized retailer items (2-4 commits) ** acceptance criteria -- llm suggestions are generated only for unresolved observed products +- llm suggestions are generated only for unresolved normalized retailer items - llm outputs are stored as suggestions, not auto-applied truth - reviewer can approve/edit/reject suggestions - approved decisions are persisted into canonical/link files