ben/scrape-giant
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

data-model prep for refactor, removing observed layer

Browse Source
This commit is contained in:
ben
2026-03-18 15:15:29 -04:00
parent c13d144418
commit e74253f6fb
2 changed files with 147 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files

40
pm/data-model.org
View File

@@ -26,14 +26,15 @@ Goals:
*** Review/Combined data:
- catalog of reviewed products
- links from normalized line items to catalog
- links from normalized retailer items to catalog
- human review state for unresolved cases
* Pipeline
Each step can be run alone if its dependents exist.
Each retail provider script must produce deterministic line-item outputs and
must not group items into product identities before review.
Each retail provider script must produce deterministic line-item outputs, and
normalization may assign within-retailer product identity only when the
retailer itself provides strong evidence.
Key:
- (1) input
@@ -56,7 +57,8 @@ Strictly dependent on Collect method and output.
- Extract quantity, size, pack, pricing, variant
- Add discount line items to product line items using upc/retail_item_id and concurrence
- Cleanup naming to facilitate later matching
- Do not group line items into retailer-level product identities here
- Assign retailer-level `normalized_item_id` only when evidence is deterministic
- Never use fuzzy or semantic matching here
- (1) collected items from each retailer
- (2) collected visits from each retailer
- [1] normalized items from each retailer
@@ -64,8 +66,9 @@ Strictly dependent on Collect method and output.
** 3. Review/Combine (Canonicalization)
Decide whether two normalized retailer items are "the same product";
match items across retailers using algo/logic and human review.
Create catalog linked to normalized items.
- Group line items only here, when review/combine logic explicitly decides they belong together
Create catalog linked to normalized retailer items.
- Review operates on distinct `normalized_item_id` values, not individual purchase rows
- Cross-retailer identity decisions happen only here
- Asking human to create a canonical/catalog item with:
- friendly/catalog_name: "bell pepper"; "milk"
- category: "produce"; "dairy"
@@ -74,7 +77,7 @@ Create catalog linked to normalized items.
- Then link the group of items to that catalog item.
- (1) normalized items from each retailer
- [1] review queue of items to be reviewed
- [2] catalog (lookup table) of confirmed normalized line items and catalog_id
- [2] catalog (lookup table) of confirmed normalized retailer items and catalog_id
- [3] purchase list of normalized items , pivot-ready
** Unresolved Issues
@@ -106,7 +109,7 @@ data/
normalized_items.csv
review/
review_queue.csv # Human review queue for unresolved matching/parsing cases.
product_links.csv # Links from normalized line items to catalog items.
product_links.csv # Links from normalized retailer items to catalog items.
catalog.csv # Cross-retailer product catalog entities used for comparison.
purchases.csv
#+end_example
@@ -173,7 +176,8 @@ One row per order/visit/receipt.
** `data/<retailer-method>/normalized_items.csv`
One row per retailer line item after deterministic parsing. Preserve raw
fields from `collected_items.csv` and add parsed fields that make later review
and grouping easier. Normalization does not assign product identity.
and grouping easier. Normalization may assign retailer-level identity when the
evidence is deterministic and retailer-scoped.
| key | definition |
|----------------------------+------------------------------------------------------------------|
@@ -181,6 +185,8 @@ and grouping easier. Normalization does not assign product identity.
| `order_id` PK | retailer order id |
| `line_no` PK | line number within order |
| `normalized_row_id` | stable row key, typically `<retailer>:<order_id>:<line_no>` |
| `normalized_item_id` | stable retailer-level item identity when deterministic grouping is supported |
| `normalization_basis` | basis used to assign `normalized_item_id` |
| `retailer_item_id` | retailer-native item id |
| `item_name` | raw retailer item name |
| `item_name_norm` | normalized retailer item name |
@@ -213,18 +219,19 @@ and grouping easier. Normalization does not assign product identity.
| `parse_notes` | optional non-fatal parser notes |
Notes:
- `normalized_row_id` is the only required identity at this stage.
- Many normalized rows may later be grouped together during review/combine, but that grouping is not persisted here.
- `normalized_row_id` identifies the purchase row; `normalized_item_id` identifies a repeated retailer item when strong retailer evidence supports grouping.
- Valid `normalization_basis` values should be explicit, e.g. `exact_upc`, `exact_retailer_item_id`, `exact_name_size_pack`, or `approved_retailer_alias`.
- Do not use fuzzy or semantic matching to assign `normalized_item_id`.
- Discount/coupon rows may remain as standalone normalized rows for auditability even when their amounts are attached to a purchased row via `matched_discount_amount`.
- Cross-retailer identity is handled later in review/combine via `catalog.csv` and `product_links.csv`.
** `data/review/product_links.csv`
One row per review-approved link from a normalized row to a catalog item.
Many normalized rows may link to the same catalog item.
One row per review-approved link from a normalized retailer item to a catalog item.
Many normalized retailer items may link to the same catalog item.
| key | definition |
|-------------------------+---------------------------------------------|
| `normalized_row_id` PK | normalized retailer line-item id |
| `normalized_item_id` PK | normalized retailer item id |
| `catalog_id` PK | linked catalog product id |
| `link_method` | `manual`, `exact_upc`, `exact_name_size`, etc. |
| `link_confidence` | optional confidence label |
@@ -241,6 +248,7 @@ One row per issue needing human review.
| `review_id` PK | stable review row id |
| `queue_type` | `link_candidate`, `parse_issue`, `catalog_cleanup` |
| `retailer` | retailer slug when applicable |
| `normalized_item_id` | normalized retailer item id when review is item-level |
| `normalized_row_id` | normalized row id when review is row-specific |
| `catalog_id` | candidate canonical id |
| `reason_code` | machine-readable review reason |
@@ -281,7 +289,7 @@ Notes:
- `catalog_name` should come from review-approved naming, not raw retailer strings.
** `data/purchases.csv`
One row per purchased item (i.e., `row_type=item` from normalized layer), with
One row per purchased item (i.e., `is_item`==true from normalized layer), with
catalog attributes denormalized in and discounts already applied.
| key | definition |
@@ -291,6 +299,7 @@ catalog attributes denormalized in and discounts already applied.
| `order_id` | retailer order id |
| `line_no` | line number within order |
| `normalized_row_id` | `<retailer>:<order_id>:<line_no>` |
| `normalized_item_id` | retailer-level normalized item identity |
| `catalog_id` | linked catalog product id |
| `catalog_name` | catalog product name for analysis |
| `catalog_product_type` | broader product family (e.g., `egg`, `milk`) |
@@ -332,5 +341,6 @@ Notes:
- `line_total` preserves retailer truth; `net_line_total` is what you actually paid.
- catalog fields are denormalized in to make pivoting trivial.
- no discount/coupon rows exist here; their effects are carried via `matched_discount_amount`.
- review/link decisions should apply at the `normalized_item_id` level, then fan out to all purchase rows sharing that id.
* /

124
pm/tasks.org
View File

@@ -1,3 +1,5 @@
#+title: Scrape-Giant Task Log
* [X] t1.1: harden giant receipt fetch cli (2-4 commits)
** acceptance criteria
- giant scraper runs from cli with prompts or env-backed defaults for `user_id` and `loyalty`
@@ -470,10 +472,128 @@ refactor canonical generation so product identity is cleaner, duplicate canonica
** notes
- Removed weak exact-name auto-canonical creation so ambiguous products stay in review instead of generating junk canonicals.
- Canonical display names are now cleaned of obvious punctuation and packaging noise, but I kept the cleanup conservative rather than adding a broad fuzzy merge layer.
* [ ] 1t.10: add optional llm-assisted suggestion workflow for unresolved products (2-4 commits)
* [ ] t1.14: refactor retailer collection into the new data model (2-4 commits)
move Giant and Costco collection into the new collect structure and make both retailers emit the same collected schemas
** Acceptance Criteria
1. create retailer-specific collect scripts in the target naming pattern, e.g.:
- collect_giant_web.py
- collect_costco_web.py
2. collected outputs conform to pm/data-model.org:
- data/<retailer-method>/raw/...
- data/<retailer-method>/collected_orders.csv
- data/<retailer-method>/collected_items.csv
3. current Giant and Costco raw acquisition behavior is preserved during the move
4. collected schemas preserve retailer truth and provenance:
- no interpretation beyond basic flattening
- raw_order_path/raw_history_path remain usable
- unknown values remain blank rather than guessed
5. old paths should be removed or deprecated
6. collect_* scripts do not depend on any normalize/review files or scripts
- pm note: this is a path/schema refactor, not a parsing rewrite
** evidence
- commit:
- tests:
- datetime:
** notes
* [ ] t1.14.1: refactor retailer normalization into the new normalized_items schema (3-5 commits)
make Giant and Costco emit the shared normalized line-item schema without introducing cross-retailer identity logic
** Acceptance Criteria
1. create retailer-specific normalize scripts in the target naming pattern, e.g.:
- normalize_giant_web.py
- normalize_costco_web.py
2. normalized outputs conform to pm/data-model.org:
- data/<retailer-method>/normalized_items.csv
- one row per collected line item
- normalized_row_id is stable and present
- normalized_item_id is stable, present, and represents retailer-level identity reused across repeated purchase rows when deterministic retailer evidence is sufficient
- normalized_quantity and normalized_quantity_unit
- repeated rows for the same retailer product resolve to the same normalized_item_id only when supported by deterministic retailer evidence, e.g. exact upc, exact retailer_item_id, exact cleaned name + same size/pack
- normalization_basis is explicit
3. Giant normalization preserves current useful parsing:
- normalized item name
- size/unit/pack parsing
- fee/store-brand flags
- derived price fields
4. Costco normalization preserves current useful parsing:
- normalized item name
- size/unit/pack parsing
- explicit discount matching using retailer-specific logic
- matched_discount_amount and net_line_total
5. both normalizers preserve raw retailer truth:
- line_total is never overwritten
- unknown values remain blank rather than guessed
6. no cross-retailer identity assignment occurs in normalization
7. normalize never uses fuzzy or semantic matching to assign normalized_item_id
- pm note: prefer explicit retailer-specific code paths over generic normalization helpers unless the duplication is truly mechanical
- pm note: normalization may resolve retailer-level identity, but not catalog identity
- pm note: normalized_item_id is the only retailer-level grouping identity; do not introduce observed_products or a second grouping artifact
** evidence
- commit:
- tests:
- datetime:
** notes
* [ ] t1.15: refactor review/combine pipeline around normalized_item_id and catalog links (4-8 commits)
replace the old observed/canonical workflow with a review-first pipeline that uses normalized_item_id as the retailer-level review unit and links it to catalog items
** Acceptance Criteria
1. refactor review outputs to conform to pm/data-model.org:
- data/review/review_queue.csv
- data/review/product_links.csv
- data/catalog.csv
- data/purchases.csv
2. review logic uses normalized_item_id as the upstream retailer-level review identity:
- no dependency on observed_product_id
- no dependency on products_observed.csv
- one review/link decision applies to all purchase rows sharing the same normalized_item_id
3. product_links.csv stores review-approved links from normalized_item_id to catalog_id
- one row per approved retailer-level identity to catalog mapping
4. catalog.csv entries are review-first and conservative:
- no auto-creation from weak normalized names alone
- names come from reviewed catalog naming, not raw retailer strings
- packaging/count is not embedded in catalog_name unless essential to identity
- catalog_name/product_type/category/brand/variant may be blank until reviewed; blank is preferred to guessed
5. purchases.csv remains pivot-ready and retains:
- raw item name
- normalized item name
- normalized_row_id (not for review)
- normalized_item_id
- catalog_id
- catalog fields
- raw line_total
- matched_discount_amount and net_line_total when present
- derived price fields and their bases
6. terminal review flow remains simple and usable:
- reviewer sees one grouped retailer item identity (normalized_item_id) with count and list of matches, not one prompt per purchase row; use existing pattern as a template
- link to existing catalog item
- create new catalog item
- exclude
- skip
7. pipeline accounting remains valid after the refactor:
- unresolved items are visible
- missing items are not silently dropped
8. pm note: prefer a better manual review loop over aggressive automatic grouping. initial manual data entry is expected, and should resolve over time
9. pm note: keep review/combine auditable; each catalog link should be explainable from normalized rows and review state
** evidence
- commit:
- tests:
- datetime:
** notes
* [ ] 1t.10: add optional llm-assisted suggestion workflow for unresolved normalized retailer items (2-4 commits)
** acceptance criteria
- llm suggestions are generated only for unresolved observed products
- llm suggestions are generated only for unresolved normalized retailer items
- llm outputs are stored as suggestions, not auto-applied truth
- reviewer can approve/edit/reject suggestions
- approved decisions are persisted into canonical/link files