ben/scrape-giant
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

data-model prep for refactor, removing observed layer

Browse Source
This commit is contained in:
ben
2026-03-18 15:15:29 -04:00
parent c13d144418
commit e74253f6fb
2 changed files with 147 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files

40
pm/data-model.org
View File

@@ -26,14 +26,15 @@ Goals:
*** Review/Combined data:
- catalog of reviewed products
- links from normalized line items to catalog
- links from normalized retailer items to catalog
- human review state for unresolved cases
* Pipeline
Each step can be run alone if its dependents exist.
Each retail provider script must produce deterministic line-item outputs and
must not group items into product identities before review.
Each retail provider script must produce deterministic line-item outputs, and
normalization may assign within-retailer product identity only when the
retailer itself provides strong evidence.
Key:
- (1) input
@@ -56,7 +57,8 @@ Strictly dependent on Collect method and output.
- Extract quantity, size, pack, pricing, variant
- Add discount line items to product line items using upc/retail_item_id and concurrence
- Cleanup naming to facilitate later matching
- Do not group line items into retailer-level product identities here
- Assign retailer-level `normalized_item_id` only when evidence is deterministic
- Never use fuzzy or semantic matching here
- (1) collected items from each retailer
- (2) collected visits from each retailer
- [1] normalized items from each retailer
@@ -64,8 +66,9 @@ Strictly dependent on Collect method and output.
** 3. Review/Combine (Canonicalization)
Decide whether two normalized retailer items are "the same product";
match items across retailers using algo/logic and human review.
Create catalog linked to normalized items.
- Group line items only here, when review/combine logic explicitly decides they belong together
Create catalog linked to normalized retailer items.
- Review operates on distinct `normalized_item_id` values, not individual purchase rows
- Cross-retailer identity decisions happen only here
- Asking human to create a canonical/catalog item with:
- friendly/catalog_name: "bell pepper"; "milk"
- category: "produce"; "dairy"
@@ -74,7 +77,7 @@ Create catalog linked to normalized items.
- Then link the group of items to that catalog item.
- (1) normalized items from each retailer
- [1] review queue of items to be reviewed
- [2] catalog (lookup table) of confirmed normalized line items and catalog_id
- [2] catalog (lookup table) of confirmed normalized retailer items and catalog_id
- [3] purchase list of normalized items , pivot-ready
** Unresolved Issues
@@ -106,7 +109,7 @@ data/
normalized_items.csv
review/
review_queue.csv # Human review queue for unresolved matching/parsing cases.
product_links.csv # Links from normalized line items to catalog items.
product_links.csv # Links from normalized retailer items to catalog items.
catalog.csv # Cross-retailer product catalog entities used for comparison.
purchases.csv
#+end_example
@@ -173,7 +176,8 @@ One row per order/visit/receipt.
** `data/<retailer-method>/normalized_items.csv`
One row per retailer line item after deterministic parsing. Preserve raw
fields from `collected_items.csv` and add parsed fields that make later review
and grouping easier. Normalization does not assign product identity.
and grouping easier. Normalization may assign retailer-level identity when the
evidence is deterministic and retailer-scoped.
| key | definition |
|----------------------------+------------------------------------------------------------------|
@@ -181,6 +185,8 @@ and grouping easier. Normalization does not assign product identity.
| `order_id` PK | retailer order id |
| `line_no` PK | line number within order |
| `normalized_row_id` | stable row key, typically `<retailer>:<order_id>:<line_no>` |
| `normalized_item_id` | stable retailer-level item identity when deterministic grouping is supported |
| `normalization_basis` | basis used to assign `normalized_item_id` |
| `retailer_item_id` | retailer-native item id |
| `item_name` | raw retailer item name |
| `item_name_norm` | normalized retailer item name |
@@ -213,18 +219,19 @@ and grouping easier. Normalization does not assign product identity.
| `parse_notes` | optional non-fatal parser notes |
Notes:
- `normalized_row_id` is the only required identity at this stage.
- Many normalized rows may later be grouped together during review/combine, but that grouping is not persisted here.
- `normalized_row_id` identifies the purchase row; `normalized_item_id` identifies a repeated retailer item when strong retailer evidence supports grouping.
- Valid `normalization_basis` values should be explicit, e.g. `exact_upc`, `exact_retailer_item_id`, `exact_name_size_pack`, or `approved_retailer_alias`.
- Do not use fuzzy or semantic matching to assign `normalized_item_id`.
- Discount/coupon rows may remain as standalone normalized rows for auditability even when their amounts are attached to a purchased row via `matched_discount_amount`.
- Cross-retailer identity is handled later in review/combine via `catalog.csv` and `product_links.csv`.
** `data/review/product_links.csv`
One row per review-approved link from a normalized row to a catalog item.
Many normalized rows may link to the same catalog item.
One row per review-approved link from a normalized retailer item to a catalog item.
Many normalized retailer items may link to the same catalog item.
| key | definition |
|-------------------------+---------------------------------------------|
| `normalized_row_id` PK | normalized retailer line-item id |
| `normalized_item_id` PK | normalized retailer item id |
| `catalog_id` PK | linked catalog product id |
| `link_method` | `manual`, `exact_upc`, `exact_name_size`, etc. |
| `link_confidence` | optional confidence label |
@@ -241,6 +248,7 @@ One row per issue needing human review.
| `review_id` PK | stable review row id |
| `queue_type` | `link_candidate`, `parse_issue`, `catalog_cleanup` |
| `retailer` | retailer slug when applicable |
| `normalized_item_id` | normalized retailer item id when review is item-level |
| `normalized_row_id` | normalized row id when review is row-specific |
| `catalog_id` | candidate canonical id |
| `reason_code` | machine-readable review reason |
@@ -281,7 +289,7 @@ Notes:
- `catalog_name` should come from review-approved naming, not raw retailer strings.
** `data/purchases.csv`
One row per purchased item (i.e., `row_type=item` from normalized layer), with
One row per purchased item (i.e., `is_item`==true from normalized layer), with
catalog attributes denormalized in and discounts already applied.
| key | definition |
@@ -291,6 +299,7 @@ catalog attributes denormalized in and discounts already applied.
| `order_id` | retailer order id |
| `line_no` | line number within order |
| `normalized_row_id` | `<retailer>:<order_id>:<line_no>` |
| `normalized_item_id` | retailer-level normalized item identity |
| `catalog_id` | linked catalog product id |
| `catalog_name` | catalog product name for analysis |
| `catalog_product_type` | broader product family (e.g., `egg`, `milk`) |
@@ -332,5 +341,6 @@ Notes:
- `line_total` preserves retailer truth; `net_line_total` is what you actually paid.
- catalog fields are denormalized in to make pivoting trivial.
- no discount/coupon rows exist here; their effects are carried via `matched_discount_amount`.
- review/link decisions should apply at the `normalized_item_id` level, then fan out to all purchase rows sharing that id.
* /