301 lines
13 KiB
Org Mode
301 lines
13 KiB
Org Mode
* grocery data model and file layout
|
|
|
|
This document defines the shared file layout and stable CSV schemas for the
|
|
grocery pipeline. The goal is to keep retailer-specific ingest separate from
|
|
cross-retailer product modeling so Giant-specific quirks do not become the
|
|
system of record.
|
|
|
|
** design rules
|
|
|
|
- Raw retailer exports remain the source of truth.
|
|
- Retailer parsing is isolated to retailer-specific files and ids.
|
|
- Cross-retailer product layers begin only after retailer-specific enrichment.
|
|
- CSV schemas are stable and additive: new columns may be appended, but
|
|
existing columns should not be repurposed.
|
|
- Unknown values should be left blank rather than guessed.
|
|
|
|
** directory layout
|
|
|
|
Use one top-level data root:
|
|
|
|
#+begin_example
|
|
data/
|
|
giant/
|
|
raw/
|
|
history.json
|
|
orders/
|
|
<order_id>.json
|
|
orders.csv
|
|
items_raw.csv
|
|
items_enriched.csv
|
|
products_observed.csv
|
|
costco/
|
|
raw/
|
|
...
|
|
orders.csv
|
|
items_raw.csv
|
|
items_enriched.csv
|
|
products_observed.csv
|
|
shared/
|
|
products_canonical.csv
|
|
product_links.csv
|
|
review_queue.csv
|
|
#+end_example
|
|
|
|
** layer responsibilities
|
|
|
|
- `data/<retailer>/raw/`
|
|
Stores unmodified retailer payloads exactly as fetched.
|
|
- `data/<retailer>/orders.csv`
|
|
One row per retailer order or visit, flattened from raw order data.
|
|
- `data/<retailer>/items_raw.csv`
|
|
One row per retailer line item, preserving retailer-native values needed for
|
|
reruns and debugging.
|
|
- `data/<retailer>/items_enriched.csv`
|
|
Parsed retailer line items with normalized fields and derived guesses, still
|
|
retailer-specific.
|
|
- `data/<retailer>/products_observed.csv`
|
|
Distinct retailer-facing observed products aggregated from enriched items.
|
|
- `data/shared/products_canonical.csv`
|
|
Cross-retailer canonical product entities used for comparison.
|
|
- `data/shared/product_links.csv`
|
|
Links from retailer observed products to canonical products.
|
|
- `data/shared/review_queue.csv`
|
|
Human review queue for unresolved or low-confidence matching/parsing cases.
|
|
|
|
** retailer-specific versus shared
|
|
|
|
Retailer-specific:
|
|
|
|
- raw json payloads
|
|
- retailer order ids
|
|
- retailer line numbers
|
|
- retailer category ids and names
|
|
- retailer item names
|
|
- retailer image urls
|
|
- parsed guesses derived from one retailer feed
|
|
- observed products scoped to one retailer
|
|
|
|
Shared:
|
|
|
|
- canonical products
|
|
- observed-to-canonical links
|
|
- human review state for unresolved cases
|
|
- comparison-ready normalized quantity basis fields
|
|
|
|
Observed products are the boundary between retailer-specific parsing and
|
|
cross-retailer canonicalization. Nothing upstream of `products_observed.csv`
|
|
should require knowledge of another retailer.
|
|
|
|
** schema: `data/<retailer>/orders.csv`
|
|
|
|
One row per order or visit.
|
|
|
|
| column | meaning |
|
|
|-
|
|
| `retailer` | retailer slug such as `giant` |
|
|
| `order_id` | retailer order or visit id |
|
|
| `order_date` | order date in `YYYY-MM-DD` when available |
|
|
| `delivery_date` | fulfillment date in `YYYY-MM-DD` when available |
|
|
| `service_type` | retailer service type such as `INSTORE` |
|
|
| `order_total` | order total as provided by retailer |
|
|
| `payment_method` | retailer payment label |
|
|
| `total_item_count` | total line count or item count from retailer |
|
|
| `total_savings` | total savings as provided by retailer |
|
|
| `your_savings_total` | savings field from retailer when present |
|
|
| `coupons_discounts_total` | coupon/discount total from retailer |
|
|
| `store_name` | retailer store name |
|
|
| `store_number` | retailer store number |
|
|
| `store_address1` | street address |
|
|
| `store_city` | city |
|
|
| `store_state` | state or province |
|
|
| `store_zipcode` | postal code |
|
|
| `refund_order` | retailer refund flag |
|
|
| `ebt_order` | retailer EBT flag |
|
|
| `raw_history_path` | relative path to source history payload |
|
|
| `raw_order_path` | relative path to source order payload |
|
|
|
|
Primary key:
|
|
|
|
- (`retailer`, `order_id`)
|
|
|
|
** schema: `data/<retailer>/items_raw.csv`
|
|
|
|
One row per retailer line item.
|
|
|
|
| column | meaning |
|
|
|------------------+-----------------------------------------|
|
|
| `retailer` | retailer slug |
|
|
| `order_id` | retailer order id |
|
|
| `line_no` | stable line number within order export |
|
|
| `order_date` | copied from order when available |
|
|
| `pod_id` | retailer pod/item id |
|
|
| `item_name` | raw retailer item name |
|
|
| `upc` | retailer UPC or PLU value |
|
|
| `category_id` | retailer category id |
|
|
| `category` | retailer category description |
|
|
| `qty` | retailer quantity field |
|
|
| `unit` | retailer unit code such as `EA` or `LB` |
|
|
| `unit_price` | retailer unit price field |
|
|
| `line_total` | retailer extended price field |
|
|
| `picked_weight` | retailer picked weight field |
|
|
| `mvp_savings` | retailer savings field |
|
|
| `reward_savings` | retailer rewards savings field |
|
|
| `coupon_savings` | retailer coupon savings field |
|
|
| `coupon_price` | retailer coupon price field |
|
|
| `image_url` | raw retailer image url when present |
|
|
| `raw_order_path` | relative path to source order payload |
|
|
|
|
Primary key:
|
|
|
|
- (`retailer`, `order_id`, `line_no`)
|
|
|
|
** schema: `data/<retailer>/items_enriched.csv`
|
|
|
|
One row per retailer line item after deterministic parsing. Preserve the raw
|
|
fields from `items_raw.csv` and add parsed fields.
|
|
|
|
| column | meaning |
|
|
|---------------------+-------------------------------------------------------------|
|
|
| `retailer` | retailer slug |
|
|
| `order_id` | retailer order id |
|
|
| `line_no` | line number within order |
|
|
| `observed_item_key` | stable row key, typically `<retailer>:<order_id>:<line_no>` |
|
|
| `item_name` | raw retailer item name |
|
|
| `item_name_norm` | normalized item name |
|
|
| `brand_guess` | parsed brand guess |
|
|
| `variant` | parsed variant text |
|
|
| `size_value` | parsed numeric size value |
|
|
| `size_unit` | parsed size unit such as `oz`, `lb`, `fl_oz` |
|
|
| `pack_qty` | parsed pack or count guess |
|
|
| `measure_type` | `each`, `weight`, `volume`, `count`, or blank |
|
|
| `is_store_brand` | store-brand guess |
|
|
| `is_fee` | fee or non-product flag |
|
|
| `price_per_each` | derived per-each price when supported |
|
|
| `price_per_lb` | derived per-pound price when supported |
|
|
| `price_per_oz` | derived per-ounce price when supported |
|
|
| `image_url` | best available retailer image url |
|
|
| `parse_version` | parser version string for reruns |
|
|
| `parse_notes` | optional non-fatal parser notes |
|
|
|
|
Primary key:
|
|
|
|
- (`retailer`, `order_id`, `line_no`)
|
|
|
|
** schema: `data/<retailer>/products_observed.csv`
|
|
|
|
One row per distinct retailer-facing observed product.
|
|
|
|
| column | meaning |
|
|
|-------------------------------+----------------------------------------------------------------|
|
|
| `observed_product_id` | stable observed product id |
|
|
| `retailer` | retailer slug |
|
|
| `observed_key` | deterministic grouping key used to create the observed product |
|
|
| `representative_upc` | best representative UPC/PLU |
|
|
| `representative_item_name` | representative raw retailer name |
|
|
| `representative_name_norm` | representative normalized name |
|
|
| `representative_brand` | representative brand guess |
|
|
| `representative_variant` | representative variant |
|
|
| `representative_size_value` | representative size value |
|
|
| `representative_size_unit` | representative size unit |
|
|
| `representative_pack_qty` | representative pack/count |
|
|
| `representative_measure_type` | representative measure type |
|
|
| `representative_image_url` | representative image url |
|
|
| `is_store_brand` | representative store-brand flag |
|
|
| `is_fee` | representative fee flag |
|
|
| `first_seen_date` | first order date seen |
|
|
| `last_seen_date` | last order date seen |
|
|
| `times_seen` | number of enriched item rows grouped here |
|
|
| `example_order_id` | one example retailer order id |
|
|
| `example_item_name` | one example raw item name |
|
|
|
|
Primary key:
|
|
|
|
- (`observed_product_id`)
|
|
|
|
** schema: `data/shared/products_canonical.csv`
|
|
|
|
One row per cross-retailer canonical product.
|
|
|
|
| column | meaning |
|
|
|----------------------------+--------------------------------------------------|
|
|
| `canonical_product_id` | stable canonical product id |
|
|
| `canonical_name` | canonical human-readable name |
|
|
| `product_type` | broad class such as `apple`, `milk`, `trash_bag` |
|
|
| `brand` | canonical brand when applicable |
|
|
| `variant` | canonical variant |
|
|
| `size_value` | normalized size value |
|
|
| `size_unit` | normalized size unit |
|
|
| `pack_qty` | normalized pack/count |
|
|
| `measure_type` | normalized measure type |
|
|
| `normalized_quantity` | numeric comparison basis value |
|
|
| `normalized_quantity_unit` | basis unit such as `oz`, `lb`, `count` |
|
|
| `notes` | optional human notes |
|
|
| `created_at` | creation timestamp or date |
|
|
| `updated_at` | last update timestamp or date |
|
|
|
|
Primary key:
|
|
|
|
- (`canonical_product_id`)
|
|
|
|
** schema: `data/shared/product_links.csv`
|
|
|
|
One row per observed-to-canonical relationship.
|
|
|
|
| column | meaning |
|
|
|-
|
|
| `observed_product_id` | retailer observed product id |
|
|
| `canonical_product_id` | linked canonical product id |
|
|
| `link_method` | `manual`, `exact_upc`, `exact_name`, etc. |
|
|
| `link_confidence` | optional confidence label |
|
|
| `review_status` | `pending`, `approved`, `rejected`, or blank |
|
|
| `reviewed_by` | reviewer id or initials |
|
|
| `reviewed_at` | review timestamp or date |
|
|
| `link_notes` | optional notes |
|
|
|
|
Primary key:
|
|
|
|
- (`observed_product_id`, `canonical_product_id`)
|
|
|
|
** schema: `data/shared/review_queue.csv`
|
|
|
|
One row per issue needing human review.
|
|
|
|
| column | meaning |
|
|
|-
|
|
| `review_id` | stable review row id |
|
|
| `queue_type` | `observed_product`, `link_candidate`, `parse_issue` |
|
|
| `retailer` | retailer slug when applicable |
|
|
| `observed_product_id` | observed product id when applicable |
|
|
| `canonical_product_id` | candidate canonical id when applicable |
|
|
| `reason_code` | machine-readable review reason |
|
|
| `priority` | optional priority label |
|
|
| `raw_item_names` | compact list of example raw names |
|
|
| `normalized_names` | compact list of example normalized names |
|
|
| `upc` | example UPC/PLU |
|
|
| `image_url` | example image url |
|
|
| `example_prices` | compact list of example prices |
|
|
| `seen_count` | count of related rows |
|
|
| `status` | `pending`, `approved`, `rejected`, `deferred` |
|
|
| `resolution_notes` | reviewer notes |
|
|
| `created_at` | creation timestamp or date |
|
|
| `updated_at` | last update timestamp or date |
|
|
|
|
Primary key:
|
|
|
|
- (`review_id`)
|
|
|
|
** current giant mapping
|
|
|
|
Current scraper outputs map to the new layout as follows:
|
|
|
|
- `giant_output/raw/history.json` -> `data/giant/raw/history.json`
|
|
- `giant_output/raw/<order_id>.json` -> `data/giant/raw/orders/<order_id>.json`
|
|
- `giant_output/orders.csv` -> `data/giant/orders.csv`
|
|
- `giant_output/items.csv` -> `data/giant/items_raw.csv`
|
|
|
|
Current Giant raw order payloads already expose fields needed for future
|
|
enrichment, including `image`, `itemName`, `primUpcCd`, `lbEachCd`,
|
|
`unitPrice`, `groceryAmount`, and `totalPickedWeight`.
|
|
|