scrape-giant/pm/tasks.org

 #+title: Scrape-Giant Task Log
#+STARTUP: overview
* [X] t1.1: harden giant receipt fetch cli (2-4 commits)
** acceptance criteria
- giant scraper runs from cli with prompts or env-backed defaults for `user_id` and `loyalty`
- script reuses current browser session via firefox cookies + `curl_cffi`
- script only fetches unseen orders
- script appends to `orders.csv` and `items.csv` without duplicating prior visits
- script prints a note that giant only exposes the most recent 50 visits

** notes
- keep this giant-specific
- no canonical product logic here
- raw json archive remains source of truth

** evidence
- commit: `d57b9cf` on branch `cx`
- tests: `./venv/bin/python -m unittest discover -s tests`; `./venv/bin/python scraper.py --help`; verified `.env` loading via `scraper.load_config()`
- date: 2026-03-14

* [X] t1.2: define grocery data model and file layout (1-2 commits)
** acceptance criteria
- decide and document the files/directories for:
  - retailer raw exports
  - enriched line items
  - observed products
  - canonical products
  - product links
- define stable column schemas for each file
- explicitly separate retailer-specific parsing from cross-retailer canonicalization

** notes
- this is the guardrail task so we don't make giant-specific hacks the system of record
- keep schema minimal but extensible

** evidence
- commit: `42dbae1` on branch `cx`
- tests: reviewed `giant_output/raw/history.json`, one sample raw order json, `giant_output/orders.csv`, `giant_output/items.csv`; documented schemas in `pm/data-model.org`
- date: 2026-03-15

* [X] t1.3: build giant parser/enricher from raw json (2-4 commits)
** acceptance criteria
- parser reads giant raw order json files
- outputs `items_enriched.csv`
- preserves core raw values plus parsed fields such as:
  - normalized item name
  - image url
  - size value/unit guesses
  - pack/count guesses
  - fee/store-brand flags
  - per-unit/per-weight derived price where possible
- parser is deterministic and rerunnable

** notes
- do not attempt canonical cross-store matching yet
- parser should preserve ambiguity rather than hallucinating precision

** evidence
- commit: `14f2cc2` on branch `cx`
- tests: `./venv/bin/python -m unittest discover -s tests`; `./venv/bin/python enrich_giant.py`; verified `giant_output/items_enriched.csv` on real raw data
- date: 2026-03-16

* [X] t1.4: generate observed-product layer from enriched items (2-3 commits)

** acceptance criteria
- distinct observed products are generated from enriched giant items
- each observed product has a stable `observed_product_id`
- observed products aggregate:
  - first seen / last seen
  - times seen
  - representative upc
  - representative image url
  - representative normalized name
- outputs `products_observed.csv`

** notes
- observed product is retailer-facing, not yet canonical
- likely key is some combo of retailer + upc + normalized name

** evidence
- commit: `dc39214` on branch `cx`
- tests: `./venv/bin/python -m unittest discover -s tests`; `./venv/bin/python build_observed_products.py`; verified `giant_output/products_observed.csv`
- date: 2026-03-16

* [X] t1.5: build review queue for unresolved or low-confidence products (1-3 commits)

** acceptance criteria
- produce a review file containing observed products needing manual review
- include enough context to review quickly:
  - raw names
  - parsed names
  - upc
  - image url
  - example prices
  - seen count
- reviewed status can be stored and reused

** notes
- this is where human-in-the-loop starts
- optimize for “approve once, remember forever”

** evidence
- commit: `9b13ec3` on branch `cx`
- tests: `./venv/bin/python -m unittest discover -s tests`; `./venv/bin/python build_review_queue.py`; verified `giant_output/review_queue.csv`
- date: 2026-03-16

* [X] t1.6: create canonical product layer and observed→canonical links (2-4 commits)

** acceptance criteria
- define and create `products_canonical.csv`
- define and create `product_links.csv`
- support linking one or more observed products to one canonical product
- canonical product schema supports food-cost comparison fields such as:
  - product type
  - variant
  - size
  - measure type
  - normalized quantity basis

** notes
- this is the first cross-retailer abstraction layer
- do not require llm assistance for v1

** evidence
- commit: `347cd44` on branch `cx`
- tests: `./venv/bin/python -m unittest discover -s tests`; `./venv/bin/python build_canonical_layer.py`; verified seeded `giant_output/products_canonical.csv` and `giant_output/product_links.csv`
- date: 2026-03-16

* [X] t1.7: implement auto-link rules for easy matches (2-3 commits)

** acceptance criteria
- auto-link can match observed products to canonical products using deterministic rules
- rules include at least:
  - exact upc
  - exact normalized name
  - exact size/unit match where available
- low-confidence cases remain unlinked for review

** notes
- keep the rules conservative
- false positives are worse than unresolved items

** evidence
- commit: `385a31c` on branch `cx`
- tests: `./venv/bin/python -m unittest discover -s tests`; `./venv/bin/python build_canonical_layer.py`; verified auto-linked `giant_output/products_canonical.csv` and `giant_output/product_links.csv`
- date: 2026-03-16

* [X] t1.8: support costco raw ingest path (2-5 commits)

** acceptance criteria
- add a costco-specific raw ingest/export path
- fetch costco receipt summary and receipt detail payloads from graphql endpoint
- persist raw json under `costco_output/raw/orders.csv` and `./items.csv`, same format as giant
- costco-native identifiers such as `transactionBarcode` as order id and `itemNumber` as retailer item id
- preserve discount/coupon rows rather than dropping

** notes
- focus on raw costco acquisistion and flattening
- do not force costco identifiers into `upc`
- bearer/auth values should come from local env, not source

** evidence
- commit: `da00288` on branch `cx`
- tests: `./venv/bin/python -m unittest discover -s tests`; `./venv/bin/python scrape_costco.py --help`; verified `costco_output/raw/*.json`, `costco_output/orders.csv`, and `costco_output/items.csv` from the local sample payload
- date: 2026-03-16

* [X] t1.8.1: support costco parser/enricher path (2-4 commits)

** acceptance criteria
- add a costco-specific enrich step producing `costco_output/items_enriched.csv`
- output rows into the same shared enriched schema family as Giant
- support costco-specific parsing for:
  - `itemDescription01` + `itemDescription02`
  - `itemNumber` as `retailer_item_id`
  - discount lines / negative rows
  - common size patterns such as `25#`, `48 OZ`, `2/24 OZ`, `6-PACK`
- preserve obvious unknowns as blank rather than guessed values

** notes
- this is the real schema compatibility proof, not raw ingest alone
- expect weaker identifiers than Giant

** evidence
- commit: `da00288` on branch `cx`
- tests: `./venv/bin/python -m unittest discover -s tests`; `./venv/bin/python enrich_costco.py`; verified `costco_output/items_enriched.csv`
- date: 2026-03-16
* [X] t1.8.2: validate cross-retailer observed/canonical flow (1-3 commits)

** acceptance criteria
- feed Giant and Costco enriched rows through the same observed/canonical pipeline
- confirm at least one product class can exist as:
  - Giant observed product
  - Costco observed product
  - one shared canonical product
- document the exact example used for proof

** notes
- keep this to one or two well-behaved product classes first
- apples, eggs, bananas, or flour are better than weird prepared foods

** evidence
- commit: `da00288` on branch `cx`
- tests: `./venv/bin/python -m unittest discover -s tests`; `./venv/bin/python validate_cross_retailer_flow.py`; proof example: Giant `FRESH BANANA` and Costco `BANANAS 3 LB / 1.36 KG` share one canonical in `combined_output/proof_examples.csv`
- date: 2026-03-16
* [X] t1.8.3: extend shared schema for retailer-native ids and adjustment lines (1-2 commits)

** acceptance criteria
- add shared fields needed for non-upc retailers, including:
  - `retailer_item_id`
  - `is_discount_line`
  - `is_coupon_line` or equivalent if needed
- keep `upc` nullable across the pipeline
- update downstream builders/tests to accept retailers with blank `upc`

** notes
- this prevents costco from becoming a schema hack
- do this once instead of sprinkling exceptions everywhere

** evidence
- commit: `9497565` on branch `cx`
- tests: `./venv/bin/python -m unittest discover -s tests`; verified shared enriched fields in `giant_output/items_enriched.csv` and `costco_output/items_enriched.csv`
- date: 2026-03-16
* [X] t1.8.4: verify and correct costco receipt enumeration (1–2 commits)

** acceptance criteria
- confirm graphql summary query returns all expected receipts
- compare `inWarehouse` count vs number of `receipts` returned
- widen or parameterize date window if necessary; website shows receipts in 3-month windows
- persist request metadata (`startDate`, `endDate`, `documentType`, `documentSubType`)
- emit warning when receipt counts mismatch

** notes
- goal is to confirm we are enumerating all receipts before parsing
- do not expand schema or parser logic in this task
- keep changes limited to summary query handling and diagnostics

** evidence
- commit: `ac82fa6` on branch `cx`
- tests: `./venv/bin/python -m unittest discover -s tests`; `./venv/bin/python scrape_costco.py --help`; reviewed the sample Costco summary request in `pm/scrape-giant.org` against `costco_output/raw/summary.json` and added 3-month window chunking plus mismatch diagnostics
- date: 2026-03-16
* [X] t1.8.5: refactor costco scraper auth and UX with giant scraper

** acceptance criteria
- remove manual auth env vars
- load costco cookies from firefox session
- require only logged-in browser
- replace start/end date flags with --months-back
- maintain same raw output structure
- ensure summary_lookup keys are collision-safe by using a composite key (transactionBarcode + transactionDateTime) instead of transactionBarcode alone 

** notes
- align Costco acquisition ergonomics with the Giant scraper
- keep downstream Costco parsing and shared schemas unchanged

** evidence
- commit: `c0054dc` on branch `cx`
- tests: `./venv/bin/python -m unittest discover -s tests`; `./venv/bin/python scrape_costco.py --help`; verified Costco summary/detail flattening now uses composite receipt keys in unit tests
- date: 2026-03-16
* [X] t1.8.6: add browser session helper (2-4 commits)

** acceptance criteria
- create a separate Python module/script that extracts firefox browser session data needed for giant and costco scrapers.
- support Firefox and Costco first, including:
  - loading cookies via existing browser-cookie approach
  - reading browser storage needed for dynamic auth headers (e.g. Costco bearer token)
  - copying locked browser sqlite/db files to a temp location before reading when necessary
- expose a small interface usable by scrapers, e.g. cookie jar + storage/header values
- keep retailer-specific parsing of extracted session data outside the low-level browser access layer
- structure the helper so Chromium-family browser support can be added later without changing scraper call sites

** notes
- goal is to replace manual `.env` copying of volatile browser-derived auth data
- session bootstrap only, not full browser automation
- prefer one shared helper over retailer-specific ad hoc storage reads
- Firefox only; Chromium support later

** evidence
- commit: `7789c2e` on branch `cx`
- tests: `./venv/bin/python -m unittest discover -s tests`; `./venv/bin/python scrape_giant.py --help`; `./venv/bin/python scrape_costco.py --help`; verified Firefox storage token extraction and locked-db copy behavior in unit tests
- date: 2026-03-16
* [X] t1.8.7: simplify costco session bootstrap and remove over-abstraction (2-4 commits)

** acceptance criteria
- make `scrape_costco.py` readable end-to-end without tracing through multiple partial bootstrap layers
- keep `browser_session.py` limited to low-level browser data access only:
  - firefox profile discovery
  - cookie loading
  - storage reads
  - sqlite copy/read helpers
- remove or sharply reduce `retailer_sessions.py` so retailer-specific header extraction lives with the retailer scraper or in a very small retailer-specific helper
- make session bootstrap flow explicit and linear:
  - load browser context
  - extract costco auth values
  - build request headers
  - build requests session
- eliminate inconsistent/obsolete function signatures and dead call paths (e.g. mixed `build_session(...)` calling conventions, stale fallback branches, mismatched `build_headers(...)` args)
- add one focused bootstrap debug print showing whether cookies, authorization, client id, and client identifier were found
- preserve current working behavior where available; this is a refactor/clarification task, not a feature expansion task

** notes
- goal is to restore concern separation and debuggability
- prefer obvious retailer-specific code over “generic” helpers that guess and obscure control flow
- browser access can stay shared; retailer auth mapping should be explicit
- no new heuristics in this task

** evidence
- commit: `d7a0329` on branch `cx`
- tests: `./venv/bin/python -m unittest discover -s tests`; `./venv/bin/python scrape_costco.py --help`; verified explicit Costco session bootstrap flow in `scrape_costco.py` and low-level-only browser access in `browser_session.py`
- date: 2026-03-16
* [X] t1.9: build pivot-ready normalized purchase log and comparison metrics (2-4 commits)

** acceptance criteria
- produce a flat `purchases.csv` suitable for excel pivot tables and pivot charts
- each purchase row preserves:
  - purchase date
  - retailer
  - order id
  - raw item name
  - normalized item name
  - canonical item id when resolved
  - quantity / unit
  - line total
  - store/location info where available
- derive normalized comparison fields where possible on enriched or observed product rows:
  - `price_per_lb`
  - `price_per_oz`
  - `price_per_each`
  - `price_per_count`
- preserve the source basis used to derive each metric, e.g.:
  - parsed size/unit
  - receipt weight
  - explicit count/pack
- emit nulls when basis is unknown, conflicting, or ambiguous
- support pivot-friendly analysis of purchase frequency and item cost over time
- document at least one Giant vs Costco comparison example using the normalized metrics

** notes
- compute metrics as close to the raw observation as possible
- canonical layer can aggregate later, but should not invent missing unit economics
- unit discipline matters more than coverage
- raw item name must be retained for audit/debugging

** evidence
- commit: `be1bf63` on branch `cx`
- tests: `./venv/bin/python -m unittest discover -s tests`; `./venv/bin/python build_purchases.py`; verified `combined_output/purchases.csv` and `combined_output/comparison_examples.csv` on the current Giant + Costco dataset
- date: 2026-03-16

* [X] t1.11: define review and item-resolution workflow for unresolved products (2-3 commits)

** acceptance criteria
- define the persistent files used to resolve unknown items, including:
  - review queue
  - canonical item catalog
  - alias / mapping layer if separate
- specify how unresolved items move from `review_queue.csv` into the final normalized purchase log
- define the manual resolution workflow, including:
  - what the human edits
  - what script is rerun afterward
  - how resolved mappings are persisted for future runs
- ensure resolved items are positively identified into stable canonical item ids rather than one-off text substitutions
- document how raw item name, normalized item name, and canonical item id are all retained

** notes
- goal is “approve once, reuse forever”
- keep the workflow simple and auditable
- manual review is fine; the important part is making it durable and rerunnable

** evidence
- commit: `c7dad54` on branch `cx`
- tests: `./venv/bin/python -m unittest discover -s tests`; `./venv/bin/python build_purchases.py`; `./venv/bin/python review_products.py --refresh-only`; verified `combined_output/review_queue.csv`, `combined_output/review_resolutions.csv` workflow, and `combined_output/canonical_catalog.csv`
- date: 2026-03-16
* [X] t1.12: simplify review process display
Clearly show current state separate from proposed future state.
** acceptance criteria
1. Display position in review queue, e.g., (1/22)
2. Display compact header with observed_product under review, queue position, and canonical decision, e.g.: "Resolve [n] observed product group [name]  and associated items to canonical_name [name]? (\n [n] matched items)"
3. color-code outputs based on info, input/prompt, warning/error
   1. color action menu/requests for input differently from display text; do not color individual options separately
   2. "no canonical_name suggestions found" is informational, not a warning/error.
4. update action menu `[x]exclude` to `e[x]clude`
5. on each review item, display a list of all matched items to be linked, sorted by descending date:
   1. YYYY-mm-dd, price, raw item name, normalized item name, upc, retailer
   2. image URL, if exists
   3. Sample:
6. on each review item, suggest (but do not auto-apply) up to 3 likely existing canonicals using determinstic rules, e.g:
   1. exact normalized name match
   2. prefix/contains match on canonical name
   3. exact UPC
7. Sample Entry:
#+begin_comment
 Review 7/22: Resolve observed_product MIXED PEPPER to canonical_name [__]?
 2 matched items:
  [1] 2026-03-12 | 7.49 | MIXED PEPPER 6-PACK | MIXED PEPPER | [upc] | costco | [img_url]
  [2] [YYYY-mm-dd] | [price] | [raw_name] | [observed_name] | [upc] | [retailer] | [img_url]
 2 canonical suggestions found:
  [1] BELL PEPPERS, PRODUCE
  [2] PEPPER, SPICES
#+end_comment
8. When link is selected, users should be able to select the number of the item in the list, e.g.:
#+begin_comment
  Select the canonical_name to associate [n] items with:
   [1] GRB GRADU PCH PUF1. | gcan_01b0d623aa02
   [2] BTB CHICKEN         | gcan_0201f0feb749
   [3] LIME                | gcan_02074d9e7359
#+end_comment
9. Add confirmation to link selection with instructions, "[n] [observed_name] and future observed_name matches will be associated with [canonical_name], is this ok?
     actions: [Y]es  [n]o  [b]ack  [s]kip  [q]uit

- reinforce project terminology such as raw_name, observed_name, canonical_name   

** evidence
- commit: `7b8141c`, `d39497c`
- tests: `./venv/bin/python -m unittest discover -s tests`; `./venv/bin/python -m unittest tests.test_review_workflow tests.test_purchases`; `./venv/bin/python review_products.py --help`; verified compact review header, numbered matched-item display, informational no-suggestion state, numbered canonical selection, and confirmation flow
- date: 2026-03-17

** notes
- The key improvement was shifting the prompt from system metadata to reviewer intent: one observed_product, its matched retailer rows, and one canonical_name decision.
- Numbered canonical selection plus confirmation worked better than free-text id entry and should reduce accidental links.
- Deterministic suggestions remain intentionally conservative; they speed up common cases, but unresolved items still depend on human review by design.

* [X] t1.13.1 pipeline accountability and stage visibility (1-2 commits)
add simple accounting so we can see what survives or drops at each pipeline stage

** AC
1. emit counts for raw, enriched, combined/observed, review-queued, canonical-linked, and final purchase-log rows
2. report unresolved and dropped item counts explicitly
3. make it easy to verify that missing items were intentionally left in review rather than silently lost
- pm note: simple text/json/csv summary is sufficient; trust and visibility matter more than presentation

** evidence
- commit: `967e19e`
- tests: `./venv/bin/python -m unittest discover -s tests`; `./venv/bin/python report_pipeline_status.py --help`; `./venv/bin/python report_pipeline_status.py`; verified `combined_output/pipeline_status.csv` and `combined_output/pipeline_status.json`
- date: 2026-03-17

** notes
- Added a single explicit status script instead of threading counters through every pipeline step; this keeps the pipeline simple while still making row survival visible.
- The most useful check here is `unresolved_not_in_review_rows`; when it is non-zero, we know we have a real accounting bug rather than normal unresolved work.

* [X] t1.13.2 costco discount matching and net pricing in enrich_costco (2-3 commits)
refactor costco enrichment so discount lines are matched to purchased items and net pricing is preserved

** AC
1. detect costco discount/coupon rows like `/<retailer_item_id>` and match them to purchased items within the same order
2. preserve raw discount rows for auditability while also carrying matched discount values onto the purchased item row
3. add explicit fields for discount-adjusted pricing, e.g. `matched_discount_amount` and `net_line_total` (or equivalent)
4. preserve original raw receipt amounts (`line_total`) without overwriting them
- pm note: keep this retailer-specific and explicit; do not introduce generic discount heuristics

** evidence
- commit: `56a03bc`
- tests: `./venv/bin/python -m unittest discover -s tests`; `./venv/bin/python enrich_costco.py`; verified matched Costco discount rows now populate `matched_discount_amount` and `net_line_total` while preserving raw `line_total`
- date: 2026-03-17

** notes
- Kept this retailer-specific and literal: only discount rows with `/<retailer_item_id>` are matched, and only within the same order.
- Raw discount rows are still preserved for auditability; the purchased row now carries the matched adjustment separately rather than overwriting the original amount.
* [X] t1.13.3 canonical cleanup and review-first product identity (3-4 commits)
refactor canonical generation so product identity is cleaner, duplicate canonicals are reduced, and unresolved items stay in review instead of spawning junk canonicals

** AC
1. stop auto-creating new canonical products from weak normalized names alone; unresolved items remain in `review_queue.csv`
2. canonical names are based on stable product identity rather than noisy observed titles
3. packaging/count/size tokens are removed from canonical names when they belong in structured fields (`pack_qty`, `size_value`, `size_unit`)
4. consolidate obvious duplicate canonicals (e.g. egg/lime cases) and ensure final outputs retain raw item name, normalized item name, and canonical item id
- pm note: prefer conservative canonical creation and a better manual review loop over aggressive auto-unification

** evidence
- commit: `08e2a86`
- tests: `./venv/bin/python -m unittest discover -s tests`; `./venv/bin/python build_purchases.py`; `./venv/bin/python review_products.py --refresh-only`; verified weaker exact-name cases now remain unresolved in `combined_output/review_queue.csv` and canonical names are cleaned before auto-catalog creation
- date: 2026-03-17

** notes
- Removed weak exact-name auto-canonical creation so ambiguous products stay in review instead of generating junk canonicals.
- Canonical display names are now cleaned of obvious punctuation and packaging noise, but I kept the cleanup conservative rather than adding a broad fuzzy merge layer.
* [X] t1.14: refactor retailer collection into the new data model (2-4 commits)
move Giant and Costco collection into the new collect structure and make both retailers emit the same collected schemas

** Acceptance Criteria
1. create retailer-specific collect scripts in the target naming pattern, e.g.:
  - collect_giant_web.py
  - collect_costco_web.py
2. collected outputs conform to pm/data-model.org:
  - data/<retailer-method>/raw/...
  - data/<retailer-method>/collected_orders.csv
  - data/<retailer-method>/collected_items.csv
3. current Giant and Costco raw acquisition behavior is preserved during the move
4. collected schemas preserve retailer truth and provenance:
  - no interpretation beyond basic flattening
  - raw_order_path/raw_history_path remain usable
  - unknown values remain blank rather than guessed
5. old paths should be removed or deprecated
6. collect_* scripts do not depend on any normalize/review files or scripts
- pm note: this is a path/schema refactor, not a parsing rewrite

** evidence
- commit: `48c6eaf`
- tests: `./venv/bin/python -m unittest tests.test_scraper tests.test_costco_pipeline tests.test_browser_session`; `./venv/bin/python collect_giant_web.py --help`; `./venv/bin/python collect_costco_web.py --help`; `./venv/bin/python scrape_giant.py --help`; `./venv/bin/python scrape_costco.py --help`
- datetime: 2026-03-18

** notes
- Kept this as a path/schema move, not a parsing rewrite: the existing Giant and Costco collection behavior remains in place behind new `collect_*` entry points.
- Added lightweight deprecation nudges on the legacy `scrape_*` commands rather than removing them immediately, so the move is inspectable and low-risk.
- The main schema fix was on Giant collection, which was missing retailer/provenance/audit fields that Costco collection already carried.

* [X] t1.14.1: refactor retailer normalization into the new normalized_items schema (3-5 commits)
make Giant and Costco emit the shared normalized line-item schema without introducing cross-retailer identity logic

** Acceptance Criteria
1. create retailer-specific normalize scripts in the target naming pattern, e.g.:
   - normalize_giant_web.py
   - normalize_costco_web.py
2. normalized outputs conform to pm/data-model.org:
   - data/<retailer-method>/normalized_items.csv
   - one row per collected line item
   - normalized_row_id is stable and present
   - normalized_item_id is stable, present, and represents retailer-level identity reused across repeated purchase rows when deterministic retailer evidence is sufficient
   - normalized_quantity and normalized_quantity_unit
   - repeated rows for the same retailer product resolve to the same normalized_item_id only when supported by deterministic retailer evidence, e.g. exact upc, exact retailer_item_id, exact cleaned name + same size/pack
   - normalization_basis is explicit
3. Giant normalization preserves current useful parsing:
   - normalized item name
   - size/unit/pack parsing
   - fee/store-brand flags
   - derived price fields
4. Costco normalization preserves current useful parsing:
   - normalized item name
   - size/unit/pack parsing
   - explicit discount matching using retailer-specific logic
   - matched_discount_amount and net_line_total
5. both normalizers preserve raw retailer truth:
   - line_total is never overwritten
   - unknown values remain blank rather than guessed
6. no cross-retailer identity assignment occurs in normalization
7. normalize never uses fuzzy or semantic matching to assign normalized_item_id

- pm note: prefer explicit retailer-specific code paths over generic normalization helpers unless the duplication is truly mechanical
- pm note: normalization may resolve retailer-level identity, but not catalog identity
- pm note: normalized_item_id is the only retailer-level grouping identity; do not introduce observed_products or a second grouping artifact
** evidence
- commit: `9064de5`
- tests: `./venv/bin/python -m unittest discover -s tests`; `./venv/bin/python -m unittest tests.test_enrich_giant tests.test_costco_pipeline tests.test_purchases`; `./venv/bin/python normalize_giant_web.py --help`; `./venv/bin/python normalize_costco_web.py --help`; `./venv/bin/python enrich_giant.py --help`; `./venv/bin/python enrich_costco.py --help`
- datetime: 2026-03-18

** notes
- Kept the existing Giant and Costco parsing logic intact and added the new normalized schema fields in place, rather than rewriting the enrichers from scratch.
- `normalized_item_id` is always present, but it only collapses repeated rows when the evidence is strong; otherwise it falls back to row-level identity via `normalized_row_id`.
- Added `normalize_*` entry points for the new data-model layout while leaving the legacy `enrich_*` commands available during the transition.

* [X] t1.14.2: finalize filesystem and schema alignment for the refactor (2-4 commits)
bring on-disk outputs fully into the target `data/` structure without changing retailer behavior

** Acceptance Criteria
1. retailer data directories conform to pm/data-model.org:
   - `data/giant-web/raw/...`
   - `data/giant-web/collected_orders.csv`
   - `data/giant-web/collected_items.csv`
   - `data/giant-web/normalized_items.csv`
   - `data/costco-web/raw/...`
   - `data/costco-web/collected_orders.csv`
   - `data/costco-web/collected_items.csv`
   - `data/costco-web/normalized_items.csv`
2. review/combine outputs are moved or rewritten into the target review paths:
   - `data/review/review_queue.csv`
   - `data/review/product_links.csv`
   - `data/review/review_resolutions.csv`
   - `data/review/purchases.csv`
   - `data/review/pipeline_status.csv`
   - `data/review/pipeline_status.json`
3. old transitional output paths are either:
   - removed from active script defaults, or
   - left as explicit compatibility shims with clear deprecation notes
4. no recollection is required if existing raw files and collected csvs can be moved/copied losslessly into the new structure
5. no schema information is lost during the move:
   - raw paths still resolve
   - collected/normalized csvs still open with the expected headers
6. README and task/docs reflect the final active paths
- pm note: prefer moving/adapting existing files over recollecting from retailers unless a real data loss or schema mismatch forces recollection
- pm note: this is a structure-alignment task, not a retailer parsing task

** evidence
- commit: `d2e6f2a`
- tests: `./venv/bin/python -m unittest discover -s tests`; `./venv/bin/python build_purchases.py`; `./venv/bin/python review_products.py --refresh-only`; `./venv/bin/python report_pipeline_status.py`; `./venv/bin/python build_purchases.py --help`; `./venv/bin/python review_products.py --help`; `./venv/bin/python report_pipeline_status.py --help`; verified `data/giant-web/collected_orders.csv`, `data/giant-web/collected_items.csv`, `data/costco-web/collected_orders.csv`, `data/costco-web/collected_items.csv`, `data/catalog.csv`, and archived transitional review outputs under `data/review/archive/`
- datetime: [2026-03-20 10:04:15 EDT]

** notes
- No recollection was needed; existing raw and collected exports were adapted in place and moved into the target names.
- Updated the active script defaults to point at `data/...` so the code and on-disk layout now agree.
- Kept obviously obsolete review artifacts, but moved them under `data/review/archive/` instead of deleting them outright.

* [X] t1.14.3: retailer-specific Costco normalization cleanup (2-4 commits)
tighten Costco-specific normalization so normalized item names are cleaner and deterministic retailer grouping is less noisy

** Acceptance Criteria
1. improve Costco item-name cleanup for obvious non-identity noise, such as:
   - trailing slash fragments
   - code tokens and receipt-format artifacts
   - duplicated measurement fragments already captured in structured fields
2. preserve deterministic normalization rules only:
   - exact retailer_item_id
   - exact cleaned name + same size/pack when needed
   - approved retailer alias
   - no fuzzy or semantic matching
3. normalized Costco names improve on known bad examples, e.g.:
   - `MANDARIN /` -> cleaner normalized item name
   - `LIFE 6'TABLE ... /` -> cleaner normalized item name
4. cleanup does not overwrite retailer truth:
   - raw `item_name` is unchanged
   - parsed `size_value`, `size_unit`, `pack_qty`, and pricing fields remain intact
5. discount-row behavior remains correct:
   - matched discount rows still populate `matched_discount_amount`
   - `net_line_total` remains correct
   - discount rows remain auditable
6. add regression tests for the cleaned Costco examples and any new parsing rules
- pm note: keep this explicitly Costco-specific; do not introduce a generic cleanup framework
- pm note: prefer a short allowlist/blocklist of known receipt artifacts over broad heuristics

** evidence
- commit: `bcec6b3`
- tests: `./venv/bin/python -m unittest discover -s tests`; `./venv/bin/python -m unittest tests.test_costco_pipeline`; `./venv/bin/python normalize_costco_web.py`; verified live cleaned examples in `data/costco-web/normalized_items.csv`, including `MANDARINS 2.27 KG / 5 LBS -> MANDARIN` and `LIFE 6'TABLE MDL #80873U - T12/H3/P36 -> LIFE 6'TABLE MDL`
- datetime: 2026-03-20 11:09:32 EDT

** notes
- Kept this explicitly Costco-specific and narrow: the cleanup removes known logistics/code artifacts and orphan slash tokens without introducing fuzzy naming logic.
- The structured parsing still owns size/pack extraction, so name cleanup can safely strip dual-unit and logistics fragments after those fields are parsed.
- Discount-line behavior remains unchanged; this task only cleaned normalized names and preserved the existing audit trail.

* [X] t1.15: refactor review/combine pipeline around normalized_item_id and catalog links (4-8 commits)
replace the old observed/canonical workflow with a review-first pipeline that uses normalized_item_id as the retailer-level review unit and links it to catalog items

** Acceptance Criteria
1. refactor review outputs to conform to pm/data-model.org:
   - data/review/review_queue.csv
   - data/review/product_links.csv
   - data/catalog.csv
   - data/purchases.csv
2. review logic uses normalized_item_id as the upstream retailer-level review identity:
   - no dependency on observed_product_id
   - no dependency on products_observed.csv
   - one review/link decision applies to all purchase rows sharing the same normalized_item_id
3. product_links.csv stores review-approved links from normalized_item_id to catalog_id
   - one row per approved retailer-level identity to catalog mapping
4. catalog.csv entries are review-first and conservative:
   - no auto-creation from weak normalized names alone
   - names come from reviewed catalog naming, not raw retailer strings
   - packaging/count is not embedded in catalog_name unless essential to identity
   - catalog_name/product_type/category/brand/variant may be blank until reviewed; blank is preferred to guessed
5. purchases.csv remains pivot-ready and retains:
   - raw item name
   - normalized item name
   - normalized_row_id (not for review)
   - normalized_item_id
   - catalog_id
   - catalog fields
   - raw line_total
   - matched_discount_amount and net_line_total when present
   - derived price fields and their bases
6. terminal review flow remains simple and usable:
   - reviewer sees one grouped retailer item identity (normalized_item_id) with count and list of matches, not one prompt per purchase row; use existing pattern as a template
   - link to existing catalog item
   - create new catalog item
   - exclude
   - skip
7. pipeline accounting remains valid after the refactor:
   - unresolved items are visible
   - missing items are not silently dropped
8. pm note: prefer a better manual review loop over aggressive automatic grouping. initial manual data entry is expected, and should resolve over time
9. pm note: keep review/combine auditable; each catalog link should be explainable from normalized rows and review state

** evidence
- commit: `9104781`
- tests: `./venv/bin/python -m unittest discover -s tests`; `./venv/bin/python build_purchases.py`; `./venv/bin/python review_products.py --refresh-only`; `./venv/bin/python report_pipeline_status.py`; `./venv/bin/python build_purchases.py --help`; `./venv/bin/python review_products.py --help`; `./venv/bin/python report_pipeline_status.py --help`
- datetime: 2026-03-20 11:27:12 EDT

** notes
- The old observed/canonical auto-layer is no longer in the active review/combine path. `build_purchases.py`, `review_products.py`, and `report_pipeline_status.py` now operate on `normalized_item_id`, `catalog_id`, and `catalog_name`.
- I kept the review CLI shape intentionally close to the pre-refactor flow so the project only changed its identity model, not the operator workflow.
- Existing auto-generated catalog rows are no longer carried forward by default; only deliberate catalog entries survive. That keeps the new `catalog.csv` conservative, but it also means prior observed-based auto-links do not migrate into the new model.
- Live rerun after the refactor produced `627` purchase rows, `387` review-queue rows, `407` distinct normalized items, `0` linked normalized items, and `0` unresolved rows missing from the review queue.

* [X] t1.16: cleanup review process and format

** acceptance criteria
1. Add intro text explaining:
   1. catalog name: unique product including variant but not packaging, eg "whole milk", "sharp cheddar cheese"
   2. product type: general product you would like to compare to, eg "milk", "cheese"
   3. category: eg "dairy"
2. Reformat input per item
   1. Change matched item field display order
   2. Add count of distinct normalized_item_ids and total purchase rows already linked to the catalog item
   3. Add option to select catalog suggestion directly
   #+begin_comment
    Review 7/22: MIXED PEPPER 6-PK
    2 matched items:
     - MIXED PEPPER 6-PK | costco | 2026-03-12 | 7.49 | [img_url]
     - [raw_name] |  [retailer] | [YYYY-mm-dd] | [price] | [img_url]  
    2 catalog suggestions found:
     [1] bell pepper, pepper, produce (42 items)
     [2] ground pepper, spice, baking (1 item)
   [#] link to suggestion [n]ew [s]kip e[x]clude [q]uit > 
   #+end_comment
3. When creating new, ask for input in catalog_name, product_type, category order
   1. enter to accept blank value
4. Each reviewed item is saved after user input, not at the end of the script.
   1. on new creation, create entry in catalog.csv and create entry in product_links.csv
   2. on link existing, create entry in product_links.csv
   3. update review_queue.csv status for item immediately after action
5. linking operates at normalized_item_id level, not per normalized_row_id
6. ensure catalog.csv and product_links.csv are human-editable and consistent so manual correction is possible without tooling
   

** evidence
- commit: `975d44b`
- tests: `./venv/bin/python -m unittest discover -s tests`; `./venv/bin/python review_products.py --refresh-only`; `./venv/bin/python review_products.py --help`
- datetime: 2026-03-20 12:45:25 EDT

** notes
- The main flow change is operational rather than architectural: each review decision now persists immediately to `review_resolutions.csv`, `catalog.csv`, `product_links.csv`, and the on-disk `review_queue.csv`.
- Direct numeric selection works well for suggestion-heavy review, while `[l]ink existing` remains available as a fallback when the suggestion list is empty or incomplete.
- I kept the review data model unchanged from `t1.15`; this task only tightened the prompt format, field order, and save behavior.

* [X] t1.16.1: add catalog search flow to review ui (2-3 commits)
enable fast lookup of catalog items during review via tokenized search and replace manual list scanning

** acceptance criteria
1. replace `[l]ink existing` with `[f]ind` in review prompt:
   - `[#] link to suggestion  [f]ind  [n]ew  [s]kip   [x]exclude  [q]uit >`
2. implement search flow:
   - on `s`, prompt: `search: `
   - tokenize input using same normalization rules as suggestion matching
   - return ranked list of catalog items where tokens overlap with:
     - catalog_name
     - product_type
     - variant
   - display results in same numbered format as suggestions:
     [1] flour, flour, baking (12 items, 48 rows)
3. allow direct selection from search results:
   - when user inputs number, immediately creates approved resolution and product_links rows
   - returns to next review item
4. reuse match logic used for suggestion matching; no new matching system introduced
   - future improvements to matching logic will therefore apply in both places
5. search results exclude already-linked current normalized_item_id target
6. fallback behavior:
   - if no results, print `no matches found`
   - allow retry or return to main prompt
7. keep interaction tight:
   - no full catalog dump
   - max ~10 results returned
   - sorted by simple score (token overlap count)
8. persistence:
   - selected link writes immediately to `product_links.csv`
   - no buffering until script end

- pm note: optimize for speed over correctness; this is a manual assist tool, not a ranking system
- pm note: improve manual lookup flow only, don't retool or create a second algorithm
** evidence
- commit: `f93b9aa`
- tests: `./venv/bin/python -m unittest discover -s tests`; `./venv/bin/python review_products.py --help`; `./venv/bin/python review_products.py --refresh-only`
- datetime: 2026-03-20 13:34:57 EDT

** notes
- The search path reuses the same lightweight token matching rules as suggestion ranking, so there is still only one matching system to maintain.
- Direct numeric suggestion-pick remains the fastest happy path; search is the fallback when suggestions are sparse or missing.
- Search intentionally optimizes for manual speed rather than smart ranking: simple token overlap, max 10 rows, and immediate persistence on selection.
- Follow-up fix: search moved to `[f]ind` so `[s]kip` remains available at the main prompt.

* [X] t1.17: fix normalized quantity derivation and carry it through purchases (2-4 commits)
correct and document deterministic normalized quantity fields so unit-cost analysis works across package sizes

** Acceptance Criteria
1. populate and validate `normalized_quantity` and `normalized_quantity_unit` in `data/<retailer-method>/normalized_items.csv`
   - these columns already exist and must be corrected rather than reintroduced
2. carry `normalized_quantity` and `normalized_quantity_unit` through to `data/review/purchases.csv`
3. derive normalized quantity deterministically from existing parsed fields only:
   - `qty`
   - `pack_qty`
   - `size_value`
   - `size_unit`
   - `measure_type`
4. prefer the best deterministic basis rather than falling back to `each` too early:
   - count items when count is explicit
   - weight items when parsed weight is explicit
   - volume items when parsed volume is explicit
   - `each` only when no better basis is available
5. handle common cases explicitly, including totals derived from deterministic patterns such as:
   - `18 count`
   - `5 lb`
   - `64 oz`
   - `2 each`
6. preserve blanks when no reliable normalized quantity basis can be derived
7. existing `normalized_item_id` values remain stable; this task must not change retailer-level grouping identity
8. document the derivation rules and any intentional conversions or non-conversions in `pm/data-model.org` or task notes
   - if unit conversions are allowed, they must be explicit and minimal
- pm note: keep this deterministic and conservative; do not introduce fuzzy inference
- pm note: if `lb <-> oz` or volume conversions are used, document them directly rather than hiding them in code
- pm note: this task enables cost analysis and charting, not catalog/review changes

** evidence
- commit: `d25448b`
- tests: `./venv/bin/python -m unittest discover -s tests`; `./venv/bin/python normalize_giant_web.py`; `./venv/bin/python normalize_costco_web.py`; `./venv/bin/python build_purchases.py`
- datetime: 2026-03-21 21:02:21 EDT

** notes
- The missing purchases fields were a carry-through bug: normalization had `normalized_quantity` and `normalized_quantity_unit`, but `build_purchases.py` never wrote them into `data/review/purchases.csv`.
- Normalized quantity now prefers explicit package basis over `each`, so rows like `PEPSI 6PK 7.5Z` resolve to `90 oz` and `KS ALMND BAR US 1.74QTS` purchased twice resolves to `3.48 qt`.
- The derivation stays conservative and does not convert units during normalization; parsed units such as `oz`, `lb`, `qt`, and `count` are preserved as-is.
* [ ] t1.18: add regression tests for known quantity/price failures (1-2 commits)
capture the currently broken comparison cases before changing normalization or purchases logic

** acceptance criteria
1. when generating `data/purchases.csv`, add `effective_price` = `effective_total` / `normalized_quantity`
2. define `effective_price` behavior explicitly from the covered cases:
   - use `net_line_total` when present and non-zero, else use `line_total`
   - divide by `normalized_quantity` when `normalized_quantity > 0`
   - leave blank when no valid denominator exists
   - never emit `0` or divide-by-zero for missing-basis cases
   - `effective_price` only comparable within same `normalized_quantity_unit` unless later analysis converts the units
3. ensure the new tests assert the intended `effective_price` behavior for the known banana, ice, and beef patty examples
4. add tests covering known broken cases:
   - giant bananas produce non-blank effective price
   - giant bagged ice produces non-zero effective price
   - costco bananas retain correct effective price
   - beef patty comparison rows preserve expected quantity basis behavior
5. tests fail against current broken behavior and document the expected outcome
6. include at least one assertion that effective_price is blank rather than `0` or divide-by-zero when no denominator exists
7. pm note: this task should only add tests/fixtures and not change business logic
** pm identified problems
we have a few problems to scope. looks like:
1. normalize_giant_web not always propagating weight data to price_per
2. effective_price calc needs more robust matching algo (my excel hack is clearly not engouh)
```
catalog_name	banana										
Average of effective_price	Column Labels										
Row Labels	8/6/2024	12/6/2024	12/12/2024	1/7/2025	1/24/2025	2/16/2025	2/20/2025	6/25/2025	2/14/2026	3/12/2026	Grand Total
Jan				#DIV/0!	0.496666667						#DIV/0!
Feb						#DIV/0!	#DIV/0!		0.496666667		#DIV/0!
Mar										0.496666667	0.496666667
Jun								#DIV/0!			#DIV/0!
Aug	0.496666667										0.496666667
Dec		#DIV/0!	#DIV/0!								#DIV/0!
Grand Total	0.496666667	#DIV/0!	#DIV/0!	#DIV/0!	0.496666667	#DIV/0!	#DIV/0!	#DIV/0!	0.496666667	0.496666667	#DIV/0!

purchase_date	retailer	normalized_item_name	catalog_name	category	product_type	qty	unit	normalized_quantity	normalized_quantity_unit	pack_qty	size_value	size_unit	measure_type	line_total	unit_price	net_line_total	price_per_each	price_per_each_basis	price_per_count	price_per_count_basis	price_per_lb	price_per_lb_basis	price_per_oz	price_per_oz_basis	effective_price
8/6/2024	costco	BANANA	banana	produce	banana	1	E	3	lb		3	lb	weight	1.49	1.49	1.49	1.49	line_total_over_qty			0.4967	parsed_size_lb	0.031	parsed_size_lb_to_oz	0.496666667
12/6/2024	giant	BANANA	banana	produce	banana	1	LB						weight	0.99	0.99		0.99	line_total_over_qty			0.5893	picked_weight_lb	0.0368	picked_weight_lb_to_oz	#DIV/0!
12/12/2024	giant	BANANA	banana	produce	banana	1	LB						weight	1.37	1.37		1.37	line_total_over_qty			0.5905	picked_weight_lb	0.0369	picked_weight_lb_to_oz	#DIV/0!
1/7/2025	giant	BANANA	banana	produce	banana	1	LB						weight	1.44	1.44		1.44	line_total_over_qty			0.5902	picked_weight_lb	0.0369	picked_weight_lb_to_oz	#DIV/0!
1/24/2025	costco	BANANA	banana	produce	banana	1	E	3	lb		3	lb	weight	1.49	1.49	1.49	1.49	line_total_over_qty			0.4967	parsed_size_lb	0.031	parsed_size_lb_to_oz	0.496666667
2/16/2025	giant	BANANA	banana	produce	banana	2	LB						weight	2.54	1.27		1.27	line_total_over_qty			0.588	picked_weight_lb	0.0367	picked_weight_lb_to_oz	#DIV/0!
2/20/2025	giant	BANANA	banana	produce	banana	1	LB						weight	1.4	1.4		1.4	line_total_over_qty			0.5907	picked_weight_lb	0.0369	picked_weight_lb_to_oz	#DIV/0!
6/25/2025	giant	BANANA	banana	produce	banana	1	LB						weight	1.29	1.29		1.29	line_total_over_qty			0.589	picked_weight_lb	0.0368	picked_weight_lb_to_oz	#DIV/0!
2/14/2026	costco	BANANA	banana	produce	banana	1	E	3	lb		3	lb	weight	1.49	1.49	1.49	1.49	line_total_over_qty			0.4967	parsed_size_lb	0.031	parsed_size_lb_to_oz	0.496666667
3/12/2026	costco	BANANA	banana	produce	banana	2	E	6	lb		3	lb	weight	2.98	1.49	2.98	1.49	line_total_over_qty			0.4967	parsed_size_lb	0.031	parsed_size_lb_to_oz	0.496666667

purchase_date	retailer	normalized_item_name	catalog_name	category	product_type	qty	unit	normalized_quantity	normalized_quantity_unit	pack_qty	size_value	size_unit	measure_type	line_total	unit_price	net_line_total	price_per_each	price_per_each_basis	price_per_count	price_per_count_basis	price_per_lb	price_per_lb_basis	price_per_oz	price_per_oz_basis	effective_price
9/9/2023	costco	BEEF PATTIES 6# BAG	beef patty	meat	hamburger	1	E	1	each				each	26.99	26.99	26.99	26.99	line_total_over_qty							26.99
11/26/2025	giant	80% PATTIES PK12	beef patty	meat	hamburger	1	LB						weight	10.05	10.05		10.05	line_total_over_qty			7.7907	picked_weight_lb	0.4869	picked_weight_lb_to_oz	#DIV/0!

purchase_date	retailer	normalized_item_name	catalog_name	category	product_type	qty	unit	normalized_quantity	normalized_quantity_unit	pack_qty	size_value	size_unit	measure_type	line_total	unit_price	net_line_total	price_per_each	price_per_each_basis	price_per_count	price_per_count_basis	price_per_lb	price_per_lb_basis	price_per_oz	price_per_oz_basis	effective_price
5/26/2025	giant	BAGGED ICE	bagged ice cubes	frozen	ice	2	EA	40	lb		20	lb	weight	9.98	4.99		4.99	line_total_over_qty			0.2495	parsed_size_lb	0.0156	parsed_size_lb_to_oz	0
6/12/2025	giant	BAG ICE CUBED	bagged ice cubes	frozen	ice	1	EA	10	lb		10	lb	weight	3.49	3.49		3.49	line_total_over_qty			0.349	parsed_size_lb	0.0218	parsed_size_lb_to_oz	0
9/13/2025	giant	BAGGED ICE	bagged ice cubes	frozen	ice	2	EA	20	lb		10	lb	weight	6.98	3.49		3.49	line_total_over_qty			0.349	parsed_size_lb	0.0218	parsed_size_lb_to_oz	0
10/10/2025	giant	BAGGED ICE	bagged ice cubes	frozen	ice	1	EA	20	lb		20	lb	weight	4.99	4.99		4.99	line_total_over_qty			0.2495	parsed_size_lb	0.0156	parsed_size_lb_to_oz	0
```
** evidence
- commit:
- tests:
- datetime:

** notes

* [ ] t1.18.1: fix effective price calculation precedence and blank handling (1-3 commits)
correct purchases/effective price logic for the known broken cases using existing normalized fields

** acceptance criteria
1. effective_price uses explicit numerator precedence:
   - prefer `net_line_total`
   - fallback to `line_total`
2. effective_price uses `normalized_quantity` when present and > 0
3. effective_price is blank when no valid denominator exists
4. effective_price is never written as `0` or divide-by-zero for missing-basis cases
5. existing regression tests for bananas and ice pass
- pm note: keep this limited to calculation logic; do not broaden into catalog or review changes

** evidence
- commit:
- tests:
- datetime:

** notes


* [ ] t1.18.2: fix giant normalization quantity carry-through for weight-based items (1-3 commits)
ensure giant normalization emits usable normalized quantity for known weight-based cases

** acceptance criteria
1. giant bananas populate normalized quantity and unit from deterministic weight basis
2. giant weight-based items that already produce `price_per_lb` also carry enough quantity basis for effective price calculation where supported
3. existing regression tests pass without changing normalized_item_id behavior
4. blanks are preserved only when no deterministic quantity basis exists
- pm note: this task is about normalization carry-through, not fuzzy matching or catalog cleanup

** evidence
- commit:
- tests:
- datetime:

** notes
* [ ] t1.10: add optional llm-assisted suggestion workflow for unresolved normalized retailer items (2-4 commits)

** acceptance criteria
- llm suggestions are generated only for unresolved normalized retailer items
- llm outputs are stored as suggestions, not auto-applied truth
- reviewer can approve/edit/reject suggestions
- approved decisions are persisted into canonical/link files

** notes
- bounded assistant, not autonomous goblin
- image urls may become useful here

** evidence
- commit:
- tests:
- date: