updated readme and prep for next phase

README.md

@@ -1,103 +1,113 @@
 # scrape-giant
 
-Small grocery-history pipeline for Giant receipts.
+CLI to pull purchase history from Giant and Costco websites and refine into a single product catalog for external analysis.
 
-The project currently does four things:
+Run each script step-by-step from the terminal.
 
-1. scrape Giant in-store order history from an active Firefox session
+## What It Does
-2. enrich raw line items into a deterministic `items_enriched.csv`
-3. aggregate retailer-facing observed products and build a manual review queue
-4. create a first-pass canonical product layer plus conservative auto-links
 
-The work so far is Giant-specific on the ingest side and intentionally simple on
+1. `scrape_giant.py`: download Giant orders and items
-the shared product-model side.
+2. `enrich_giant.py`: normalize Giant line items
+3. `scrape_costco.py`: download Costco orders and items
+4. `enrich_costco.py`: normalize Costco line items
+5. `build_purchases.py`: combine retailer outputs into one purchase table
+6. `review_products.py`: review unresolved product matches in the terminal
 
-## Current flow
+## Requirements
 
-Run the commands from the repo root with the project venv active, or call them
+- Python 3.10+
-directly through `./venv/bin/python`.
+- Firefox installed with active Giant and Costco sessions
+
+## Install
 
 ```bash
-./venv/bin/python scraper.py
+python -m venv venv
-./venv/bin/python enrich_giant.py
+./venv/scripts/activate
-./venv/bin/python build_observed_products.py
+pip install -r requirements.txt
-./venv/bin/python build_review_queue.py
-./venv/bin/python build_canonical_layer.py
 ```
 
-## Inputs
+## Optional `.env`
 
-- Firefox cookies for `giantfood.com`
+Current version works best with `.env` in the project root.  The scraper will prompt for these values if they are not found in the current browser session.  
-- `GIANT_USER_ID` and `GIANT_LOYALTY_NUMBER` in `.env`, shell env, or prompts
+- `scrape_giant` prompts if `GIANT_USER_ID` or `GIANT_LOYALTY_NUMBER` is missing.
-- Giant raw order payloads in `giant_output/raw/`
+- `scrape_costco` tries `.env` first, then Firefox local storage for session-backed values; `COSTCO_CLIENT_IDENTIFIER` should still be set explicitly.
 
-## Outputs
+```env
+GIANT_USER_ID=...
+GIANT_LOYALTY_NUMBER=...
 
-Current generated files live under `giant_output/`:
+COSTCO_X_AUTHORIZATION=...
+COSTCO_X_WCS_CLIENTID=...
+COSTCO_CLIENT_IDENTIFIER=...
+```
 
-- `orders.csv`: flattened visit/order rows from the Giant history API
+## Run Order
-- `items.csv`: flattened raw line items from fetched order detail payloads
-- `items_enriched.csv`: deterministic parsed/enriched line items
-- `products_observed.csv`: retailer-facing observed product groups
-- `review_queue.csv`: products needing manual review
-- `products_canonical.csv`: shared canonical product rows
-- `product_links.csv`: observed-to-canonical links
 
-Raw json remains the source of truth:
+Run the pipeline in this order:
 
-- `giant_output/raw/history.json`
+```bash
-- `giant_output/raw/<order_id>.json`
+python scrape_giant.py
+python enrich_giant.py
+python scrape_costco.py
+python enrich_costco.py
+python build_purchases.py
+python review_products.py
+python build_purchases.py
+```
 
-## Scripts
+Why run `build_purchases.py` twice:
+- first pass builds the current combined dataset and review queue inputs
+- `review_products.py` writes durable review decisions
+- second pass reapplies those decisions into the purchase output
 
-- `scraper.py`: fetches Giant history/detail payloads and updates `orders.csv` and `items.csv`
+If you only want to refresh the queue without reviewing interactively:
-- `enrich_giant.py`: reads raw Giant order json and writes `items_enriched.csv`
-- `build_observed_products.py`: groups enriched rows into `products_observed.csv`
-- `build_review_queue.py`: generates `review_queue.csv` and preserves review status on reruns
-- `build_canonical_layer.py`: builds `products_canonical.csv` and `product_links.csv`
 
-## Notes on the current model
+```bash
+python review_products.py --refresh-only
+```
 
-- Observed products are retailer-specific: Giant, Costco.
+## Key Outputs
-- Canonical products are the first cross-retailer layer.
-- Auto-linking is conservative:
-  exact UPC first, then exact normalized name plus exact size/unit context, then
-  exact normalized name when there is no size context to conflict.
-- Fee rows are excluded from auto-linking.
-- Unknown values are left blank instead of guessed.
 
-## Verification
+Giant:
+- `giant_output/orders.csv`
+- `giant_output/items.csv`
+- `giant_output/items_enriched.csv`
 
-Run the test suite with:
+Costco:
+- `costco_output/orders.csv`
+- `costco_output/items.csv`
+- `costco_output/items_enriched.csv`
+
+Combined:
+- `combined_output/purchases.csv`
+- `combined_output/review_queue.csv`
+- `combined_output/review_resolutions.csv`
+- `combined_output/canonical_catalog.csv`
+- `combined_output/product_links.csv`
+- `combined_output/comparison_examples.csv`
+
+## Review Workflow
+
+Run `review_products.py` to cleanup unresolved or weakly unified items:
+- link an item to an existing canonical product
+- create a new canonical product
+- exclude an item
+- skip it for later
+Decisions are saved and reused on later runs.
+
+## Notes
+- This project is designed around fragile retailer scraping flows, so the code favors explicit retailer-specific steps over heavy abstraction.
+- `scrape_giant.py` and `scrape_costco.py` are meant to work as standalone acquisition scripts.
+- `validate_cross_retailer_flow.py` is a proof/check script, not a required production step.
+
+## Test
 
 ```bash
 ./venv/bin/python -m unittest discover -s tests
 ```
 
-Useful one-off rebuilds:
+## Project Docs
 
-```bash
+- `pm/tasks.org`: task tracking
-./venv/bin/python enrich_giant.py
+- `pm/data-model.org`: current data model notes
-./venv/bin/python build_observed_products.py
+- `pm/review-workflow.org`: review and resolution workflow
-./venv/bin/python build_review_queue.py
-./venv/bin/python build_canonical_layer.py
-```
-
-## Project docs
-
-- `pm/tasks.org`: task log and evidence
-- `pm/data-model.org`: file layout and schema decisions
-
-## Status
-
-Completed through `t1.7`:
-
-- Giant receipt fetch CLI
-- data model and file layout
-- Giant parser/enricher
-- observed products
-- review queue
-- canonical layer scaffold
-- conservative auto-link rules
-
-Next planned task is `t1.8`: add a Costco raw ingest path.

browser_session.py

@@ -0,0 +1,129 @@
+import configparser
+import os
+import shutil
+import sqlite3
+import tempfile
+from pathlib import Path
+
+import browser_cookie3
+
+
+def find_firefox_profile_dir():
+    profiles_ini = firefox_profiles_root() / "profiles.ini"
+    parser = configparser.RawConfigParser()
+    if not profiles_ini.exists():
+        raise FileNotFoundError(f"Firefox profiles.ini not found at {profiles_ini}")
+
+    parser.read(profiles_ini, encoding="utf-8")
+    profiles = []
+    for section in parser.sections():
+        if not section.startswith("Profile"):
+            continue
+        path_value = parser.get(section, "Path", fallback="")
+        if not path_value:
+            continue
+        is_relative = parser.getboolean(section, "IsRelative", fallback=True)
+        profile_path = (
+            profiles_ini.parent / path_value if is_relative else Path(path_value)
+        )
+        profiles.append(
+            (
+                parser.getboolean(section, "Default", fallback=False),
+                profile_path,
+            )
+        )
+
+    if not profiles:
+        raise FileNotFoundError("No Firefox profiles found in profiles.ini")
+
+    profiles.sort(key=lambda item: (not item[0], str(item[1])))
+    return profiles[0][1]
+
+
+def firefox_profiles_root():
+    if os.name == "nt":
+        appdata = os.getenv("APPDATA", "").strip()
+        if not appdata:
+            raise FileNotFoundError("APPDATA is not set")
+        return Path(appdata) / "Mozilla" / "Firefox"
+    return Path.home() / ".mozilla" / "firefox"
+
+
+def load_firefox_cookies(domain_name, profile_dir):
+    cookie_file = Path(profile_dir) / "cookies.sqlite"
+    return browser_cookie3.firefox(cookie_file=str(cookie_file), domain_name=domain_name)
+
+
+def read_firefox_local_storage(profile_dir, origin_filter):
+    storage_root = profile_dir / "storage" / "default"
+    if not storage_root.exists():
+        return {}
+
+    for ls_path in storage_root.glob("*/ls/data.sqlite"):
+        origin = decode_firefox_origin(ls_path.parents[1].name)
+        if origin_filter.lower() not in origin.lower():
+            continue
+        return {
+            stringify_sql_value(row[0]): stringify_sql_value(row[1])
+            for row in query_sqlite(ls_path, "SELECT key, value FROM data")
+        }
+    return {}
+
+
+def read_firefox_webapps_store(profile_dir, origin_filter):
+    webapps_path = profile_dir / "webappsstore.sqlite"
+    if not webapps_path.exists():
+        return {}
+
+    values = {}
+    for row in query_sqlite(
+        webapps_path,
+        "SELECT originKey, key, value FROM webappsstore2",
+    ):
+        origin = stringify_sql_value(row[0])
+        if origin_filter.lower() not in origin.lower():
+            continue
+        values[stringify_sql_value(row[1])] = stringify_sql_value(row[2])
+    return values
+
+def query_sqlite(path, query):
+    copied_path = copy_sqlite_to_temp(path)
+    connection = None
+    cursor = None
+    try:
+        connection = sqlite3.connect(copied_path)
+        cursor = connection.cursor()
+        cursor.execute(query)
+        rows = cursor.fetchall()
+        return rows
+    except sqlite3.OperationalError:
+        return []
+    finally:
+        if cursor is not None:
+            cursor.close()
+        if connection is not None:
+            connection.close()
+        copied_path.unlink(missing_ok=True)
+
+
+def copy_sqlite_to_temp(path):
+    fd, tmp = tempfile.mkstemp(suffix=".sqlite")
+    os.close(fd)
+    shutil.copyfile(path, tmp)
+    return Path(tmp)
+
+def decode_firefox_origin(raw_origin):
+    origin = raw_origin.split("^", 1)[0]
+    return origin.replace("+++", "://")
+
+def stringify_sql_value(value):
+    if value is None:
+        return ""
+    if isinstance(value, bytes):
+        for encoding in ("utf-8", "utf-16-le", "utf-16"):
+            try:
+                return value.decode(encoding)
+            except UnicodeDecodeError:
+                continue
+        return value.decode("utf-8", errors="ignore")
+    return str(value)

build_purchases.py

@@ -0,0 +1,414 @@
+from decimal import Decimal
+from pathlib import Path
+
+import click
+
+import build_canonical_layer
+import build_observed_products
+import validate_cross_retailer_flow
+from enrich_giant import format_decimal, to_decimal
+from layer_helpers import read_csv_rows, stable_id, write_csv_rows
+
+
+PURCHASE_FIELDS = [
+    "purchase_date",
+    "retailer",
+    "order_id",
+    "line_no",
+    "observed_item_key",
+    "observed_product_id",
+    "canonical_product_id",
+    "review_status",
+    "resolution_action",
+    "raw_item_name",
+    "normalized_item_name",
+    "image_url",
+    "retailer_item_id",
+    "upc",
+    "qty",
+    "unit",
+    "pack_qty",
+    "size_value",
+    "size_unit",
+    "measure_type",
+    "line_total",
+    "unit_price",
+    "store_name",
+    "store_number",
+    "store_city",
+    "store_state",
+    "price_per_each",
+    "price_per_each_basis",
+    "price_per_count",
+    "price_per_count_basis",
+    "price_per_lb",
+    "price_per_lb_basis",
+    "price_per_oz",
+    "price_per_oz_basis",
+    "is_discount_line",
+    "is_coupon_line",
+    "is_fee",
+    "raw_order_path",
+]
+
+EXAMPLE_FIELDS = [
+    "example_name",
+    "canonical_product_id",
+    "giant_purchase_date",
+    "giant_raw_item_name",
+    "giant_price_per_lb",
+    "costco_purchase_date",
+    "costco_raw_item_name",
+    "costco_price_per_lb",
+    "notes",
+]
+
+CATALOG_FIELDS = [
+    "canonical_product_id",
+    "canonical_name",
+    "category",
+    "product_type",
+    "brand",
+    "variant",
+    "size_value",
+    "size_unit",
+    "pack_qty",
+    "measure_type",
+    "notes",
+    "created_at",
+    "updated_at",
+]
+
+RESOLUTION_FIELDS = [
+    "observed_product_id",
+    "canonical_product_id",
+    "resolution_action",
+    "status",
+    "resolution_notes",
+    "reviewed_at",
+]
+
+
+def decimal_or_zero(value):
+    return to_decimal(value) or Decimal("0")
+
+
+def derive_metrics(row):
+    line_total = to_decimal(row.get("line_total"))
+    qty = to_decimal(row.get("qty"))
+    pack_qty = to_decimal(row.get("pack_qty"))
+    size_value = to_decimal(row.get("size_value"))
+    picked_weight = to_decimal(row.get("picked_weight"))
+    size_unit = row.get("size_unit", "")
+
+    price_per_each = row.get("price_per_each", "")
+    price_per_lb = row.get("price_per_lb", "")
+    price_per_oz = row.get("price_per_oz", "")
+    price_per_count = ""
+
+    basis_each = ""
+    basis_count = ""
+    basis_lb = ""
+    basis_oz = ""
+
+    if price_per_each:
+        basis_each = "line_total_over_qty"
+    elif line_total is not None and qty not in (None, 0):
+        price_per_each = format_decimal(line_total / qty)
+        basis_each = "line_total_over_qty"
+
+    if line_total is not None and pack_qty not in (None, 0):
+        total_count = pack_qty * (qty or Decimal("1"))
+        if total_count not in (None, 0):
+            price_per_count = format_decimal(line_total / total_count)
+            basis_count = "line_total_over_pack_qty"
+
+    if picked_weight not in (None, 0):
+        price_per_lb = format_decimal(line_total / picked_weight) if line_total is not None else ""
+        price_per_oz = (
+            format_decimal((line_total / picked_weight) / Decimal("16"))
+            if line_total is not None
+            else ""
+        )
+        basis_lb = "picked_weight_lb"
+        basis_oz = "picked_weight_lb_to_oz"
+    elif line_total is not None and size_value not in (None, 0):
+        total_units = size_value * (pack_qty or Decimal("1")) * (qty or Decimal("1"))
+        if size_unit == "lb" and total_units not in (None, 0):
+            per_lb = line_total / total_units
+            price_per_lb = format_decimal(per_lb)
+            price_per_oz = format_decimal(per_lb / Decimal("16"))
+            basis_lb = "parsed_size_lb"
+            basis_oz = "parsed_size_lb_to_oz"
+        elif size_unit == "oz" and total_units not in (None, 0):
+            per_oz = line_total / total_units
+            price_per_oz = format_decimal(per_oz)
+            price_per_lb = format_decimal(per_oz * Decimal("16"))
+            basis_lb = "parsed_size_oz_to_lb"
+            basis_oz = "parsed_size_oz"
+
+    return {
+        "price_per_each": price_per_each,
+        "price_per_each_basis": basis_each,
+        "price_per_count": price_per_count,
+        "price_per_count_basis": basis_count,
+        "price_per_lb": price_per_lb,
+        "price_per_lb_basis": basis_lb,
+        "price_per_oz": price_per_oz,
+        "price_per_oz_basis": basis_oz,
+    }
+
+
+def order_lookup(rows, retailer):
+    return {
+        (retailer, row["order_id"]): row
+        for row in rows
+    }
+
+
+def read_optional_csv_rows(path):
+    path = Path(path)
+    if not path.exists():
+        return []
+    return read_csv_rows(path)
+
+
+def load_resolution_lookup(resolution_rows):
+    lookup = {}
+    for row in resolution_rows:
+        if not row.get("observed_product_id"):
+            continue
+        lookup[row["observed_product_id"]] = row
+    return lookup
+
+
+def merge_catalog_rows(existing_rows, auto_rows):
+    merged = {}
+    for row in auto_rows + existing_rows:
+        canonical_product_id = row.get("canonical_product_id", "")
+        if canonical_product_id:
+            merged[canonical_product_id] = row
+    return sorted(merged.values(), key=lambda row: row["canonical_product_id"])
+
+
+def catalog_row_from_canonical(row):
+    return {
+        "canonical_product_id": row.get("canonical_product_id", ""),
+        "canonical_name": row.get("canonical_name", ""),
+        "category": row.get("category", ""),
+        "product_type": row.get("product_type", ""),
+        "brand": row.get("brand", ""),
+        "variant": row.get("variant", ""),
+        "size_value": row.get("size_value", ""),
+        "size_unit": row.get("size_unit", ""),
+        "pack_qty": row.get("pack_qty", ""),
+        "measure_type": row.get("measure_type", ""),
+        "notes": row.get("notes", ""),
+        "created_at": row.get("created_at", ""),
+        "updated_at": row.get("updated_at", ""),
+    }
+
+
+def build_link_state(enriched_rows):
+    observed_rows = build_observed_products.build_observed_products(enriched_rows)
+    canonical_rows, link_rows = build_canonical_layer.build_canonical_layer(observed_rows)
+    giant_row, costco_row = validate_cross_retailer_flow.find_proof_pair(observed_rows)
+    canonical_rows, link_rows, _proof_rows = validate_cross_retailer_flow.merge_proof_pair(
+        canonical_rows,
+        link_rows,
+        giant_row,
+        costco_row,
+    )
+
+    observed_id_by_key = {
+        row["observed_key"]: row["observed_product_id"] for row in observed_rows
+    }
+    canonical_id_by_observed = {
+        row["observed_product_id"]: row["canonical_product_id"] for row in link_rows
+    }
+    return observed_rows, canonical_rows, link_rows, observed_id_by_key, canonical_id_by_observed
+
+
+def build_purchase_rows(
+    giant_enriched_rows,
+    costco_enriched_rows,
+    giant_orders,
+    costco_orders,
+    resolution_rows,
+):
+    all_enriched_rows = giant_enriched_rows + costco_enriched_rows
+    (
+        observed_rows,
+        canonical_rows,
+        link_rows,
+        observed_id_by_key,
+        canonical_id_by_observed,
+    ) = build_link_state(all_enriched_rows)
+    resolution_lookup = load_resolution_lookup(resolution_rows)
+    for observed_product_id, resolution in resolution_lookup.items():
+        action = resolution.get("resolution_action", "")
+        status = resolution.get("status", "")
+        if status != "approved":
+            continue
+        if action in {"link", "create"} and resolution.get("canonical_product_id"):
+            canonical_id_by_observed[observed_product_id] = resolution["canonical_product_id"]
+        elif action == "exclude":
+            canonical_id_by_observed[observed_product_id] = ""
+    orders_by_id = {}
+    orders_by_id.update(order_lookup(giant_orders, "giant"))
+    orders_by_id.update(order_lookup(costco_orders, "costco"))
+
+    purchase_rows = []
+    for row in sorted(
+        all_enriched_rows,
+        key=lambda item: (item["order_date"], item["retailer"], item["order_id"], int(item["line_no"])),
+    ):
+        observed_key = build_observed_products.build_observed_key(row)
+        observed_product_id = observed_id_by_key.get(observed_key, "")
+        order_row = orders_by_id.get((row["retailer"], row["order_id"]), {})
+        metrics = derive_metrics(row)
+        resolution = resolution_lookup.get(observed_product_id, {})
+        purchase_rows.append(
+            {
+                "purchase_date": row["order_date"],
+                "retailer": row["retailer"],
+                "order_id": row["order_id"],
+                "line_no": row["line_no"],
+                "observed_item_key": row["observed_item_key"],
+                "observed_product_id": observed_product_id,
+                "canonical_product_id": canonical_id_by_observed.get(observed_product_id, ""),
+                "review_status": resolution.get("status", ""),
+                "resolution_action": resolution.get("resolution_action", ""),
+                "raw_item_name": row["item_name"],
+                "normalized_item_name": row["item_name_norm"],
+                "image_url": row.get("image_url", ""),
+                "retailer_item_id": row["retailer_item_id"],
+                "upc": row["upc"],
+                "qty": row["qty"],
+                "unit": row["unit"],
+                "pack_qty": row["pack_qty"],
+                "size_value": row["size_value"],
+                "size_unit": row["size_unit"],
+                "measure_type": row["measure_type"],
+                "line_total": row["line_total"],
+                "unit_price": row["unit_price"],
+                "store_name": order_row.get("store_name", ""),
+                "store_number": order_row.get("store_number", ""),
+                "store_city": order_row.get("store_city", ""),
+                "store_state": order_row.get("store_state", ""),
+                "is_discount_line": row["is_discount_line"],
+                "is_coupon_line": row["is_coupon_line"],
+                "is_fee": row["is_fee"],
+                "raw_order_path": row["raw_order_path"],
+                **metrics,
+            }
+        )
+    return purchase_rows, observed_rows, canonical_rows, link_rows
+
+
+def apply_manual_resolutions_to_links(link_rows, resolution_rows):
+    link_by_observed = {row["observed_product_id"]: dict(row) for row in link_rows}
+    for resolution in resolution_rows:
+        if resolution.get("status") != "approved":
+            continue
+        observed_product_id = resolution.get("observed_product_id", "")
+        action = resolution.get("resolution_action", "")
+        if not observed_product_id:
+            continue
+        if action == "exclude":
+            link_by_observed.pop(observed_product_id, None)
+            continue
+        if action in {"link", "create"} and resolution.get("canonical_product_id"):
+            link_by_observed[observed_product_id] = {
+                "observed_product_id": observed_product_id,
+                "canonical_product_id": resolution["canonical_product_id"],
+                "link_method": f"manual_{action}",
+                "link_confidence": "high",
+                "review_status": resolution.get("status", ""),
+                "reviewed_by": "",
+                "reviewed_at": resolution.get("reviewed_at", ""),
+                "link_notes": resolution.get("resolution_notes", ""),
+            }
+    return sorted(link_by_observed.values(), key=lambda row: row["observed_product_id"])
+
+
+def build_comparison_examples(purchase_rows):
+    giant_banana = None
+    costco_banana = None
+    for row in purchase_rows:
+        if row.get("normalized_item_name") != "BANANA":
+            continue
+        if not row.get("canonical_product_id"):
+            continue
+        if row["retailer"] == "giant" and row.get("price_per_lb"):
+            giant_banana = row
+        if row["retailer"] == "costco" and row.get("price_per_lb"):
+            costco_banana = row
+
+    if not giant_banana or not costco_banana:
+        return []
+
+    return [
+        {
+            "example_name": "banana_price_per_lb",
+            "canonical_product_id": giant_banana["canonical_product_id"],
+            "giant_purchase_date": giant_banana["purchase_date"],
+            "giant_raw_item_name": giant_banana["raw_item_name"],
+            "giant_price_per_lb": giant_banana["price_per_lb"],
+            "costco_purchase_date": costco_banana["purchase_date"],
+            "costco_raw_item_name": costco_banana["raw_item_name"],
+            "costco_price_per_lb": costco_banana["price_per_lb"],
+            "notes": "Example comparison using normalized price_per_lb across Giant and Costco",
+        }
+    ]
+
+
+@click.command()
+@click.option("--giant-items-enriched-csv", default="giant_output/items_enriched.csv", show_default=True)
+@click.option("--costco-items-enriched-csv", default="costco_output/items_enriched.csv", show_default=True)
+@click.option("--giant-orders-csv", default="giant_output/orders.csv", show_default=True)
+@click.option("--costco-orders-csv", default="costco_output/orders.csv", show_default=True)
+@click.option("--resolutions-csv", default="combined_output/review_resolutions.csv", show_default=True)
+@click.option("--catalog-csv", default="combined_output/canonical_catalog.csv", show_default=True)
+@click.option("--links-csv", default="combined_output/product_links.csv", show_default=True)
+@click.option("--output-csv", default="combined_output/purchases.csv", show_default=True)
+@click.option("--examples-csv", default="combined_output/comparison_examples.csv", show_default=True)
+def main(
+    giant_items_enriched_csv,
+    costco_items_enriched_csv,
+    giant_orders_csv,
+    costco_orders_csv,
+    resolutions_csv,
+    catalog_csv,
+    links_csv,
+    output_csv,
+    examples_csv,
+):
+    resolution_rows = read_optional_csv_rows(resolutions_csv)
+    purchase_rows, _observed_rows, canonical_rows, link_rows = build_purchase_rows(
+        read_csv_rows(giant_items_enriched_csv),
+        read_csv_rows(costco_items_enriched_csv),
+        read_csv_rows(giant_orders_csv),
+        read_csv_rows(costco_orders_csv),
+        resolution_rows,
+    )
+    existing_catalog_rows = read_optional_csv_rows(catalog_csv)
+    merged_catalog_rows = merge_catalog_rows(
+        existing_catalog_rows,
+        [catalog_row_from_canonical(row) for row in canonical_rows],
+    )
+    link_rows = apply_manual_resolutions_to_links(link_rows, resolution_rows)
+    example_rows = build_comparison_examples(purchase_rows)
+    write_csv_rows(catalog_csv, merged_catalog_rows, CATALOG_FIELDS)
+    write_csv_rows(links_csv, link_rows, build_canonical_layer.LINK_FIELDS)
+    write_csv_rows(output_csv, purchase_rows, PURCHASE_FIELDS)
+    write_csv_rows(examples_csv, example_rows, EXAMPLE_FIELDS)
+    click.echo(
+        f"wrote {len(purchase_rows)} purchase rows to {output_csv}, "
+        f"{len(merged_catalog_rows)} catalog rows to {catalog_csv}, "
+        f"and {len(example_rows)} comparison examples to {examples_csv}"
+    )
+
+
+if __name__ == "__main__":
+    main()

enrich_costco.py

@@ -213,9 +213,11 @@ def parse_costco_item(order_id, order_date, raw_path, line_no, item):
 
 def iter_costco_rows(raw_dir):
     for path in discover_json_files(raw_dir):
-        if path.name == "summary.json":
+        if path.name in {"summary.json", "summary_requests.json"}:
             continue
         payload = json.loads(path.read_text(encoding="utf-8"))
+        if not isinstance(payload, dict):
+            continue
         receipts = payload.get("data", {}).get("receiptsWithCounts", {}).get("receipts", [])
         for receipt in receipts:
             order_id = receipt["transactionBarcode"]

pm/review-workflow.org

@@ -0,0 +1,73 @@
+* review and item-resolution workflow
+
+This document defines the durable review workflow for unresolved observed
+products.
+
+** persistent files
+
+- `combined_output/purchases.csv`
+  Flat normalized purchase log. This is the review input because it retains:
+  - raw item name
+  - normalized item name
+  - observed product id
+  - canonical product id when resolved
+  - retailer/order/date/price context
+- `combined_output/review_queue.csv`
+  Current unresolved observed products grouped for review.
+- `combined_output/review_resolutions.csv`
+  Durable mapping decisions from observed products to canonical products.
+- `combined_output/canonical_catalog.csv`
+  Durable canonical item catalog used by manual review and later purchase-log
+  rebuilds.
+
+There is no separate alias file in v1. `review_resolutions.csv` is the mapping
+layer from observed products to canonical product ids.
+
+** workflow
+
+1. Run `build_purchases.py`
+   This refreshes the purchase log and seeds/updates the canonical catalog from
+   current auto-linked canonical rows.
+2. Run `review_products.py`
+   This rebuilds `review_queue.csv` from unresolved purchase rows and prompts in
+   the terminal for one observed product at a time.
+3. Choose one of:
+   - link to existing canonical
+   - create new canonical
+   - exclude
+   - skip
+4. `review_products.py` writes decisions immediately to:
+   - `review_resolutions.csv`
+   - `canonical_catalog.csv` when a new canonical item is created
+5. Rerun `build_purchases.py`
+   This reapplies approved resolutions so the final normalized purchase log now
+   carries the reviewed `canonical_product_id`.
+
+** what the human edits
+
+The primary interface is terminal prompts in `review_products.py`.
+
+The human provides:
+- existing canonical id when linking
+- canonical name/category/product type when creating a new canonical item
+- optional resolution notes
+
+The generated CSVs remain editable by hand if needed, but the intended workflow
+is terminal-first.
+
+** durability
+
+- Resolutions are keyed by `observed_product_id`, not by one-off text
+  substitution.
+- Canonical products are keyed by stable `canonical_product_id`.
+- Future runs reuse approved mappings through `review_resolutions.csv`.
+
+** retention of audit fields
+
+The final `purchases.csv` retains:
+- `raw_item_name`
+- `normalized_item_name`
+- `canonical_product_id`
+
+This preserves the raw receipt description, the deterministic parser output, and
+the human-approved canonical identity in one flat purchase log.

pm/tasks.org

@@ -254,9 +254,71 @@
 - commit: `c0054dc` on branch `cx`
 - tests: `./venv/bin/python -m unittest discover -s tests`; `./venv/bin/python scrape_costco.py --help`; verified Costco summary/detail flattening now uses composite receipt keys in unit tests
 - date: 2026-03-16
-* [ ] t1.9: compute normalized comparison metrics (2-4 commits)
+* [X] t1.8.6: add browser session helper (2-4 commits)
 
 ** acceptance criteria
+- create a separate Python module/script that extracts firefox browser session data needed for giant and costco scrapers.
+- support Firefox and Costco first, including:
+  - loading cookies via existing browser-cookie approach
+  - reading browser storage needed for dynamic auth headers (e.g. Costco bearer token)
+  - copying locked browser sqlite/db files to a temp location before reading when necessary
+- expose a small interface usable by scrapers, e.g. cookie jar + storage/header values
+- keep retailer-specific parsing of extracted session data outside the low-level browser access layer
+- structure the helper so Chromium-family browser support can be added later without changing scraper call sites
+
+** notes
+- goal is to replace manual `.env` copying of volatile browser-derived auth data
+- session bootstrap only, not full browser automation
+- prefer one shared helper over retailer-specific ad hoc storage reads
+- Firefox only; Chromium support later
+
+** evidence
+- commit: `7789c2e` on branch `cx`
+- tests: `./venv/bin/python -m unittest discover -s tests`; `./venv/bin/python scrape_giant.py --help`; `./venv/bin/python scrape_costco.py --help`; verified Firefox storage token extraction and locked-db copy behavior in unit tests
+- date: 2026-03-16
+* [X] t1.8.7: simplify costco session bootstrap and remove over-abstraction (2-4 commits)
+
+** acceptance criteria
+- make `scrape_costco.py` readable end-to-end without tracing through multiple partial bootstrap layers
+- keep `browser_session.py` limited to low-level browser data access only:
+  - firefox profile discovery
+  - cookie loading
+  - storage reads
+  - sqlite copy/read helpers
+- remove or sharply reduce `retailer_sessions.py` so retailer-specific header extraction lives with the retailer scraper or in a very small retailer-specific helper
+- make session bootstrap flow explicit and linear:
+  - load browser context
+  - extract costco auth values
+  - build request headers
+  - build requests session
+- eliminate inconsistent/obsolete function signatures and dead call paths (e.g. mixed `build_session(...)` calling conventions, stale fallback branches, mismatched `build_headers(...)` args)
+- add one focused bootstrap debug print showing whether cookies, authorization, client id, and client identifier were found
+- preserve current working behavior where available; this is a refactor/clarification task, not a feature expansion task
+
+** notes
+- goal is to restore concern separation and debuggability
+- prefer obvious retailer-specific code over “generic” helpers that guess and obscure control flow
+- browser access can stay shared; retailer auth mapping should be explicit
+- no new heuristics in this task
+
+** evidence
+- commit: `d7a0329` on branch `cx`
+- tests: `./venv/bin/python -m unittest discover -s tests`; `./venv/bin/python scrape_costco.py --help`; verified explicit Costco session bootstrap flow in `scrape_costco.py` and low-level-only browser access in `browser_session.py`
+- date: 2026-03-16
+* [X] t1.9: build pivot-ready normalized purchase log and comparison metrics (2-4 commits)
+
+** acceptance criteria
+- produce a flat `purchases.csv` suitable for excel pivot tables and pivot charts
+- each purchase row preserves:
+  - purchase date
+  - retailer
+  - order id
+  - raw item name
+  - normalized item name
+  - canonical item id when resolved
+  - quantity / unit
+  - line total
+  - store/location info where available
 - derive normalized comparison fields where possible on enriched or observed product rows:
   - `price_per_lb`
   - `price_per_oz`
@@ -267,17 +329,92 @@
   - receipt weight
   - explicit count/pack
 - emit nulls when basis is unknown, conflicting, or ambiguous
+- support pivot-friendly analysis of purchase frequency and item cost over time
 - document at least one Giant vs Costco comparison example using the normalized metrics
 
 ** notes
 - compute metrics as close to the raw observation as possible
 - canonical layer can aggregate later, but should not invent missing unit economics
 - unit discipline matters more than coverage
+- raw item name must be retained for audit/debugging
 
 ** evidence
-- commit:
+- commit: `be1bf63` on branch `cx`
-- tests:
+- tests: `./venv/bin/python -m unittest discover -s tests`; `./venv/bin/python build_purchases.py`; verified `combined_output/purchases.csv` and `combined_output/comparison_examples.csv` on the current Giant + Costco dataset
-- date:
+- date: 2026-03-16
+
+* [X] t1.11: define review and item-resolution workflow for unresolved products (2-3 commits)
+
+** acceptance criteria
+- define the persistent files used to resolve unknown items, including:
+  - review queue
+  - canonical item catalog
+  - alias / mapping layer if separate
+- specify how unresolved items move from `review_queue.csv` into the final normalized purchase log
+- define the manual resolution workflow, including:
+  - what the human edits
+  - what script is rerun afterward
+  - how resolved mappings are persisted for future runs
+- ensure resolved items are positively identified into stable canonical item ids rather than one-off text substitutions
+- document how raw item name, normalized item name, and canonical item id are all retained
+
+** notes
+- goal is “approve once, reuse forever”
+- keep the workflow simple and auditable
+- manual review is fine; the important part is making it durable and rerunnable
+
+** evidence
+- commit: `c7dad54` on branch `cx`
+- tests: `./venv/bin/python -m unittest discover -s tests`; `./venv/bin/python build_purchases.py`; `./venv/bin/python review_products.py --refresh-only`; verified `combined_output/review_queue.csv`, `combined_output/review_resolutions.csv` workflow, and `combined_output/canonical_catalog.csv`
+- date: 2026-03-16
+* [X] t1.12: simplify review process display
+Clearly show current state separate from proposed future state.
+** acceptance criteria
+1. Display position in review queue, e.g., (1/22)
+2. Display compact header with observed_product under review, queue position, and canonical decision, e.g.: "Resolve [n] observed product group [name]  and associated items to canonical_name [name]? (\n [n] matched items)"
+3. color-code outputs based on info, input/prompt, warning/error
+   1. color action menu/requests for input differently from display text; do not color individual options separately
+   2. "no canonical_name suggestions found" is informational, not a warning/error.
+4. update action menu `[x]exclude` to `e[x]clude`
+5. on each review item, display a list of all matched items to be linked, sorted by descending date:
+   1. YYYY-mm-dd, price, raw item name, normalized item name, upc, retailer
+   2. image URL, if exists
+   3. Sample:
+6. on each review item, suggest (but do not auto-apply) up to 3 likely existing canonicals using determinstic rules, e.g:
+   1. exact normalized name match
+   2. prefix/contains match on canonical name
+   3. exact UPC
+7. Sample Entry:
+#+begin_comment
+ Review 7/22: Resolve observed_product MIXED PEPPER to canonical_name [__]?
+ 2 matched items:
+  [1] 2026-03-12 | 7.49 | MIXED PEPPER 6-PACK | MIXED PEPPER | [upc] | costco | [img_url]
+  [2] [YYYY-mm-dd] | [price] | [raw_name] | [observed_name] | [upc] | [retailer] | [img_url]
+ 2 canonical suggestions found:
+  [1] BELL PEPPERS, PRODUCE
+  [2] PEPPER, SPICES
+#+end_comment
+8. When link is selected, users should be able to select the number of the item in the list, e.g.:
+#+begin_comment
+  Select the canonical_name to associate [n] items with:
+   [1] GRB GRADU PCH PUF1. | gcan_01b0d623aa02
+   [2] BTB CHICKEN         | gcan_0201f0feb749
+   [3] LIME                | gcan_02074d9e7359
+#+end_comment
+9. Add confirmation to link selection with instructions, "[n] [observed_name] and future observed_name matches will be associated with [canonical_name], is this ok?
+     actions: [Y]es  [n]o  [b]ack  [s]kip  [q]uit
+
+- reinforce project terminology such as raw_name, observed_name, canonical_name   
+
+** evidence
+- commit: `7b8141c`, `d39497c`
+- tests: `./venv/bin/python -m unittest discover -s tests`; `./venv/bin/python -m unittest tests.test_review_workflow tests.test_purchases`; `./venv/bin/python review_products.py --help`; verified compact review header, numbered matched-item display, informational no-suggestion state, numbered canonical selection, and confirmation flow
+- date: 2026-03-17
+
+** notes
+- The key improvement was shifting the prompt from system metadata to reviewer intent: one observed_product, its matched retailer rows, and one canonical_name decision.
+- Numbered canonical selection plus confirmation worked better than free-text id entry and should reduce accidental links.
+- Deterministic suggestions remain intentionally conservative; they speed up common cases, but unresolved items still depend on human review by design.
 
 * [ ] t1.10: add optional llm-assisted suggestion workflow for unresolved products (2-4 commits)
 

review_products.py

@@ -0,0 +1,426 @@
+from collections import defaultdict
+from datetime import date
+
+import click
+
+import build_purchases
+from layer_helpers import compact_join, stable_id, write_csv_rows
+
+
+QUEUE_FIELDS = [
+    "review_id",
+    "retailer",
+    "observed_product_id",
+    "canonical_product_id",
+    "reason_code",
+    "priority",
+    "raw_item_names",
+    "normalized_names",
+    "upc_values",
+    "example_prices",
+    "seen_count",
+    "status",
+    "resolution_action",
+    "resolution_notes",
+    "created_at",
+    "updated_at",
+]
+
+
+def build_review_queue(purchase_rows, resolution_rows):
+    by_observed = defaultdict(list)
+    resolution_lookup = build_purchases.load_resolution_lookup(resolution_rows)
+
+    for row in purchase_rows:
+        observed_product_id = row.get("observed_product_id", "")
+        if not observed_product_id:
+            continue
+        by_observed[observed_product_id].append(row)
+
+    today_text = str(date.today())
+    queue_rows = []
+    for observed_product_id, rows in sorted(by_observed.items()):
+        current_resolution = resolution_lookup.get(observed_product_id, {})
+        if current_resolution.get("status") == "approved":
+            continue
+        unresolved_rows = [row for row in rows if not row.get("canonical_product_id")]
+        if not unresolved_rows:
+            continue
+
+        retailers = sorted({row["retailer"] for row in rows})
+        review_id = stable_id("rvw", observed_product_id)
+        queue_rows.append(
+            {
+                "review_id": review_id,
+                "retailer": " | ".join(retailers),
+                "observed_product_id": observed_product_id,
+                "canonical_product_id": current_resolution.get("canonical_product_id", ""),
+                "reason_code": "missing_canonical_link",
+                "priority": "high",
+                "raw_item_names": compact_join(
+                    sorted({row["raw_item_name"] for row in rows if row["raw_item_name"]}),
+                    limit=8,
+                ),
+                "normalized_names": compact_join(
+                    sorted(
+                        {
+                            row["normalized_item_name"]
+                            for row in rows
+                            if row["normalized_item_name"]
+                        }
+                    ),
+                    limit=8,
+                ),
+                "upc_values": compact_join(
+                    sorted({row["upc"] for row in rows if row["upc"]}),
+                    limit=8,
+                ),
+                "example_prices": compact_join(
+                    sorted({row["line_total"] for row in rows if row["line_total"]}),
+                    limit=8,
+                ),
+                "seen_count": str(len(rows)),
+                "status": current_resolution.get("status", "pending"),
+                "resolution_action": current_resolution.get("resolution_action", ""),
+                "resolution_notes": current_resolution.get("resolution_notes", ""),
+                "created_at": current_resolution.get("reviewed_at", today_text),
+                "updated_at": today_text,
+            }
+        )
+    return queue_rows
+
+
+def save_resolution_rows(path, rows):
+    write_csv_rows(path, rows, build_purchases.RESOLUTION_FIELDS)
+
+
+def save_catalog_rows(path, rows):
+    write_csv_rows(path, rows, build_purchases.CATALOG_FIELDS)
+
+
+INFO_COLOR = "cyan"
+PROMPT_COLOR = "bright_yellow"
+WARNING_COLOR = "magenta"
+
+
+def sort_related_items(rows):
+    return sorted(
+        rows,
+        key=lambda row: (
+            row.get("purchase_date", ""),
+            row.get("order_id", ""),
+            int(row.get("line_no", "0") or "0"),
+        ),
+        reverse=True,
+    )
+
+
+def build_canonical_suggestions(related_rows, catalog_rows, limit=3):
+    normalized_names = {
+        row.get("normalized_item_name", "").strip().upper()
+        for row in related_rows
+        if row.get("normalized_item_name", "").strip()
+    }
+    upcs = {
+        row.get("upc", "").strip()
+        for row in related_rows
+        if row.get("upc", "").strip()
+    }
+    suggestions = []
+    seen_ids = set()
+
+    def add_matches(rows, reason):
+        for row in rows:
+            canonical_product_id = row.get("canonical_product_id", "")
+            if not canonical_product_id or canonical_product_id in seen_ids:
+                continue
+            seen_ids.add(canonical_product_id)
+            suggestions.append(
+                {
+                    "canonical_product_id": canonical_product_id,
+                    "canonical_name": row.get("canonical_name", ""),
+                    "reason": reason,
+                }
+            )
+            if len(suggestions) >= limit:
+                return True
+        return False
+
+    exact_upc_rows = [
+        row
+        for row in catalog_rows
+        if row.get("upc", "").strip() and row.get("upc", "").strip() in upcs
+    ]
+    if add_matches(exact_upc_rows, "exact upc"):
+        return suggestions
+
+    exact_name_rows = [
+        row
+        for row in catalog_rows
+        if row.get("canonical_name", "").strip().upper() in normalized_names
+    ]
+    if add_matches(exact_name_rows, "exact normalized name"):
+        return suggestions
+
+    contains_rows = []
+    for row in catalog_rows:
+        canonical_name = row.get("canonical_name", "").strip().upper()
+        if not canonical_name:
+            continue
+        for normalized_name in normalized_names:
+            if normalized_name in canonical_name or canonical_name in normalized_name:
+                contains_rows.append(row)
+                break
+    add_matches(contains_rows, "canonical name contains match")
+    return suggestions
+
+
+def build_display_lines(queue_row, related_rows):
+    lines = []
+    for index, row in enumerate(sort_related_items(related_rows), start=1):
+        lines.append(
+            " [{index}] {purchase_date} | {line_total} | {raw_item_name} | {normalized_item_name} | "
+            "{upc} | {retailer}".format(
+                index=index,
+                purchase_date=row.get("purchase_date", ""),
+                line_total=row.get("line_total", ""),
+                raw_item_name=row.get("raw_item_name", ""),
+                normalized_item_name=row.get("normalized_item_name", ""),
+                upc=row.get("upc", ""),
+                retailer=row.get("retailer", ""),
+            )
+        )
+        if row.get("image_url"):
+            lines.append(f"     {row['image_url']}")
+    if not lines:
+        lines.append(" [1] no matched item rows found")
+    return lines
+
+
+def observed_name(queue_row, related_rows):
+    if queue_row.get("normalized_names"):
+        return queue_row["normalized_names"].split(" | ")[0]
+    for row in related_rows:
+        if row.get("normalized_item_name"):
+            return row["normalized_item_name"]
+    return queue_row.get("observed_product_id", "")
+
+
+def choose_existing_canonical(display_rows, observed_label, matched_count):
+    click.secho(
+        f"Select the canonical_name to associate {matched_count} items with:",
+        fg=INFO_COLOR,
+    )
+    for index, row in enumerate(display_rows, start=1):
+        click.echo(f"  [{index}] {row['canonical_name']} | {row['canonical_product_id']}")
+    choice = click.prompt(
+        click.style("selection", fg=PROMPT_COLOR),
+        type=click.IntRange(1, len(display_rows)),
+    )
+    chosen_row = display_rows[choice - 1]
+    click.echo(
+        f'{matched_count} "{observed_label}" items and future matches will be associated '
+        f'with "{chosen_row["canonical_name"]}".'
+    )
+    click.secho(
+        "actions: [y]es  [n]o  [b]ack  [s]kip  [q]uit",
+        fg=PROMPT_COLOR,
+    )
+    confirm = click.prompt(
+        click.style("confirm", fg=PROMPT_COLOR),
+        type=click.Choice(["y", "n", "b", "s", "q"]),
+    )
+    if confirm == "y":
+        return chosen_row["canonical_product_id"], ""
+    if confirm == "s":
+        return "", "skip"
+    if confirm == "q":
+        return "", "quit"
+    return "", "back"
+
+
+def prompt_resolution(queue_row, related_rows, catalog_rows, queue_index, queue_total):
+    suggestions = build_canonical_suggestions(related_rows, catalog_rows)
+    observed_label = observed_name(queue_row, related_rows)
+    matched_count = len(related_rows)
+    click.echo("")
+    click.secho(
+        f"Review {queue_index}/{queue_total}: Resolve observed_product {observed_label} "
+        "to canonical_name [__]?",
+        fg=INFO_COLOR,
+    )
+    click.echo(f"{matched_count} matched items:")
+    for line in build_display_lines(queue_row, related_rows):
+        click.echo(line)
+    if suggestions:
+        click.echo(f"{len(suggestions)} canonical suggestions found:")
+        for index, suggestion in enumerate(suggestions, start=1):
+            click.echo(f" [{index}] {suggestion['canonical_name']}")
+    else:
+        click.echo("no canonical_name suggestions found")
+    click.secho(
+        "[l]ink existing  [n]ew canonical  e[x]clude  [s]kip  [q]uit:",
+        fg=PROMPT_COLOR,
+    )
+    action = click.prompt(
+        "",
+        type=click.Choice(["l", "n", "x", "s", "q"]),
+        prompt_suffix=" ",
+    )
+    if action == "q":
+        return None, None
+    if action == "s":
+        return {
+            "observed_product_id": queue_row["observed_product_id"],
+            "canonical_product_id": "",
+            "resolution_action": "skip",
+            "status": "pending",
+            "resolution_notes": queue_row.get("resolution_notes", ""),
+            "reviewed_at": str(date.today()),
+        }, None
+    if action == "x":
+        notes = click.prompt(
+            click.style("exclude notes", fg=PROMPT_COLOR),
+            default="",
+            show_default=False,
+        )
+        return {
+            "observed_product_id": queue_row["observed_product_id"],
+            "canonical_product_id": "",
+            "resolution_action": "exclude",
+            "status": "approved",
+            "resolution_notes": notes,
+            "reviewed_at": str(date.today()),
+        }, None
+    if action == "l":
+        display_rows = suggestions or [
+            {
+                "canonical_product_id": row["canonical_product_id"],
+                "canonical_name": row["canonical_name"],
+                "reason": "catalog sample",
+            }
+            for row in catalog_rows[:10]
+        ]
+        while True:
+            canonical_product_id, outcome = choose_existing_canonical(
+                display_rows,
+                observed_label,
+                matched_count,
+            )
+            if outcome == "skip":
+                return {
+                    "observed_product_id": queue_row["observed_product_id"],
+                    "canonical_product_id": "",
+                    "resolution_action": "skip",
+                    "status": "pending",
+                    "resolution_notes": queue_row.get("resolution_notes", ""),
+                    "reviewed_at": str(date.today()),
+                }, None
+            if outcome == "quit":
+                return None, None
+            if outcome == "back":
+                continue
+            break
+        notes = click.prompt(click.style("link notes", fg=PROMPT_COLOR), default="", show_default=False)
+        return {
+            "observed_product_id": queue_row["observed_product_id"],
+            "canonical_product_id": canonical_product_id,
+            "resolution_action": "link",
+            "status": "approved",
+            "resolution_notes": notes,
+            "reviewed_at": str(date.today()),
+        }, None
+
+    canonical_name = click.prompt(click.style("canonical name", fg=PROMPT_COLOR), type=str)
+    category = click.prompt(
+        click.style("category", fg=PROMPT_COLOR),
+        default="",
+        show_default=False,
+    )
+    product_type = click.prompt(
+        click.style("product type", fg=PROMPT_COLOR),
+        default="",
+        show_default=False,
+    )
+    notes = click.prompt(
+        click.style("notes", fg=PROMPT_COLOR),
+        default="",
+        show_default=False,
+    )
+    canonical_product_id = stable_id("gcan", f"manual|{canonical_name}|{category}|{product_type}")
+    canonical_row = {
+        "canonical_product_id": canonical_product_id,
+        "canonical_name": canonical_name,
+        "category": category,
+        "product_type": product_type,
+        "brand": "",
+        "variant": "",
+        "size_value": "",
+        "size_unit": "",
+        "pack_qty": "",
+        "measure_type": "",
+        "notes": notes,
+        "created_at": str(date.today()),
+        "updated_at": str(date.today()),
+    }
+    resolution_row = {
+        "observed_product_id": queue_row["observed_product_id"],
+        "canonical_product_id": canonical_product_id,
+        "resolution_action": "create",
+        "status": "approved",
+        "resolution_notes": notes,
+        "reviewed_at": str(date.today()),
+    }
+    return resolution_row, canonical_row
+
+
+@click.command()
+@click.option("--purchases-csv", default="combined_output/purchases.csv", show_default=True)
+@click.option("--queue-csv", default="combined_output/review_queue.csv", show_default=True)
+@click.option("--resolutions-csv", default="combined_output/review_resolutions.csv", show_default=True)
+@click.option("--catalog-csv", default="combined_output/canonical_catalog.csv", show_default=True)
+@click.option("--limit", default=0, show_default=True, type=int)
+@click.option("--refresh-only", is_flag=True, help="Only rebuild review_queue.csv without prompting.")
+def main(purchases_csv, queue_csv, resolutions_csv, catalog_csv, limit, refresh_only):
+    purchase_rows = build_purchases.read_optional_csv_rows(purchases_csv)
+    resolution_rows = build_purchases.read_optional_csv_rows(resolutions_csv)
+    catalog_rows = build_purchases.read_optional_csv_rows(catalog_csv)
+    queue_rows = build_review_queue(purchase_rows, resolution_rows)
+    write_csv_rows(queue_csv, queue_rows, QUEUE_FIELDS)
+    click.echo(f"wrote {len(queue_rows)} rows to {queue_csv}")
+
+    if refresh_only:
+        return
+
+    resolution_lookup = build_purchases.load_resolution_lookup(resolution_rows)
+    catalog_by_id = {row["canonical_product_id"]: row for row in catalog_rows if row.get("canonical_product_id")}
+    rows_by_observed = defaultdict(list)
+    for row in purchase_rows:
+        observed_product_id = row.get("observed_product_id", "")
+        if observed_product_id:
+            rows_by_observed[observed_product_id].append(row)
+    reviewed = 0
+    for index, queue_row in enumerate(queue_rows, start=1):
+        if limit and reviewed >= limit:
+            break
+        related_rows = rows_by_observed.get(queue_row["observed_product_id"], [])
+        result = prompt_resolution(queue_row, related_rows, catalog_rows, index, len(queue_rows))
+        if result == (None, None):
+            break
+        resolution_row, canonical_row = result
+        resolution_lookup[resolution_row["observed_product_id"]] = resolution_row
+        if canonical_row and canonical_row["canonical_product_id"] not in catalog_by_id:
+            catalog_by_id[canonical_row["canonical_product_id"]] = canonical_row
+            catalog_rows.append(canonical_row)
+        reviewed += 1
+
+    save_resolution_rows(resolutions_csv, sorted(resolution_lookup.values(), key=lambda row: row["observed_product_id"]))
+    save_catalog_rows(catalog_csv, sorted(catalog_by_id.values(), key=lambda row: row["canonical_product_id"]))
+    click.echo(
+        f"saved {len(resolution_lookup)} resolution rows to {resolutions_csv} "
+        f"and {len(catalog_by_id)} catalog rows to {catalog_csv}"
+    )
+
+
+if __name__ == "__main__":
+    main()

scrape-click.py

@@ -1,5 +0,0 @@
-from scraper import main
-
-
-if __name__ == "__main__":
-    main()

scrape_costco.py

@@ -1,14 +1,22 @@
+import os
 import csv
 import json
 import time
+import re
+from pathlib import Path
 from calendar import monthrange
 from datetime import datetime, timedelta
-from pathlib import Path
+from dotenv import load_dotenv
-
 import click
-import browser_cookie3
 from curl_cffi import requests
 
+from browser_session import (
+    find_firefox_profile_dir,
+    load_firefox_cookies,
+    read_firefox_local_storage,
+    read_firefox_webapps_store,
+)
+
 BASE_URL = "https://ecom-api.costco.com/ebusiness/order/v1/orders/graphql"
 RETAILER = "costco"
 
@@ -21,7 +29,7 @@ query receiptsWithCounts($startDate: String!, $endDate: String!, $documentType:
     gasAndCarWash
     receipts {
       warehouseName
-      receiptType
+      receiptType 
       documentType
       transactionDateTime
       transactionBarcode
@@ -207,9 +215,21 @@ ITEM_FIELDS = [
     "is_coupon_line",
 ]
 
+COSTCO_STORAGE_ORIGIN = "costco.com"
+COSTCO_ID_TOKEN_STORAGE_KEY = "idToken"
+COSTCO_CLIENT_ID_STORAGE_KEY = "clientID"
 
-def build_headers():
+def load_config():
+    load_dotenv()
     return {
+        "authorization": os.getenv("COSTCO_X_AUTHORIZATION", "").strip(),
+        "client_id": os.getenv("COSTCO_X_WCS_CLIENTID", "").strip(),
+        "client_identifier": os.getenv("COSTCO_CLIENT_IDENTIFIER", "").strip(),
+    }
+
+
+def build_headers(auth_headers):
+    headers = {
         "accept": "*/*",
         "content-type": "application/json-patch+json",
         "costco.service": "restOrders",
@@ -221,12 +241,53 @@ def build_headers():
             "Gecko/20100101 Firefox/148.0"
         ),
     }
+    headers.update(auth_headers)
+    return headers
 
 
-def build_session():
+def load_costco_browser_headers(profile_dir, authorization, client_id, client_identifier):
+    local_storage = read_firefox_local_storage(profile_dir, COSTCO_STORAGE_ORIGIN)
+    webapps_store = read_firefox_webapps_store(profile_dir, COSTCO_STORAGE_ORIGIN)
+    auth_header = authorization.strip() if authorization else ""
+    if client_id:
+        client_id = client_id.strip()
+    if client_identifier:
+        client_identifier = client_identifier.strip()
+
+    if not auth_header:
+        id_token = (
+            local_storage.get(COSTCO_ID_TOKEN_STORAGE_KEY, "").strip()
+            or webapps_store.get(COSTCO_ID_TOKEN_STORAGE_KEY, "").strip()
+        )
+        if id_token:
+            auth_header = f"Bearer {id_token}"
+
+    client_id = client_id or (
+        local_storage.get(COSTCO_CLIENT_ID_STORAGE_KEY, "").strip()
+        or webapps_store.get(COSTCO_CLIENT_ID_STORAGE_KEY, "").strip()
+    )
+
+    if not auth_header:
+        raise click.ClickException(
+            "could not find Costco auth token; set COSTCO_X_AUTHORIZATION or load Firefox idToken"
+        )
+    if not client_id or not client_identifier:
+        raise click.ClickException(
+            "missing Costco client ids; set COSTCO_X_WCS_CLIENTID and COSTCO_CLIENT_IDENTIFIER"
+        )
+
+    return {
+        "costco-x-authorization": auth_header,
+        "costco-x-wcs-clientId": client_id,
+        "client-identifier": client_identifier,
+    }
+
+
+def build_session(profile_dir, auth_headers):
     session = requests.Session()
-    session.cookies.update(browser_cookie3.firefox(domain_name="costco.com"))
+    session.cookies.update(load_firefox_cookies(".costco.com", profile_dir))
-    session.headers.update(build_headers())
+    session.headers.update(build_headers(auth_headers))
+    session.headers.update(auth_headers)
     return session
 
 
@@ -244,7 +305,7 @@ def graphql_post(session, query, variables):
             last_response = response
             if response.status_code == 200:
                 return response.json()
-            click.echo(f"retry {attempt + 1}/3 status={response.status_code}")
+            click.echo(f"retry {attempt + 1}/3 status={response.status_code} body={response.text[:500]}")
         except Exception as exc:  # pragma: no cover - network error path
             click.echo(f"retry {attempt + 1}/3 error={exc}")
         time.sleep(3)
@@ -254,6 +315,8 @@ def graphql_post(session, query, variables):
 
     raise RuntimeError("failed to fetch Costco GraphQL payload")
 
+def safe_filename(value):
+    return re.sub(r'[<>:"/\\|?*]+', "-", str(value))
 
 def summary_receipts(payload):
     return payload.get("data", {}).get("receiptsWithCounts", {}).get("receipts", [])
@@ -423,7 +486,7 @@ def flatten_costco_data(summary_payload, detail_payloads, raw_dir):
                 for row in summary_row.get("couponArray", []) or []
                 if row.get("upcnumberCoupon")
             }
-            raw_order_path = raw_dir / f"{receipt_id or order_id}.json"
+            raw_order_path = raw_dir / f"{safe_filename(receipt_id or order_id)}.json"
 
             orders.append(
                 {
@@ -568,20 +631,53 @@ def write_csv(path, rows, fieldnames):
 )
 @click.option(
     "--months-back",
-    default=3,
+    default=36,
     show_default=True,
     type=int,
     help="How many months of receipts to enumerate back from today.",
 )
-def main(outdir, document_type, document_sub_type, window_days, months_back):
+@click.option(
+    "--firefox-profile-dir",
+    default=None,
+    help="Firefox profile directory to use for cookies and session storage.",
+)
+def main(
+    outdir,
+    document_type,
+    document_sub_type,
+    window_days,
+    months_back,
+    firefox_profile_dir,
+):
     outdir = Path(outdir)
     raw_dir = outdir / "raw"
-    try:
+    config = load_config()
-        session = build_session()
+
-    except Exception as exc:
+    profile_dir = Path(firefox_profile_dir) if firefox_profile_dir else None
-        raise click.ClickException(
+    if profile_dir is None:
-            f"failed to load Costco Firefox cookies: {exc}"
+        try:
-        ) from exc
+            profile_dir = find_firefox_profile_dir()
+        except Exception:
+            profile_dir = click.prompt(
+                "Firefox profile dir",
+                type=click.Path(exists=True, file_okay=False, path_type=Path),
+            )
+
+    auth_headers = load_costco_browser_headers(
+        profile_dir,
+        authorization=config["authorization"],
+        client_id=config["client_id"],
+        client_identifier=config["client_identifier"],
+    )
+    session = build_session(profile_dir, auth_headers)
+    click.echo(
+        "session bootstrap: "
+        f"cookies={True} "
+        f"authorization={bool(auth_headers.get('costco-x-authorization'))} "
+        f"client_id={bool(auth_headers.get('costco-x-wcs-clientId'))} "
+        f"client_identifier={bool(auth_headers.get('client-identifier'))}"
+    )
+
     start_date, end_date = resolve_date_range(months_back)
 
     summary_payload, request_metadata = fetch_summary_windows(
@@ -607,7 +703,7 @@ def main(outdir, document_type, document_sub_type, window_days, months_back):
             {"barcode": barcode, "documentType": "warehouse"},
         )
         detail_payloads.append(detail_payload)
-        write_json(raw_dir / f"{receipt_id}.json", detail_payload)
+        write_json(raw_dir / f"{safe_filename(receipt_id)}.json", detail_payload)
 
     orders, items = flatten_costco_data(summary_payload, detail_payloads, raw_dir)
     write_csv(outdir / "orders.csv", orders, ORDER_FIELDS)
@@ -617,3 +713,5 @@ def main(outdir, document_type, document_sub_type, window_days, months_back):
 
 if __name__ == "__main__":
     main()
+
+ 

scrape_giant.py

@@ -0,0 +1,333 @@
+import csv
+import json
+import os
+import time
+from pathlib import Path
+
+import click
+from dotenv import load_dotenv
+from curl_cffi import requests
+
+from browser_session import find_firefox_profile_dir, load_firefox_cookies
+
+
+BASE = "https://giantfood.com"
+ACCOUNT_PAGE = f"{BASE}/account/history/invoice/in-store"
+
+ORDER_FIELDS = [
+    "order_id",
+    "order_date",
+    "delivery_date",
+    "service_type",
+    "order_total",
+    "payment_method",
+    "total_item_count",
+    "total_savings",
+    "your_savings_total",
+    "coupons_discounts_total",
+    "store_name",
+    "store_number",
+    "store_address1",
+    "store_city",
+    "store_state",
+    "store_zipcode",
+    "refund_order",
+    "ebt_order",
+]
+
+ITEM_FIELDS = [
+    "order_id",
+    "order_date",
+    "line_no",
+    "pod_id",
+    "item_name",
+    "upc",
+    "category_id",
+    "category",
+    "qty",
+    "unit",
+    "unit_price",
+    "line_total",
+    "picked_weight",
+    "mvp_savings",
+    "reward_savings",
+    "coupon_savings",
+    "coupon_price",
+]
+
+
+def load_config():
+    if load_dotenv is not None:
+        load_dotenv()
+
+    return {
+        "user_id": os.getenv("GIANT_USER_ID", "").strip(),
+        "loyalty": os.getenv("GIANT_LOYALTY_NUMBER", "").strip(),
+    }
+
+
+def build_session():
+    profile_dir = find_firefox_profile_dir()
+    session = requests.Session()
+    session.cookies.update(load_firefox_cookies("giantfood.com", profile_dir))
+    session.headers.update(
+        {
+            "user-agent": (
+                "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:148.0) "
+                "Gecko/20100101 Firefox/148.0"
+            ),
+            "accept": "application/json, text/plain, */*",
+            "accept-language": "en-US,en;q=0.9",
+            "referer": ACCOUNT_PAGE,
+        }
+    )
+    return session
+
+
+def safe_get(session, url, **kwargs):
+    last_response = None
+
+    for attempt in range(3):
+        try:
+            response = session.get(
+                url,
+                impersonate="firefox",
+                timeout=30,
+                **kwargs,
+            )
+            last_response = response
+
+            if response.status_code == 200:
+                return response
+
+            click.echo(f"retry {attempt + 1}/3 status={response.status_code}")
+        except Exception as exc:  # pragma: no cover - network error path
+            click.echo(f"retry {attempt + 1}/3 error={exc}")
+
+        time.sleep(3)
+
+    if last_response is not None:
+        last_response.raise_for_status()
+
+    raise RuntimeError(f"failed to fetch {url}")
+
+
+def get_history(session, user_id, loyalty):
+    response = safe_get(
+        session,
+        f"{BASE}/api/v6.0/user/{user_id}/order/history",
+        params={"filter": "instore", "loyaltyNumber": loyalty},
+    )
+    return response.json()
+
+
+def get_order_detail(session, user_id, order_id):
+    response = safe_get(
+        session,
+        f"{BASE}/api/v6.0/user/{user_id}/order/history/detail/{order_id}",
+        params={"isInStore": "true"},
+    )
+    return response.json()
+
+
+def flatten_orders(history, details):
+    orders = []
+    items = []
+    history_lookup = {record["orderId"]: record for record in history.get("records", [])}
+
+    for detail in details:
+        order_id = str(detail["orderId"])
+        history_row = history_lookup.get(detail["orderId"], {})
+        pickup = detail.get("pup", {})
+
+        orders.append(
+            {
+                "order_id": order_id,
+                "order_date": detail.get("orderDate"),
+                "delivery_date": detail.get("deliveryDate"),
+                "service_type": history_row.get("serviceType"),
+                "order_total": detail.get("orderTotal"),
+                "payment_method": detail.get("paymentMethod"),
+                "total_item_count": detail.get("totalItemCount"),
+                "total_savings": detail.get("totalSavings"),
+                "your_savings_total": detail.get("yourSavingsTotal"),
+                "coupons_discounts_total": detail.get("couponsDiscountsTotal"),
+                "store_name": pickup.get("storeName"),
+                "store_number": pickup.get("aholdStoreNumber"),
+                "store_address1": pickup.get("storeAddress1"),
+                "store_city": pickup.get("storeCity"),
+                "store_state": pickup.get("storeState"),
+                "store_zipcode": pickup.get("storeZipcode"),
+                "refund_order": detail.get("refundOrder"),
+                "ebt_order": detail.get("ebtOrder"),
+            }
+        )
+
+        for line_no, item in enumerate(detail.get("items", []), start=1):
+            items.append(
+                {
+                    "order_id": order_id,
+                    "order_date": detail.get("orderDate"),
+                    "line_no": str(line_no),
+                    "pod_id": item.get("podId"),
+                    "item_name": item.get("itemName"),
+                    "upc": item.get("primUpcCd"),
+                    "category_id": item.get("categoryId"),
+                    "category": item.get("categoryDesc"),
+                    "qty": item.get("shipQy"),
+                    "unit": item.get("lbEachCd"),
+                    "unit_price": item.get("unitPrice"),
+                    "line_total": item.get("groceryAmount"),
+                    "picked_weight": item.get("totalPickedWeight"),
+                    "mvp_savings": item.get("mvpSavings"),
+                    "reward_savings": item.get("rewardSavings"),
+                    "coupon_savings": item.get("couponSavings"),
+                    "coupon_price": item.get("couponPrice"),
+                }
+            )
+
+    return orders, items
+
+
+def normalize_row(row, fieldnames):
+    return {field: stringify(row.get(field)) for field in fieldnames}
+
+
+def stringify(value):
+    if value is None:
+        return ""
+    return str(value)
+
+
+def read_csv_rows(path):
+    if not path.exists():
+        return [], []
+
+    with path.open(newline="", encoding="utf-8") as handle:
+        reader = csv.DictReader(handle)
+        fieldnames = reader.fieldnames or []
+        return fieldnames, list(reader)
+
+
+def read_existing_order_ids(path):
+    _, rows = read_csv_rows(path)
+    return {row["order_id"] for row in rows if row.get("order_id")}
+
+
+def merge_rows(existing_rows, new_rows, subset):
+    merged = []
+    row_index = {}
+
+    for row in existing_rows + new_rows:
+        key = tuple(stringify(row.get(field)) for field in subset)
+        normalized = dict(row)
+        if key in row_index:
+            merged[row_index[key]] = normalized
+        else:
+            row_index[key] = len(merged)
+            merged.append(normalized)
+
+    return merged
+
+
+def append_dedup(path, new_rows, subset, fieldnames):
+    existing_fieldnames, existing_rows = read_csv_rows(path)
+    all_fieldnames = list(dict.fromkeys(existing_fieldnames + fieldnames))
+
+    merged = merge_rows(
+        [normalize_row(row, all_fieldnames) for row in existing_rows],
+        [normalize_row(row, all_fieldnames) for row in new_rows],
+        subset=subset,
+    )
+
+    with path.open("w", newline="", encoding="utf-8") as handle:
+        writer = csv.DictWriter(handle, fieldnames=all_fieldnames)
+        writer.writeheader()
+        writer.writerows(merged)
+
+    return merged
+
+
+def write_json(path, payload):
+    path.write_text(json.dumps(payload, indent=2), encoding="utf-8")
+
+
+@click.command()
+@click.option("--user-id", default=None, help="Giant user id.")
+@click.option("--loyalty", default=None, help="Giant loyalty number.")
+@click.option(
+    "--outdir",
+    default="giant_output",
+    show_default=True,
+    help="Directory for raw json and csv outputs.",
+)
+@click.option(
+    "--sleep-seconds",
+    default=1.5,
+    show_default=True,
+    type=float,
+    help="Delay between order detail requests.",
+)
+def main(user_id, loyalty, outdir, sleep_seconds):
+    config = load_config()
+    user_id = user_id or config["user_id"] or click.prompt("Giant user id", type=str)
+    loyalty = loyalty or config["loyalty"] or click.prompt(
+        "Giant loyalty number", type=str
+    )
+
+    outdir = Path(outdir)
+    rawdir = outdir / "raw"
+    rawdir.mkdir(parents=True, exist_ok=True)
+
+    orders_csv = outdir / "orders.csv"
+    items_csv = outdir / "items.csv"
+    existing_order_ids = read_existing_order_ids(orders_csv)
+
+    session = build_session()
+    history = get_history(session, user_id, loyalty)
+    write_json(rawdir / "history.json", history)
+
+    records = history.get("records", [])
+    click.echo(f"history returned {len(records)} visits; Giant exposes only the most recent 50")
+
+    unseen_records = [
+        record
+        for record in records
+        if stringify(record.get("orderId")) not in existing_order_ids
+    ]
+    click.echo(
+        f"found {len(unseen_records)} unseen visits "
+        f"({len(existing_order_ids)} already stored)"
+    )
+
+    details = []
+    for index, record in enumerate(unseen_records, start=1):
+        order_id = stringify(record.get("orderId"))
+        click.echo(f"[{index}/{len(unseen_records)}] fetching {order_id}")
+        detail = get_order_detail(session, user_id, order_id)
+        write_json(rawdir / f"{order_id}.json", detail)
+        details.append(detail)
+        if index < len(unseen_records):
+            time.sleep(sleep_seconds)
+
+    orders, items = flatten_orders(history, details)
+    merged_orders = append_dedup(
+        orders_csv,
+        orders,
+        subset=["order_id"],
+        fieldnames=ORDER_FIELDS,
+    )
+    merged_items = append_dedup(
+        items_csv,
+        items,
+        subset=["order_id", "line_no"],
+        fieldnames=ITEM_FIELDS,
+    )
+    click.echo(
+        f"wrote {len(orders)} new orders / {len(items)} new items "
+        f"({len(merged_orders)} total orders, {len(merged_items)} total items)"
+    )
+
+
+if __name__ == "__main__":
+    main()

scraper.py

@@ -1,341 +1,4 @@
-import csv
+from scrape_giant import *  # noqa: F401,F403
-import json
-import os
-import time
-from pathlib import Path
-from dotenv import load_dotenv
-import browser_cookie3
-from curl_cffi import requests
-import click
-
-
-BASE = "https://giantfood.com"
-ACCOUNT_PAGE = f"{BASE}/account/history/invoice/in-store"
-
-ORDER_FIELDS = [
-    "order_id",
-    "order_date",
-    "delivery_date",
-    "service_type",
-    "order_total",
-    "payment_method",
-    "total_item_count",
-    "total_savings",
-    "your_savings_total",
-    "coupons_discounts_total",
-    "store_name",
-    "store_number",
-    "store_address1",
-    "store_city",
-    "store_state",
-    "store_zipcode",
-    "refund_order",
-    "ebt_order",
-]
-
-ITEM_FIELDS = [
-    "order_id",
-    "order_date",
-    "line_no",
-    "pod_id",
-    "item_name",
-    "upc",
-    "category_id",
-    "category",
-    "qty",
-    "unit",
-    "unit_price",
-    "line_total",
-    "picked_weight",
-    "mvp_savings",
-    "reward_savings",
-    "coupon_savings",
-    "coupon_price",
-]
-
-
-def load_config():
-    if load_dotenv is not None:
-        load_dotenv()
-
-    return {
-        "user_id": os.getenv("GIANT_USER_ID", "").strip(),
-        "loyalty": os.getenv("GIANT_LOYALTY_NUMBER", "").strip(),
-    }
-
-
-def build_session():
-    session = requests.Session()
-    session.cookies.update(browser_cookie3.firefox(domain_name="giantfood.com"))
-    session.headers.update(
-        {
-            "user-agent": (
-                "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:148.0) "
-                "Gecko/20100101 Firefox/148.0"
-            ),
-            "accept": "application/json, text/plain, */*",
-            "accept-language": "en-US,en;q=0.9",
-            "referer": ACCOUNT_PAGE,
-        }
-    )
-    return session
-
-
-def safe_get(session, url, **kwargs):
-    last_response = None
-
-    for attempt in range(3):
-        try:
-            response = session.get(
-                url,
-                impersonate="firefox",
-                timeout=30,
-                **kwargs,
-            )
-            last_response = response
-
-            if response.status_code == 200:
-                return response
-
-            click.echo(f"retry {attempt + 1}/3 status={response.status_code}")
-        except Exception as exc:  # pragma: no cover - network error path
-            click.echo(f"retry {attempt + 1}/3 error={exc}")
-
-        time.sleep(3)
-
-    if last_response is not None:
-        last_response.raise_for_status()
-
-    raise RuntimeError(f"failed to fetch {url}")
-
-
-def get_history(session, user_id, loyalty):
-    response = safe_get(
-        session,
-        f"{BASE}/api/v6.0/user/{user_id}/order/history",
-        params={"filter": "instore", "loyaltyNumber": loyalty},
-    )
-    return response.json()
-
-
-def get_order_detail(session, user_id, order_id):
-    response = safe_get(
-        session,
-        f"{BASE}/api/v6.0/user/{user_id}/order/history/detail/{order_id}",
-        params={"isInStore": "true"},
-    )
-    return response.json()
-
-
-def flatten_orders(history, details):
-    orders = []
-    items = []
-    history_lookup = {record["orderId"]: record for record in history.get("records", [])}
-
-    for detail in details:
-        order_id = str(detail["orderId"])
-        history_row = history_lookup.get(detail["orderId"], {})
-        pickup = detail.get("pup", {})
-
-        orders.append(
-            {
-                "order_id": order_id,
-                "order_date": detail.get("orderDate"),
-                "delivery_date": detail.get("deliveryDate"),
-                "service_type": history_row.get("serviceType"),
-                "order_total": detail.get("orderTotal"),
-                "payment_method": detail.get("paymentMethod"),
-                "total_item_count": detail.get("totalItemCount"),
-                "total_savings": detail.get("totalSavings"),
-                "your_savings_total": detail.get("yourSavingsTotal"),
-                "coupons_discounts_total": detail.get("couponsDiscountsTotal"),
-                "store_name": pickup.get("storeName"),
-                "store_number": pickup.get("aholdStoreNumber"),
-                "store_address1": pickup.get("storeAddress1"),
-                "store_city": pickup.get("storeCity"),
-                "store_state": pickup.get("storeState"),
-                "store_zipcode": pickup.get("storeZipcode"),
-                "refund_order": detail.get("refundOrder"),
-                "ebt_order": detail.get("ebtOrder"),
-            }
-        )
-
-        for line_no, item in enumerate(detail.get("items", []), start=1):
-            items.append(
-                {
-                    "order_id": order_id,
-                    "order_date": detail.get("orderDate"),
-                    "line_no": str(line_no),
-                    "pod_id": item.get("podId"),
-                    "item_name": item.get("itemName"),
-                    "upc": item.get("primUpcCd"),
-                    "category_id": item.get("categoryId"),
-                    "category": item.get("categoryDesc"),
-                    "qty": item.get("shipQy"),
-                    "unit": item.get("lbEachCd"),
-                    "unit_price": item.get("unitPrice"),
-                    "line_total": item.get("groceryAmount"),
-                    "picked_weight": item.get("totalPickedWeight"),
-                    "mvp_savings": item.get("mvpSavings"),
-                    "reward_savings": item.get("rewardSavings"),
-                    "coupon_savings": item.get("couponSavings"),
-                    "coupon_price": item.get("couponPrice"),
-                }
-            )
-
-    return orders, items
-
-
-def normalize_row(row, fieldnames):
-    return {field: stringify(row.get(field)) for field in fieldnames}
-
-
-def stringify(value):
-    if value is None:
-        return ""
-    return str(value)
-
-
-def read_csv_rows(path):
-    if not path.exists():
-        return [], []
-
-    with path.open(newline="", encoding="utf-8") as handle:
-        reader = csv.DictReader(handle)
-        fieldnames = reader.fieldnames or []
-        return fieldnames, list(reader)
-
-
-def read_existing_order_ids(path):
-    _, rows = read_csv_rows(path)
-    return {row["order_id"] for row in rows if row.get("order_id")}
-
-
-def merge_rows(existing_rows, new_rows, subset):
-    merged = []
-    row_index = {}
-
-    for row in existing_rows + new_rows:
-        key = tuple(stringify(row.get(field)) for field in subset)
-        normalized = dict(row)
-        if key in row_index:
-            merged[row_index[key]] = normalized
-        else:
-            row_index[key] = len(merged)
-            merged.append(normalized)
-
-    return merged
-
-
-def append_dedup(path, new_rows, subset, fieldnames):
-    existing_fieldnames, existing_rows = read_csv_rows(path)
-    all_fieldnames = list(dict.fromkeys(existing_fieldnames + fieldnames))
-
-    merged = merge_rows(
-        [normalize_row(row, all_fieldnames) for row in existing_rows],
-        [normalize_row(row, all_fieldnames) for row in new_rows],
-        subset=subset,
-    )
-
-    with path.open("w", newline="", encoding="utf-8") as handle:
-        writer = csv.DictWriter(handle, fieldnames=all_fieldnames)
-        writer.writeheader()
-        writer.writerows(merged)
-
-    return merged
-
-
-def write_json(path, payload):
-    path.write_text(json.dumps(payload, indent=2), encoding="utf-8")
-
-
-@click.command()
-@click.option("--user-id", default=None, help="Giant user id.")
-@click.option("--loyalty", default=None, help="Giant loyalty number.")
-@click.option(
-    "--outdir",
-    default="giant_output",
-    show_default=True,
-    help="Directory for raw json and csv outputs.",
-)
-@click.option(
-    "--sleep-seconds",
-    default=1.5,
-    show_default=True,
-    type=float,
-    help="Delay between order detail requests.",
-)
-def main(user_id, loyalty, outdir, sleep_seconds):
-    config = load_config()
-    user_id = user_id or config["user_id"] or click.prompt("Giant user id", type=str)
-    loyalty = loyalty or config["loyalty"] or click.prompt(
-        "Giant loyalty number", type=str
-    )
-
-    outdir = Path(outdir)
-    rawdir = outdir / "raw"
-    rawdir.mkdir(parents=True, exist_ok=True)
-
-    orders_csv = outdir / "orders.csv"
-    items_csv = outdir / "items.csv"
-
-    click.echo("Using cookies from your current Firefox profile.")
-    click.echo(f"Open Giant here, confirm you're logged in, then return: {ACCOUNT_PAGE}")
-    click.pause(info="Press any key once Giant is open and logged in")
-
-    session = build_session()
-
-    click.echo("Fetching order history...")
-    history = get_history(session, user_id, loyalty)
-    write_json(rawdir / "history.json", history)
-
-    records = history.get("records", [])
-    click.echo(f"History returned {len(records)} visits.")
-    click.echo(
-        "Note: Giant appears to expose only the most recent 50 visits, "
-        "so run this periodically if you want full continuity."
-    )
-
-    history_order_ids = [str(record["orderId"]) for record in records]
-    existing_order_ids = read_existing_order_ids(orders_csv)
-    new_order_ids = [order_id for order_id in history_order_ids if order_id not in existing_order_ids]
-
-    click.echo(f"Existing orders in csv: {len(existing_order_ids)}")
-    click.echo(f"New orders to fetch: {len(new_order_ids)}")
-
-    if not new_order_ids:
-        click.echo("No new orders found. Done.")
-        return
-
-    details = []
-    for order_id in new_order_ids:
-        click.echo(f"Fetching {order_id}")
-        detail = get_order_detail(session, user_id, order_id)
-        details.append(detail)
-        write_json(rawdir / f"{order_id}.json", detail)
-        time.sleep(sleep_seconds)
-
-    click.echo("Flattening new data...")
-    orders, items = flatten_orders(history, details)
-
-    all_orders = append_dedup(
-        orders_csv,
-        orders,
-        subset=["order_id"],
-        fieldnames=ORDER_FIELDS,
-    )
-    all_items = append_dedup(
-        items_csv,
-        items,
-        subset=["order_id", "line_no", "item_name", "upc", "line_total"],
-        fieldnames=ITEM_FIELDS,
-    )
-
-    click.echo("Done.")
-    click.echo(f"Orders csv: {orders_csv}")
-    click.echo(f"Items csv:  {items_csv}")
-    click.echo(f"Total orders stored: {len(all_orders)}")
-    click.echo(f"Total item rows stored: {len(all_items)}")
 
 
 if __name__ == "__main__":

tests/test_browser_session.py

@@ -0,0 +1,155 @@
+import sqlite3
+import tempfile
+import unittest
+from pathlib import Path
+from unittest import mock
+
+import browser_session
+import scrape_costco
+
+
+class BrowserSessionTests(unittest.TestCase):
+    def test_read_firefox_local_storage_reads_copied_sqlite(self):
+        with tempfile.TemporaryDirectory() as tmpdir:
+            profile_dir = Path(tmpdir) / "abcd.default-release"
+            ls_dir = profile_dir / "storage" / "default" / "https+++www.costco.com" / "ls"
+            ls_dir.mkdir(parents=True)
+            db_path = ls_dir / "data.sqlite"
+
+            with sqlite3.connect(db_path) as connection:
+                connection.execute("CREATE TABLE data (key TEXT, value TEXT)")
+                connection.execute(
+                    "INSERT INTO data (key, value) VALUES (?, ?)",
+                    ("costco-x-wcs-clientId", "4900eb1f-0c10-4bd9-99c3-c59e6c1ecebf"),
+                )
+
+            values = browser_session.read_firefox_local_storage(
+                profile_dir,
+                origin_filter="costco.com",
+            )
+
+            self.assertEqual(
+                "4900eb1f-0c10-4bd9-99c3-c59e6c1ecebf",
+                values["costco-x-wcs-clientId"],
+            )
+
+    def test_load_costco_browser_headers_reads_id_token_and_client_id(self):
+        with tempfile.TemporaryDirectory() as tmpdir:
+            profile_dir = Path(tmpdir)
+            storage_dir = profile_dir / "storage" / "default" / "https+++www.costco.com" / "ls"
+            storage_dir.mkdir(parents=True)
+            db_path = storage_dir / "data.sqlite"
+
+            with sqlite3.connect(db_path) as connection:
+                connection.execute("CREATE TABLE data (key TEXT, value TEXT)")
+                connection.execute(
+                    "INSERT INTO data (key, value) VALUES (?, ?)",
+                    ("idToken", "header.payload.signature"),
+                )
+                connection.execute(
+                    "INSERT INTO data (key, value) VALUES (?, ?)",
+                    ("clientID", "4900eb1f-0c10-4bd9-99c3-c59e6c1ecebf"),
+                )
+
+            headers = scrape_costco.load_costco_browser_headers(
+                profile_dir,
+                authorization="",
+                client_id="",
+                client_identifier="481b1aec-aa3b-454b-b81b-48187e28f205",
+            )
+
+        self.assertEqual("Bearer header.payload.signature", headers["costco-x-authorization"])
+        self.assertEqual(
+            "4900eb1f-0c10-4bd9-99c3-c59e6c1ecebf",
+            headers["costco-x-wcs-clientId"],
+        )
+        self.assertEqual(
+            "481b1aec-aa3b-454b-b81b-48187e28f205",
+            headers["client-identifier"],
+        )
+
+    def test_load_costco_browser_headers_prefers_env_values(self):
+        with tempfile.TemporaryDirectory() as tmpdir:
+            profile_dir = Path(tmpdir)
+            storage_dir = profile_dir / "storage" / "default" / "https+++www.costco.com" / "ls"
+            storage_dir.mkdir(parents=True)
+            db_path = storage_dir / "data.sqlite"
+
+            with sqlite3.connect(db_path) as connection:
+                connection.execute("CREATE TABLE data (key TEXT, value TEXT)")
+                connection.execute(
+                    "INSERT INTO data (key, value) VALUES (?, ?)",
+                    ("idToken", "storage.payload.signature"),
+                )
+                connection.execute(
+                    "INSERT INTO data (key, value) VALUES (?, ?)",
+                    ("clientID", "4900eb1f-0c10-4bd9-99c3-c59e6c1ecebf"),
+                )
+
+            headers = scrape_costco.load_costco_browser_headers(
+                profile_dir,
+                authorization="Bearer env.payload.signature",
+                client_id="env-client-id",
+                client_identifier="481b1aec-aa3b-454b-b81b-48187e28f205",
+            )
+
+        self.assertEqual("Bearer env.payload.signature", headers["costco-x-authorization"])
+        self.assertEqual("env-client-id", headers["costco-x-wcs-clientId"])
+
+    def test_scrape_costco_prompts_for_profile_dir_when_autodiscovery_fails(self):
+        with mock.patch.object(
+            scrape_costco,
+            "find_firefox_profile_dir",
+            side_effect=FileNotFoundError("no default profile"),
+        ), mock.patch.object(
+            scrape_costco.click,
+            "prompt",
+            return_value=Path("/tmp/profile"),
+        ) as mocked_prompt, mock.patch.object(
+            scrape_costco,
+            "load_config",
+            return_value={
+                "authorization": "",
+                "client_id": "4900eb1f-0c10-4bd9-99c3-c59e6c1ecebf",
+                "client_identifier": "481b1aec-aa3b-454b-b81b-48187e28f205",
+            },
+        ), mock.patch.object(
+            scrape_costco,
+            "load_costco_browser_headers",
+            return_value={
+                "costco-x-authorization": "Bearer header.payload.signature",
+                "costco-x-wcs-clientId": "4900eb1f-0c10-4bd9-99c3-c59e6c1ecebf",
+                "client-identifier": "481b1aec-aa3b-454b-b81b-48187e28f205",
+            },
+        ), mock.patch.object(
+            scrape_costco,
+            "build_session",
+            return_value=object(),
+        ), mock.patch.object(
+            scrape_costco,
+            "fetch_summary_windows",
+            return_value=(
+                {"data": {"receiptsWithCounts": {"receipts": []}}},
+                [],
+            ),
+        ), mock.patch.object(
+            scrape_costco,
+            "write_json",
+        ), mock.patch.object(
+            scrape_costco,
+            "write_csv",
+        ):
+            scrape_costco.main.callback(
+                outdir="/tmp/costco_output",
+                document_type="all",
+                document_sub_type="all",
+                window_days=92,
+                months_back=3,
+                firefox_profile_dir=None,
+            )
+
+        mocked_prompt.assert_called_once()
+
+
+if __name__ == "__main__":
+    unittest.main()

tests/test_costco_pipeline.py

@@ -234,7 +234,7 @@ class CostcoPipelineTests(unittest.TestCase):
 
         self.assertEqual("VISA", orders[0]["payment_method"])
         self.assertEqual("true", items[0]["is_coupon_line"])
-        self.assertIn("dup::2026-03-12T16:16:00.json", items[0]["raw_order_path"])
+        self.assertIn("dup-2026-03-12T16-16-00.json", items[0]["raw_order_path"])
 
     def test_costco_enricher_parses_size_pack_and_discount(self):
         row = enrich_costco.parse_costco_item(
@@ -411,6 +411,26 @@ class CostcoPipelineTests(unittest.TestCase):
             ]
 
             with mock.patch.object(
+                scrape_costco,
+                "load_config",
+                return_value={
+                    "authorization": "",
+                    "client_id": "4900eb1f-0c10-4bd9-99c3-c59e6c1ecebf",
+                    "client_identifier": "481b1aec-aa3b-454b-b81b-48187e28f205",
+                },
+            ), mock.patch.object(
+                scrape_costco,
+                "find_firefox_profile_dir",
+                return_value=Path("/tmp/profile"),
+            ), mock.patch.object(
+                scrape_costco,
+                "load_costco_browser_headers",
+                return_value={
+                    "costco-x-authorization": "Bearer header.payload.signature",
+                    "costco-x-wcs-clientId": "4900eb1f-0c10-4bd9-99c3-c59e6c1ecebf",
+                    "client-identifier": "481b1aec-aa3b-454b-b81b-48187e28f205",
+                },
+            ), mock.patch.object(
                 scrape_costco, "build_session", return_value=object()
             ), mock.patch.object(
                 scrape_costco,
@@ -427,6 +447,7 @@ class CostcoPipelineTests(unittest.TestCase):
                     document_sub_type="all",
                     window_days=92,
                     months_back=3,
+                    firefox_profile_dir=None,
                 )
 
             metadata_path = outdir / "raw" / "summary_requests.json"

tests/test_purchases.py

@@ -0,0 +1,301 @@
+import csv
+import tempfile
+import unittest
+from pathlib import Path
+
+import build_purchases
+import enrich_costco
+
+
+class PurchaseLogTests(unittest.TestCase):
+    def test_derive_metrics_prefers_picked_weight_and_pack_count(self):
+        metrics = build_purchases.derive_metrics(
+            {
+                "line_total": "4.00",
+                "qty": "1",
+                "pack_qty": "4",
+                "size_value": "",
+                "size_unit": "",
+                "picked_weight": "2",
+                "price_per_each": "",
+                "price_per_lb": "",
+                "price_per_oz": "",
+            }
+        )
+
+        self.assertEqual("4", metrics["price_per_each"])
+        self.assertEqual("1", metrics["price_per_count"])
+        self.assertEqual("2", metrics["price_per_lb"])
+        self.assertEqual("0.125", metrics["price_per_oz"])
+        self.assertEqual("picked_weight_lb", metrics["price_per_lb_basis"])
+
+    def test_build_purchase_rows_maps_canonical_ids(self):
+        fieldnames = enrich_costco.OUTPUT_FIELDS
+        giant_row = {field: "" for field in fieldnames}
+        giant_row.update(
+            {
+                "retailer": "giant",
+                "order_id": "g1",
+                "line_no": "1",
+                "observed_item_key": "giant:g1:1",
+                "order_date": "2026-03-01",
+                "item_name": "FRESH BANANA",
+                "item_name_norm": "BANANA",
+                "image_url": "https://example.test/banana.jpg",
+                "retailer_item_id": "100",
+                "upc": "4011",
+                "qty": "1",
+                "unit": "LB",
+                "line_total": "1.29",
+                "unit_price": "1.29",
+                "measure_type": "weight",
+                "price_per_lb": "1.29",
+                "raw_order_path": "giant_output/raw/g1.json",
+                "is_discount_line": "false",
+                "is_coupon_line": "false",
+                "is_fee": "false",
+            }
+        )
+        costco_row = {field: "" for field in fieldnames}
+        costco_row.update(
+            {
+                "retailer": "costco",
+                "order_id": "c1",
+                "line_no": "1",
+                "observed_item_key": "costco:c1:1",
+                "order_date": "2026-03-12",
+                "item_name": "BANANAS 3 LB / 1.36 KG",
+                "item_name_norm": "BANANA",
+                "retailer_item_id": "30669",
+                "qty": "1",
+                "unit": "E",
+                "line_total": "2.98",
+                "unit_price": "2.98",
+                "size_value": "3",
+                "size_unit": "lb",
+                "measure_type": "weight",
+                "price_per_lb": "0.9933",
+                "raw_order_path": "costco_output/raw/c1.json",
+                "is_discount_line": "false",
+                "is_coupon_line": "false",
+                "is_fee": "false",
+            }
+        )
+        giant_orders = [
+            {
+                "order_id": "g1",
+                "store_name": "Giant",
+                "store_number": "42",
+                "store_city": "Springfield",
+                "store_state": "VA",
+            }
+        ]
+        costco_orders = [
+            {
+                "order_id": "c1",
+                "store_name": "MT VERNON",
+                "store_number": "1115",
+                "store_city": "ALEXANDRIA",
+                "store_state": "VA",
+            }
+        ]
+
+        rows, _observed, _canon, _links = build_purchases.build_purchase_rows(
+            [giant_row],
+            [costco_row],
+            giant_orders,
+            costco_orders,
+            [],
+        )
+
+        self.assertEqual(2, len(rows))
+        self.assertTrue(all(row["canonical_product_id"] for row in rows))
+        self.assertEqual({"giant", "costco"}, {row["retailer"] for row in rows})
+        self.assertEqual("https://example.test/banana.jpg", rows[0]["image_url"])
+
+    def test_main_writes_purchase_and_example_csvs(self):
+        with tempfile.TemporaryDirectory() as tmpdir:
+            giant_items = Path(tmpdir) / "giant_items.csv"
+            costco_items = Path(tmpdir) / "costco_items.csv"
+            giant_orders = Path(tmpdir) / "giant_orders.csv"
+            costco_orders = Path(tmpdir) / "costco_orders.csv"
+            resolutions_csv = Path(tmpdir) / "review_resolutions.csv"
+            catalog_csv = Path(tmpdir) / "canonical_catalog.csv"
+            links_csv = Path(tmpdir) / "product_links.csv"
+            purchases_csv = Path(tmpdir) / "combined" / "purchases.csv"
+            examples_csv = Path(tmpdir) / "combined" / "comparison_examples.csv"
+
+            fieldnames = enrich_costco.OUTPUT_FIELDS
+            giant_row = {field: "" for field in fieldnames}
+            giant_row.update(
+                {
+                    "retailer": "giant",
+                    "order_id": "g1",
+                    "line_no": "1",
+                    "observed_item_key": "giant:g1:1",
+                    "order_date": "2026-03-01",
+                    "item_name": "FRESH BANANA",
+                    "item_name_norm": "BANANA",
+                    "retailer_item_id": "100",
+                    "upc": "4011",
+                    "qty": "1",
+                    "unit": "LB",
+                    "line_total": "1.29",
+                    "unit_price": "1.29",
+                    "measure_type": "weight",
+                    "price_per_lb": "1.29",
+                    "raw_order_path": "giant_output/raw/g1.json",
+                    "is_discount_line": "false",
+                    "is_coupon_line": "false",
+                    "is_fee": "false",
+                }
+            )
+            costco_row = {field: "" for field in fieldnames}
+            costco_row.update(
+                {
+                    "retailer": "costco",
+                    "order_id": "c1",
+                    "line_no": "1",
+                    "observed_item_key": "costco:c1:1",
+                    "order_date": "2026-03-12",
+                    "item_name": "BANANAS 3 LB / 1.36 KG",
+                    "item_name_norm": "BANANA",
+                    "retailer_item_id": "30669",
+                    "qty": "1",
+                    "unit": "E",
+                    "line_total": "2.98",
+                    "unit_price": "2.98",
+                    "size_value": "3",
+                    "size_unit": "lb",
+                    "measure_type": "weight",
+                    "price_per_lb": "0.9933",
+                    "raw_order_path": "costco_output/raw/c1.json",
+                    "is_discount_line": "false",
+                    "is_coupon_line": "false",
+                    "is_fee": "false",
+                }
+            )
+
+            for path, source_rows in [
+                (giant_items, [giant_row]),
+                (costco_items, [costco_row]),
+            ]:
+                with path.open("w", newline="", encoding="utf-8") as handle:
+                    writer = csv.DictWriter(handle, fieldnames=fieldnames)
+                    writer.writeheader()
+                    writer.writerows(source_rows)
+
+            order_fields = ["order_id", "store_name", "store_number", "store_city", "store_state"]
+            for path, source_rows in [
+                (
+                    giant_orders,
+                    [
+                        {
+                            "order_id": "g1",
+                            "store_name": "Giant",
+                            "store_number": "42",
+                            "store_city": "Springfield",
+                            "store_state": "VA",
+                        }
+                    ],
+                ),
+                (
+                    costco_orders,
+                    [
+                        {
+                            "order_id": "c1",
+                            "store_name": "MT VERNON",
+                            "store_number": "1115",
+                            "store_city": "ALEXANDRIA",
+                            "store_state": "VA",
+                        }
+                    ],
+                ),
+            ]:
+                with path.open("w", newline="", encoding="utf-8") as handle:
+                    writer = csv.DictWriter(handle, fieldnames=order_fields)
+                    writer.writeheader()
+                    writer.writerows(source_rows)
+
+            build_purchases.main.callback(
+                giant_items_enriched_csv=str(giant_items),
+                costco_items_enriched_csv=str(costco_items),
+                giant_orders_csv=str(giant_orders),
+                costco_orders_csv=str(costco_orders),
+                resolutions_csv=str(resolutions_csv),
+                catalog_csv=str(catalog_csv),
+                links_csv=str(links_csv),
+                output_csv=str(purchases_csv),
+                examples_csv=str(examples_csv),
+            )
+
+            self.assertTrue(purchases_csv.exists())
+            self.assertTrue(examples_csv.exists())
+            with purchases_csv.open(newline="", encoding="utf-8") as handle:
+                purchase_rows = list(csv.DictReader(handle))
+            with examples_csv.open(newline="", encoding="utf-8") as handle:
+                example_rows = list(csv.DictReader(handle))
+            self.assertEqual(2, len(purchase_rows))
+            self.assertEqual(1, len(example_rows))
+
+    def test_build_purchase_rows_applies_manual_resolution(self):
+        fieldnames = enrich_costco.OUTPUT_FIELDS
+        giant_row = {field: "" for field in fieldnames}
+        giant_row.update(
+            {
+                "retailer": "giant",
+                "order_id": "g1",
+                "line_no": "1",
+                "observed_item_key": "giant:g1:1",
+                "order_date": "2026-03-01",
+                "item_name": "SB BAGGED ICE 20LB",
+                "item_name_norm": "BAGGED ICE",
+                "retailer_item_id": "100",
+                "upc": "",
+                "qty": "1",
+                "unit": "EA",
+                "line_total": "3.50",
+                "unit_price": "3.50",
+                "measure_type": "each",
+                "raw_order_path": "giant_output/raw/g1.json",
+                "is_discount_line": "false",
+                "is_coupon_line": "false",
+                "is_fee": "false",
+            }
+        )
+        observed_rows, _canonical_rows, _link_rows, _observed_id_by_key, _canonical_by_observed = (
+            build_purchases.build_link_state([giant_row])
+        )
+        observed_product_id = observed_rows[0]["observed_product_id"]
+        rows, _observed, _canon, _links = build_purchases.build_purchase_rows(
+            [giant_row],
+            [],
+            [
+                {
+                    "order_id": "g1",
+                    "store_name": "Giant",
+                    "store_number": "42",
+                    "store_city": "Springfield",
+                    "store_state": "VA",
+                }
+            ],
+            [],
+            [
+                {
+                    "observed_product_id": observed_product_id,
+                    "canonical_product_id": "gcan_manual_ice",
+                    "resolution_action": "create",
+                    "status": "approved",
+                    "resolution_notes": "manual ice merge",
+                    "reviewed_at": "2026-03-16",
+                }
+            ],
+        )
+
+        self.assertEqual("gcan_manual_ice", rows[0]["canonical_product_id"])
+        self.assertEqual("approved", rows[0]["review_status"])
+        self.assertEqual("create", rows[0]["resolution_action"])
+
+
+if __name__ == "__main__":
+    unittest.main()

tests/test_review_workflow.py

@@ -0,0 +1,409 @@
+import csv
+import tempfile
+import unittest
+from pathlib import Path
+from unittest import mock
+
+from click.testing import CliRunner
+
+import review_products
+
+
+class ReviewWorkflowTests(unittest.TestCase):
+    def test_build_review_queue_groups_unresolved_purchases(self):
+        queue_rows = review_products.build_review_queue(
+            [
+                {
+                    "observed_product_id": "gobs_1",
+                    "canonical_product_id": "",
+                    "retailer": "giant",
+                    "raw_item_name": "SB BAGGED ICE 20LB",
+                    "normalized_item_name": "BAGGED ICE",
+                    "upc": "",
+                    "line_total": "3.50",
+                },
+                {
+                    "observed_product_id": "gobs_1",
+                    "canonical_product_id": "",
+                    "retailer": "giant",
+                    "raw_item_name": "SB BAG ICE CUBED 10LB",
+                    "normalized_item_name": "BAG ICE",
+                    "upc": "",
+                    "line_total": "2.50",
+                },
+            ],
+            [],
+        )
+
+        self.assertEqual(1, len(queue_rows))
+        self.assertEqual("gobs_1", queue_rows[0]["observed_product_id"])
+        self.assertIn("SB BAGGED ICE 20LB", queue_rows[0]["raw_item_names"])
+
+    def test_build_canonical_suggestions_prefers_upc_then_name(self):
+        suggestions = review_products.build_canonical_suggestions(
+            [
+                {
+                    "normalized_item_name": "MIXED PEPPER",
+                    "upc": "12345",
+                }
+            ],
+            [
+                {
+                    "canonical_product_id": "gcan_1",
+                    "canonical_name": "MIXED PEPPER",
+                    "upc": "",
+                },
+                {
+                    "canonical_product_id": "gcan_2",
+                    "canonical_name": "MIXED PEPPER 6 PACK",
+                    "upc": "12345",
+                },
+            ],
+        )
+
+        self.assertEqual("gcan_2", suggestions[0]["canonical_product_id"])
+        self.assertEqual("exact upc", suggestions[0]["reason"])
+        self.assertEqual("gcan_1", suggestions[1]["canonical_product_id"])
+
+    def test_review_products_displays_position_items_and_suggestions(self):
+        with tempfile.TemporaryDirectory() as tmpdir:
+            purchases_csv = Path(tmpdir) / "purchases.csv"
+            queue_csv = Path(tmpdir) / "review_queue.csv"
+            resolutions_csv = Path(tmpdir) / "review_resolutions.csv"
+            catalog_csv = Path(tmpdir) / "canonical_catalog.csv"
+
+            purchase_fields = [
+                "purchase_date",
+                "retailer",
+                "order_id",
+                "line_no",
+                "observed_product_id",
+                "canonical_product_id",
+                "raw_item_name",
+                "normalized_item_name",
+                "image_url",
+                "upc",
+                "line_total",
+            ]
+            with purchases_csv.open("w", newline="", encoding="utf-8") as handle:
+                writer = csv.DictWriter(handle, fieldnames=purchase_fields)
+                writer.writeheader()
+                writer.writerows(
+                    [
+                        {
+                            "purchase_date": "2026-03-14",
+                            "retailer": "costco",
+                            "order_id": "c2",
+                            "line_no": "2",
+                            "observed_product_id": "gobs_mix",
+                            "canonical_product_id": "",
+                            "raw_item_name": "MIXED PEPPER 6-PACK",
+                            "normalized_item_name": "MIXED PEPPER",
+                            "image_url": "",
+                            "upc": "",
+                            "line_total": "7.49",
+                        },
+                        {
+                            "purchase_date": "2026-03-12",
+                            "retailer": "costco",
+                            "order_id": "c1",
+                            "line_no": "1",
+                            "observed_product_id": "gobs_mix",
+                            "canonical_product_id": "",
+                            "raw_item_name": "MIXED PEPPER 6-PACK",
+                            "normalized_item_name": "MIXED PEPPER",
+                            "image_url": "https://example.test/mixed-pepper.jpg",
+                            "upc": "",
+                            "line_total": "6.99",
+                        },
+                    ]
+                )
+
+            with catalog_csv.open("w", newline="", encoding="utf-8") as handle:
+                writer = csv.DictWriter(handle, fieldnames=review_products.build_purchases.CATALOG_FIELDS)
+                writer.writeheader()
+                writer.writerow(
+                    {
+                        "canonical_product_id": "gcan_mix",
+                        "canonical_name": "MIXED PEPPER",
+                        "category": "produce",
+                        "product_type": "pepper",
+                        "brand": "",
+                        "variant": "",
+                        "size_value": "",
+                        "size_unit": "",
+                        "pack_qty": "",
+                        "measure_type": "",
+                        "notes": "",
+                        "created_at": "",
+                        "updated_at": "",
+                    }
+                )
+
+            runner = CliRunner()
+            result = runner.invoke(
+                review_products.main,
+                [
+                    "--purchases-csv",
+                    str(purchases_csv),
+                    "--queue-csv",
+                    str(queue_csv),
+                    "--resolutions-csv",
+                    str(resolutions_csv),
+                    "--catalog-csv",
+                    str(catalog_csv),
+                ],
+                input="q\n",
+                color=True,
+            )
+
+            self.assertEqual(0, result.exit_code)
+            self.assertIn("Review 1/1: Resolve observed_product MIXED PEPPER to canonical_name [__]?", result.output)
+            self.assertIn("2 matched items:", result.output)
+            self.assertIn("[l]ink existing  [n]ew canonical  e[x]clude  [s]kip  [q]uit:", result.output)
+            first_item = result.output.index("[1] 2026-03-14 | 7.49")
+            second_item = result.output.index("[2] 2026-03-12 | 6.99")
+            self.assertLess(first_item, second_item)
+            self.assertIn("https://example.test/mixed-pepper.jpg", result.output)
+            self.assertIn("1 canonical suggestions found:", result.output)
+            self.assertIn("[1] MIXED PEPPER", result.output)
+            self.assertIn("\x1b[", result.output)
+
+    def test_review_products_no_suggestions_is_informational(self):
+        with tempfile.TemporaryDirectory() as tmpdir:
+            purchases_csv = Path(tmpdir) / "purchases.csv"
+            queue_csv = Path(tmpdir) / "review_queue.csv"
+            resolutions_csv = Path(tmpdir) / "review_resolutions.csv"
+            catalog_csv = Path(tmpdir) / "canonical_catalog.csv"
+
+            with purchases_csv.open("w", newline="", encoding="utf-8") as handle:
+                writer = csv.DictWriter(
+                    handle,
+                    fieldnames=[
+                        "purchase_date",
+                        "retailer",
+                        "order_id",
+                        "line_no",
+                        "observed_product_id",
+                        "canonical_product_id",
+                        "raw_item_name",
+                        "normalized_item_name",
+                        "image_url",
+                        "upc",
+                        "line_total",
+                    ],
+                )
+                writer.writeheader()
+                writer.writerow(
+                    {
+                        "purchase_date": "2026-03-14",
+                        "retailer": "giant",
+                        "order_id": "g1",
+                        "line_no": "1",
+                        "observed_product_id": "gobs_ice",
+                        "canonical_product_id": "",
+                        "raw_item_name": "SB BAGGED ICE 20LB",
+                        "normalized_item_name": "BAGGED ICE",
+                        "image_url": "",
+                        "upc": "",
+                        "line_total": "3.50",
+                    }
+                )
+
+            with catalog_csv.open("w", newline="", encoding="utf-8") as handle:
+                writer = csv.DictWriter(handle, fieldnames=review_products.build_purchases.CATALOG_FIELDS)
+                writer.writeheader()
+
+            result = CliRunner().invoke(
+                review_products.main,
+                [
+                    "--purchases-csv",
+                    str(purchases_csv),
+                    "--queue-csv",
+                    str(queue_csv),
+                    "--resolutions-csv",
+                    str(resolutions_csv),
+                    "--catalog-csv",
+                    str(catalog_csv),
+                ],
+                input="q\n",
+                color=True,
+            )
+
+            self.assertEqual(0, result.exit_code)
+            self.assertIn("no canonical_name suggestions found", result.output)
+
+    def test_link_existing_uses_numbered_selection_and_confirmation(self):
+        with tempfile.TemporaryDirectory() as tmpdir:
+            purchases_csv = Path(tmpdir) / "purchases.csv"
+            queue_csv = Path(tmpdir) / "review_queue.csv"
+            resolutions_csv = Path(tmpdir) / "review_resolutions.csv"
+            catalog_csv = Path(tmpdir) / "canonical_catalog.csv"
+
+            with purchases_csv.open("w", newline="", encoding="utf-8") as handle:
+                writer = csv.DictWriter(
+                    handle,
+                    fieldnames=[
+                        "purchase_date",
+                        "retailer",
+                        "order_id",
+                        "line_no",
+                        "observed_product_id",
+                        "canonical_product_id",
+                        "raw_item_name",
+                        "normalized_item_name",
+                        "image_url",
+                        "upc",
+                        "line_total",
+                    ],
+                )
+                writer.writeheader()
+                writer.writerows(
+                    [
+                        {
+                            "purchase_date": "2026-03-14",
+                            "retailer": "costco",
+                            "order_id": "c2",
+                            "line_no": "2",
+                            "observed_product_id": "gobs_mix",
+                            "canonical_product_id": "",
+                            "raw_item_name": "MIXED PEPPER 6-PACK",
+                            "normalized_item_name": "MIXED PEPPER",
+                            "image_url": "",
+                            "upc": "",
+                            "line_total": "7.49",
+                        },
+                        {
+                            "purchase_date": "2026-03-12",
+                            "retailer": "costco",
+                            "order_id": "c1",
+                            "line_no": "1",
+                            "observed_product_id": "gobs_mix",
+                            "canonical_product_id": "",
+                            "raw_item_name": "MIXED PEPPER 6-PACK",
+                            "normalized_item_name": "MIXED PEPPER",
+                            "image_url": "",
+                            "upc": "",
+                            "line_total": "6.99",
+                        },
+                    ]
+                )
+
+            with catalog_csv.open("w", newline="", encoding="utf-8") as handle:
+                writer = csv.DictWriter(handle, fieldnames=review_products.build_purchases.CATALOG_FIELDS)
+                writer.writeheader()
+                writer.writerow(
+                    {
+                        "canonical_product_id": "gcan_mix",
+                        "canonical_name": "MIXED PEPPER",
+                        "category": "",
+                        "product_type": "",
+                        "brand": "",
+                        "variant": "",
+                        "size_value": "",
+                        "size_unit": "",
+                        "pack_qty": "",
+                        "measure_type": "",
+                        "notes": "",
+                        "created_at": "",
+                        "updated_at": "",
+                    }
+                )
+
+            result = CliRunner().invoke(
+                review_products.main,
+                [
+                    "--purchases-csv",
+                    str(purchases_csv),
+                    "--queue-csv",
+                    str(queue_csv),
+                    "--resolutions-csv",
+                    str(resolutions_csv),
+                    "--catalog-csv",
+                    str(catalog_csv),
+                    "--limit",
+                    "1",
+                ],
+                input="l\n1\ny\nlinked by test\n",
+                color=True,
+            )
+
+            self.assertEqual(0, result.exit_code)
+            self.assertIn("Select the canonical_name to associate 2 items with:", result.output)
+            self.assertIn('[1] MIXED PEPPER | gcan_mix', result.output)
+            self.assertIn('2 "MIXED PEPPER" items and future matches will be associated with "MIXED PEPPER".', result.output)
+            self.assertIn("actions: [y]es  [n]o  [b]ack  [s]kip  [q]uit", result.output)
+            with resolutions_csv.open(newline="", encoding="utf-8") as handle:
+                rows = list(csv.DictReader(handle))
+            self.assertEqual("gcan_mix", rows[0]["canonical_product_id"])
+            self.assertEqual("link", rows[0]["resolution_action"])
+
+    def test_review_products_creates_canonical_and_resolution(self):
+        with tempfile.TemporaryDirectory() as tmpdir:
+            purchases_csv = Path(tmpdir) / "purchases.csv"
+            queue_csv = Path(tmpdir) / "review_queue.csv"
+            resolutions_csv = Path(tmpdir) / "review_resolutions.csv"
+            catalog_csv = Path(tmpdir) / "canonical_catalog.csv"
+
+            with purchases_csv.open("w", newline="", encoding="utf-8") as handle:
+                writer = csv.DictWriter(
+                    handle,
+                    fieldnames=[
+                        "purchase_date",
+                        "observed_product_id",
+                        "canonical_product_id",
+                        "retailer",
+                        "raw_item_name",
+                        "normalized_item_name",
+                        "image_url",
+                        "upc",
+                        "line_total",
+                        "order_id",
+                        "line_no",
+                    ],
+                )
+                writer.writeheader()
+                writer.writerow(
+                    {
+                        "purchase_date": "2026-03-15",
+                        "observed_product_id": "gobs_ice",
+                        "canonical_product_id": "",
+                        "retailer": "giant",
+                        "raw_item_name": "SB BAGGED ICE 20LB",
+                        "normalized_item_name": "BAGGED ICE",
+                        "image_url": "",
+                        "upc": "",
+                        "line_total": "3.50",
+                        "order_id": "g1",
+                        "line_no": "1",
+                    }
+                )
+
+            with mock.patch.object(
+                review_products.click,
+                "prompt",
+                side_effect=["n", "ICE", "frozen", "ice", "manual merge", "q"],
+            ):
+                review_products.main.callback(
+                    purchases_csv=str(purchases_csv),
+                    queue_csv=str(queue_csv),
+                    resolutions_csv=str(resolutions_csv),
+                    catalog_csv=str(catalog_csv),
+                    limit=1,
+                    refresh_only=False,
+                )
+
+            self.assertTrue(queue_csv.exists())
+            self.assertTrue(resolutions_csv.exists())
+            self.assertTrue(catalog_csv.exists())
+            with resolutions_csv.open(newline="", encoding="utf-8") as handle:
+                resolution_rows = list(csv.DictReader(handle))
+            with catalog_csv.open(newline="", encoding="utf-8") as handle:
+                catalog_rows = list(csv.DictReader(handle))
+            self.assertEqual("create", resolution_rows[0]["resolution_action"])
+            self.assertEqual("approved", resolution_rows[0]["status"])
+            self.assertEqual("ICE", catalog_rows[0]["canonical_name"])
+
+
+if __name__ == "__main__":
+    unittest.main()