Compare commits
1 Commits
d20a131e04
...
master
| Author | SHA1 | Date | |
|---|---|---|---|
| bf2934f487 |
1
.gitignore
vendored
1
.gitignore
vendored
@@ -21,6 +21,7 @@ env/
|
|||||||
|
|
||||||
# --- project private data ---
|
# --- project private data ---
|
||||||
/private/
|
/private/
|
||||||
|
giant_output/
|
||||||
|
|
||||||
# --- django ---
|
# --- django ---
|
||||||
db.sqlite3
|
db.sqlite3
|
||||||
|
|||||||
103
README.md
103
README.md
@@ -1,103 +0,0 @@
|
|||||||
# scrape-giant
|
|
||||||
|
|
||||||
Small grocery-history pipeline for Giant receipts.
|
|
||||||
|
|
||||||
The project currently does four things:
|
|
||||||
|
|
||||||
1. scrape Giant in-store order history from an active Firefox session
|
|
||||||
2. enrich raw line items into a deterministic `items_enriched.csv`
|
|
||||||
3. aggregate retailer-facing observed products and build a manual review queue
|
|
||||||
4. create a first-pass canonical product layer plus conservative auto-links
|
|
||||||
|
|
||||||
The work so far is Giant-specific on the ingest side and intentionally simple on
|
|
||||||
the shared product-model side.
|
|
||||||
|
|
||||||
## Current flow
|
|
||||||
|
|
||||||
Run the commands from the repo root with the project venv active, or call them
|
|
||||||
directly through `./venv/bin/python`.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
./venv/bin/python scraper.py
|
|
||||||
./venv/bin/python enrich_giant.py
|
|
||||||
./venv/bin/python build_observed_products.py
|
|
||||||
./venv/bin/python build_review_queue.py
|
|
||||||
./venv/bin/python build_canonical_layer.py
|
|
||||||
```
|
|
||||||
|
|
||||||
## Inputs
|
|
||||||
|
|
||||||
- Firefox cookies for `giantfood.com`
|
|
||||||
- `GIANT_USER_ID` and `GIANT_LOYALTY_NUMBER` in `.env`, shell env, or prompts
|
|
||||||
- Giant raw order payloads in `giant_output/raw/`
|
|
||||||
|
|
||||||
## Outputs
|
|
||||||
|
|
||||||
Current generated files live under `giant_output/`:
|
|
||||||
|
|
||||||
- `orders.csv`: flattened visit/order rows from the Giant history API
|
|
||||||
- `items.csv`: flattened raw line items from fetched order detail payloads
|
|
||||||
- `items_enriched.csv`: deterministic parsed/enriched line items
|
|
||||||
- `products_observed.csv`: retailer-facing observed product groups
|
|
||||||
- `review_queue.csv`: products needing manual review
|
|
||||||
- `products_canonical.csv`: shared canonical product rows
|
|
||||||
- `product_links.csv`: observed-to-canonical links
|
|
||||||
|
|
||||||
Raw json remains the source of truth:
|
|
||||||
|
|
||||||
- `giant_output/raw/history.json`
|
|
||||||
- `giant_output/raw/<order_id>.json`
|
|
||||||
|
|
||||||
## Scripts
|
|
||||||
|
|
||||||
- `scraper.py`: fetches Giant history/detail payloads and updates `orders.csv` and `items.csv`
|
|
||||||
- `enrich_giant.py`: reads raw Giant order json and writes `items_enriched.csv`
|
|
||||||
- `build_observed_products.py`: groups enriched rows into `products_observed.csv`
|
|
||||||
- `build_review_queue.py`: generates `review_queue.csv` and preserves review status on reruns
|
|
||||||
- `build_canonical_layer.py`: builds `products_canonical.csv` and `product_links.csv`
|
|
||||||
|
|
||||||
## Notes on the current model
|
|
||||||
|
|
||||||
- Observed products are retailer-specific: Giant, Costco.
|
|
||||||
- Canonical products are the first cross-retailer layer.
|
|
||||||
- Auto-linking is conservative:
|
|
||||||
exact UPC first, then exact normalized name plus exact size/unit context, then
|
|
||||||
exact normalized name when there is no size context to conflict.
|
|
||||||
- Fee rows are excluded from auto-linking.
|
|
||||||
- Unknown values are left blank instead of guessed.
|
|
||||||
|
|
||||||
## Verification
|
|
||||||
|
|
||||||
Run the test suite with:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
./venv/bin/python -m unittest discover -s tests
|
|
||||||
```
|
|
||||||
|
|
||||||
Useful one-off rebuilds:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
./venv/bin/python enrich_giant.py
|
|
||||||
./venv/bin/python build_observed_products.py
|
|
||||||
./venv/bin/python build_review_queue.py
|
|
||||||
./venv/bin/python build_canonical_layer.py
|
|
||||||
```
|
|
||||||
|
|
||||||
## Project docs
|
|
||||||
|
|
||||||
- `pm/tasks.org`: task log and evidence
|
|
||||||
- `pm/data-model.org`: file layout and schema decisions
|
|
||||||
|
|
||||||
## Status
|
|
||||||
|
|
||||||
Completed through `t1.7`:
|
|
||||||
|
|
||||||
- Giant receipt fetch CLI
|
|
||||||
- data model and file layout
|
|
||||||
- Giant parser/enricher
|
|
||||||
- observed products
|
|
||||||
- review queue
|
|
||||||
- canonical layer scaffold
|
|
||||||
- conservative auto-link rules
|
|
||||||
|
|
||||||
Next planned task is `t1.8`: add a Costco raw ingest path.
|
|
||||||
23
agents.md
23
agents.md
@@ -1,23 +0,0 @@
|
|||||||
# agent rules
|
|
||||||
|
|
||||||
## priorities
|
|
||||||
- optimize for simplicity, boringness, and long-term maintainability
|
|
||||||
- prefer minimal diffs; avoid refactors unless required for the active task
|
|
||||||
|
|
||||||
## tech stack
|
|
||||||
- python; pandas or polars
|
|
||||||
- file storage: json and csv, no sqlite or databases
|
|
||||||
- do not add new dependencies unless explicitly approved; if unavoidable, document justification in the active task notes
|
|
||||||
|
|
||||||
## workflow
|
|
||||||
- prefer direct argv commands (no bash -lc / compound shell chains) unless necessary
|
|
||||||
- work on ONE task at a time unless explicitly instructed otherwise
|
|
||||||
- at the start of work, state the task id you are executing
|
|
||||||
- do not start work unless a task id is specified; if missing, choose the earliest unchecked task and say so
|
|
||||||
- propose incremental steps
|
|
||||||
- always include basic tests for core logic
|
|
||||||
- when you complete a task:
|
|
||||||
- mark it [x] in pm/tasks.md
|
|
||||||
- fill in evidence with commit hash + commands run
|
|
||||||
- never mark complete unless acceptance criteria are met
|
|
||||||
- include date and time (HH:MM)
|
|
||||||
@@ -1,212 +0,0 @@
|
|||||||
import click
|
|
||||||
|
|
||||||
from layer_helpers import read_csv_rows, representative_value, stable_id, write_csv_rows
|
|
||||||
|
|
||||||
|
|
||||||
CANONICAL_FIELDS = [
|
|
||||||
"canonical_product_id",
|
|
||||||
"canonical_name",
|
|
||||||
"product_type",
|
|
||||||
"brand",
|
|
||||||
"variant",
|
|
||||||
"size_value",
|
|
||||||
"size_unit",
|
|
||||||
"pack_qty",
|
|
||||||
"measure_type",
|
|
||||||
"normalized_quantity",
|
|
||||||
"normalized_quantity_unit",
|
|
||||||
"notes",
|
|
||||||
"created_at",
|
|
||||||
"updated_at",
|
|
||||||
]
|
|
||||||
|
|
||||||
LINK_FIELDS = [
|
|
||||||
"observed_product_id",
|
|
||||||
"canonical_product_id",
|
|
||||||
"link_method",
|
|
||||||
"link_confidence",
|
|
||||||
"review_status",
|
|
||||||
"reviewed_by",
|
|
||||||
"reviewed_at",
|
|
||||||
"link_notes",
|
|
||||||
]
|
|
||||||
|
|
||||||
|
|
||||||
def to_float(value):
|
|
||||||
try:
|
|
||||||
return float(value)
|
|
||||||
except (TypeError, ValueError):
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def normalized_quantity(row):
|
|
||||||
size_value = to_float(row.get("representative_size_value"))
|
|
||||||
pack_qty = to_float(row.get("representative_pack_qty")) or 1.0
|
|
||||||
size_unit = row.get("representative_size_unit", "")
|
|
||||||
measure_type = row.get("representative_measure_type", "")
|
|
||||||
|
|
||||||
if size_value is not None and size_unit:
|
|
||||||
return format(size_value * pack_qty, "g"), size_unit
|
|
||||||
|
|
||||||
if row.get("representative_pack_qty") and measure_type == "count":
|
|
||||||
return row["representative_pack_qty"], "count"
|
|
||||||
|
|
||||||
if measure_type == "each":
|
|
||||||
return "1", "each"
|
|
||||||
|
|
||||||
return "", ""
|
|
||||||
|
|
||||||
|
|
||||||
def auto_link_rule(observed_row):
|
|
||||||
if observed_row.get("is_fee") == "true":
|
|
||||||
return "", "", ""
|
|
||||||
|
|
||||||
if observed_row.get("representative_upc"):
|
|
||||||
return (
|
|
||||||
"exact_upc",
|
|
||||||
f"upc={observed_row['representative_upc']}",
|
|
||||||
"high",
|
|
||||||
)
|
|
||||||
|
|
||||||
if (
|
|
||||||
observed_row.get("representative_name_norm")
|
|
||||||
and observed_row.get("representative_size_value")
|
|
||||||
and observed_row.get("representative_size_unit")
|
|
||||||
):
|
|
||||||
return (
|
|
||||||
"exact_name_size",
|
|
||||||
"|".join(
|
|
||||||
[
|
|
||||||
f"name={observed_row['representative_name_norm']}",
|
|
||||||
f"size={observed_row['representative_size_value']}",
|
|
||||||
f"unit={observed_row['representative_size_unit']}",
|
|
||||||
f"pack={observed_row['representative_pack_qty']}",
|
|
||||||
f"measure={observed_row['representative_measure_type']}",
|
|
||||||
]
|
|
||||||
),
|
|
||||||
"high",
|
|
||||||
)
|
|
||||||
|
|
||||||
if (
|
|
||||||
observed_row.get("representative_name_norm")
|
|
||||||
and not observed_row.get("representative_size_value")
|
|
||||||
and not observed_row.get("representative_size_unit")
|
|
||||||
and not observed_row.get("representative_pack_qty")
|
|
||||||
):
|
|
||||||
return (
|
|
||||||
"exact_name",
|
|
||||||
"|".join(
|
|
||||||
[
|
|
||||||
f"name={observed_row['representative_name_norm']}",
|
|
||||||
f"measure={observed_row['representative_measure_type']}",
|
|
||||||
]
|
|
||||||
),
|
|
||||||
"medium",
|
|
||||||
)
|
|
||||||
|
|
||||||
return "", "", ""
|
|
||||||
|
|
||||||
|
|
||||||
def canonical_row_for_group(canonical_product_id, group_rows, link_method):
|
|
||||||
quantity_value, quantity_unit = normalized_quantity(
|
|
||||||
{
|
|
||||||
"representative_size_value": representative_value(
|
|
||||||
group_rows, "representative_size_value"
|
|
||||||
),
|
|
||||||
"representative_size_unit": representative_value(
|
|
||||||
group_rows, "representative_size_unit"
|
|
||||||
),
|
|
||||||
"representative_pack_qty": representative_value(
|
|
||||||
group_rows, "representative_pack_qty"
|
|
||||||
),
|
|
||||||
"representative_measure_type": representative_value(
|
|
||||||
group_rows, "representative_measure_type"
|
|
||||||
),
|
|
||||||
}
|
|
||||||
)
|
|
||||||
return {
|
|
||||||
"canonical_product_id": canonical_product_id,
|
|
||||||
"canonical_name": representative_value(group_rows, "representative_name_norm"),
|
|
||||||
"product_type": "",
|
|
||||||
"brand": representative_value(group_rows, "representative_brand"),
|
|
||||||
"variant": representative_value(group_rows, "representative_variant"),
|
|
||||||
"size_value": representative_value(group_rows, "representative_size_value"),
|
|
||||||
"size_unit": representative_value(group_rows, "representative_size_unit"),
|
|
||||||
"pack_qty": representative_value(group_rows, "representative_pack_qty"),
|
|
||||||
"measure_type": representative_value(group_rows, "representative_measure_type"),
|
|
||||||
"normalized_quantity": quantity_value,
|
|
||||||
"normalized_quantity_unit": quantity_unit,
|
|
||||||
"notes": f"auto-linked via {link_method}",
|
|
||||||
"created_at": "",
|
|
||||||
"updated_at": "",
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def build_canonical_layer(observed_rows):
|
|
||||||
canonical_rows = []
|
|
||||||
link_rows = []
|
|
||||||
groups = {}
|
|
||||||
|
|
||||||
for observed_row in sorted(observed_rows, key=lambda row: row["observed_product_id"]):
|
|
||||||
link_method, group_key, confidence = auto_link_rule(observed_row)
|
|
||||||
if not group_key:
|
|
||||||
continue
|
|
||||||
|
|
||||||
canonical_product_id = stable_id("gcan", f"{link_method}|{group_key}")
|
|
||||||
groups.setdefault(canonical_product_id, {"method": link_method, "rows": []})
|
|
||||||
groups[canonical_product_id]["rows"].append(observed_row)
|
|
||||||
link_rows.append(
|
|
||||||
{
|
|
||||||
"observed_product_id": observed_row["observed_product_id"],
|
|
||||||
"canonical_product_id": canonical_product_id,
|
|
||||||
"link_method": link_method,
|
|
||||||
"link_confidence": confidence,
|
|
||||||
"review_status": "",
|
|
||||||
"reviewed_by": "",
|
|
||||||
"reviewed_at": "",
|
|
||||||
"link_notes": "",
|
|
||||||
}
|
|
||||||
)
|
|
||||||
|
|
||||||
for canonical_product_id, group in sorted(groups.items()):
|
|
||||||
canonical_rows.append(
|
|
||||||
canonical_row_for_group(
|
|
||||||
canonical_product_id, group["rows"], group["method"]
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
return canonical_rows, link_rows
|
|
||||||
|
|
||||||
|
|
||||||
@click.command()
|
|
||||||
@click.option(
|
|
||||||
"--observed-csv",
|
|
||||||
default="giant_output/products_observed.csv",
|
|
||||||
show_default=True,
|
|
||||||
help="Path to observed product rows.",
|
|
||||||
)
|
|
||||||
@click.option(
|
|
||||||
"--canonical-csv",
|
|
||||||
default="giant_output/products_canonical.csv",
|
|
||||||
show_default=True,
|
|
||||||
help="Path to canonical product output.",
|
|
||||||
)
|
|
||||||
@click.option(
|
|
||||||
"--links-csv",
|
|
||||||
default="giant_output/product_links.csv",
|
|
||||||
show_default=True,
|
|
||||||
help="Path to observed-to-canonical link output.",
|
|
||||||
)
|
|
||||||
def main(observed_csv, canonical_csv, links_csv):
|
|
||||||
observed_rows = read_csv_rows(observed_csv)
|
|
||||||
canonical_rows, link_rows = build_canonical_layer(observed_rows)
|
|
||||||
write_csv_rows(canonical_csv, canonical_rows, CANONICAL_FIELDS)
|
|
||||||
write_csv_rows(links_csv, link_rows, LINK_FIELDS)
|
|
||||||
click.echo(
|
|
||||||
f"wrote {len(canonical_rows)} canonical rows to {canonical_csv} and "
|
|
||||||
f"{len(link_rows)} links to {links_csv}"
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,147 +0,0 @@
|
|||||||
from collections import defaultdict
|
|
||||||
|
|
||||||
import click
|
|
||||||
|
|
||||||
from layer_helpers import (
|
|
||||||
compact_join,
|
|
||||||
distinct_values,
|
|
||||||
first_nonblank,
|
|
||||||
read_csv_rows,
|
|
||||||
representative_value,
|
|
||||||
stable_id,
|
|
||||||
write_csv_rows,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
OUTPUT_FIELDS = [
|
|
||||||
"observed_product_id",
|
|
||||||
"retailer",
|
|
||||||
"observed_key",
|
|
||||||
"representative_upc",
|
|
||||||
"representative_item_name",
|
|
||||||
"representative_name_norm",
|
|
||||||
"representative_brand",
|
|
||||||
"representative_variant",
|
|
||||||
"representative_size_value",
|
|
||||||
"representative_size_unit",
|
|
||||||
"representative_pack_qty",
|
|
||||||
"representative_measure_type",
|
|
||||||
"representative_image_url",
|
|
||||||
"is_store_brand",
|
|
||||||
"is_fee",
|
|
||||||
"first_seen_date",
|
|
||||||
"last_seen_date",
|
|
||||||
"times_seen",
|
|
||||||
"example_order_id",
|
|
||||||
"example_item_name",
|
|
||||||
"raw_name_examples",
|
|
||||||
"normalized_name_examples",
|
|
||||||
"example_prices",
|
|
||||||
"distinct_item_names_count",
|
|
||||||
"distinct_upcs_count",
|
|
||||||
]
|
|
||||||
|
|
||||||
|
|
||||||
def build_observed_key(row):
|
|
||||||
if row.get("upc"):
|
|
||||||
return "|".join(
|
|
||||||
[
|
|
||||||
row["retailer"],
|
|
||||||
f"upc={row['upc']}",
|
|
||||||
f"name={row['item_name_norm']}",
|
|
||||||
]
|
|
||||||
)
|
|
||||||
|
|
||||||
return "|".join(
|
|
||||||
[
|
|
||||||
row["retailer"],
|
|
||||||
f"name={row['item_name_norm']}",
|
|
||||||
f"size={row['size_value']}",
|
|
||||||
f"unit={row['size_unit']}",
|
|
||||||
f"pack={row['pack_qty']}",
|
|
||||||
f"measure={row['measure_type']}",
|
|
||||||
f"store_brand={row['is_store_brand']}",
|
|
||||||
f"fee={row['is_fee']}",
|
|
||||||
]
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def build_observed_products(rows):
|
|
||||||
grouped = defaultdict(list)
|
|
||||||
for row in rows:
|
|
||||||
grouped[build_observed_key(row)].append(row)
|
|
||||||
|
|
||||||
observed_rows = []
|
|
||||||
for observed_key, group_rows in sorted(grouped.items()):
|
|
||||||
ordered = sorted(
|
|
||||||
group_rows,
|
|
||||||
key=lambda row: (row["order_date"], row["order_id"], int(row["line_no"])),
|
|
||||||
)
|
|
||||||
observed_rows.append(
|
|
||||||
{
|
|
||||||
"observed_product_id": stable_id("gobs", observed_key),
|
|
||||||
"retailer": ordered[0]["retailer"],
|
|
||||||
"observed_key": observed_key,
|
|
||||||
"representative_upc": representative_value(ordered, "upc"),
|
|
||||||
"representative_item_name": representative_value(ordered, "item_name"),
|
|
||||||
"representative_name_norm": representative_value(
|
|
||||||
ordered, "item_name_norm"
|
|
||||||
),
|
|
||||||
"representative_brand": representative_value(ordered, "brand_guess"),
|
|
||||||
"representative_variant": representative_value(ordered, "variant"),
|
|
||||||
"representative_size_value": representative_value(ordered, "size_value"),
|
|
||||||
"representative_size_unit": representative_value(ordered, "size_unit"),
|
|
||||||
"representative_pack_qty": representative_value(ordered, "pack_qty"),
|
|
||||||
"representative_measure_type": representative_value(
|
|
||||||
ordered, "measure_type"
|
|
||||||
),
|
|
||||||
"representative_image_url": first_nonblank(ordered, "image_url"),
|
|
||||||
"is_store_brand": representative_value(ordered, "is_store_brand"),
|
|
||||||
"is_fee": representative_value(ordered, "is_fee"),
|
|
||||||
"first_seen_date": ordered[0]["order_date"],
|
|
||||||
"last_seen_date": ordered[-1]["order_date"],
|
|
||||||
"times_seen": str(len(ordered)),
|
|
||||||
"example_order_id": ordered[0]["order_id"],
|
|
||||||
"example_item_name": ordered[0]["item_name"],
|
|
||||||
"raw_name_examples": compact_join(
|
|
||||||
distinct_values(ordered, "item_name"), limit=4
|
|
||||||
),
|
|
||||||
"normalized_name_examples": compact_join(
|
|
||||||
distinct_values(ordered, "item_name_norm"), limit=4
|
|
||||||
),
|
|
||||||
"example_prices": compact_join(
|
|
||||||
distinct_values(ordered, "line_total"), limit=4
|
|
||||||
),
|
|
||||||
"distinct_item_names_count": str(
|
|
||||||
len(distinct_values(ordered, "item_name"))
|
|
||||||
),
|
|
||||||
"distinct_upcs_count": str(len(distinct_values(ordered, "upc"))),
|
|
||||||
}
|
|
||||||
)
|
|
||||||
|
|
||||||
observed_rows.sort(key=lambda row: row["observed_product_id"])
|
|
||||||
return observed_rows
|
|
||||||
|
|
||||||
|
|
||||||
@click.command()
|
|
||||||
@click.option(
|
|
||||||
"--items-enriched-csv",
|
|
||||||
default="giant_output/items_enriched.csv",
|
|
||||||
show_default=True,
|
|
||||||
help="Path to enriched Giant item rows.",
|
|
||||||
)
|
|
||||||
@click.option(
|
|
||||||
"--output-csv",
|
|
||||||
default="giant_output/products_observed.csv",
|
|
||||||
show_default=True,
|
|
||||||
help="Path to observed product output.",
|
|
||||||
)
|
|
||||||
def main(items_enriched_csv, output_csv):
|
|
||||||
rows = read_csv_rows(items_enriched_csv)
|
|
||||||
observed_rows = build_observed_products(rows)
|
|
||||||
write_csv_rows(output_csv, observed_rows, OUTPUT_FIELDS)
|
|
||||||
click.echo(f"wrote {len(observed_rows)} rows to {output_csv}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,168 +0,0 @@
|
|||||||
from collections import defaultdict
|
|
||||||
from datetime import date
|
|
||||||
|
|
||||||
import click
|
|
||||||
|
|
||||||
from layer_helpers import compact_join, distinct_values, read_csv_rows, stable_id, write_csv_rows
|
|
||||||
|
|
||||||
|
|
||||||
OUTPUT_FIELDS = [
|
|
||||||
"review_id",
|
|
||||||
"queue_type",
|
|
||||||
"retailer",
|
|
||||||
"observed_product_id",
|
|
||||||
"canonical_product_id",
|
|
||||||
"reason_code",
|
|
||||||
"priority",
|
|
||||||
"raw_item_names",
|
|
||||||
"normalized_names",
|
|
||||||
"upc",
|
|
||||||
"image_url",
|
|
||||||
"example_prices",
|
|
||||||
"seen_count",
|
|
||||||
"status",
|
|
||||||
"resolution_notes",
|
|
||||||
"created_at",
|
|
||||||
"updated_at",
|
|
||||||
]
|
|
||||||
|
|
||||||
|
|
||||||
def existing_review_state(path):
|
|
||||||
try:
|
|
||||||
rows = read_csv_rows(path)
|
|
||||||
except FileNotFoundError:
|
|
||||||
return {}
|
|
||||||
return {row["review_id"]: row for row in rows}
|
|
||||||
|
|
||||||
|
|
||||||
def review_reasons(observed_row):
|
|
||||||
reasons = []
|
|
||||||
if observed_row["is_fee"] == "true":
|
|
||||||
return reasons
|
|
||||||
if observed_row["distinct_upcs_count"] not in {"", "0", "1"}:
|
|
||||||
reasons.append(("multiple_upcs", "high"))
|
|
||||||
if observed_row["distinct_item_names_count"] not in {"", "0", "1"}:
|
|
||||||
reasons.append(("multiple_raw_names", "medium"))
|
|
||||||
if not observed_row["representative_image_url"]:
|
|
||||||
reasons.append(("missing_image", "medium"))
|
|
||||||
if not observed_row["representative_upc"]:
|
|
||||||
reasons.append(("missing_upc", "high"))
|
|
||||||
if not observed_row["representative_name_norm"]:
|
|
||||||
reasons.append(("missing_normalized_name", "high"))
|
|
||||||
return reasons
|
|
||||||
|
|
||||||
|
|
||||||
def build_review_queue(observed_rows, item_rows, existing_rows, today_text):
|
|
||||||
by_observed = defaultdict(list)
|
|
||||||
for row in item_rows:
|
|
||||||
observed_id = row.get("observed_product_id", "")
|
|
||||||
if observed_id:
|
|
||||||
by_observed[observed_id].append(row)
|
|
||||||
|
|
||||||
queue_rows = []
|
|
||||||
for observed_row in observed_rows:
|
|
||||||
reasons = review_reasons(observed_row)
|
|
||||||
if not reasons:
|
|
||||||
continue
|
|
||||||
|
|
||||||
related_items = by_observed.get(observed_row["observed_product_id"], [])
|
|
||||||
raw_names = compact_join(distinct_values(related_items, "item_name"), limit=5)
|
|
||||||
norm_names = compact_join(
|
|
||||||
distinct_values(related_items, "item_name_norm"), limit=5
|
|
||||||
)
|
|
||||||
example_prices = compact_join(
|
|
||||||
distinct_values(related_items, "line_total"), limit=5
|
|
||||||
)
|
|
||||||
|
|
||||||
for reason_code, priority in reasons:
|
|
||||||
review_id = stable_id(
|
|
||||||
"rvw",
|
|
||||||
f"{observed_row['observed_product_id']}|{reason_code}",
|
|
||||||
)
|
|
||||||
prior = existing_rows.get(review_id, {})
|
|
||||||
queue_rows.append(
|
|
||||||
{
|
|
||||||
"review_id": review_id,
|
|
||||||
"queue_type": "observed_product",
|
|
||||||
"retailer": observed_row["retailer"],
|
|
||||||
"observed_product_id": observed_row["observed_product_id"],
|
|
||||||
"canonical_product_id": prior.get("canonical_product_id", ""),
|
|
||||||
"reason_code": reason_code,
|
|
||||||
"priority": priority,
|
|
||||||
"raw_item_names": raw_names,
|
|
||||||
"normalized_names": norm_names,
|
|
||||||
"upc": observed_row["representative_upc"],
|
|
||||||
"image_url": observed_row["representative_image_url"],
|
|
||||||
"example_prices": example_prices,
|
|
||||||
"seen_count": observed_row["times_seen"],
|
|
||||||
"status": prior.get("status", "pending"),
|
|
||||||
"resolution_notes": prior.get("resolution_notes", ""),
|
|
||||||
"created_at": prior.get("created_at", today_text),
|
|
||||||
"updated_at": today_text,
|
|
||||||
}
|
|
||||||
)
|
|
||||||
|
|
||||||
queue_rows.sort(key=lambda row: (row["priority"], row["reason_code"], row["review_id"]))
|
|
||||||
return queue_rows
|
|
||||||
|
|
||||||
|
|
||||||
def attach_observed_ids(item_rows, observed_rows):
|
|
||||||
observed_by_key = {row["observed_key"]: row["observed_product_id"] for row in observed_rows}
|
|
||||||
attached = []
|
|
||||||
for row in item_rows:
|
|
||||||
observed_key = "|".join(
|
|
||||||
[
|
|
||||||
row["retailer"],
|
|
||||||
f"upc={row['upc']}",
|
|
||||||
f"name={row['item_name_norm']}",
|
|
||||||
]
|
|
||||||
) if row.get("upc") else "|".join(
|
|
||||||
[
|
|
||||||
row["retailer"],
|
|
||||||
f"name={row['item_name_norm']}",
|
|
||||||
f"size={row['size_value']}",
|
|
||||||
f"unit={row['size_unit']}",
|
|
||||||
f"pack={row['pack_qty']}",
|
|
||||||
f"measure={row['measure_type']}",
|
|
||||||
f"store_brand={row['is_store_brand']}",
|
|
||||||
f"fee={row['is_fee']}",
|
|
||||||
]
|
|
||||||
)
|
|
||||||
enriched = dict(row)
|
|
||||||
enriched["observed_product_id"] = observed_by_key.get(observed_key, "")
|
|
||||||
attached.append(enriched)
|
|
||||||
return attached
|
|
||||||
|
|
||||||
|
|
||||||
@click.command()
|
|
||||||
@click.option(
|
|
||||||
"--observed-csv",
|
|
||||||
default="giant_output/products_observed.csv",
|
|
||||||
show_default=True,
|
|
||||||
help="Path to observed product rows.",
|
|
||||||
)
|
|
||||||
@click.option(
|
|
||||||
"--items-enriched-csv",
|
|
||||||
default="giant_output/items_enriched.csv",
|
|
||||||
show_default=True,
|
|
||||||
help="Path to enriched Giant item rows.",
|
|
||||||
)
|
|
||||||
@click.option(
|
|
||||||
"--output-csv",
|
|
||||||
default="giant_output/review_queue.csv",
|
|
||||||
show_default=True,
|
|
||||||
help="Path to review queue output.",
|
|
||||||
)
|
|
||||||
def main(observed_csv, items_enriched_csv, output_csv):
|
|
||||||
observed_rows = read_csv_rows(observed_csv)
|
|
||||||
item_rows = read_csv_rows(items_enriched_csv)
|
|
||||||
item_rows = attach_observed_ids(item_rows, observed_rows)
|
|
||||||
existing_rows = existing_review_state(output_csv)
|
|
||||||
today_text = str(date.today())
|
|
||||||
queue_rows = build_review_queue(observed_rows, item_rows, existing_rows, today_text)
|
|
||||||
write_csv_rows(output_csv, queue_rows, OUTPUT_FIELDS)
|
|
||||||
click.echo(f"wrote {len(queue_rows)} rows to {output_csv}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
426
enrich_giant.py
426
enrich_giant.py
@@ -1,426 +0,0 @@
|
|||||||
import csv
|
|
||||||
import json
|
|
||||||
import re
|
|
||||||
from decimal import Decimal, InvalidOperation, ROUND_HALF_UP
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import click
|
|
||||||
|
|
||||||
|
|
||||||
PARSER_VERSION = "giant-enrich-v1"
|
|
||||||
RETAILER = "giant"
|
|
||||||
DEFAULT_INPUT_DIR = Path("giant_output/raw")
|
|
||||||
DEFAULT_OUTPUT_CSV = Path("giant_output/items_enriched.csv")
|
|
||||||
|
|
||||||
OUTPUT_FIELDS = [
|
|
||||||
"retailer",
|
|
||||||
"order_id",
|
|
||||||
"line_no",
|
|
||||||
"observed_item_key",
|
|
||||||
"order_date",
|
|
||||||
"pod_id",
|
|
||||||
"item_name",
|
|
||||||
"upc",
|
|
||||||
"category_id",
|
|
||||||
"category",
|
|
||||||
"qty",
|
|
||||||
"unit",
|
|
||||||
"unit_price",
|
|
||||||
"line_total",
|
|
||||||
"picked_weight",
|
|
||||||
"mvp_savings",
|
|
||||||
"reward_savings",
|
|
||||||
"coupon_savings",
|
|
||||||
"coupon_price",
|
|
||||||
"image_url",
|
|
||||||
"raw_order_path",
|
|
||||||
"item_name_norm",
|
|
||||||
"brand_guess",
|
|
||||||
"variant",
|
|
||||||
"size_value",
|
|
||||||
"size_unit",
|
|
||||||
"pack_qty",
|
|
||||||
"measure_type",
|
|
||||||
"is_store_brand",
|
|
||||||
"is_fee",
|
|
||||||
"price_per_each",
|
|
||||||
"price_per_lb",
|
|
||||||
"price_per_oz",
|
|
||||||
"parse_version",
|
|
||||||
"parse_notes",
|
|
||||||
]
|
|
||||||
|
|
||||||
STORE_BRAND_PREFIXES = {
|
|
||||||
"SB": "SB",
|
|
||||||
"NP": "NP",
|
|
||||||
}
|
|
||||||
|
|
||||||
ABBREVIATIONS = {
|
|
||||||
"APPLE": "APPLE",
|
|
||||||
"APPLES": "APPLES",
|
|
||||||
"APLE": "APPLE",
|
|
||||||
"BASIL": "BASIL",
|
|
||||||
"BLK": "BLACK",
|
|
||||||
"BNLS": "BONELESS",
|
|
||||||
"BRWN": "BROWN",
|
|
||||||
"CARROTS": "CARROTS",
|
|
||||||
"CHDR": "CHEDDAR",
|
|
||||||
"CHICKEN": "CHICKEN",
|
|
||||||
"CHOC": "CHOCOLATE",
|
|
||||||
"CHS": "CHEESE",
|
|
||||||
"CHSE": "CHEESE",
|
|
||||||
"CHZ": "CHEESE",
|
|
||||||
"CILANTRO": "CILANTRO",
|
|
||||||
"CKI": "COOKIE",
|
|
||||||
"CRSHD": "CRUSHED",
|
|
||||||
"FLR": "FLOUR",
|
|
||||||
"FRSH": "FRESH",
|
|
||||||
"GALA": "GALA",
|
|
||||||
"GRAHM": "GRAHAM",
|
|
||||||
"HOT": "HOT",
|
|
||||||
"HRSRDSH": "HORSERADISH",
|
|
||||||
"IMP": "IMPORTED",
|
|
||||||
"IQF": "IQF",
|
|
||||||
"LENTILS": "LENTILS",
|
|
||||||
"LG": "LARGE",
|
|
||||||
"MLK": "MILK",
|
|
||||||
"MSTRD": "MUSTARD",
|
|
||||||
"ONION": "ONION",
|
|
||||||
"ORG": "ORGANIC",
|
|
||||||
"PEPPER": "PEPPER",
|
|
||||||
"PEPPERS": "PEPPERS",
|
|
||||||
"POT": "POTATO",
|
|
||||||
"POTATO": "POTATO",
|
|
||||||
"PPR": "PEPPER",
|
|
||||||
"RICOTTA": "RICOTTA",
|
|
||||||
"ROASTER": "ROASTER",
|
|
||||||
"ROTINI": "ROTINI",
|
|
||||||
"SCE": "SAUCE",
|
|
||||||
"SLC": "SLICED",
|
|
||||||
"SPINCH": "SPINACH",
|
|
||||||
"SPNC": "SPINACH",
|
|
||||||
"SPINACH": "SPINACH",
|
|
||||||
"SQZ": "SQUEEZE",
|
|
||||||
"SWT": "SWEET",
|
|
||||||
"THYME": "THYME",
|
|
||||||
"TOM": "TOMATO",
|
|
||||||
"TOMS": "TOMATOES",
|
|
||||||
"TRTL": "TORTILLA",
|
|
||||||
"VEG": "VEGETABLE",
|
|
||||||
"VINEGAR": "VINEGAR",
|
|
||||||
"WHT": "WHITE",
|
|
||||||
"WHOLE": "WHOLE",
|
|
||||||
"YLW": "YELLOW",
|
|
||||||
"YLWGLD": "YELLOW_GOLD",
|
|
||||||
}
|
|
||||||
|
|
||||||
FEE_PATTERNS = [
|
|
||||||
re.compile(r"\bBAG CHARGE\b"),
|
|
||||||
re.compile(r"\bDISC AT TOTAL\b"),
|
|
||||||
]
|
|
||||||
|
|
||||||
SIZE_RE = re.compile(r"(?<![A-Z0-9])(\d+(?:\.\d+)?)(?:\s*)(OZ|Z|LB|LBS|ML|L|FZ|FL OZ|QT|PT|GAL|GA)\b")
|
|
||||||
PACK_RE = re.compile(r"(?<![A-Z0-9])(\d+(?:\.\d+)?)(?:\s*)(CT|PK|PKG|PACK)\b")
|
|
||||||
|
|
||||||
|
|
||||||
def to_decimal(value):
|
|
||||||
if value in ("", None):
|
|
||||||
return None
|
|
||||||
|
|
||||||
try:
|
|
||||||
return Decimal(str(value))
|
|
||||||
except (InvalidOperation, ValueError):
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def format_decimal(value, places=4):
|
|
||||||
if value is None:
|
|
||||||
return ""
|
|
||||||
|
|
||||||
quant = Decimal("1").scaleb(-places)
|
|
||||||
normalized = value.quantize(quant, rounding=ROUND_HALF_UP).normalize()
|
|
||||||
return format(normalized, "f")
|
|
||||||
|
|
||||||
|
|
||||||
def normalize_whitespace(value):
|
|
||||||
return " ".join(str(value or "").strip().split())
|
|
||||||
|
|
||||||
|
|
||||||
def clean_item_name(name):
|
|
||||||
cleaned = normalize_whitespace(name).upper()
|
|
||||||
cleaned = re.sub(r"^\+", "", cleaned)
|
|
||||||
cleaned = re.sub(r"^PLU#\d+\s*", "", cleaned)
|
|
||||||
cleaned = cleaned.replace("#", " ")
|
|
||||||
return normalize_whitespace(cleaned)
|
|
||||||
|
|
||||||
|
|
||||||
def extract_store_brand_prefix(cleaned_name):
|
|
||||||
for prefix, brand in STORE_BRAND_PREFIXES.items():
|
|
||||||
if cleaned_name == prefix or cleaned_name.startswith(f"{prefix} "):
|
|
||||||
return prefix, brand
|
|
||||||
return "", ""
|
|
||||||
|
|
||||||
|
|
||||||
def extract_image_url(item):
|
|
||||||
image = item.get("image")
|
|
||||||
if isinstance(image, dict):
|
|
||||||
for key in ["xlarge", "large", "medium", "small"]:
|
|
||||||
value = image.get(key)
|
|
||||||
if value:
|
|
||||||
return value
|
|
||||||
if isinstance(image, str):
|
|
||||||
return image
|
|
||||||
return ""
|
|
||||||
|
|
||||||
|
|
||||||
def parse_size_and_pack(cleaned_name):
|
|
||||||
size_value = ""
|
|
||||||
size_unit = ""
|
|
||||||
pack_qty = ""
|
|
||||||
|
|
||||||
size_matches = list(SIZE_RE.finditer(cleaned_name))
|
|
||||||
if size_matches:
|
|
||||||
match = size_matches[-1]
|
|
||||||
size_value = normalize_number(match.group(1))
|
|
||||||
size_unit = normalize_unit(match.group(2))
|
|
||||||
|
|
||||||
pack_matches = list(PACK_RE.finditer(cleaned_name))
|
|
||||||
if pack_matches:
|
|
||||||
match = pack_matches[-1]
|
|
||||||
pack_qty = normalize_number(match.group(1))
|
|
||||||
|
|
||||||
return size_value, size_unit, pack_qty
|
|
||||||
|
|
||||||
|
|
||||||
def normalize_number(value):
|
|
||||||
decimal = to_decimal(value)
|
|
||||||
if decimal is None:
|
|
||||||
return ""
|
|
||||||
return format(decimal.normalize(), "f")
|
|
||||||
|
|
||||||
|
|
||||||
def normalize_unit(unit):
|
|
||||||
collapsed = normalize_whitespace(unit).upper()
|
|
||||||
return {
|
|
||||||
"Z": "oz",
|
|
||||||
"OZ": "oz",
|
|
||||||
"FZ": "fl_oz",
|
|
||||||
"FL OZ": "fl_oz",
|
|
||||||
"LB": "lb",
|
|
||||||
"LBS": "lb",
|
|
||||||
"ML": "ml",
|
|
||||||
"L": "l",
|
|
||||||
"QT": "qt",
|
|
||||||
"PT": "pt",
|
|
||||||
"GAL": "gal",
|
|
||||||
"GA": "gal",
|
|
||||||
}.get(collapsed, collapsed.lower())
|
|
||||||
|
|
||||||
|
|
||||||
def strip_measure_tokens(cleaned_name):
|
|
||||||
without_sizes = SIZE_RE.sub(" ", cleaned_name)
|
|
||||||
without_measures = PACK_RE.sub(" ", without_sizes)
|
|
||||||
return normalize_whitespace(without_measures)
|
|
||||||
|
|
||||||
|
|
||||||
def expand_token(token):
|
|
||||||
return ABBREVIATIONS.get(token, token)
|
|
||||||
|
|
||||||
|
|
||||||
def normalize_item_name(cleaned_name):
|
|
||||||
prefix, _brand = extract_store_brand_prefix(cleaned_name)
|
|
||||||
base = cleaned_name
|
|
||||||
if prefix:
|
|
||||||
base = normalize_whitespace(base[len(prefix):])
|
|
||||||
|
|
||||||
base = strip_measure_tokens(base)
|
|
||||||
expanded_tokens = [expand_token(token) for token in base.split()]
|
|
||||||
expanded = " ".join(token for token in expanded_tokens if token)
|
|
||||||
return normalize_whitespace(expanded)
|
|
||||||
|
|
||||||
|
|
||||||
def guess_measure_type(item, size_unit, pack_qty):
|
|
||||||
unit = normalize_whitespace(item.get("lbEachCd")).upper()
|
|
||||||
picked_weight = to_decimal(item.get("totalPickedWeight"))
|
|
||||||
qty = to_decimal(item.get("shipQy"))
|
|
||||||
|
|
||||||
if unit == "LB" or (picked_weight is not None and picked_weight > 0 and unit != "EA"):
|
|
||||||
return "weight"
|
|
||||||
if size_unit in {"lb", "oz"}:
|
|
||||||
return "weight"
|
|
||||||
if size_unit in {"ml", "l", "qt", "pt", "gal", "fl_oz"}:
|
|
||||||
return "volume"
|
|
||||||
if pack_qty:
|
|
||||||
return "count"
|
|
||||||
if unit == "EA" or (qty is not None and qty > 0):
|
|
||||||
return "each"
|
|
||||||
return ""
|
|
||||||
|
|
||||||
|
|
||||||
def is_fee_item(cleaned_name):
|
|
||||||
return any(pattern.search(cleaned_name) for pattern in FEE_PATTERNS)
|
|
||||||
|
|
||||||
|
|
||||||
def derive_prices(item, measure_type, size_value="", size_unit="", pack_qty=""):
|
|
||||||
qty = to_decimal(item.get("shipQy"))
|
|
||||||
line_total = to_decimal(item.get("groceryAmount"))
|
|
||||||
picked_weight = to_decimal(item.get("totalPickedWeight"))
|
|
||||||
parsed_size = to_decimal(size_value)
|
|
||||||
parsed_pack = to_decimal(pack_qty) or Decimal("1")
|
|
||||||
|
|
||||||
price_per_each = ""
|
|
||||||
price_per_lb = ""
|
|
||||||
price_per_oz = ""
|
|
||||||
|
|
||||||
if line_total is None:
|
|
||||||
return price_per_each, price_per_lb, price_per_oz
|
|
||||||
|
|
||||||
if measure_type == "each" and qty not in (None, Decimal("0")):
|
|
||||||
price_per_each = format_decimal(line_total / qty)
|
|
||||||
|
|
||||||
if measure_type == "count" and qty not in (None, Decimal("0")):
|
|
||||||
price_per_each = format_decimal(line_total / qty)
|
|
||||||
|
|
||||||
if measure_type == "weight" and picked_weight not in (None, Decimal("0")):
|
|
||||||
per_lb = line_total / picked_weight
|
|
||||||
price_per_lb = format_decimal(per_lb)
|
|
||||||
price_per_oz = format_decimal(per_lb / Decimal("16"))
|
|
||||||
return price_per_each, price_per_lb, price_per_oz
|
|
||||||
|
|
||||||
if measure_type == "weight" and parsed_size not in (None, Decimal("0")) and qty not in (None, Decimal("0")):
|
|
||||||
total_units = qty * parsed_pack * parsed_size
|
|
||||||
if size_unit == "lb":
|
|
||||||
per_lb = line_total / total_units
|
|
||||||
price_per_lb = format_decimal(per_lb)
|
|
||||||
price_per_oz = format_decimal(per_lb / Decimal("16"))
|
|
||||||
elif size_unit == "oz":
|
|
||||||
per_oz = line_total / total_units
|
|
||||||
price_per_oz = format_decimal(per_oz)
|
|
||||||
price_per_lb = format_decimal(per_oz * Decimal("16"))
|
|
||||||
|
|
||||||
return price_per_each, price_per_lb, price_per_oz
|
|
||||||
|
|
||||||
|
|
||||||
def parse_item(order_id, order_date, raw_path, line_no, item):
|
|
||||||
cleaned_name = clean_item_name(item.get("itemName", ""))
|
|
||||||
size_value, size_unit, pack_qty = parse_size_and_pack(cleaned_name)
|
|
||||||
prefix, brand_guess = extract_store_brand_prefix(cleaned_name)
|
|
||||||
normalized_name = normalize_item_name(cleaned_name)
|
|
||||||
measure_type = guess_measure_type(item, size_unit, pack_qty)
|
|
||||||
price_per_each, price_per_lb, price_per_oz = derive_prices(
|
|
||||||
item,
|
|
||||||
measure_type,
|
|
||||||
size_value=size_value,
|
|
||||||
size_unit=size_unit,
|
|
||||||
pack_qty=pack_qty,
|
|
||||||
)
|
|
||||||
is_fee = is_fee_item(cleaned_name)
|
|
||||||
parse_notes = []
|
|
||||||
|
|
||||||
if prefix:
|
|
||||||
parse_notes.append(f"store_brand_prefix={prefix}")
|
|
||||||
if is_fee:
|
|
||||||
parse_notes.append("fee_item")
|
|
||||||
if size_value and not size_unit:
|
|
||||||
parse_notes.append("size_without_unit")
|
|
||||||
|
|
||||||
return {
|
|
||||||
"retailer": RETAILER,
|
|
||||||
"order_id": str(order_id),
|
|
||||||
"line_no": str(line_no),
|
|
||||||
"observed_item_key": f"{RETAILER}:{order_id}:{line_no}",
|
|
||||||
"order_date": normalize_whitespace(order_date),
|
|
||||||
"pod_id": stringify(item.get("podId")),
|
|
||||||
"item_name": stringify(item.get("itemName")),
|
|
||||||
"upc": stringify(item.get("primUpcCd")),
|
|
||||||
"category_id": stringify(item.get("categoryId")),
|
|
||||||
"category": stringify(item.get("categoryDesc")),
|
|
||||||
"qty": stringify(item.get("shipQy")),
|
|
||||||
"unit": stringify(item.get("lbEachCd")),
|
|
||||||
"unit_price": stringify(item.get("unitPrice")),
|
|
||||||
"line_total": stringify(item.get("groceryAmount")),
|
|
||||||
"picked_weight": stringify(item.get("totalPickedWeight")),
|
|
||||||
"mvp_savings": stringify(item.get("mvpSavings")),
|
|
||||||
"reward_savings": stringify(item.get("rewardSavings")),
|
|
||||||
"coupon_savings": stringify(item.get("couponSavings")),
|
|
||||||
"coupon_price": stringify(item.get("couponPrice")),
|
|
||||||
"image_url": extract_image_url(item),
|
|
||||||
"raw_order_path": raw_path.as_posix(),
|
|
||||||
"item_name_norm": normalized_name,
|
|
||||||
"brand_guess": brand_guess,
|
|
||||||
"variant": "",
|
|
||||||
"size_value": size_value,
|
|
||||||
"size_unit": size_unit,
|
|
||||||
"pack_qty": pack_qty,
|
|
||||||
"measure_type": measure_type,
|
|
||||||
"is_store_brand": "true" if bool(prefix) else "false",
|
|
||||||
"is_fee": "true" if is_fee else "false",
|
|
||||||
"price_per_each": price_per_each,
|
|
||||||
"price_per_lb": price_per_lb,
|
|
||||||
"price_per_oz": price_per_oz,
|
|
||||||
"parse_version": PARSER_VERSION,
|
|
||||||
"parse_notes": ";".join(parse_notes),
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def stringify(value):
|
|
||||||
if value is None:
|
|
||||||
return ""
|
|
||||||
return str(value)
|
|
||||||
|
|
||||||
|
|
||||||
def iter_order_rows(raw_dir):
|
|
||||||
for path in sorted(raw_dir.glob("*.json")):
|
|
||||||
if path.name == "history.json":
|
|
||||||
continue
|
|
||||||
|
|
||||||
payload = json.loads(path.read_text(encoding="utf-8"))
|
|
||||||
order_id = payload.get("orderId", path.stem)
|
|
||||||
order_date = payload.get("orderDate", "")
|
|
||||||
|
|
||||||
for line_no, item in enumerate(payload.get("items", []), start=1):
|
|
||||||
yield parse_item(order_id, order_date, path, line_no, item)
|
|
||||||
|
|
||||||
|
|
||||||
def build_items_enriched(raw_dir):
|
|
||||||
rows = list(iter_order_rows(raw_dir))
|
|
||||||
rows.sort(key=lambda row: (row["order_date"], row["order_id"], int(row["line_no"])))
|
|
||||||
return rows
|
|
||||||
|
|
||||||
|
|
||||||
def write_csv(path, rows):
|
|
||||||
path.parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
with path.open("w", newline="", encoding="utf-8") as handle:
|
|
||||||
writer = csv.DictWriter(handle, fieldnames=OUTPUT_FIELDS)
|
|
||||||
writer.writeheader()
|
|
||||||
writer.writerows(rows)
|
|
||||||
|
|
||||||
|
|
||||||
@click.command()
|
|
||||||
@click.option(
|
|
||||||
"--input-dir",
|
|
||||||
default=str(DEFAULT_INPUT_DIR),
|
|
||||||
show_default=True,
|
|
||||||
help="Directory containing Giant raw order json files.",
|
|
||||||
)
|
|
||||||
@click.option(
|
|
||||||
"--output-csv",
|
|
||||||
default=str(DEFAULT_OUTPUT_CSV),
|
|
||||||
show_default=True,
|
|
||||||
help="CSV path for enriched Giant item rows.",
|
|
||||||
)
|
|
||||||
def main(input_dir, output_csv):
|
|
||||||
raw_dir = Path(input_dir)
|
|
||||||
output_path = Path(output_csv)
|
|
||||||
|
|
||||||
if not raw_dir.exists():
|
|
||||||
raise click.ClickException(f"input dir does not exist: {raw_dir}")
|
|
||||||
|
|
||||||
rows = build_items_enriched(raw_dir)
|
|
||||||
write_csv(output_path, rows)
|
|
||||||
|
|
||||||
click.echo(f"wrote {len(rows)} rows to {output_path}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,54 +0,0 @@
|
|||||||
import csv
|
|
||||||
import hashlib
|
|
||||||
from collections import Counter
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
|
|
||||||
def read_csv_rows(path):
|
|
||||||
path = Path(path)
|
|
||||||
with path.open(newline="", encoding="utf-8") as handle:
|
|
||||||
return list(csv.DictReader(handle))
|
|
||||||
|
|
||||||
|
|
||||||
def write_csv_rows(path, rows, fieldnames):
|
|
||||||
path = Path(path)
|
|
||||||
path.parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
with path.open("w", newline="", encoding="utf-8") as handle:
|
|
||||||
writer = csv.DictWriter(handle, fieldnames=fieldnames)
|
|
||||||
writer.writeheader()
|
|
||||||
writer.writerows(rows)
|
|
||||||
|
|
||||||
|
|
||||||
def stable_id(prefix, raw_key):
|
|
||||||
digest = hashlib.sha1(str(raw_key).encode("utf-8")).hexdigest()[:12]
|
|
||||||
return f"{prefix}_{digest}"
|
|
||||||
|
|
||||||
|
|
||||||
def first_nonblank(rows, field):
|
|
||||||
for row in rows:
|
|
||||||
value = row.get(field, "")
|
|
||||||
if value:
|
|
||||||
return value
|
|
||||||
return ""
|
|
||||||
|
|
||||||
|
|
||||||
def representative_value(rows, field):
|
|
||||||
values = [row.get(field, "") for row in rows if row.get(field, "")]
|
|
||||||
if not values:
|
|
||||||
return ""
|
|
||||||
counts = Counter(values)
|
|
||||||
return sorted(counts.items(), key=lambda item: (-item[1], item[0]))[0][0]
|
|
||||||
|
|
||||||
|
|
||||||
def distinct_values(rows, field):
|
|
||||||
return sorted({row.get(field, "") for row in rows if row.get(field, "")})
|
|
||||||
|
|
||||||
|
|
||||||
def compact_join(values, limit=3):
|
|
||||||
unique = []
|
|
||||||
seen = set()
|
|
||||||
for value in values:
|
|
||||||
if value and value not in seen:
|
|
||||||
seen.add(value)
|
|
||||||
unique.append(value)
|
|
||||||
return " | ".join(unique[:limit])
|
|
||||||
@@ -1,300 +0,0 @@
|
|||||||
* grocery data model and file layout
|
|
||||||
|
|
||||||
This document defines the shared file layout and stable CSV schemas for the
|
|
||||||
grocery pipeline. The goal is to keep retailer-specific ingest separate from
|
|
||||||
cross-retailer product modeling so Giant-specific quirks do not become the
|
|
||||||
system of record.
|
|
||||||
|
|
||||||
** design rules
|
|
||||||
|
|
||||||
- Raw retailer exports remain the source of truth.
|
|
||||||
- Retailer parsing is isolated to retailer-specific files and ids.
|
|
||||||
- Cross-retailer product layers begin only after retailer-specific enrichment.
|
|
||||||
- CSV schemas are stable and additive: new columns may be appended, but
|
|
||||||
existing columns should not be repurposed.
|
|
||||||
- Unknown values should be left blank rather than guessed.
|
|
||||||
|
|
||||||
** directory layout
|
|
||||||
|
|
||||||
Use one top-level data root:
|
|
||||||
|
|
||||||
#+begin_example
|
|
||||||
data/
|
|
||||||
giant/
|
|
||||||
raw/
|
|
||||||
history.json
|
|
||||||
orders/
|
|
||||||
<order_id>.json
|
|
||||||
orders.csv
|
|
||||||
items_raw.csv
|
|
||||||
items_enriched.csv
|
|
||||||
products_observed.csv
|
|
||||||
costco/
|
|
||||||
raw/
|
|
||||||
...
|
|
||||||
orders.csv
|
|
||||||
items_raw.csv
|
|
||||||
items_enriched.csv
|
|
||||||
products_observed.csv
|
|
||||||
shared/
|
|
||||||
products_canonical.csv
|
|
||||||
product_links.csv
|
|
||||||
review_queue.csv
|
|
||||||
#+end_example
|
|
||||||
|
|
||||||
** layer responsibilities
|
|
||||||
|
|
||||||
- `data/<retailer>/raw/`
|
|
||||||
Stores unmodified retailer payloads exactly as fetched.
|
|
||||||
- `data/<retailer>/orders.csv`
|
|
||||||
One row per retailer order or visit, flattened from raw order data.
|
|
||||||
- `data/<retailer>/items_raw.csv`
|
|
||||||
One row per retailer line item, preserving retailer-native values needed for
|
|
||||||
reruns and debugging.
|
|
||||||
- `data/<retailer>/items_enriched.csv`
|
|
||||||
Parsed retailer line items with normalized fields and derived guesses, still
|
|
||||||
retailer-specific.
|
|
||||||
- `data/<retailer>/products_observed.csv`
|
|
||||||
Distinct retailer-facing observed products aggregated from enriched items.
|
|
||||||
- `data/shared/products_canonical.csv`
|
|
||||||
Cross-retailer canonical product entities used for comparison.
|
|
||||||
- `data/shared/product_links.csv`
|
|
||||||
Links from retailer observed products to canonical products.
|
|
||||||
- `data/shared/review_queue.csv`
|
|
||||||
Human review queue for unresolved or low-confidence matching/parsing cases.
|
|
||||||
|
|
||||||
** retailer-specific versus shared
|
|
||||||
|
|
||||||
Retailer-specific:
|
|
||||||
|
|
||||||
- raw json payloads
|
|
||||||
- retailer order ids
|
|
||||||
- retailer line numbers
|
|
||||||
- retailer category ids and names
|
|
||||||
- retailer item names
|
|
||||||
- retailer image urls
|
|
||||||
- parsed guesses derived from one retailer feed
|
|
||||||
- observed products scoped to one retailer
|
|
||||||
|
|
||||||
Shared:
|
|
||||||
|
|
||||||
- canonical products
|
|
||||||
- observed-to-canonical links
|
|
||||||
- human review state for unresolved cases
|
|
||||||
- comparison-ready normalized quantity basis fields
|
|
||||||
|
|
||||||
Observed products are the boundary between retailer-specific parsing and
|
|
||||||
cross-retailer canonicalization. Nothing upstream of `products_observed.csv`
|
|
||||||
should require knowledge of another retailer.
|
|
||||||
|
|
||||||
** schema: `data/<retailer>/orders.csv`
|
|
||||||
|
|
||||||
One row per order or visit.
|
|
||||||
|
|
||||||
| column | meaning |
|
|
||||||
|-
|
|
||||||
| `retailer` | retailer slug such as `giant` |
|
|
||||||
| `order_id` | retailer order or visit id |
|
|
||||||
| `order_date` | order date in `YYYY-MM-DD` when available |
|
|
||||||
| `delivery_date` | fulfillment date in `YYYY-MM-DD` when available |
|
|
||||||
| `service_type` | retailer service type such as `INSTORE` |
|
|
||||||
| `order_total` | order total as provided by retailer |
|
|
||||||
| `payment_method` | retailer payment label |
|
|
||||||
| `total_item_count` | total line count or item count from retailer |
|
|
||||||
| `total_savings` | total savings as provided by retailer |
|
|
||||||
| `your_savings_total` | savings field from retailer when present |
|
|
||||||
| `coupons_discounts_total` | coupon/discount total from retailer |
|
|
||||||
| `store_name` | retailer store name |
|
|
||||||
| `store_number` | retailer store number |
|
|
||||||
| `store_address1` | street address |
|
|
||||||
| `store_city` | city |
|
|
||||||
| `store_state` | state or province |
|
|
||||||
| `store_zipcode` | postal code |
|
|
||||||
| `refund_order` | retailer refund flag |
|
|
||||||
| `ebt_order` | retailer EBT flag |
|
|
||||||
| `raw_history_path` | relative path to source history payload |
|
|
||||||
| `raw_order_path` | relative path to source order payload |
|
|
||||||
|
|
||||||
Primary key:
|
|
||||||
|
|
||||||
- (`retailer`, `order_id`)
|
|
||||||
|
|
||||||
** schema: `data/<retailer>/items_raw.csv`
|
|
||||||
|
|
||||||
One row per retailer line item.
|
|
||||||
|
|
||||||
| column | meaning |
|
|
||||||
|------------------+-----------------------------------------|
|
|
||||||
| `retailer` | retailer slug |
|
|
||||||
| `order_id` | retailer order id |
|
|
||||||
| `line_no` | stable line number within order export |
|
|
||||||
| `order_date` | copied from order when available |
|
|
||||||
| `pod_id` | retailer pod/item id |
|
|
||||||
| `item_name` | raw retailer item name |
|
|
||||||
| `upc` | retailer UPC or PLU value |
|
|
||||||
| `category_id` | retailer category id |
|
|
||||||
| `category` | retailer category description |
|
|
||||||
| `qty` | retailer quantity field |
|
|
||||||
| `unit` | retailer unit code such as `EA` or `LB` |
|
|
||||||
| `unit_price` | retailer unit price field |
|
|
||||||
| `line_total` | retailer extended price field |
|
|
||||||
| `picked_weight` | retailer picked weight field |
|
|
||||||
| `mvp_savings` | retailer savings field |
|
|
||||||
| `reward_savings` | retailer rewards savings field |
|
|
||||||
| `coupon_savings` | retailer coupon savings field |
|
|
||||||
| `coupon_price` | retailer coupon price field |
|
|
||||||
| `image_url` | raw retailer image url when present |
|
|
||||||
| `raw_order_path` | relative path to source order payload |
|
|
||||||
|
|
||||||
Primary key:
|
|
||||||
|
|
||||||
- (`retailer`, `order_id`, `line_no`)
|
|
||||||
|
|
||||||
** schema: `data/<retailer>/items_enriched.csv`
|
|
||||||
|
|
||||||
One row per retailer line item after deterministic parsing. Preserve the raw
|
|
||||||
fields from `items_raw.csv` and add parsed fields.
|
|
||||||
|
|
||||||
| column | meaning |
|
|
||||||
|---------------------+-------------------------------------------------------------|
|
|
||||||
| `retailer` | retailer slug |
|
|
||||||
| `order_id` | retailer order id |
|
|
||||||
| `line_no` | line number within order |
|
|
||||||
| `observed_item_key` | stable row key, typically `<retailer>:<order_id>:<line_no>` |
|
|
||||||
| `item_name` | raw retailer item name |
|
|
||||||
| `item_name_norm` | normalized item name |
|
|
||||||
| `brand_guess` | parsed brand guess |
|
|
||||||
| `variant` | parsed variant text |
|
|
||||||
| `size_value` | parsed numeric size value |
|
|
||||||
| `size_unit` | parsed size unit such as `oz`, `lb`, `fl_oz` |
|
|
||||||
| `pack_qty` | parsed pack or count guess |
|
|
||||||
| `measure_type` | `each`, `weight`, `volume`, `count`, or blank |
|
|
||||||
| `is_store_brand` | store-brand guess |
|
|
||||||
| `is_fee` | fee or non-product flag |
|
|
||||||
| `price_per_each` | derived per-each price when supported |
|
|
||||||
| `price_per_lb` | derived per-pound price when supported |
|
|
||||||
| `price_per_oz` | derived per-ounce price when supported |
|
|
||||||
| `image_url` | best available retailer image url |
|
|
||||||
| `parse_version` | parser version string for reruns |
|
|
||||||
| `parse_notes` | optional non-fatal parser notes |
|
|
||||||
|
|
||||||
Primary key:
|
|
||||||
|
|
||||||
- (`retailer`, `order_id`, `line_no`)
|
|
||||||
|
|
||||||
** schema: `data/<retailer>/products_observed.csv`
|
|
||||||
|
|
||||||
One row per distinct retailer-facing observed product.
|
|
||||||
|
|
||||||
| column | meaning |
|
|
||||||
|-------------------------------+----------------------------------------------------------------|
|
|
||||||
| `observed_product_id` | stable observed product id |
|
|
||||||
| `retailer` | retailer slug |
|
|
||||||
| `observed_key` | deterministic grouping key used to create the observed product |
|
|
||||||
| `representative_upc` | best representative UPC/PLU |
|
|
||||||
| `representative_item_name` | representative raw retailer name |
|
|
||||||
| `representative_name_norm` | representative normalized name |
|
|
||||||
| `representative_brand` | representative brand guess |
|
|
||||||
| `representative_variant` | representative variant |
|
|
||||||
| `representative_size_value` | representative size value |
|
|
||||||
| `representative_size_unit` | representative size unit |
|
|
||||||
| `representative_pack_qty` | representative pack/count |
|
|
||||||
| `representative_measure_type` | representative measure type |
|
|
||||||
| `representative_image_url` | representative image url |
|
|
||||||
| `is_store_brand` | representative store-brand flag |
|
|
||||||
| `is_fee` | representative fee flag |
|
|
||||||
| `first_seen_date` | first order date seen |
|
|
||||||
| `last_seen_date` | last order date seen |
|
|
||||||
| `times_seen` | number of enriched item rows grouped here |
|
|
||||||
| `example_order_id` | one example retailer order id |
|
|
||||||
| `example_item_name` | one example raw item name |
|
|
||||||
|
|
||||||
Primary key:
|
|
||||||
|
|
||||||
- (`observed_product_id`)
|
|
||||||
|
|
||||||
** schema: `data/shared/products_canonical.csv`
|
|
||||||
|
|
||||||
One row per cross-retailer canonical product.
|
|
||||||
|
|
||||||
| column | meaning |
|
|
||||||
|----------------------------+--------------------------------------------------|
|
|
||||||
| `canonical_product_id` | stable canonical product id |
|
|
||||||
| `canonical_name` | canonical human-readable name |
|
|
||||||
| `product_type` | broad class such as `apple`, `milk`, `trash_bag` |
|
|
||||||
| `brand` | canonical brand when applicable |
|
|
||||||
| `variant` | canonical variant |
|
|
||||||
| `size_value` | normalized size value |
|
|
||||||
| `size_unit` | normalized size unit |
|
|
||||||
| `pack_qty` | normalized pack/count |
|
|
||||||
| `measure_type` | normalized measure type |
|
|
||||||
| `normalized_quantity` | numeric comparison basis value |
|
|
||||||
| `normalized_quantity_unit` | basis unit such as `oz`, `lb`, `count` |
|
|
||||||
| `notes` | optional human notes |
|
|
||||||
| `created_at` | creation timestamp or date |
|
|
||||||
| `updated_at` | last update timestamp or date |
|
|
||||||
|
|
||||||
Primary key:
|
|
||||||
|
|
||||||
- (`canonical_product_id`)
|
|
||||||
|
|
||||||
** schema: `data/shared/product_links.csv`
|
|
||||||
|
|
||||||
One row per observed-to-canonical relationship.
|
|
||||||
|
|
||||||
| column | meaning |
|
|
||||||
|-
|
|
||||||
| `observed_product_id` | retailer observed product id |
|
|
||||||
| `canonical_product_id` | linked canonical product id |
|
|
||||||
| `link_method` | `manual`, `exact_upc`, `exact_name`, etc. |
|
|
||||||
| `link_confidence` | optional confidence label |
|
|
||||||
| `review_status` | `pending`, `approved`, `rejected`, or blank |
|
|
||||||
| `reviewed_by` | reviewer id or initials |
|
|
||||||
| `reviewed_at` | review timestamp or date |
|
|
||||||
| `link_notes` | optional notes |
|
|
||||||
|
|
||||||
Primary key:
|
|
||||||
|
|
||||||
- (`observed_product_id`, `canonical_product_id`)
|
|
||||||
|
|
||||||
** schema: `data/shared/review_queue.csv`
|
|
||||||
|
|
||||||
One row per issue needing human review.
|
|
||||||
|
|
||||||
| column | meaning |
|
|
||||||
|-
|
|
||||||
| `review_id` | stable review row id |
|
|
||||||
| `queue_type` | `observed_product`, `link_candidate`, `parse_issue` |
|
|
||||||
| `retailer` | retailer slug when applicable |
|
|
||||||
| `observed_product_id` | observed product id when applicable |
|
|
||||||
| `canonical_product_id` | candidate canonical id when applicable |
|
|
||||||
| `reason_code` | machine-readable review reason |
|
|
||||||
| `priority` | optional priority label |
|
|
||||||
| `raw_item_names` | compact list of example raw names |
|
|
||||||
| `normalized_names` | compact list of example normalized names |
|
|
||||||
| `upc` | example UPC/PLU |
|
|
||||||
| `image_url` | example image url |
|
|
||||||
| `example_prices` | compact list of example prices |
|
|
||||||
| `seen_count` | count of related rows |
|
|
||||||
| `status` | `pending`, `approved`, `rejected`, `deferred` |
|
|
||||||
| `resolution_notes` | reviewer notes |
|
|
||||||
| `created_at` | creation timestamp or date |
|
|
||||||
| `updated_at` | last update timestamp or date |
|
|
||||||
|
|
||||||
Primary key:
|
|
||||||
|
|
||||||
- (`review_id`)
|
|
||||||
|
|
||||||
** current giant mapping
|
|
||||||
|
|
||||||
Current scraper outputs map to the new layout as follows:
|
|
||||||
|
|
||||||
- `giant_output/raw/history.json` -> `data/giant/raw/history.json`
|
|
||||||
- `giant_output/raw/<order_id>.json` -> `data/giant/raw/orders/<order_id>.json`
|
|
||||||
- `giant_output/orders.csv` -> `data/giant/orders.csv`
|
|
||||||
- `giant_output/items.csv` -> `data/giant/items_raw.csv`
|
|
||||||
|
|
||||||
Current Giant raw order payloads already expose fields needed for future
|
|
||||||
enrichment, including `image`, `itemName`, `primUpcCd`, `lbEachCd`,
|
|
||||||
`unitPrice`, `groceryAmount`, and `totalPickedWeight`.
|
|
||||||
|
|
||||||
File diff suppressed because one or more lines are too long
151
pm/tasks.org
151
pm/tasks.org
@@ -1,4 +1,4 @@
|
|||||||
* [X] t1.1: harden giant receipt fetch cli (2-4 commits)
|
* [ ] t1.1: harden giant receipt fetch cli (2-4 commits)
|
||||||
** acceptance criteria
|
** acceptance criteria
|
||||||
- giant scraper runs from cli with prompts or env-backed defaults for `user_id` and `loyalty`
|
- giant scraper runs from cli with prompts or env-backed defaults for `user_id` and `loyalty`
|
||||||
- script reuses current browser session via firefox cookies + `curl_cffi`
|
- script reuses current browser session via firefox cookies + `curl_cffi`
|
||||||
@@ -12,11 +12,11 @@
|
|||||||
- raw json archive remains source of truth
|
- raw json archive remains source of truth
|
||||||
|
|
||||||
** evidence
|
** evidence
|
||||||
- commit: `d57b9cf` on branch `cx`
|
- commit:
|
||||||
- tests: `./venv/bin/python -m unittest discover -s tests`; `./venv/bin/python scraper.py --help`; verified `.env` loading via `scraper.load_config()`
|
- tests:
|
||||||
- date: 2026-03-14
|
- date:
|
||||||
|
|
||||||
* [X] t1.2: define grocery data model and file layout (1-2 commits)
|
* [ ] t1.2: define grocery data model and file layout (1-2 commits)
|
||||||
** acceptance criteria
|
** acceptance criteria
|
||||||
- decide and document the files/directories for:
|
- decide and document the files/directories for:
|
||||||
- retailer raw exports
|
- retailer raw exports
|
||||||
@@ -28,15 +28,15 @@
|
|||||||
- explicitly separate retailer-specific parsing from cross-retailer canonicalization
|
- explicitly separate retailer-specific parsing from cross-retailer canonicalization
|
||||||
|
|
||||||
** notes
|
** notes
|
||||||
- this is the guardrail task so we don't make giant-specific hacks the system of record
|
- this is the guardrail task so we don’t make giant-specific hacks the system of record
|
||||||
- keep schema minimal but extensible
|
- keep schema minimal but extensible
|
||||||
|
|
||||||
** evidence
|
** evidence
|
||||||
- commit: `42dbae1` on branch `cx`
|
- commit:
|
||||||
- tests: reviewed `giant_output/raw/history.json`, one sample raw order json, `giant_output/orders.csv`, `giant_output/items.csv`; documented schemas in `pm/data-model.org`
|
- tests:
|
||||||
- date: 2026-03-15
|
- date:
|
||||||
|
|
||||||
* [X] t1.3: build giant parser/enricher from raw json (2-4 commits)
|
* [ ] t1.3: build giant parser/enricher from raw json (2-4 commits)
|
||||||
** acceptance criteria
|
** acceptance criteria
|
||||||
- parser reads giant raw order json files
|
- parser reads giant raw order json files
|
||||||
- outputs `items_enriched.csv`
|
- outputs `items_enriched.csv`
|
||||||
@@ -54,11 +54,11 @@
|
|||||||
- parser should preserve ambiguity rather than hallucinating precision
|
- parser should preserve ambiguity rather than hallucinating precision
|
||||||
|
|
||||||
** evidence
|
** evidence
|
||||||
- commit: `14f2cc2` on branch `cx`
|
- commit:
|
||||||
- tests: `./venv/bin/python -m unittest discover -s tests`; `./venv/bin/python enrich_giant.py`; verified `giant_output/items_enriched.csv` on real raw data
|
- tests:
|
||||||
- date: 2026-03-16
|
- date:
|
||||||
|
|
||||||
* [X] t1.4: generate observed-product layer from enriched items (2-3 commits)
|
* [ ] t1.4: generate observed-product layer from enriched items (2-3 commits)
|
||||||
|
|
||||||
** acceptance criteria
|
** acceptance criteria
|
||||||
- distinct observed products are generated from enriched giant items
|
- distinct observed products are generated from enriched giant items
|
||||||
@@ -76,11 +76,11 @@
|
|||||||
- likely key is some combo of retailer + upc + normalized name
|
- likely key is some combo of retailer + upc + normalized name
|
||||||
|
|
||||||
** evidence
|
** evidence
|
||||||
- commit: `dc39214` on branch `cx`
|
- commit:
|
||||||
- tests: `./venv/bin/python -m unittest discover -s tests`; `./venv/bin/python build_observed_products.py`; verified `giant_output/products_observed.csv`
|
- tests:
|
||||||
- date: 2026-03-16
|
- date:
|
||||||
|
|
||||||
* [X] t1.5: build review queue for unresolved or low-confidence products (1-3 commits)
|
* [ ] t1.5: build review queue for unresolved or low-confidence products (1-3 commits)
|
||||||
|
|
||||||
** acceptance criteria
|
** acceptance criteria
|
||||||
- produce a review file containing observed products needing manual review
|
- produce a review file containing observed products needing manual review
|
||||||
@@ -98,11 +98,11 @@
|
|||||||
- optimize for “approve once, remember forever”
|
- optimize for “approve once, remember forever”
|
||||||
|
|
||||||
** evidence
|
** evidence
|
||||||
- commit: `9b13ec3` on branch `cx`
|
- commit:
|
||||||
- tests: `./venv/bin/python -m unittest discover -s tests`; `./venv/bin/python build_review_queue.py`; verified `giant_output/review_queue.csv`
|
- tests:
|
||||||
- date: 2026-03-16
|
- date:
|
||||||
|
|
||||||
* [X] t1.6: create canonical product layer and observed→canonical links (2-4 commits)
|
* [ ] t1.6: create canonical product layer and observed→canonical links (2-4 commits)
|
||||||
|
|
||||||
** acceptance criteria
|
** acceptance criteria
|
||||||
- define and create `products_canonical.csv`
|
- define and create `products_canonical.csv`
|
||||||
@@ -120,11 +120,11 @@
|
|||||||
- do not require llm assistance for v1
|
- do not require llm assistance for v1
|
||||||
|
|
||||||
** evidence
|
** evidence
|
||||||
- commit: `347cd44` on branch `cx`
|
- commit:
|
||||||
- tests: `./venv/bin/python -m unittest discover -s tests`; `./venv/bin/python build_canonical_layer.py`; verified seeded `giant_output/products_canonical.csv` and `giant_output/product_links.csv`
|
- tests:
|
||||||
- date: 2026-03-16
|
- date:
|
||||||
|
|
||||||
* [X] t1.7: implement auto-link rules for easy matches (2-3 commits)
|
* [ ] t1.7: implement auto-link rules for easy matches (2-3 commits)
|
||||||
|
|
||||||
** acceptance criteria
|
** acceptance criteria
|
||||||
- auto-link can match observed products to canonical products using deterministic rules
|
- auto-link can match observed products to canonical products using deterministic rules
|
||||||
@@ -139,104 +139,43 @@
|
|||||||
- false positives are worse than unresolved items
|
- false positives are worse than unresolved items
|
||||||
|
|
||||||
** evidence
|
** evidence
|
||||||
- commit: `385a31c` on branch `cx`
|
- commit:
|
||||||
- tests: `./venv/bin/python -m unittest discover -s tests`; `./venv/bin/python build_canonical_layer.py`; verified auto-linked `giant_output/products_canonical.csv` and `giant_output/product_links.csv`
|
- tests:
|
||||||
- date: 2026-03-16
|
- date:
|
||||||
|
|
||||||
* [ ] t1.8: support costco raw ingest path (2-5 commits)
|
* [ ] t1.8: support costco raw ingest path (2-5 commits)
|
||||||
|
|
||||||
** acceptance criteria
|
** acceptance criteria
|
||||||
- add a costco-specific raw ingest/export path
|
- add a costco-specific raw ingest/export path
|
||||||
- fetch costco receipt summary and receipt detail payloads from graphql endpoint
|
- output costco line items into the same shared raw/enriched schema family
|
||||||
- persist raw json under `costco_output/raw/orders.csv` and `./items.csv`, same format as giant
|
|
||||||
- costco-native identifiers such as `transactionBarcode` as order id and `itemNumber` as retailer item id
|
|
||||||
- preserve discount/coupon rows rather than dropping
|
|
||||||
|
|
||||||
** notes
|
|
||||||
- focus on raw costco acquisistion and flattening
|
|
||||||
- do not force costco identifiers into `upc`
|
|
||||||
- bearer/auth values should come from local env, not source
|
|
||||||
|
|
||||||
** evidence
|
|
||||||
- commit:
|
|
||||||
- tests:
|
|
||||||
- date:
|
|
||||||
|
|
||||||
* [ ] t1.8.1: support costco parser/enricher path (2-4 commits)
|
|
||||||
|
|
||||||
** acceptance criteria
|
|
||||||
- add a costco-specific enrich step producing `costco_output/items_enriched.csv`
|
|
||||||
- output rows into the same shared enriched schema family as Giant
|
|
||||||
- support costco-specific parsing for:
|
|
||||||
- `itemDescription01` + `itemDescription02`
|
|
||||||
- `itemNumber` as `retailer_item_id`
|
|
||||||
- discount lines / negative rows
|
|
||||||
- common size patterns such as `25#`, `48 OZ`, `2/24 OZ`, `6-PACK`
|
|
||||||
- preserve obvious unknowns as blank rather than guessed values
|
|
||||||
|
|
||||||
** notes
|
|
||||||
- this is the real schema compatibility proof, not raw ingest alone
|
|
||||||
- expect weaker identifiers than Giant
|
|
||||||
|
|
||||||
** evidence
|
|
||||||
- commit:
|
|
||||||
- tests:
|
|
||||||
- date:
|
|
||||||
* [ ] t1.8.2: validate cross-retailer observed/canonical flow (1-3 commits)
|
|
||||||
|
|
||||||
** acceptance criteria
|
|
||||||
- feed Giant and Costco enriched rows through the same observed/canonical pipeline
|
|
||||||
- confirm at least one product class can exist as:
|
- confirm at least one product class can exist as:
|
||||||
- Giant observed product
|
- giant observed product
|
||||||
- Costco observed product
|
- costco observed product
|
||||||
- one shared canonical product
|
- one shared canonical product
|
||||||
- document the exact example used for proof
|
|
||||||
|
|
||||||
** notes
|
** notes
|
||||||
- keep this to one or two well-behaved product classes first
|
- this is the proof that the architecture generalizes
|
||||||
- apples, eggs, bananas, or flour are better than weird prepared foods
|
- don’t chase perfection before the second retailer lands
|
||||||
|
|
||||||
** evidence
|
** evidence
|
||||||
- commit:
|
- commit:
|
||||||
- tests:
|
- tests:
|
||||||
- date:
|
- date:
|
||||||
* [ ] t1.8.3: extend shared schema for retailer-native ids and adjustment lines (1-2 commits)
|
|
||||||
|
* [ ] t1.9: compute normalized comparison metrics (2-3 commits)
|
||||||
|
|
||||||
** acceptance criteria
|
** acceptance criteria
|
||||||
- add shared fields needed for non-upc retailers, including:
|
- derive normalized comparison fields where possible:
|
||||||
- `retailer_item_id`
|
- price per lb
|
||||||
- `is_discount_line`
|
- price per oz
|
||||||
- `is_coupon_line` or equivalent if needed
|
- price per each
|
||||||
- keep `upc` nullable across the pipeline
|
- price per count
|
||||||
- update downstream builders/tests to accept retailers with blank `upc`
|
- metrics are attached at canonical or linked-observed level as appropriate
|
||||||
|
- emit obvious nulls when basis is unknown rather than inventing values
|
||||||
|
|
||||||
** notes
|
** notes
|
||||||
- this prevents costco from becoming a schema hack
|
- this is where “gala apples 5 lb bag vs other gala apples” becomes possible
|
||||||
- do this once instead of sprinkling exceptions everywhere
|
- units discipline matters a lot here
|
||||||
|
|
||||||
** evidence
|
|
||||||
- commit:
|
|
||||||
- tests:
|
|
||||||
- date:
|
|
||||||
* [ ] t1.9: compute normalized comparison metrics (2-4 commits)
|
|
||||||
|
|
||||||
** acceptance criteria
|
|
||||||
- derive normalized comparison fields where possible on enriched or observed product rows:
|
|
||||||
- `price_per_lb`
|
|
||||||
- `price_per_oz`
|
|
||||||
- `price_per_each`
|
|
||||||
- `price_per_count`
|
|
||||||
- preserve the source basis used to derive each metric, e.g.:
|
|
||||||
- parsed size/unit
|
|
||||||
- receipt weight
|
|
||||||
- explicit count/pack
|
|
||||||
- emit nulls when basis is unknown, conflicting, or ambiguous
|
|
||||||
- document at least one Giant vs Costco comparison example using the normalized metrics
|
|
||||||
|
|
||||||
** notes
|
|
||||||
- compute metrics as close to the raw observation as possible
|
|
||||||
- canonical layer can aggregate later, but should not invent missing unit economics
|
|
||||||
- unit discipline matters more than coverage
|
|
||||||
|
|
||||||
** evidence
|
** evidence
|
||||||
- commit:
|
- commit:
|
||||||
|
|||||||
BIN
requirements.txt
BIN
requirements.txt
Binary file not shown.
251
scrape-click.py
251
scrape-click.py
@@ -1,4 +1,253 @@
|
|||||||
from scraper import main
|
import json
|
||||||
|
import time
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import browser_cookie3
|
||||||
|
import click
|
||||||
|
import pandas as pd
|
||||||
|
from curl_cffi import requests
|
||||||
|
from dotenv import load_dotenv
|
||||||
|
import os
|
||||||
|
|
||||||
|
|
||||||
|
BASE = "https://giantfood.com"
|
||||||
|
ACCOUNT_PAGE = f"{BASE}/account/history/invoice/in-store"
|
||||||
|
|
||||||
|
|
||||||
|
def load_config():
|
||||||
|
load_dotenv()
|
||||||
|
return {
|
||||||
|
"user_id": os.getenv("GIANT_USER_ID", "").strip(),
|
||||||
|
"loyalty": os.getenv("GIANT_LOYALTY_NUMBER", "").strip(),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def build_session():
|
||||||
|
s = requests.Session()
|
||||||
|
s.cookies.update(browser_cookie3.firefox(domain_name="giantfood.com"))
|
||||||
|
s.headers.update({
|
||||||
|
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:148.0) Gecko/20100101 Firefox/148.0",
|
||||||
|
"accept": "application/json, text/plain, */*",
|
||||||
|
"accept-language": "en-US,en;q=0.9",
|
||||||
|
"referer": ACCOUNT_PAGE,
|
||||||
|
})
|
||||||
|
return s
|
||||||
|
|
||||||
|
|
||||||
|
def safe_get(session, url, **kwargs):
|
||||||
|
last_response = None
|
||||||
|
|
||||||
|
for attempt in range(3):
|
||||||
|
try:
|
||||||
|
r = session.get(
|
||||||
|
url,
|
||||||
|
impersonate="firefox",
|
||||||
|
timeout=30,
|
||||||
|
**kwargs,
|
||||||
|
)
|
||||||
|
last_response = r
|
||||||
|
|
||||||
|
if r.status_code == 200:
|
||||||
|
return r
|
||||||
|
|
||||||
|
click.echo(f"retry {attempt + 1}/3 status={r.status_code}")
|
||||||
|
except Exception as e:
|
||||||
|
click.echo(f"retry {attempt + 1}/3 error={e}")
|
||||||
|
|
||||||
|
time.sleep(3)
|
||||||
|
|
||||||
|
if last_response is not None:
|
||||||
|
last_response.raise_for_status()
|
||||||
|
|
||||||
|
raise RuntimeError(f"failed to fetch {url}")
|
||||||
|
|
||||||
|
|
||||||
|
def get_history(session, user_id, loyalty):
|
||||||
|
url = f"{BASE}/api/v6.0/user/{user_id}/order/history"
|
||||||
|
r = safe_get(
|
||||||
|
session,
|
||||||
|
url,
|
||||||
|
params={
|
||||||
|
"filter": "instore",
|
||||||
|
"loyaltyNumber": loyalty,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
return r.json()
|
||||||
|
|
||||||
|
|
||||||
|
def get_order_detail(session, user_id, order_id):
|
||||||
|
url = f"{BASE}/api/v6.0/user/{user_id}/order/history/detail/{order_id}"
|
||||||
|
r = safe_get(
|
||||||
|
session,
|
||||||
|
url,
|
||||||
|
params={"isInStore": "true"},
|
||||||
|
)
|
||||||
|
return r.json()
|
||||||
|
|
||||||
|
|
||||||
|
def flatten_orders(history, details):
|
||||||
|
orders = []
|
||||||
|
items = []
|
||||||
|
|
||||||
|
history_lookup = {
|
||||||
|
r["orderId"]: r
|
||||||
|
for r in history.get("records", [])
|
||||||
|
}
|
||||||
|
|
||||||
|
for d in details:
|
||||||
|
hist = history_lookup.get(d["orderId"], {})
|
||||||
|
pup = d.get("pup", {})
|
||||||
|
|
||||||
|
orders.append({
|
||||||
|
"order_id": d["orderId"],
|
||||||
|
"order_date": d.get("orderDate"),
|
||||||
|
"delivery_date": d.get("deliveryDate"),
|
||||||
|
"service_type": hist.get("serviceType"),
|
||||||
|
"order_total": d.get("orderTotal"),
|
||||||
|
"payment_method": d.get("paymentMethod"),
|
||||||
|
"total_item_count": d.get("totalItemCount"),
|
||||||
|
"total_savings": d.get("totalSavings"),
|
||||||
|
"your_savings_total": d.get("yourSavingsTotal"),
|
||||||
|
"coupons_discounts_total": d.get("couponsDiscountsTotal"),
|
||||||
|
"store_name": pup.get("storeName"),
|
||||||
|
"store_number": pup.get("aholdStoreNumber"),
|
||||||
|
"store_address1": pup.get("storeAddress1"),
|
||||||
|
"store_city": pup.get("storeCity"),
|
||||||
|
"store_state": pup.get("storeState"),
|
||||||
|
"store_zipcode": pup.get("storeZipcode"),
|
||||||
|
"refund_order": d.get("refundOrder"),
|
||||||
|
"ebt_order": d.get("ebtOrder"),
|
||||||
|
})
|
||||||
|
|
||||||
|
for i, item in enumerate(d.get("items", []), start=1):
|
||||||
|
items.append({
|
||||||
|
"order_id": d["orderId"],
|
||||||
|
"order_date": d.get("orderDate"),
|
||||||
|
"line_no": i,
|
||||||
|
"pod_id": item.get("podId"),
|
||||||
|
"item_name": item.get("itemName"),
|
||||||
|
"upc": item.get("primUpcCd"),
|
||||||
|
"category_id": item.get("categoryId"),
|
||||||
|
"category": item.get("categoryDesc"),
|
||||||
|
"qty": item.get("shipQy"),
|
||||||
|
"unit": item.get("lbEachCd"),
|
||||||
|
"unit_price": item.get("unitPrice"),
|
||||||
|
"line_total": item.get("groceryAmount"),
|
||||||
|
"picked_weight": item.get("totalPickedWeight"),
|
||||||
|
"mvp_savings": item.get("mvpSavings"),
|
||||||
|
"reward_savings": item.get("rewardSavings"),
|
||||||
|
"coupon_savings": item.get("couponSavings"),
|
||||||
|
"coupon_price": item.get("couponPrice"),
|
||||||
|
})
|
||||||
|
|
||||||
|
return pd.DataFrame(orders), pd.DataFrame(items)
|
||||||
|
|
||||||
|
|
||||||
|
def read_existing_order_ids(orders_csv: Path) -> set[str]:
|
||||||
|
if not orders_csv.exists():
|
||||||
|
return set()
|
||||||
|
|
||||||
|
try:
|
||||||
|
df = pd.read_csv(orders_csv, dtype={"order_id": str})
|
||||||
|
if "order_id" not in df.columns:
|
||||||
|
return set()
|
||||||
|
return set(df["order_id"].dropna().astype(str))
|
||||||
|
except Exception:
|
||||||
|
return set()
|
||||||
|
|
||||||
|
|
||||||
|
def append_dedup(existing_path: Path, new_df: pd.DataFrame, subset: list[str]) -> pd.DataFrame:
|
||||||
|
if existing_path.exists():
|
||||||
|
old_df = pd.read_csv(existing_path, dtype=str)
|
||||||
|
combined = pd.concat([old_df, new_df.astype(str)], ignore_index=True)
|
||||||
|
else:
|
||||||
|
combined = new_df.astype(str).copy()
|
||||||
|
|
||||||
|
combined = combined.drop_duplicates(subset=subset, keep="last")
|
||||||
|
combined.to_csv(existing_path, index=False)
|
||||||
|
return combined
|
||||||
|
|
||||||
|
|
||||||
|
@click.command()
|
||||||
|
@click.option("--user-id", default=None, help="giant user id")
|
||||||
|
@click.option("--loyalty", default=None, help="giant loyalty number")
|
||||||
|
@click.option("--outdir", default="giant_output", show_default=True, help="output directory")
|
||||||
|
@click.option("--sleep-seconds", default=1.5, show_default=True, type=float, help="delay between detail requests")
|
||||||
|
def main(user_id, loyalty, outdir, sleep_seconds):
|
||||||
|
cfg = load_config()
|
||||||
|
|
||||||
|
user_id = user_id or cfg["user_id"] or click.prompt("giant user id", type=str)
|
||||||
|
loyalty = loyalty or cfg["loyalty"] or click.prompt("giant loyalty number", type=str)
|
||||||
|
|
||||||
|
outdir = Path(outdir)
|
||||||
|
rawdir = outdir / "raw"
|
||||||
|
rawdir.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
orders_csv = outdir / "orders.csv"
|
||||||
|
items_csv = outdir / "items.csv"
|
||||||
|
|
||||||
|
click.echo("using cookies from your current firefox profile.")
|
||||||
|
click.echo(f"open giant here, make sure you're logged in, then return: {ACCOUNT_PAGE}")
|
||||||
|
click.pause(info="press any key once giant is open and logged in")
|
||||||
|
|
||||||
|
session = build_session()
|
||||||
|
|
||||||
|
click.echo("fetching order history...")
|
||||||
|
history = get_history(session, user_id, loyalty)
|
||||||
|
|
||||||
|
(rawdir / "history.json").write_text(
|
||||||
|
json.dumps(history, indent=2),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
|
||||||
|
records = history.get("records", [])
|
||||||
|
click.echo(f"history returned {len(records)} visits")
|
||||||
|
click.echo("tip: giant appears to expose only the most recent 50 visits, so run this periodically if you want full continuity.")
|
||||||
|
|
||||||
|
history_order_ids = [str(r["orderId"]) for r in records]
|
||||||
|
existing_order_ids = read_existing_order_ids(orders_csv)
|
||||||
|
new_order_ids = [oid for oid in history_order_ids if oid not in existing_order_ids]
|
||||||
|
|
||||||
|
click.echo(f"existing orders in csv: {len(existing_order_ids)}")
|
||||||
|
click.echo(f"new orders to fetch: {len(new_order_ids)}")
|
||||||
|
|
||||||
|
if not new_order_ids:
|
||||||
|
click.echo("no new orders found. done.")
|
||||||
|
return
|
||||||
|
|
||||||
|
details = []
|
||||||
|
for order_id in new_order_ids:
|
||||||
|
click.echo(f"fetching {order_id}")
|
||||||
|
d = get_order_detail(session, user_id, order_id)
|
||||||
|
details.append(d)
|
||||||
|
|
||||||
|
(rawdir / f"{order_id}.json").write_text(
|
||||||
|
json.dumps(d, indent=2),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
|
||||||
|
time.sleep(sleep_seconds)
|
||||||
|
|
||||||
|
click.echo("flattening new data...")
|
||||||
|
orders_df, items_df = flatten_orders(history, details)
|
||||||
|
|
||||||
|
orders_all = append_dedup(
|
||||||
|
orders_csv,
|
||||||
|
orders_df,
|
||||||
|
subset=["order_id"],
|
||||||
|
)
|
||||||
|
|
||||||
|
items_all = append_dedup(
|
||||||
|
items_csv,
|
||||||
|
items_df,
|
||||||
|
subset=["order_id", "line_no", "item_name", "upc", "line_total"],
|
||||||
|
)
|
||||||
|
|
||||||
|
click.echo("done")
|
||||||
|
click.echo(f"orders csv: {orders_csv}")
|
||||||
|
click.echo(f"items csv: {items_csv}")
|
||||||
|
click.echo(f"total orders stored: {len(orders_all)}")
|
||||||
|
click.echo(f"total item rows stored: {len(items_all)}")
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
|
|||||||
347
scraper.py
347
scraper.py
@@ -1,84 +1,29 @@
|
|||||||
import csv
|
|
||||||
import json
|
import json
|
||||||
import os
|
|
||||||
import time
|
import time
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from dotenv import load_dotenv
|
|
||||||
import browser_cookie3
|
import browser_cookie3
|
||||||
|
import pandas as pd
|
||||||
from curl_cffi import requests
|
from curl_cffi import requests
|
||||||
import click
|
|
||||||
|
|
||||||
|
|
||||||
BASE = "https://giantfood.com"
|
BASE = "https://giantfood.com"
|
||||||
ACCOUNT_PAGE = f"{BASE}/account/history/invoice/in-store"
|
ACCOUNT_PAGE = f"{BASE}/account/history/invoice/in-store"
|
||||||
|
|
||||||
ORDER_FIELDS = [
|
USER_ID = "369513017"
|
||||||
"order_id",
|
LOYALTY = "440155630880"
|
||||||
"order_date",
|
|
||||||
"delivery_date",
|
|
||||||
"service_type",
|
|
||||||
"order_total",
|
|
||||||
"payment_method",
|
|
||||||
"total_item_count",
|
|
||||||
"total_savings",
|
|
||||||
"your_savings_total",
|
|
||||||
"coupons_discounts_total",
|
|
||||||
"store_name",
|
|
||||||
"store_number",
|
|
||||||
"store_address1",
|
|
||||||
"store_city",
|
|
||||||
"store_state",
|
|
||||||
"store_zipcode",
|
|
||||||
"refund_order",
|
|
||||||
"ebt_order",
|
|
||||||
]
|
|
||||||
|
|
||||||
ITEM_FIELDS = [
|
|
||||||
"order_id",
|
|
||||||
"order_date",
|
|
||||||
"line_no",
|
|
||||||
"pod_id",
|
|
||||||
"item_name",
|
|
||||||
"upc",
|
|
||||||
"category_id",
|
|
||||||
"category",
|
|
||||||
"qty",
|
|
||||||
"unit",
|
|
||||||
"unit_price",
|
|
||||||
"line_total",
|
|
||||||
"picked_weight",
|
|
||||||
"mvp_savings",
|
|
||||||
"reward_savings",
|
|
||||||
"coupon_savings",
|
|
||||||
"coupon_price",
|
|
||||||
]
|
|
||||||
|
|
||||||
|
|
||||||
def load_config():
|
|
||||||
if load_dotenv is not None:
|
|
||||||
load_dotenv()
|
|
||||||
|
|
||||||
return {
|
|
||||||
"user_id": os.getenv("GIANT_USER_ID", "").strip(),
|
|
||||||
"loyalty": os.getenv("GIANT_LOYALTY_NUMBER", "").strip(),
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def build_session():
|
def build_session():
|
||||||
session = requests.Session()
|
s = requests.Session()
|
||||||
session.cookies.update(browser_cookie3.firefox(domain_name="giantfood.com"))
|
s.cookies.update(browser_cookie3.firefox(domain_name="giantfood.com"))
|
||||||
session.headers.update(
|
s.headers.update({
|
||||||
{
|
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:148.0) Gecko/20100101 Firefox/148.0",
|
||||||
"user-agent": (
|
|
||||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:148.0) "
|
|
||||||
"Gecko/20100101 Firefox/148.0"
|
|
||||||
),
|
|
||||||
"accept": "application/json, text/plain, */*",
|
"accept": "application/json, text/plain, */*",
|
||||||
"accept-language": "en-US,en;q=0.9",
|
"accept-language": "en-US,en;q=0.9",
|
||||||
"referer": ACCOUNT_PAGE,
|
"referer": ACCOUNT_PAGE,
|
||||||
}
|
})
|
||||||
)
|
return s
|
||||||
return session
|
|
||||||
|
|
||||||
|
|
||||||
def safe_get(session, url, **kwargs):
|
def safe_get(session, url, **kwargs):
|
||||||
@@ -86,20 +31,20 @@ def safe_get(session, url, **kwargs):
|
|||||||
|
|
||||||
for attempt in range(3):
|
for attempt in range(3):
|
||||||
try:
|
try:
|
||||||
response = session.get(
|
r = session.get(
|
||||||
url,
|
url,
|
||||||
impersonate="firefox",
|
impersonate="firefox",
|
||||||
timeout=30,
|
timeout=30,
|
||||||
**kwargs,
|
**kwargs,
|
||||||
)
|
)
|
||||||
last_response = response
|
last_response = r
|
||||||
|
|
||||||
if response.status_code == 200:
|
if r.status_code == 200:
|
||||||
return response
|
return r
|
||||||
|
|
||||||
click.echo(f"retry {attempt + 1}/3 status={response.status_code}")
|
print(f"retry {attempt + 1}/3 status={r.status_code}")
|
||||||
except Exception as exc: # pragma: no cover - network error path
|
except Exception as e:
|
||||||
click.echo(f"retry {attempt + 1}/3 error={exc}")
|
print(f"retry {attempt + 1}/3 error={e}")
|
||||||
|
|
||||||
time.sleep(3)
|
time.sleep(3)
|
||||||
|
|
||||||
@@ -109,63 +54,68 @@ def safe_get(session, url, **kwargs):
|
|||||||
raise RuntimeError(f"failed to fetch {url}")
|
raise RuntimeError(f"failed to fetch {url}")
|
||||||
|
|
||||||
|
|
||||||
def get_history(session, user_id, loyalty):
|
def get_history(session):
|
||||||
response = safe_get(
|
url = f"{BASE}/api/v6.0/user/{USER_ID}/order/history"
|
||||||
|
r = safe_get(
|
||||||
session,
|
session,
|
||||||
f"{BASE}/api/v6.0/user/{user_id}/order/history",
|
url,
|
||||||
params={"filter": "instore", "loyaltyNumber": loyalty},
|
params={
|
||||||
|
"filter": "instore",
|
||||||
|
"loyaltyNumber": LOYALTY,
|
||||||
|
},
|
||||||
)
|
)
|
||||||
return response.json()
|
return r.json()
|
||||||
|
|
||||||
|
|
||||||
def get_order_detail(session, user_id, order_id):
|
def get_order_detail(session, order_id):
|
||||||
response = safe_get(
|
url = f"{BASE}/api/v6.0/user/{USER_ID}/order/history/detail/{order_id}"
|
||||||
|
r = safe_get(
|
||||||
session,
|
session,
|
||||||
f"{BASE}/api/v6.0/user/{user_id}/order/history/detail/{order_id}",
|
url,
|
||||||
params={"isInStore": "true"},
|
params={"isInStore": "true"},
|
||||||
)
|
)
|
||||||
return response.json()
|
return r.json()
|
||||||
|
|
||||||
|
|
||||||
def flatten_orders(history, details):
|
def flatten_orders(history, details):
|
||||||
orders = []
|
orders = []
|
||||||
items = []
|
items = []
|
||||||
history_lookup = {record["orderId"]: record for record in history.get("records", [])}
|
|
||||||
|
|
||||||
for detail in details:
|
history_lookup = {
|
||||||
order_id = str(detail["orderId"])
|
r["orderId"]: r
|
||||||
history_row = history_lookup.get(detail["orderId"], {})
|
for r in history.get("records", [])
|
||||||
pickup = detail.get("pup", {})
|
|
||||||
|
|
||||||
orders.append(
|
|
||||||
{
|
|
||||||
"order_id": order_id,
|
|
||||||
"order_date": detail.get("orderDate"),
|
|
||||||
"delivery_date": detail.get("deliveryDate"),
|
|
||||||
"service_type": history_row.get("serviceType"),
|
|
||||||
"order_total": detail.get("orderTotal"),
|
|
||||||
"payment_method": detail.get("paymentMethod"),
|
|
||||||
"total_item_count": detail.get("totalItemCount"),
|
|
||||||
"total_savings": detail.get("totalSavings"),
|
|
||||||
"your_savings_total": detail.get("yourSavingsTotal"),
|
|
||||||
"coupons_discounts_total": detail.get("couponsDiscountsTotal"),
|
|
||||||
"store_name": pickup.get("storeName"),
|
|
||||||
"store_number": pickup.get("aholdStoreNumber"),
|
|
||||||
"store_address1": pickup.get("storeAddress1"),
|
|
||||||
"store_city": pickup.get("storeCity"),
|
|
||||||
"store_state": pickup.get("storeState"),
|
|
||||||
"store_zipcode": pickup.get("storeZipcode"),
|
|
||||||
"refund_order": detail.get("refundOrder"),
|
|
||||||
"ebt_order": detail.get("ebtOrder"),
|
|
||||||
}
|
}
|
||||||
)
|
|
||||||
|
|
||||||
for line_no, item in enumerate(detail.get("items", []), start=1):
|
for d in details:
|
||||||
items.append(
|
hist = history_lookup.get(d["orderId"], {})
|
||||||
{
|
pup = d.get("pup", {})
|
||||||
"order_id": order_id,
|
|
||||||
"order_date": detail.get("orderDate"),
|
orders.append({
|
||||||
"line_no": str(line_no),
|
"order_id": d["orderId"],
|
||||||
|
"order_date": d.get("orderDate"),
|
||||||
|
"delivery_date": d.get("deliveryDate"),
|
||||||
|
"service_type": hist.get("serviceType"),
|
||||||
|
"order_total": d.get("orderTotal"),
|
||||||
|
"payment_method": d.get("paymentMethod"),
|
||||||
|
"total_item_count": d.get("totalItemCount"),
|
||||||
|
"total_savings": d.get("totalSavings"),
|
||||||
|
"your_savings_total": d.get("yourSavingsTotal"),
|
||||||
|
"coupons_discounts_total": d.get("couponsDiscountsTotal"),
|
||||||
|
"store_name": pup.get("storeName"),
|
||||||
|
"store_number": pup.get("aholdStoreNumber"),
|
||||||
|
"store_address1": pup.get("storeAddress1"),
|
||||||
|
"store_city": pup.get("storeCity"),
|
||||||
|
"store_state": pup.get("storeState"),
|
||||||
|
"store_zipcode": pup.get("storeZipcode"),
|
||||||
|
"refund_order": d.get("refundOrder"),
|
||||||
|
"ebt_order": d.get("ebtOrder"),
|
||||||
|
})
|
||||||
|
|
||||||
|
for i, item in enumerate(d.get("items", []), start=1):
|
||||||
|
items.append({
|
||||||
|
"order_id": d["orderId"],
|
||||||
|
"order_date": d.get("orderDate"),
|
||||||
|
"line_no": i,
|
||||||
"pod_id": item.get("podId"),
|
"pod_id": item.get("podId"),
|
||||||
"item_name": item.get("itemName"),
|
"item_name": item.get("itemName"),
|
||||||
"upc": item.get("primUpcCd"),
|
"upc": item.get("primUpcCd"),
|
||||||
@@ -180,162 +130,51 @@ def flatten_orders(history, details):
|
|||||||
"reward_savings": item.get("rewardSavings"),
|
"reward_savings": item.get("rewardSavings"),
|
||||||
"coupon_savings": item.get("couponSavings"),
|
"coupon_savings": item.get("couponSavings"),
|
||||||
"coupon_price": item.get("couponPrice"),
|
"coupon_price": item.get("couponPrice"),
|
||||||
}
|
})
|
||||||
)
|
|
||||||
|
|
||||||
return orders, items
|
return pd.DataFrame(orders), pd.DataFrame(items)
|
||||||
|
|
||||||
|
|
||||||
def normalize_row(row, fieldnames):
|
def main():
|
||||||
return {field: stringify(row.get(field)) for field in fieldnames}
|
outdir = Path("giant_output")
|
||||||
|
|
||||||
|
|
||||||
def stringify(value):
|
|
||||||
if value is None:
|
|
||||||
return ""
|
|
||||||
return str(value)
|
|
||||||
|
|
||||||
|
|
||||||
def read_csv_rows(path):
|
|
||||||
if not path.exists():
|
|
||||||
return [], []
|
|
||||||
|
|
||||||
with path.open(newline="", encoding="utf-8") as handle:
|
|
||||||
reader = csv.DictReader(handle)
|
|
||||||
fieldnames = reader.fieldnames or []
|
|
||||||
return fieldnames, list(reader)
|
|
||||||
|
|
||||||
|
|
||||||
def read_existing_order_ids(path):
|
|
||||||
_, rows = read_csv_rows(path)
|
|
||||||
return {row["order_id"] for row in rows if row.get("order_id")}
|
|
||||||
|
|
||||||
|
|
||||||
def merge_rows(existing_rows, new_rows, subset):
|
|
||||||
merged = []
|
|
||||||
row_index = {}
|
|
||||||
|
|
||||||
for row in existing_rows + new_rows:
|
|
||||||
key = tuple(stringify(row.get(field)) for field in subset)
|
|
||||||
normalized = dict(row)
|
|
||||||
if key in row_index:
|
|
||||||
merged[row_index[key]] = normalized
|
|
||||||
else:
|
|
||||||
row_index[key] = len(merged)
|
|
||||||
merged.append(normalized)
|
|
||||||
|
|
||||||
return merged
|
|
||||||
|
|
||||||
|
|
||||||
def append_dedup(path, new_rows, subset, fieldnames):
|
|
||||||
existing_fieldnames, existing_rows = read_csv_rows(path)
|
|
||||||
all_fieldnames = list(dict.fromkeys(existing_fieldnames + fieldnames))
|
|
||||||
|
|
||||||
merged = merge_rows(
|
|
||||||
[normalize_row(row, all_fieldnames) for row in existing_rows],
|
|
||||||
[normalize_row(row, all_fieldnames) for row in new_rows],
|
|
||||||
subset=subset,
|
|
||||||
)
|
|
||||||
|
|
||||||
with path.open("w", newline="", encoding="utf-8") as handle:
|
|
||||||
writer = csv.DictWriter(handle, fieldnames=all_fieldnames)
|
|
||||||
writer.writeheader()
|
|
||||||
writer.writerows(merged)
|
|
||||||
|
|
||||||
return merged
|
|
||||||
|
|
||||||
|
|
||||||
def write_json(path, payload):
|
|
||||||
path.write_text(json.dumps(payload, indent=2), encoding="utf-8")
|
|
||||||
|
|
||||||
|
|
||||||
@click.command()
|
|
||||||
@click.option("--user-id", default=None, help="Giant user id.")
|
|
||||||
@click.option("--loyalty", default=None, help="Giant loyalty number.")
|
|
||||||
@click.option(
|
|
||||||
"--outdir",
|
|
||||||
default="giant_output",
|
|
||||||
show_default=True,
|
|
||||||
help="Directory for raw json and csv outputs.",
|
|
||||||
)
|
|
||||||
@click.option(
|
|
||||||
"--sleep-seconds",
|
|
||||||
default=1.5,
|
|
||||||
show_default=True,
|
|
||||||
type=float,
|
|
||||||
help="Delay between order detail requests.",
|
|
||||||
)
|
|
||||||
def main(user_id, loyalty, outdir, sleep_seconds):
|
|
||||||
config = load_config()
|
|
||||||
user_id = user_id or config["user_id"] or click.prompt("Giant user id", type=str)
|
|
||||||
loyalty = loyalty or config["loyalty"] or click.prompt(
|
|
||||||
"Giant loyalty number", type=str
|
|
||||||
)
|
|
||||||
|
|
||||||
outdir = Path(outdir)
|
|
||||||
rawdir = outdir / "raw"
|
rawdir = outdir / "raw"
|
||||||
rawdir.mkdir(parents=True, exist_ok=True)
|
rawdir.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
orders_csv = outdir / "orders.csv"
|
|
||||||
items_csv = outdir / "items.csv"
|
|
||||||
|
|
||||||
click.echo("Using cookies from your current Firefox profile.")
|
|
||||||
click.echo(f"Open Giant here, confirm you're logged in, then return: {ACCOUNT_PAGE}")
|
|
||||||
click.pause(info="Press any key once Giant is open and logged in")
|
|
||||||
|
|
||||||
session = build_session()
|
session = build_session()
|
||||||
|
|
||||||
click.echo("Fetching order history...")
|
print("fetching order history...")
|
||||||
history = get_history(session, user_id, loyalty)
|
history = get_history(session)
|
||||||
write_json(rawdir / "history.json", history)
|
|
||||||
|
|
||||||
records = history.get("records", [])
|
(rawdir / "history.json").write_text(
|
||||||
click.echo(f"History returned {len(records)} visits.")
|
json.dumps(history, indent=2),
|
||||||
click.echo(
|
encoding="utf-8",
|
||||||
"Note: Giant appears to expose only the most recent 50 visits, "
|
|
||||||
"so run this periodically if you want full continuity."
|
|
||||||
)
|
)
|
||||||
|
|
||||||
history_order_ids = [str(record["orderId"]) for record in records]
|
order_ids = [r["orderId"] for r in history.get("records", [])]
|
||||||
existing_order_ids = read_existing_order_ids(orders_csv)
|
print(f"{len(order_ids)} orders found")
|
||||||
new_order_ids = [order_id for order_id in history_order_ids if order_id not in existing_order_ids]
|
|
||||||
|
|
||||||
click.echo(f"Existing orders in csv: {len(existing_order_ids)}")
|
|
||||||
click.echo(f"New orders to fetch: {len(new_order_ids)}")
|
|
||||||
|
|
||||||
if not new_order_ids:
|
|
||||||
click.echo("No new orders found. Done.")
|
|
||||||
return
|
|
||||||
|
|
||||||
details = []
|
details = []
|
||||||
for order_id in new_order_ids:
|
for order_id in order_ids:
|
||||||
click.echo(f"Fetching {order_id}")
|
print(f"fetching {order_id}")
|
||||||
detail = get_order_detail(session, user_id, order_id)
|
d = get_order_detail(session, order_id)
|
||||||
details.append(detail)
|
details.append(d)
|
||||||
write_json(rawdir / f"{order_id}.json", detail)
|
|
||||||
time.sleep(sleep_seconds)
|
|
||||||
|
|
||||||
click.echo("Flattening new data...")
|
(rawdir / f"{order_id}.json").write_text(
|
||||||
orders, items = flatten_orders(history, details)
|
json.dumps(d, indent=2),
|
||||||
|
encoding="utf-8",
|
||||||
all_orders = append_dedup(
|
|
||||||
orders_csv,
|
|
||||||
orders,
|
|
||||||
subset=["order_id"],
|
|
||||||
fieldnames=ORDER_FIELDS,
|
|
||||||
)
|
|
||||||
all_items = append_dedup(
|
|
||||||
items_csv,
|
|
||||||
items,
|
|
||||||
subset=["order_id", "line_no", "item_name", "upc", "line_total"],
|
|
||||||
fieldnames=ITEM_FIELDS,
|
|
||||||
)
|
)
|
||||||
|
|
||||||
click.echo("Done.")
|
time.sleep(1.5)
|
||||||
click.echo(f"Orders csv: {orders_csv}")
|
|
||||||
click.echo(f"Items csv: {items_csv}")
|
print("flattening data...")
|
||||||
click.echo(f"Total orders stored: {len(all_orders)}")
|
orders_df, items_df = flatten_orders(history, details)
|
||||||
click.echo(f"Total item rows stored: {len(all_items)}")
|
|
||||||
|
orders_df.to_csv(outdir / "orders.csv", index=False)
|
||||||
|
items_df.to_csv(outdir / "items.csv", index=False)
|
||||||
|
|
||||||
|
print("done")
|
||||||
|
print(f"{len(orders_df)} orders written to {outdir / 'orders.csv'}")
|
||||||
|
print(f"{len(items_df)} items written to {outdir / 'items.csv'}")
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
|
|||||||
@@ -1,17 +1,28 @@
|
|||||||
import unittest
|
import requests
|
||||||
|
import browser_cookie3
|
||||||
|
|
||||||
|
BASE = "https://giantfood.com"
|
||||||
|
ACCOUNT_PAGE = f"{BASE}/account/history/invoice/in-store"
|
||||||
|
|
||||||
try:
|
USER_ID = "369513017"
|
||||||
import browser_cookie3 # noqa: F401
|
LOYALTY = "440155630880"
|
||||||
import requests # noqa: F401
|
|
||||||
except ImportError as exc: # pragma: no cover - dependency-gated smoke test
|
|
||||||
browser_cookie3 = None
|
|
||||||
_IMPORT_ERROR = exc
|
|
||||||
else:
|
|
||||||
_IMPORT_ERROR = None
|
|
||||||
|
|
||||||
|
cj = browser_cookie3.firefox(domain_name="giantfood.com")
|
||||||
|
|
||||||
@unittest.skipIf(browser_cookie3 is None, f"optional smoke test dependency missing: {_IMPORT_ERROR}")
|
s = requests.Session()
|
||||||
class BrowserCookieSmokeTest(unittest.TestCase):
|
s.cookies.update(cj)
|
||||||
def test_dependencies_available(self):
|
s.headers.update({
|
||||||
self.assertIsNotNone(browser_cookie3)
|
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:148.0) Gecko/20100101 Firefox/148.0",
|
||||||
|
"accept": "application/json, text/plain, */*",
|
||||||
|
"accept-language": "en-US,en;q=0.9",
|
||||||
|
"referer": ACCOUNT_PAGE,
|
||||||
|
})
|
||||||
|
|
||||||
|
r = s.get(
|
||||||
|
f"{BASE}/api/v6.0/user/{USER_ID}/order/history",
|
||||||
|
params={"filter": "instore", "loyaltyNumber": LOYALTY},
|
||||||
|
timeout=30,
|
||||||
|
)
|
||||||
|
|
||||||
|
print(r.status_code)
|
||||||
|
print(r.text[:500])
|
||||||
|
|||||||
@@ -1,17 +1,27 @@
|
|||||||
import unittest
|
import browser_cookie3
|
||||||
|
from curl_cffi import requests
|
||||||
|
|
||||||
|
BASE = "https://giantfood.com"
|
||||||
|
ACCOUNT_PAGE = f"{BASE}/account/history/invoice/in-store"
|
||||||
|
|
||||||
try:
|
USER_ID = "369513017"
|
||||||
import browser_cookie3 # noqa: F401
|
LOYALTY = "440155630880"
|
||||||
from curl_cffi import requests # noqa: F401
|
|
||||||
except ImportError as exc: # pragma: no cover - dependency-gated smoke test
|
|
||||||
browser_cookie3 = None
|
|
||||||
_IMPORT_ERROR = exc
|
|
||||||
else:
|
|
||||||
_IMPORT_ERROR = None
|
|
||||||
|
|
||||||
|
s = requests.Session()
|
||||||
|
s.cookies.update(browser_cookie3.firefox(domain_name="giantfood.com"))
|
||||||
|
s.headers.update({
|
||||||
|
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:148.0) Gecko/20100101 Firefox/148.0",
|
||||||
|
"accept": "application/json, text/plain, */*",
|
||||||
|
"accept-language": "en-US,en;q=0.9",
|
||||||
|
"referer": ACCOUNT_PAGE,
|
||||||
|
})
|
||||||
|
|
||||||
@unittest.skipIf(browser_cookie3 is None, f"optional smoke test dependency missing: {_IMPORT_ERROR}")
|
r = s.get(
|
||||||
class CurlCffiSmokeTest(unittest.TestCase):
|
f"{BASE}/api/v6.0/user/{USER_ID}/order/history",
|
||||||
def test_dependencies_available(self):
|
params={"filter": "instore", "loyaltyNumber": LOYALTY},
|
||||||
self.assertIsNotNone(browser_cookie3)
|
impersonate="firefox",
|
||||||
|
timeout=30,
|
||||||
|
)
|
||||||
|
|
||||||
|
print(r.status_code)
|
||||||
|
print(r.text[:500])
|
||||||
|
|||||||
@@ -1,84 +0,0 @@
|
|||||||
import unittest
|
|
||||||
|
|
||||||
import build_canonical_layer
|
|
||||||
|
|
||||||
|
|
||||||
class CanonicalLayerTests(unittest.TestCase):
|
|
||||||
def test_build_canonical_layer_auto_links_exact_upc_and_name_size(self):
|
|
||||||
observed_rows = [
|
|
||||||
{
|
|
||||||
"observed_product_id": "gobs_1",
|
|
||||||
"representative_upc": "111",
|
|
||||||
"representative_name_norm": "GALA APPLE",
|
|
||||||
"representative_brand": "SB",
|
|
||||||
"representative_variant": "",
|
|
||||||
"representative_size_value": "5",
|
|
||||||
"representative_size_unit": "lb",
|
|
||||||
"representative_pack_qty": "",
|
|
||||||
"representative_measure_type": "weight",
|
|
||||||
"is_fee": "false",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"observed_product_id": "gobs_2",
|
|
||||||
"representative_upc": "111",
|
|
||||||
"representative_name_norm": "LARGE WHITE EGGS",
|
|
||||||
"representative_brand": "SB",
|
|
||||||
"representative_variant": "",
|
|
||||||
"representative_size_value": "",
|
|
||||||
"representative_size_unit": "",
|
|
||||||
"representative_pack_qty": "18",
|
|
||||||
"representative_measure_type": "count",
|
|
||||||
"is_fee": "false",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"observed_product_id": "gobs_3",
|
|
||||||
"representative_upc": "",
|
|
||||||
"representative_name_norm": "ROTINI",
|
|
||||||
"representative_brand": "",
|
|
||||||
"representative_variant": "",
|
|
||||||
"representative_size_value": "16",
|
|
||||||
"representative_size_unit": "oz",
|
|
||||||
"representative_pack_qty": "",
|
|
||||||
"representative_measure_type": "weight",
|
|
||||||
"is_fee": "false",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"observed_product_id": "gobs_4",
|
|
||||||
"representative_upc": "",
|
|
||||||
"representative_name_norm": "ROTINI",
|
|
||||||
"representative_brand": "SB",
|
|
||||||
"representative_variant": "",
|
|
||||||
"representative_size_value": "16",
|
|
||||||
"representative_size_unit": "oz",
|
|
||||||
"representative_pack_qty": "",
|
|
||||||
"representative_measure_type": "weight",
|
|
||||||
"is_fee": "false",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"observed_product_id": "gobs_5",
|
|
||||||
"representative_upc": "",
|
|
||||||
"representative_name_norm": "GL BAG CHARGE",
|
|
||||||
"representative_brand": "",
|
|
||||||
"representative_variant": "",
|
|
||||||
"representative_size_value": "",
|
|
||||||
"representative_size_unit": "",
|
|
||||||
"representative_pack_qty": "",
|
|
||||||
"representative_measure_type": "each",
|
|
||||||
"is_fee": "true",
|
|
||||||
},
|
|
||||||
]
|
|
||||||
|
|
||||||
canonicals, links = build_canonical_layer.build_canonical_layer(observed_rows)
|
|
||||||
|
|
||||||
self.assertEqual(2, len(canonicals))
|
|
||||||
self.assertEqual(4, len(links))
|
|
||||||
methods = {row["observed_product_id"]: row["link_method"] for row in links}
|
|
||||||
self.assertEqual("exact_upc", methods["gobs_1"])
|
|
||||||
self.assertEqual("exact_upc", methods["gobs_2"])
|
|
||||||
self.assertEqual("exact_name_size", methods["gobs_3"])
|
|
||||||
self.assertEqual("exact_name_size", methods["gobs_4"])
|
|
||||||
self.assertNotIn("gobs_5", methods)
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
unittest.main()
|
|
||||||
@@ -1,190 +0,0 @@
|
|||||||
import csv
|
|
||||||
import json
|
|
||||||
import tempfile
|
|
||||||
import unittest
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import enrich_giant
|
|
||||||
|
|
||||||
|
|
||||||
class EnrichGiantTests(unittest.TestCase):
|
|
||||||
def test_parse_size_and_pack_handles_pack_and_weight_tokens(self):
|
|
||||||
size_value, size_unit, pack_qty = enrich_giant.parse_size_and_pack(
|
|
||||||
"COKE CHERRY 6PK 7.5Z"
|
|
||||||
)
|
|
||||||
|
|
||||||
self.assertEqual("7.5", size_value)
|
|
||||||
self.assertEqual("oz", size_unit)
|
|
||||||
self.assertEqual("6", pack_qty)
|
|
||||||
|
|
||||||
def test_parse_item_marks_store_brand_fee_and_weight_prices(self):
|
|
||||||
row = enrich_giant.parse_item(
|
|
||||||
order_id="abc123",
|
|
||||||
order_date="2026-03-01",
|
|
||||||
raw_path=Path("raw/abc123.json"),
|
|
||||||
line_no=1,
|
|
||||||
item={
|
|
||||||
"podId": 1,
|
|
||||||
"shipQy": 1,
|
|
||||||
"totalPickedWeight": 2,
|
|
||||||
"unitPrice": 3.98,
|
|
||||||
"itemName": "+SB GALA APPLE 5 LB",
|
|
||||||
"lbEachCd": "LB",
|
|
||||||
"groceryAmount": 3.98,
|
|
||||||
"primUpcCd": "111",
|
|
||||||
"mvpSavings": 0,
|
|
||||||
"rewardSavings": 0,
|
|
||||||
"couponSavings": 0,
|
|
||||||
"couponPrice": 0,
|
|
||||||
"categoryId": "1",
|
|
||||||
"categoryDesc": "Grocery",
|
|
||||||
"image": {"large": "https://example.test/apple.jpg"},
|
|
||||||
},
|
|
||||||
)
|
|
||||||
|
|
||||||
self.assertEqual("SB", row["brand_guess"])
|
|
||||||
self.assertEqual("GALA APPLE", row["item_name_norm"])
|
|
||||||
self.assertEqual("5", row["size_value"])
|
|
||||||
self.assertEqual("lb", row["size_unit"])
|
|
||||||
self.assertEqual("weight", row["measure_type"])
|
|
||||||
self.assertEqual("true", row["is_store_brand"])
|
|
||||||
self.assertEqual("1.99", row["price_per_lb"])
|
|
||||||
self.assertEqual("0.1244", row["price_per_oz"])
|
|
||||||
self.assertEqual("https://example.test/apple.jpg", row["image_url"])
|
|
||||||
|
|
||||||
fee_row = enrich_giant.parse_item(
|
|
||||||
order_id="abc123",
|
|
||||||
order_date="2026-03-01",
|
|
||||||
raw_path=Path("raw/abc123.json"),
|
|
||||||
line_no=2,
|
|
||||||
item={
|
|
||||||
"podId": 2,
|
|
||||||
"shipQy": 1,
|
|
||||||
"totalPickedWeight": 0,
|
|
||||||
"unitPrice": 0.05,
|
|
||||||
"itemName": "GL BAG CHARGE",
|
|
||||||
"lbEachCd": "EA",
|
|
||||||
"groceryAmount": 0.05,
|
|
||||||
"primUpcCd": "",
|
|
||||||
"mvpSavings": 0,
|
|
||||||
"rewardSavings": 0,
|
|
||||||
"couponSavings": 0,
|
|
||||||
"couponPrice": 0,
|
|
||||||
"categoryId": "1",
|
|
||||||
"categoryDesc": "Grocery",
|
|
||||||
},
|
|
||||||
)
|
|
||||||
|
|
||||||
self.assertEqual("true", fee_row["is_fee"])
|
|
||||||
self.assertEqual("GL BAG CHARGE", fee_row["item_name_norm"])
|
|
||||||
|
|
||||||
def test_parse_item_derives_packaged_weight_prices_from_size_tokens(self):
|
|
||||||
row = enrich_giant.parse_item(
|
|
||||||
order_id="abc123",
|
|
||||||
order_date="2026-03-01",
|
|
||||||
raw_path=Path("raw/abc123.json"),
|
|
||||||
line_no=1,
|
|
||||||
item={
|
|
||||||
"podId": 1,
|
|
||||||
"shipQy": 2,
|
|
||||||
"totalPickedWeight": 0,
|
|
||||||
"unitPrice": 3.0,
|
|
||||||
"itemName": "PEPSI 6PK 7.5Z",
|
|
||||||
"lbEachCd": "EA",
|
|
||||||
"groceryAmount": 6.0,
|
|
||||||
"primUpcCd": "111",
|
|
||||||
"mvpSavings": 0,
|
|
||||||
"rewardSavings": 0,
|
|
||||||
"couponSavings": 0,
|
|
||||||
"couponPrice": 0,
|
|
||||||
"categoryId": "1",
|
|
||||||
"categoryDesc": "Grocery",
|
|
||||||
},
|
|
||||||
)
|
|
||||||
|
|
||||||
self.assertEqual("weight", row["measure_type"])
|
|
||||||
self.assertEqual("6", row["pack_qty"])
|
|
||||||
self.assertEqual("7.5", row["size_value"])
|
|
||||||
self.assertEqual("0.0667", row["price_per_oz"])
|
|
||||||
self.assertEqual("1.0667", row["price_per_lb"])
|
|
||||||
|
|
||||||
def test_build_items_enriched_reads_raw_order_files_and_writes_csv(self):
|
|
||||||
with tempfile.TemporaryDirectory() as tmpdir:
|
|
||||||
raw_dir = Path(tmpdir) / "raw"
|
|
||||||
raw_dir.mkdir()
|
|
||||||
(raw_dir / "history.json").write_text("{}", encoding="utf-8")
|
|
||||||
(raw_dir / "order-2.json").write_text(
|
|
||||||
json.dumps(
|
|
||||||
{
|
|
||||||
"orderId": "order-2",
|
|
||||||
"orderDate": "2026-03-02",
|
|
||||||
"items": [
|
|
||||||
{
|
|
||||||
"podId": 20,
|
|
||||||
"shipQy": 1,
|
|
||||||
"totalPickedWeight": 0,
|
|
||||||
"unitPrice": 2.99,
|
|
||||||
"itemName": "SB ROTINI 16Z",
|
|
||||||
"lbEachCd": "EA",
|
|
||||||
"groceryAmount": 2.99,
|
|
||||||
"primUpcCd": "222",
|
|
||||||
"mvpSavings": 0,
|
|
||||||
"rewardSavings": 0,
|
|
||||||
"couponSavings": 0,
|
|
||||||
"couponPrice": 0,
|
|
||||||
"categoryId": "1",
|
|
||||||
"categoryDesc": "Grocery",
|
|
||||||
"image": {"small": "https://example.test/rotini.jpg"},
|
|
||||||
}
|
|
||||||
],
|
|
||||||
}
|
|
||||||
),
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
(raw_dir / "order-1.json").write_text(
|
|
||||||
json.dumps(
|
|
||||||
{
|
|
||||||
"orderId": "order-1",
|
|
||||||
"orderDate": "2026-03-01",
|
|
||||||
"items": [
|
|
||||||
{
|
|
||||||
"podId": 10,
|
|
||||||
"shipQy": 2,
|
|
||||||
"totalPickedWeight": 0,
|
|
||||||
"unitPrice": 1.5,
|
|
||||||
"itemName": "PEPSI 6PK 7.5Z",
|
|
||||||
"lbEachCd": "EA",
|
|
||||||
"groceryAmount": 3.0,
|
|
||||||
"primUpcCd": "111",
|
|
||||||
"mvpSavings": 0,
|
|
||||||
"rewardSavings": 0,
|
|
||||||
"couponSavings": 0,
|
|
||||||
"couponPrice": 0,
|
|
||||||
"categoryId": "1",
|
|
||||||
"categoryDesc": "Grocery",
|
|
||||||
}
|
|
||||||
],
|
|
||||||
}
|
|
||||||
),
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
|
|
||||||
rows = enrich_giant.build_items_enriched(raw_dir)
|
|
||||||
output_csv = Path(tmpdir) / "items_enriched.csv"
|
|
||||||
enrich_giant.write_csv(output_csv, rows)
|
|
||||||
|
|
||||||
self.assertEqual(["order-1", "order-2"], [row["order_id"] for row in rows])
|
|
||||||
self.assertEqual("PEPSI", rows[0]["item_name_norm"])
|
|
||||||
self.assertEqual("6", rows[0]["pack_qty"])
|
|
||||||
self.assertEqual("7.5", rows[0]["size_value"])
|
|
||||||
self.assertEqual("true", rows[1]["is_store_brand"])
|
|
||||||
|
|
||||||
with output_csv.open(newline="", encoding="utf-8") as handle:
|
|
||||||
written_rows = list(csv.DictReader(handle))
|
|
||||||
|
|
||||||
self.assertEqual(2, len(written_rows))
|
|
||||||
self.assertEqual(enrich_giant.OUTPUT_FIELDS, list(written_rows[0].keys()))
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
unittest.main()
|
|
||||||
@@ -1,17 +1,66 @@
|
|||||||
import unittest
|
import requests
|
||||||
|
from playwright.sync_api import sync_playwright
|
||||||
|
|
||||||
|
BASE = "https://giantfood.com"
|
||||||
|
ACCOUNT_PAGE = f"{BASE}/account/history/invoice/in-store"
|
||||||
|
|
||||||
|
USER_ID = "369513017"
|
||||||
|
LOYALTY = "440155630880"
|
||||||
|
|
||||||
|
|
||||||
try:
|
def get_session():
|
||||||
from playwright.sync_api import sync_playwright # noqa: F401
|
with sync_playwright() as p:
|
||||||
import requests # noqa: F401
|
browser = p.firefox.launch(headless=False)
|
||||||
except ImportError as exc: # pragma: no cover - dependency-gated smoke test
|
page = browser.new_page()
|
||||||
sync_playwright = None
|
|
||||||
_IMPORT_ERROR = exc
|
page.goto(ACCOUNT_PAGE)
|
||||||
else:
|
|
||||||
_IMPORT_ERROR = None
|
print("log in manually in the browser, then press ENTER here")
|
||||||
|
input()
|
||||||
|
|
||||||
|
cookies = page.context.cookies()
|
||||||
|
ua = page.evaluate("() => navigator.userAgent")
|
||||||
|
|
||||||
|
browser.close()
|
||||||
|
|
||||||
|
s = requests.Session()
|
||||||
|
|
||||||
|
s.headers.update({
|
||||||
|
"user-agent": ua,
|
||||||
|
"accept": "application/json, text/plain, */*",
|
||||||
|
"referer": ACCOUNT_PAGE,
|
||||||
|
})
|
||||||
|
|
||||||
|
for c in cookies:
|
||||||
|
domain = c.get("domain", "").lstrip(".") or "giantfood.com"
|
||||||
|
s.cookies.set(c["name"], c["value"], domain=domain)
|
||||||
|
|
||||||
|
return s
|
||||||
|
|
||||||
|
|
||||||
@unittest.skipIf(sync_playwright is None, f"optional smoke test dependency missing: {_IMPORT_ERROR}")
|
def test_history(session):
|
||||||
class GiantLoginSmokeTest(unittest.TestCase):
|
url = f"{BASE}/api/v6.0/user/{USER_ID}/order/history"
|
||||||
def test_dependencies_available(self):
|
|
||||||
self.assertIsNotNone(sync_playwright)
|
r = session.get(
|
||||||
|
url,
|
||||||
|
params={
|
||||||
|
"filter": "instore",
|
||||||
|
"loyaltyNumber": LOYALTY,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
print("status:", r.status_code)
|
||||||
|
print()
|
||||||
|
|
||||||
|
data = r.json()
|
||||||
|
|
||||||
|
print("orders found:", len(data.get("records", [])))
|
||||||
|
print()
|
||||||
|
|
||||||
|
for rec in data.get("records", [])[:5]:
|
||||||
|
print(rec["orderId"], rec["orderDate"], rec["orderTotal"])
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
session = get_session()
|
||||||
|
test_history(session)
|
||||||
|
|||||||
@@ -1,60 +0,0 @@
|
|||||||
import unittest
|
|
||||||
|
|
||||||
import build_observed_products
|
|
||||||
|
|
||||||
|
|
||||||
class ObservedProductTests(unittest.TestCase):
|
|
||||||
def test_build_observed_products_aggregates_rows_with_same_key(self):
|
|
||||||
rows = [
|
|
||||||
{
|
|
||||||
"retailer": "giant",
|
|
||||||
"order_id": "1",
|
|
||||||
"line_no": "1",
|
|
||||||
"order_date": "2026-01-01",
|
|
||||||
"item_name": "SB GALA APPLE 5LB",
|
|
||||||
"item_name_norm": "GALA APPLE",
|
|
||||||
"upc": "111",
|
|
||||||
"brand_guess": "SB",
|
|
||||||
"variant": "",
|
|
||||||
"size_value": "5",
|
|
||||||
"size_unit": "lb",
|
|
||||||
"pack_qty": "",
|
|
||||||
"measure_type": "weight",
|
|
||||||
"image_url": "https://example.test/a.jpg",
|
|
||||||
"is_store_brand": "true",
|
|
||||||
"is_fee": "false",
|
|
||||||
"line_total": "7.99",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"retailer": "giant",
|
|
||||||
"order_id": "2",
|
|
||||||
"line_no": "1",
|
|
||||||
"order_date": "2026-01-10",
|
|
||||||
"item_name": "SB GALA APPLE 5 LB",
|
|
||||||
"item_name_norm": "GALA APPLE",
|
|
||||||
"upc": "111",
|
|
||||||
"brand_guess": "SB",
|
|
||||||
"variant": "",
|
|
||||||
"size_value": "5",
|
|
||||||
"size_unit": "lb",
|
|
||||||
"pack_qty": "",
|
|
||||||
"measure_type": "weight",
|
|
||||||
"image_url": "",
|
|
||||||
"is_store_brand": "true",
|
|
||||||
"is_fee": "false",
|
|
||||||
"line_total": "8.49",
|
|
||||||
},
|
|
||||||
]
|
|
||||||
|
|
||||||
observed = build_observed_products.build_observed_products(rows)
|
|
||||||
|
|
||||||
self.assertEqual(1, len(observed))
|
|
||||||
self.assertEqual("2", observed[0]["times_seen"])
|
|
||||||
self.assertEqual("2026-01-01", observed[0]["first_seen_date"])
|
|
||||||
self.assertEqual("2026-01-10", observed[0]["last_seen_date"])
|
|
||||||
self.assertEqual("111", observed[0]["representative_upc"])
|
|
||||||
self.assertIn("SB GALA APPLE 5LB", observed[0]["raw_name_examples"])
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
unittest.main()
|
|
||||||
@@ -1,124 +0,0 @@
|
|||||||
import tempfile
|
|
||||||
import unittest
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import build_observed_products
|
|
||||||
import build_review_queue
|
|
||||||
from layer_helpers import write_csv_rows
|
|
||||||
|
|
||||||
|
|
||||||
class ReviewQueueTests(unittest.TestCase):
|
|
||||||
def test_build_review_queue_preserves_existing_status(self):
|
|
||||||
observed_rows = [
|
|
||||||
{
|
|
||||||
"observed_product_id": "gobs_1",
|
|
||||||
"retailer": "giant",
|
|
||||||
"representative_upc": "111",
|
|
||||||
"representative_image_url": "",
|
|
||||||
"representative_name_norm": "GALA APPLE",
|
|
||||||
"times_seen": "2",
|
|
||||||
"distinct_item_names_count": "2",
|
|
||||||
"distinct_upcs_count": "1",
|
|
||||||
"is_fee": "false",
|
|
||||||
}
|
|
||||||
]
|
|
||||||
item_rows = [
|
|
||||||
{
|
|
||||||
"observed_product_id": "gobs_1",
|
|
||||||
"item_name": "SB GALA APPLE 5LB",
|
|
||||||
"item_name_norm": "GALA APPLE",
|
|
||||||
"line_total": "7.99",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"observed_product_id": "gobs_1",
|
|
||||||
"item_name": "SB GALA APPLE 5 LB",
|
|
||||||
"item_name_norm": "GALA APPLE",
|
|
||||||
"line_total": "8.49",
|
|
||||||
},
|
|
||||||
]
|
|
||||||
existing = {
|
|
||||||
build_review_queue.stable_id("rvw", "gobs_1|missing_image"): {
|
|
||||||
"status": "approved",
|
|
||||||
"resolution_notes": "looked fine",
|
|
||||||
"created_at": "2026-03-15",
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
queue = build_review_queue.build_review_queue(
|
|
||||||
observed_rows, item_rows, existing, "2026-03-16"
|
|
||||||
)
|
|
||||||
|
|
||||||
self.assertEqual(2, len(queue))
|
|
||||||
missing_image = [row for row in queue if row["reason_code"] == "missing_image"][0]
|
|
||||||
self.assertEqual("approved", missing_image["status"])
|
|
||||||
self.assertEqual("looked fine", missing_image["resolution_notes"])
|
|
||||||
|
|
||||||
def test_review_queue_main_writes_output(self):
|
|
||||||
with tempfile.TemporaryDirectory() as tmpdir:
|
|
||||||
observed_path = Path(tmpdir) / "products_observed.csv"
|
|
||||||
items_path = Path(tmpdir) / "items_enriched.csv"
|
|
||||||
output_path = Path(tmpdir) / "review_queue.csv"
|
|
||||||
|
|
||||||
observed_rows = [
|
|
||||||
{
|
|
||||||
"observed_product_id": "gobs_1",
|
|
||||||
"retailer": "giant",
|
|
||||||
"observed_key": "giant|upc=111|name=GALA APPLE",
|
|
||||||
"representative_upc": "111",
|
|
||||||
"representative_item_name": "SB GALA APPLE 5LB",
|
|
||||||
"representative_name_norm": "GALA APPLE",
|
|
||||||
"representative_brand": "SB",
|
|
||||||
"representative_variant": "",
|
|
||||||
"representative_size_value": "5",
|
|
||||||
"representative_size_unit": "lb",
|
|
||||||
"representative_pack_qty": "",
|
|
||||||
"representative_measure_type": "weight",
|
|
||||||
"representative_image_url": "",
|
|
||||||
"is_store_brand": "true",
|
|
||||||
"is_fee": "false",
|
|
||||||
"first_seen_date": "2026-01-01",
|
|
||||||
"last_seen_date": "2026-01-10",
|
|
||||||
"times_seen": "2",
|
|
||||||
"example_order_id": "1",
|
|
||||||
"example_item_name": "SB GALA APPLE 5LB",
|
|
||||||
"raw_name_examples": "SB GALA APPLE 5LB | SB GALA APPLE 5 LB",
|
|
||||||
"normalized_name_examples": "GALA APPLE",
|
|
||||||
"example_prices": "7.99 | 8.49",
|
|
||||||
"distinct_item_names_count": "2",
|
|
||||||
"distinct_upcs_count": "1",
|
|
||||||
}
|
|
||||||
]
|
|
||||||
item_rows = [
|
|
||||||
{
|
|
||||||
"retailer": "giant",
|
|
||||||
"order_id": "1",
|
|
||||||
"line_no": "1",
|
|
||||||
"item_name": "SB GALA APPLE 5LB",
|
|
||||||
"item_name_norm": "GALA APPLE",
|
|
||||||
"upc": "111",
|
|
||||||
"size_value": "5",
|
|
||||||
"size_unit": "lb",
|
|
||||||
"pack_qty": "",
|
|
||||||
"measure_type": "weight",
|
|
||||||
"is_store_brand": "true",
|
|
||||||
"is_fee": "false",
|
|
||||||
"line_total": "7.99",
|
|
||||||
}
|
|
||||||
]
|
|
||||||
|
|
||||||
write_csv_rows(
|
|
||||||
observed_path, observed_rows, build_observed_products.OUTPUT_FIELDS
|
|
||||||
)
|
|
||||||
write_csv_rows(items_path, item_rows, list(item_rows[0].keys()))
|
|
||||||
|
|
||||||
build_review_queue.main.callback(
|
|
||||||
observed_csv=str(observed_path),
|
|
||||||
items_enriched_csv=str(items_path),
|
|
||||||
output_csv=str(output_path),
|
|
||||||
)
|
|
||||||
|
|
||||||
self.assertTrue(output_path.exists())
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
unittest.main()
|
|
||||||
@@ -1,117 +0,0 @@
|
|||||||
import csv
|
|
||||||
import tempfile
|
|
||||||
import unittest
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import scraper
|
|
||||||
|
|
||||||
|
|
||||||
class ScraperTests(unittest.TestCase):
|
|
||||||
def test_flatten_orders_extracts_order_and_item_rows(self):
|
|
||||||
history = {
|
|
||||||
"records": [
|
|
||||||
{
|
|
||||||
"orderId": "abc123",
|
|
||||||
"serviceType": "PICKUP",
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
details = [
|
|
||||||
{
|
|
||||||
"orderId": "abc123",
|
|
||||||
"orderDate": "2026-03-01",
|
|
||||||
"deliveryDate": "2026-03-02",
|
|
||||||
"orderTotal": "12.34",
|
|
||||||
"paymentMethod": "VISA",
|
|
||||||
"totalItemCount": 1,
|
|
||||||
"totalSavings": "1.00",
|
|
||||||
"yourSavingsTotal": "1.00",
|
|
||||||
"couponsDiscountsTotal": "0.50",
|
|
||||||
"refundOrder": False,
|
|
||||||
"ebtOrder": False,
|
|
||||||
"pup": {
|
|
||||||
"storeName": "Giant",
|
|
||||||
"aholdStoreNumber": "42",
|
|
||||||
"storeAddress1": "123 Main",
|
|
||||||
"storeCity": "Springfield",
|
|
||||||
"storeState": "VA",
|
|
||||||
"storeZipcode": "22150",
|
|
||||||
},
|
|
||||||
"items": [
|
|
||||||
{
|
|
||||||
"podId": "pod-1",
|
|
||||||
"itemName": "Bananas",
|
|
||||||
"primUpcCd": "111",
|
|
||||||
"categoryId": "produce",
|
|
||||||
"categoryDesc": "Produce",
|
|
||||||
"shipQy": "2",
|
|
||||||
"lbEachCd": "EA",
|
|
||||||
"unitPrice": "0.59",
|
|
||||||
"groceryAmount": "1.18",
|
|
||||||
"totalPickedWeight": "",
|
|
||||||
"mvpSavings": "0.10",
|
|
||||||
"rewardSavings": "0.00",
|
|
||||||
"couponSavings": "0.00",
|
|
||||||
"couponPrice": "",
|
|
||||||
}
|
|
||||||
],
|
|
||||||
}
|
|
||||||
]
|
|
||||||
|
|
||||||
orders, items = scraper.flatten_orders(history, details)
|
|
||||||
|
|
||||||
self.assertEqual(1, len(orders))
|
|
||||||
self.assertEqual("abc123", orders[0]["order_id"])
|
|
||||||
self.assertEqual("PICKUP", orders[0]["service_type"])
|
|
||||||
self.assertEqual(1, len(items))
|
|
||||||
self.assertEqual("1", items[0]["line_no"])
|
|
||||||
self.assertEqual("Bananas", items[0]["item_name"])
|
|
||||||
|
|
||||||
def test_append_dedup_replaces_duplicate_rows_and_preserves_new_values(self):
|
|
||||||
with tempfile.TemporaryDirectory() as tmpdir:
|
|
||||||
path = Path(tmpdir) / "orders.csv"
|
|
||||||
|
|
||||||
scraper.append_dedup(
|
|
||||||
path,
|
|
||||||
[
|
|
||||||
{"order_id": "1", "order_total": "10.00"},
|
|
||||||
{"order_id": "2", "order_total": "20.00"},
|
|
||||||
],
|
|
||||||
subset=["order_id"],
|
|
||||||
fieldnames=["order_id", "order_total"],
|
|
||||||
)
|
|
||||||
|
|
||||||
merged = scraper.append_dedup(
|
|
||||||
path,
|
|
||||||
[
|
|
||||||
{"order_id": "2", "order_total": "21.50"},
|
|
||||||
{"order_id": "3", "order_total": "30.00"},
|
|
||||||
],
|
|
||||||
subset=["order_id"],
|
|
||||||
fieldnames=["order_id", "order_total"],
|
|
||||||
)
|
|
||||||
|
|
||||||
self.assertEqual(
|
|
||||||
[
|
|
||||||
{"order_id": "1", "order_total": "10.00"},
|
|
||||||
{"order_id": "2", "order_total": "21.50"},
|
|
||||||
{"order_id": "3", "order_total": "30.00"},
|
|
||||||
],
|
|
||||||
merged,
|
|
||||||
)
|
|
||||||
|
|
||||||
with path.open(newline="", encoding="utf-8") as handle:
|
|
||||||
rows = list(csv.DictReader(handle))
|
|
||||||
|
|
||||||
self.assertEqual(merged, rows)
|
|
||||||
|
|
||||||
def test_read_existing_order_ids_returns_known_ids(self):
|
|
||||||
with tempfile.TemporaryDirectory() as tmpdir:
|
|
||||||
path = Path(tmpdir) / "orders.csv"
|
|
||||||
path.write_text("order_id,order_total\n1,10.00\n2,20.00\n", encoding="utf-8")
|
|
||||||
|
|
||||||
self.assertEqual({"1", "2"}, scraper.read_existing_order_ids(path))
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
unittest.main()
|
|
||||||
Reference in New Issue
Block a user