From 6eecc186f634b6ab1b1cbac1f8f13a4da80694c5 Mon Sep 17 00:00:00 2001
From: eulaly <johnmosescarter@gmail.com>
Date: Tue, 5 May 2026 23:18:12 -0400
Subject: [PATCH] updated batch processing docs

---
 docs/{openai-batch.md => openai.md} | 20 ++++++++++++++++++++
 docs/tasks.org                      | 29 ++++++++++++++++++++++++++---
 2 files changed, 46 insertions(+), 3 deletions(-)
 rename docs/{openai-batch.md => openai.md} (92%)

diff --git a/docs/openai-batch.md b/docs/openai.md
similarity index 92%
rename from docs/openai-batch.md
rename to docs/openai.md
index 74da50d..d2ba286 100644
--- a/docs/openai-batch.md
+++ b/docs/openai.md
@@ -382,3 +382,23 @@ Expired requests will be written to your error file with the message as shown be
 {"id": "batch_req_123", "custom_id": "request-3", "response": null, "error": {"code": "batch_expired", "message": "This request could not be executed before the completion window expired."}}
 {"id": "batch_req_123", "custom_id": "request-7", "response": null, "error": {"code": "batch_expired", "message": "This request could not be executed before the completion window expired."}}
 ```
+
+
+# Pricing and Limits - Batch
+Updated 2026-05-05
+
+Price per 1M Tokens, Short Context / Limits
+TPM = Tokens per minute
+Req/rpm = Requests per minute
+TPD = Tokens per /day/, you cannot queue more than this TOTAL across all concurrent batches
+| Model        | Input ($) | Cached Input ($) | Output ($) | Token  (tpm) | Req (rpm) | Batch (tpd) |
+|--------------|-----------|------------------|------------|--------------|-----------|-------------|
+| gpt-5.5      | 2.5       | 0.25             | 15.00      | 500000       | 500       | 900000      |
+| gpt-5.4      | 1.25      | 0.13             | 7.50       | 500000       | 500       | 900000      |
+| gpt-5.4-mini | 0.375     | 0.0375           | 2.25       | 200000       | 500       | 2000000     |
+| gpt-5.4-nano | 0.10      | 0.01             | 0.625      | 200000       | 500       | 200000      |
+| gpt-4o       | 1.25      | -                | 5.00       | 500000       | 500       | 900000      |
+| gpt-4o-mini  | 0.075     | -                | 0.30       | 200000       | 500       | 2000000     |
+| gpt-o4-mini  | 0.55      | -                | 0.30       | 200000       | 500       | 2000000     |
+https://developers.openai.com/api/docs/pricing?latest-pricing=batch
+https://platform.openai.com/settings/organization/limits
diff --git a/docs/tasks.org b/docs/tasks.org
index 3c6106b..95b20b6 100644
--- a/docs/tasks.org
+++ b/docs/tasks.org
@@ -77,7 +77,7 @@ Should be run manually, separate from scraper. You may use scrapy, but are not r
   - see: ./analysis/gpt4o/forum452_unknown_gpt-4o_2026-05-05T18-48-32+00-00.jsonl
 - date: [2026-05-05 Tue 15:00]
 
-* [ ] t1.2.1: batch processing
+* [X] t1.2.1: batch processing
 Create analysis-batch.py to capture same elements as t1.2 above. 
 May need to add multiple commands to upload, check batch status, download, etc.
 Commands should all be run manually.
@@ -98,13 +98,36 @@ Reference: ./docs/openai-batch.md. openai batch output order is not guaranteed,
 - custom_id format: comment_{comment_id} — unique within a forum, stable across runs.
 - PROMPT_VERSION derived from analysis/prompt-1.txt (same file as realtime); both scripts produce matching prompt_hash in all records.
 - analysis/prompt-1.txt: system prompt as plaintext, read at import time by both scripts. Edit here to change prompt for both pipelines.
-- Tests use importlib.util to load hyphenated filenames; monkeypatch for RUNS_DIR in save/load test.
 
 ** evidence
-- commit:
+- commit: 683bfb3 (remove hyphen), f3abbef
 - tests: 18 passing (pytest tests/analysis_gpt4o_batch.py), 46 total across suite
 - datetime: [2026-05-05 Tue 17:00]
 
+* [ ] t1.2.2: Tokenizer / Batch mgmt
+openai batch analysis requires coordination - more like a job queue.
+batch script should setup queue for user to setup manually; openai api will reject subsequent batches when the total daily token limit is maxed.
+** Acceptance Criteria
+1. add token estimator utility script, probably to /analysis
+2. add MODEL_LIMITS dict to analysis_batch.py. if there are more than (n)
+   - gpt-4o (30k tpm/90k tpd batch)
+   - gpt-4o-mini (200k tpm/2M tpd batch)
+   - add models listed in docs/openai.md
+3. Auto-chunk submit: before writing the request file, walk comments, accumulate estimated tokens, and split into chunks that fit under the model's limit.
+   - Each chunk becomes its own batch submission with its own run_id.
+   - Drop --limit (or keep as hard cap override).
+   - Print all run_ids
+   - Submit the first batch only
+4. Update test script to show tokenizer output
+
+** notes
+
+** evidence
+- commit:
+- tests:
+- datetime:   
+
+   
 * [ ] X: complete proposal information
 Ensure we capture as much useful information as possible about the actual proposal - contact information, etc. what the state actually says about what was posted. 
 ** acceptance criteria