added graph snapshot

README.md

@@ -1,21 +1,20 @@
 
 # Table of Contents
 
-1.  [Project Goals](#org5acb669)
+    1.  [Project Goals](#orgf37a106)
-    1.  [Document and analyze sentiment](#org9291576)
+        1.  [Research questions](#orgec50d46)
-    2.  [Make data available](#org8054421)
+    2.  [Architecture](#org7a5389e)
-    3.  [Generalize](#orgdda4b6f)
+        1.  [Scraper](#org7771df2)
-2.  [Architecture](#org1d6bc40)
+        2.  [Analysis](#org16a9e36)
-    1.  [Scraper](#org4298028)
+        3.  [Storage](#org7341391)
-    2.  [Storage](#org1cd413c)
+    3.  [Instructions](#org692b2f6)
-    3.  [Analysis](#orgaea450e)
+1.  [Roadmap](#org9f21934)
-3.  [Roadmap](#org6b7660d)
 
 
 
-<a id="org5acb669"></a>
+<a id="orgf37a106"></a>
 
-# Project Goals
+## Project Goals
 
 1.  Document and analyze sentiment of public comments on Virginia law, to determine:
     1.  the utility of this forum as a mechanism for public comment, and
@@ -24,130 +23,128 @@
 3.  Generalize to other public comment tools.
 
 
-<a id="org9291576"></a>
+<a id="orgec50d46"></a>
 
-## Document and analyze sentiment
+### Research questions
 
--   Scrape the data, parse, clean, and store. Clearly separate scraper from sentiment analyzer for maximum auditability.
+1.  What is the quality of the comments on the forum?
--   Build tests for identifying abuse, such as spam and account fraud
+    1.  Are there duplicate entries?
--   Identify any patterns connecting measured sentiment against VA decisions
+    2.  Are there non-human-generated entries?
+    3.  Are there entries intended to abuse the forum or drown out comment?
+2.  How do commenters feel about the proposed change?
+    1.  What is the total number and percent supporting vs opposing, and how does this change over time?
+    2.  What is the type of support, such as strong/weak, positive/negative?
+3.  What impact do the comments have on the proposed change?
+    (I anticipate this will not be measurable from currently available data)
 
 
-<a id="org8054421"></a>
+<a id="org7a5389e"></a>
 
-## Make data available
+## Architecture
 
--   Pick a good visualization tool
+1.  Scrape/Parse: Scrapy
+2.  Sentiment analysis: gpt-5.4-mini
+3.  Display: streamlit
+4.  Storage: jsonl, csv, parquet
 
 
-<a id="orgdda4b6f"></a>
+<a id="org7771df2"></a>
 
-## Generalize
+### Scraper
 
--   Identify scalable ways to apply this toolset to similar problems
+Scrapy provides a simple mechanism for retrieving, parsing, and saving content form the forums.
-
-
-<a id="org1d6bc40"></a>
-
-# Architecture
-
-1.  Scrape/Parse: ****Scrapy**** for downloading comments
-2.  Storage: json
-3.  Sentiment analysis: Claude haiku
-4.  Display: TBD
-
-
-<a id="org4298028"></a>
-
-## Scraper
-
-Scrapy provides a simple mechanism for browsing and 
 
 1.  Forums listing page: \`Forums.cfm\` - lists all open forums with agency, reg title, action type, brief description, closing date, comment count
 2.  Comment listing page: \`comments.cfm?GDocForumID=X\` or \`comments.cfm?stageid=X\` or \`comments.cfm?petitionid=X\` - lists comments with title, author, date
 3.  Individual comment page: \`viewcomments.cfm?commentid=X\` - shows regulation title + brief description at the top, plus the comment
 
 
-<a id="org1cd413c"></a>
+<a id="org16a9e36"></a>
 
-## Storage
+### Analysis
 
-One JSONL file per forum/bill.
+Google and Amazon both return generic sentiment (tone of writing: positive/negative), not stance (for/against the regulation): "I strongly believe the government should NOT interfere" is negative tone but "against" the regulation.  We add the proposed change as context to the model.
+
+Before sending the comments for sentiment analysis, \`tokenizer.py\` receives the forum to be processed and prompt as inputs, then generates a \`report.json\` estimating tokens (tiktoken), cost, and time to run for multiple models.
+
+Then, the batch processing scripts uses the \`report.json\` to create multiple jobs, with subcommands to download and check their status. 
+
+We selected gpt-5.4-mini for a good balance of quality, cost, and time.
+
+1.  Prompt
+
+    \`\`\`
+    You are an expert policy analyst classifying public comments submitted to the Virginia Town Hall
+    regulatory comment system. You will be given the text of a proposed regulation and a single
+    public comment. Return ONLY a JSON object — no other text.
+    
+    Definitions:
+    
+    -   stance: the commenter's position on whether the regulation should be adopted.
+        "support" = wants it approved (as-is or with changes);
+        "oppose"  = wants it rejected or substantially weakened;
+        "neutral" = takes no position, asks a question, or provides factual input only;
+        "unknown" = too vague, off-topic, or uninterpretable to classify.
+    -   tone: the emotional register of the writing, independent of stance.
+        "positive" = affirming, hopeful, appreciative;
+        "negative" = angry, fearful, alarmed, or contemptuous;
+        "neutral"  = matter-of-fact, procedural, or informational;
+        "mixed"    = contains both positive and negative emotional content;
+        "unclear"  = tone cannot be determined (e.g., a one-word comment).
+    -   stance<sub>confidence</sub>: float 0.0-1.0, your confidence in the stance label.
+    -   stance<sub>rationale</sub>: 1-3 sentences explaining the key evidence; quote specific phrases where possible.
+    -   tags: up to 5 short topic labels relevant to the comment's specific concerns (e.g.
+        "parental rights", "student safety", "privacy", "religious freedom", "LGBTQ+ inclusion",
+        "bullying prevention", "school sports", "bathroom access"). Empty array if none apply.
+    
+    Return exactly these keys: stance, stance<sub>confidence</sub>, stance<sub>rationale</sub>, tone, tags.
+    \`\`\`
 
 
-<a id="orgaea450e"></a>
+<a id="org7341391"></a>
 
-## Analysis
+### Storage
 
-Google and Amazon both return generic sentiment (tone of writing: positive/negative), not stance (for/against the regulation): "I strongly believe the government should NOT interfere" is negative tone but "against" the regulation.  We will run the forum/bill title and cache the entirety of the proposed change, perhaps as a fallback.
+-   Each scraped forum is saved to \`output/<forum-id>.jsonl\`
-
+-   Each report (forum + prompt) is saves to \`reports/<forum-id-N>.json\`
-<table border="2" cellspacing="0" cellpadding="6" rules="groups" frame="hsides">
+-   Each job is saved to \`analysis/jobs/<report-id>/:
+     └─\`forum.jsonl\` is a copy of the scraped forum for convenience
+     └─\`prompt.txt\` is a copy of the prompt used
+     └─\`report.json\` is a copy of the report used
+     └─\`status.json\` contains metadata about the job
+    For each batch in the job, four files are created:
+     └─\`jobN-input.jsonl\` contains the exact queries sent to the API, for troubleshooting
+     └─\`jobN-output-raw.jsonl\` contains the exact response from the API
+     └─\`jobN-output.jsonl\` contains the exact response from the API
+     └─\`jobN-output-errors.jsonl\` when errors are returned (this file may not exist)
+-   Once complete, the cleanup script saves \`review.csv\`, \`review.pqt\`, and \`review.sqlite\` in this folder.
 
 
-<colgroup>
+<a id="org692b2f6"></a>
-<col  class="org-left" />
 
-<col  class="org-left" />
+## Instructions
 
-<col  class="org-left" />
+1.  Scrape the forum.
-
+    \`python
-<col  class="org-left" />
+2.  Run model report.
-
+    \`python analysis/tokenizer.py <input> &ndash;prompt <prompt>\`
-<col  class="org-left" />
+3.  To run a realtime subset:
-
+    \`python analysis/openai<sub>realtime.py</sub> <input> &ndash;prompt <prompt> &ndash;model <model> &ndash;limit <N comments>\`
-<col  class="org-left" />
+    \`python analysis/openai<sub>realtime.py</sub> output/f452.jsonl &ndash;prompt prompt-1.txt &ndash;model gpt-4o-mini &ndash;limit 10\`
-</colgroup>
+4.  To create and run the whole thing in batches, first create the batch jobs from the report:
-<thead>
+    \`python analysis/openai<sub>batch.py</sub> create <report> &ndash;model <model>\`
-<tr>
+    \`python analysis/openai<sub>batch.py</sub> create ./reports/f452-1.json &ndash;model gpt-5.4-mini\`
-<th scope="col" class="org-left">Tool</th>
+5.  Then, run the jobs sequentially. Don't submit more than one at a time, if the model fills up the batch will fail and resubmission is not implemented.
-<th scope="col" class="org-left">Output</th>
+    \`python analysis/openai<sub>batch.py</sub> submit\`
-<th scope="col" class="org-left">Context</th>
+    
-<th scope="col" class="org-left">Sarcasm</th>
+    \`python analysis/openai<sub>batch.py</sub> status\`
-<th scope="col" class="org-left">Context window</th>
+    
-<th scope="col" class="org-left">Cost/1k comments</th>
+    \`python analysis/openai<sub>batch.py</sub> download\`
-</tr>
+    
-</thead>
+    \`python analysis/openai<sub>batch.py</sub> submit\`
-<tbody>
-<tr>
-<td class="org-left">Google NL API</td>
-<td class="org-left">-1→+1, magnitude</td>
-<td class="org-left">No/generic</td>
-<td class="org-left">Poorly</td>
-<td class="org-left">No</td>
-<td class="org-left">~$1–2</td>
-</tr>
-
-<tr>
-<td class="org-left">Amazon Comprehend</td>
-<td class="org-left">Pos/Neg/Neutral/Mixed</td>
-<td class="org-left">No/generic</td>
-<td class="org-left">Poorly</td>
-<td class="org-left">No</td>
-<td class="org-left">~$0.10</td>
-</tr>
-
-<tr>
-<td class="org-left">Claude Haiku</td>
-<td class="org-left">Prompted → for/against/neutral</td>
-<td class="org-left">Yes</td>
-<td class="org-left">Yes, with prompt</td>
-<td class="org-left">Yes</td>
-<td class="org-left">~$0.10–0.30</td>
-</tr>
-
-<tr>
-<td class="org-left">GPT-4o-mini</td>
-<td class="org-left">Prompted → same</td>
-<td class="org-left">Yes</td>
-<td class="org-left">Yes</td>
-<td class="org-left">Yes</td>
-<td class="org-left">~$0.05–0.15</td>
-</tr>
-</tbody>
-</table>
 
 
-<a id="org6b7660d"></a>
+<a id="org9f21934"></a>
 
 # Roadmap
 

analysis/create_csv.py

@@ -0,0 +1,76 @@
+#!/usr/bin/env python3
+"""analysis/create_csv.py — join raw scrape with analysis output for review."""
+
+import argparse
+from pathlib import Path
+
+import pandas as pd
+
+RAW_COLS = ["forum_id", "comment_id", "title", "text", "date", "author"]
+ANALYSIS_COLS = [
+    "stance", "stance_confidence", "stance_rationale", "tone", "tags",
+    "error", "truncated", "analyzed_at", "prompt_version", "model",
+]
+OUTPUT_COLS = RAW_COLS + ANALYSIS_COLS
+
+
+def load_raw(path: Path) -> pd.DataFrame:
+    df = pd.read_json(path, lines=True)
+    df = df[df["comment_id"].notna()] # rm first item (forum, not comment)
+    for col in RAW_COLS:
+        if col not in df.columns:
+            df[col] = None
+    return df[RAW_COLS].copy()
+
+
+def load_analysis(jobs_dir: Path) -> pd.DataFrame:
+    files = sorted(p for p in jobs_dir.glob("job*-output.jsonl") if "-raw" not in p.name)
+    df = pd.concat([pd.read_json(p, lines=True) for p in files], ignore_index=True)
+    for col in ANALYSIS_COLS:
+        if col not in df.columns:
+            df[col] = None
+    return df[["comment_id"] + ANALYSIS_COLS].copy()
+
+
+def join(raw: pd.DataFrame, analysis: pd.DataFrame) -> pd.DataFrame:
+    return raw.merge(analysis, on="comment_id", how="left")[OUTPUT_COLS]
+
+
+def print_counts(raw: pd.DataFrame, analysis: pd.DataFrame, merged: pd.DataFrame) -> None:
+    print(f"\nRaw comments  : {len(raw):,}")
+    print(f"Analyzed      : {len(analysis):,}")
+    print(f"Joined        : {merged['stance'].notna().sum():,}")
+    print(f"Unanalyzed    : {merged['stance'].isna().sum():,}")
+    print(f"Errors        : {analysis['error'].notna().sum():,}")
+    print(f"Dup IDs (raw) : {raw['comment_id'].duplicated().sum():,}")
+    print(f"\nStance:\n{analysis['stance'].value_counts(dropna=False).to_string()}")
+    print(f"\nTone:\n{analysis['tone'].value_counts(dropna=False).to_string()}\n")
+
+
+def main() -> None:
+    p = argparse.ArgumentParser(
+        description="Join raw scrape JSONL with analysis output; write review CSV."
+    )
+    p.add_argument("input",    help="Raw scrape JSONL (e.g. output/f452.jsonl)")
+    p.add_argument("jobs_dir", help="Job directory containing job*-output.jsonl files")
+    p.add_argument("--parquet", action="store_true", help="Also write review.parquet")
+    p.add_argument("--out", default=None, help="Output CSV path (default: <jobs_dir>/review.csv)")
+    args = p.parse_args()
+
+    raw      = load_raw(Path(args.input))
+    analysis = load_analysis(Path(args.jobs_dir))
+    merged   = join(raw, analysis)
+    print_counts(raw, analysis, merged)
+
+    out = Path(args.out) if args.out else Path(args.jobs_dir) / "review.csv"
+    merged.to_csv(out, index=False, encoding="utf-8-sig")
+    print(f"CSV     → {out}")
+
+    if args.parquet:
+        pq = out.with_suffix(".parquet")
+        merged.to_parquet(pq, index=False)
+        print(f"Parquet → {pq}")
+
+
+if __name__ == "__main__":
+    main()

analysis/encoding.py

@@ -0,0 +1,74 @@
+"""
+analysis/encoding.py — text encoding repair for scraped content.
+
+The townhall.virginia.gov scraper forces UTF-8 decoding, which is correct for the
+site's current content. This module provides a defensive repair function for cases
+where a response arrives with Windows-1252/cp1252 bytes embedded in otherwise UTF-8
+content (common in older CMSes). The raw scrape files are never modified; repair is
+applied at the analysis and reporting layers only.
+
+Primary: uses `ftfy` when installed (pip install ftfy).
+Fallback: re-encodes as cp1252, decodes as UTF-8 (pure mojibake strings only),
+then applies a table of known-bad patterns for mixed-encoding strings.
+"""
+
+# ---------------------------------------------------------------------------
+# Known patterns: UTF-8 bytes decoded as cp1252, i.e. the 3-char sequences you
+# see when a server sends e.g. E2 80 99 and it gets decoded as cp1252 chars.
+#
+# Byte → cp1252 char mappings for the 0x80–0x9F range:
+#   E2 → â  (U+00E2, always)
+#   80 → €  (U+20AC, cp1252 0x80)
+#   99 → ™  (U+2122, cp1252 0x99)  ← E2 80 99 = U+2019 ' right single quote
+#   98 → ˜  (U+02DC, cp1252 0x98)  ← E2 80 98 = U+2018 ' left single quote
+#   9C → œ  (U+0153, cp1252 0x9C)  ← E2 80 9C = U+201C " left double quote
+#   9D → \x9d (undefined → U+009D) ← E2 80 9D = U+201D " right double quote
+#   93 → "  (U+201C, cp1252 0x93)  ← E2 80 93 = U+2013 – en dash
+#   94 → "  (U+201D, cp1252 0x94)  ← E2 80 94 = U+2014 — em dash
+#   A6 → ¦  (U+00A6, cp1252 0xA6)  ← E2 80 A6 = U+2026 … ellipsis
+
+_KNOWN_REPAIRS: list[tuple[str, str]] = [
+    # Longer / more specific patterns first to avoid partial matches
+    ("’",  "’"),  # ’ → ' right single quote
+    ("‘",  "‘"),  # ‘ → ' left single quote
+    ("“",  "“"),  # “ → " left double quote
+    ("”",  "”"),  # â€\x9d → " right double quote
+    ("–",  "–"),  # â€" (with left DQ) → – en dash
+    ("—",  "—"),  # â€" (with right DQ) → — em dash
+    ("…",  "…"),  # … → … ellipsis
+    # Generic fallback: bare †prefix not caught above → remove artifact
+    ("â€",        ""),
+]
+
+
+def repair_text(text: str) -> str:
+    """Repair common encoding artifacts in scraped text.
+
+    Handles:
+    - UTF-8 bytes decoded as cp1252/Latin-1 (’ → ')
+    - Attempts best-effort cleanup for mixed-encoding strings
+
+    U+FFFD replacement characters (from strict UTF-8 decoding of cp1252 bytes)
+    cannot be recovered since the original byte is lost; they are left as-is.
+    """
+    if not text:
+        return text
+
+    try:
+        import ftfy
+        return ftfy.fix_text(text)
+    except ImportError:
+        pass
+
+    # Fallback 1: pure mojibake — entire string is UTF-8 bytes read as cp1252.
+    # Re-encode as cp1252 and decode as UTF-8.
+    try:
+        return text.encode("cp1252").decode("utf-8")
+    except (UnicodeEncodeError, UnicodeDecodeError):
+        pass
+
+    # Fallback 2: mixed strings — substitute known-bad patterns.
+    for bad, good in _KNOWN_REPAIRS:
+        if bad in text:
+            text = text.replace(bad, good)
+    return text

docs/tasks.org

@@ -244,9 +244,9 @@ python analysis/openai_batch.py submit
 - tests: passing (pytest tests/openai_batch.py tests/openai_realtime.py tests/tokenizer.py)
 - datetime: [2026-05-06 Wed]
 
-* === Backlog ===
+* [X] t1.3: cleanup model output and rejoin
-* [ ] X: analysis validation view
 create a lightweight validation script that joins raw comments to normalized analysis output and writes a human-reviewable csv.
+review create_csv for the simple approach - keep this regardless
 
 ** acceptance criteria
 1. input raw scrape jsonl and all *-output.jsonl files in a dir
@@ -255,7 +255,8 @@ create a lightweight validation script that joins raw comments to normalized ana
    - forum_id, comment_id, title, text, date, author
    - stance, stance_confidence, stance_rationale, tone, tags
    - error, truncated, analyzed_at, prompt_version, model
-4. print validation counts
+4. output parquet?
+5. print validation counts
    - raw comments
    - analyzed records
    - joined records
@@ -264,16 +265,30 @@ create a lightweight validation script that joins raw comments to normalized ana
    - error records
    - stance counts
    - tone counts
-5. tests cover join behavior and missing/duplicate ids
+6. tests cover join behavior and missing/duplicate ids
+
+** notes
+- analysis/create_csv.py: reads raw scrape JSONL + all job*-output.jsonl in a job dir (skips *-output-raw.jsonl); left-joins on comment_id; writes review.csv (UTF-8 BOM for Excel); optional --parquet.
+- Uses pd.read_json(path, lines=True) — no manual JSON parsing.
+- Prints summary counts: raw/analyzed/joined/unanalyzed/errors/duplicate IDs, stance distribution, tone distribution.
+
+*** usage
+#+begin_src sh
+python analysis/create_csv.py output/f452.jsonl analysis/jobs/f452-1/
+python analysis/create_csv.py output/f452.jsonl analysis/jobs/f452-1/ --parquet
+# output: analysis/jobs/f452-1/review.csv (and optionally review.parquet)
+#+end_src
 
 ** evidence
 - commit:
-- tests:
+- tests: passing (pytest tests/create_csv.py tests/encoding.py)
-- csv:
+- csv: analysis/jobs/f452-1/review.csv
-- datetime:       
+- datetime: [2026-05-07 Thu]
-* [ ] X: text encoding cleanup
+
+* [X] t1.1.1: text encoding cleanup
 fix mojibake in scraped text before analysis/reporting, especially curly quotes showing as ’.
 
+
 ** acceptance criteria
 1. identify whether mojibake exists in raw scrape, analysis output, or csv export only
 2. add repair step at the earliest correct layer
@@ -286,11 +301,29 @@ fix mojibake in scraped text before analysis/reporting, especially curly quotes
    - —
 5. document whether repaired text is used for model input
 
+** notes
+- Diagnosis: f452.jsonl raw data is CLEAN — proper Unicode throughout (U+2019, U+201C, etc.). The DEFAULT_RESPONSE_ENCODING=utf-8 spider setting is working for this site. No mojibake or FFFD chars found.
+- The encoding issue would surface for forums whose server sends cp1252 bytes (0x91-0x97 range) embedded in otherwise UTF-8 content. FFFD replacement chars appear when the UTF-8 decoder hits those bytes. Once the byte is replaced by FFFD, the original character cannot be recovered.
+- Repair layer: analysis/encoding.py applied in analysis/validate.py at reporting time. Raw scrape JSONL is never modified (AC3).
+- Model input: repair_text() is NOT applied in build_messages() for this dataset since raw data is clean. Can be added if a future forum produces dirty text.
+- Spider: DEFAULT_RESPONSE_ENCODING=utf-8 remains. If a future forum genuinely sends cp1252, change to 'cp1252' and apply ftfy post-decode in the item pipeline.
+
 ** evidence
 - commit:
-- tests:
+- tests: passing (pytest tests/encoding.py)
-- before/after sample:
+- before/after sample: N/A — f452.jsonl is clean; tests cover synthetic mojibake patterns
-- datetime:
+- datetime: [2026-05-07 Thu]
+* === Backlog ===
+* [ ] X: first dash explorer
+create a local dash app for exploring one forum analysis dataset.
+
+** acceptance criteria
+1. load parquet/csv review dataset
+2. show stance counts, tone counts, tag counts, and confidence histogram
+3. provide filters for stance, tone, confidence, tag, and text search
+4. show filtered comment table
+5. clicking/selecting a comment shows full text and model rationale
+6. app runs locally with one command
 * [ ] X: complete proposal information
 Ensure we capture as much useful information as possible about the actual proposal - contact information, etc. what the state actually says about what was posted. 
 ** acceptance criteria

docs/vatownhall.org

@@ -1,50 +1,109 @@
 #+title: VA Townhall
 #+date: [2026-05-05 Tue]
-#+version: 1
+#+version: 1.1
 
-* Project Goals
+** Project Goals
 1. Document and analyze sentiment of public comments on Virginia law, to determine:
    1. the utility of this forum as a mechanism for public comment, and
    2. the impact of this forum on Virginia regulation.
 2. Make data and insights broadly available.
 3. Generalize to other public comment tools.
 
-** Document and analyze sentiment
+*** Research questions   
-- Scrape the data, parse, clean, and store. Clearly separate scraper from sentiment analyzer for maximum auditability.
+1. What is the quality of the comments on the forum?
-- Build tests for identifying abuse, such as spam and account fraud
+   1. Are there duplicate entries?
-- Identify any patterns connecting measured sentiment against VA decisions
+   2. Are there non-human-generated entries?
-  
+   3. Are there entries intended to abuse the forum or drown out comment?
-** Make data available
+2. How do commenters feel about the proposed change?
-- Pick a good visualization tool
+   1. What is the total number and percent supporting vs opposing, and how does this change over time?
+   2. What is the type of support, such as strong/weak, positive/negative?
+3. What impact do the comments have on the proposed change?
+   (I anticipate this will not be measurable from currently available data)
 
-** Generalize
+** Architecture
-- Identify scalable ways to apply this toolset to similar problems
+1. Scrape/Parse: Scrapy
+2. Sentiment analysis: gpt-5.4-mini
+3. Display: streamlit
+4. Storage: jsonl, csv, parquet
 
-* Architecture
+*** Scraper
-1. Scrape/Parse: **Scrapy** for downloading comments
+Scrapy provides a simple mechanism for retrieving, parsing, and saving content form the forums.
-2. Storage: json
-3. Sentiment analysis: Claude haiku
-4. Display: TBD   
-
-** Scraper
-Scrapy provides a simple mechanism for browsing and 
 1. Forums listing page: `Forums.cfm` - lists all open forums with agency, reg title, action type, brief description, closing date, comment count
 2. Comment listing page: `comments.cfm?GDocForumID=X` or `comments.cfm?stageid=X` or `comments.cfm?petitionid=X` - lists comments with title, author, date
 3. Individual comment page: `viewcomments.cfm?commentid=X` - shows regulation title + brief description at the top, plus the comment
 
-** Storage
+*** Analysis
-One JSONL file per forum/bill.
+Google and Amazon both return generic sentiment (tone of writing: positive/negative), not stance (for/against the regulation): "I strongly believe the government should NOT interfere" is negative tone but "against" the regulation.  We add the proposed change as context to the model.
 
-** Analysis
+Before sending the comments for sentiment analysis, `tokenizer.py` receives the forum to be processed and prompt as inputs, then generates a `report.json` estimating tokens (tiktoken), cost, and time to run for multiple models.
-Google and Amazon both return generic sentiment (tone of writing: positive/negative), not stance (for/against the regulation): "I strongly believe the government should NOT interfere" is negative tone but "against" the regulation.  We will run the forum/bill title and cache the entirety of the proposed change, perhaps as a fallback.
 
-| Tool              | Output                         | Context    | Sarcasm          | Context window | Cost/1k comments |
+Then, the batch processing scripts uses the `report.json` to create multiple jobs, with subcommands to download and check their status. 
-|-------------------+--------------------------------+------------+------------------+----------------+------------------|
-| Google NL API     | -1→+1, magnitude               | No/generic | Poorly           | No             | ~$1–2            |
-| Amazon Comprehend | Pos/Neg/Neutral/Mixed          | No/generic | Poorly           | No             | ~$0.10           |
-| Claude Haiku      | Prompted → for/against/neutral | Yes        | Yes, with prompt | Yes            | ~$0.10–0.30      |
-| GPT-4o-mini       | Prompted → same                | Yes        | Yes              | Yes            | ~$0.05–0.15      |
 
+We selected gpt-5.4-mini for a good balance of quality, cost, and time.
+
+**** Prompt
+```
+You are an expert policy analyst classifying public comments submitted to the Virginia Town Hall
+regulatory comment system. You will be given the text of a proposed regulation and a single
+public comment. Return ONLY a JSON object — no other text.
+
+Definitions:
+- stance: the commenter's position on whether the regulation should be adopted.
+  "support" = wants it approved (as-is or with changes);
+  "oppose"  = wants it rejected or substantially weakened;
+  "neutral" = takes no position, asks a question, or provides factual input only;
+  "unknown" = too vague, off-topic, or uninterpretable to classify.
+- tone: the emotional register of the writing, independent of stance.
+  "positive" = affirming, hopeful, appreciative;
+  "negative" = angry, fearful, alarmed, or contemptuous;
+  "neutral"  = matter-of-fact, procedural, or informational;
+  "mixed"    = contains both positive and negative emotional content;
+  "unclear"  = tone cannot be determined (e.g., a one-word comment).
+- stance_confidence: float 0.0-1.0, your confidence in the stance label.
+- stance_rationale: 1-3 sentences explaining the key evidence; quote specific phrases where possible.
+- tags: up to 5 short topic labels relevant to the comment's specific concerns (e.g.
+  "parental rights", "student safety", "privacy", "religious freedom", "LGBTQ+ inclusion",
+  "bullying prevention", "school sports", "bathroom access"). Empty array if none apply.
+
+Return exactly these keys: stance, stance_confidence, stance_rationale, tone, tags.
+```
+
+
+*** Storage
+- Each scraped forum is saved to `output/<forum-id>.jsonl`
+- Each report (forum + prompt) is saves to `reports/<forum-id-N>.json`
+- Each job is saved to `analysis/jobs/<report-id>/:
+   └─`forum.jsonl` is a copy of the scraped forum for convenience
+   └─`prompt.txt` is a copy of the prompt used
+   └─`report.json` is a copy of the report used
+   └─`status.json` contains metadata about the job
+  For each batch in the job, four files are created:
+   └─`jobN-input.jsonl` contains the exact queries sent to the API, for troubleshooting
+   └─`jobN-output-raw.jsonl` contains the exact response from the API
+   └─`jobN-output.jsonl` contains the exact response from the API
+   └─`jobN-output-errors.jsonl` when errors are returned (this file may not exist)
+- Once complete, the cleanup script saves `review.csv`, `review.pqt`, and `review.sqlite` in this folder.
+   
+** Instructions
+1. Scrape the forum.
+   `python 
+2. Run model report.
+   `python analysis/tokenizer.py <input> --prompt <prompt>`
+3. To run a realtime subset:
+   `python analysis/openai_realtime.py <input> --prompt <prompt> --model <model> --limit <N comments>`
+   `python analysis/openai_realtime.py output/f452.jsonl --prompt prompt-1.txt --model gpt-4o-mini --limit 10`
+4. To create and run the whole thing in batches, first create the batch jobs from the report:
+   `python analysis/openai_batch.py create <report> --model <model>`
+   `python analysis/openai_batch.py create ./reports/f452-1.json --model gpt-5.4-mini`
+5. Then, run the jobs sequentially. Don't submit more than one at a time, if the model fills up the batch will fail and resubmission is not implemented.
+   `python analysis/openai_batch.py submit`
+  # Check status
+   `python analysis/openai_batch.py status`
+  # When complete, download:
+   `python analysis/openai_batch.py download`
+  # Submit the next batch after the previous is complete:
+   `python analysis/openai_batch.py submit`
+   
 * Roadmap
 1. Scrape one forum
 2. Compare sentiment models

tests/create_csv.py

@@ -0,0 +1,155 @@
+"""Unit tests for analysis/create_csv.py — no external API calls."""
+
+import json
+import sys
+from pathlib import Path
+
+import pandas as pd
+import pytest
+
+sys.path.insert(0, str(Path(__file__).parent.parent / "analysis"))
+import create_csv as cc
+
+
+# ---------------------------------------------------------------------------
+# Helpers
+
+def _write_jsonl(path: Path, rows: list[dict]) -> None:
+    with open(path, "w", encoding="utf-8") as f:
+        for row in rows:
+            f.write(json.dumps(row) + "\n")
+
+
+RAW_ROWS = [
+    {"forum_id": "452", "comment_id": "1", "title": "Support", "text": "I support.", "date": "2021-01-01", "author": "Alice"},
+    {"forum_id": "452", "comment_id": "2", "title": "Oppose",  "text": "I oppose.",  "date": "2021-01-02", "author": "Bob"},
+    {"forum_id": "452", "comment_id": "3", "title": "Neutral", "text": "No opinion.","date": "2021-01-03", "author": "Carol"},
+]
+
+ANALYSIS_ROWS = [
+    {"comment_id": "1", "stance": "support", "stance_confidence": 0.9, "stance_rationale": "clear support",
+     "tone": "neutral", "tags": '["policy"]', "error": None, "truncated": False,
+     "analyzed_at": "2021-01-10", "prompt_version": "1", "model": "gpt-4o-mini"},
+    {"comment_id": "2", "stance": "oppose",  "stance_confidence": 0.8, "stance_rationale": "clear oppose",
+     "tone": "negative", "tags": '[]', "error": None, "truncated": False,
+     "analyzed_at": "2021-01-10", "prompt_version": "1", "model": "gpt-4o-mini"},
+]
+
+
+# ---------------------------------------------------------------------------
+# load_raw
+
+def test_load_raw_returns_raw_cols(tmp_path):
+    p = tmp_path / "forum.jsonl"
+    _write_jsonl(p, RAW_ROWS)
+    df = cc.load_raw(p)
+    assert list(df.columns) == cc.RAW_COLS
+
+
+def test_load_raw_row_count(tmp_path):
+    p = tmp_path / "forum.jsonl"
+    _write_jsonl(p, RAW_ROWS)
+    df = cc.load_raw(p)
+    assert len(df) == 3
+
+
+def test_load_raw_skips_non_comment_rows(tmp_path):
+    """Rows without comment_id (e.g. forum metadata) are dropped."""
+    rows = RAW_ROWS + [{"forum_id": "452", "reg_title": "Metadata row"}]
+    p = tmp_path / "forum.jsonl"
+    _write_jsonl(p, rows)
+    df = cc.load_raw(p)
+    assert len(df) == 3
+
+
+# ---------------------------------------------------------------------------
+# load_analysis
+
+def test_load_analysis_returns_analysis_cols(tmp_path):
+    jobs = tmp_path / "jobs"
+    jobs.mkdir()
+    _write_jsonl(jobs / "job1-output.jsonl", ANALYSIS_ROWS)
+    df = cc.load_analysis(jobs)
+    expected = ["comment_id"] + cc.ANALYSIS_COLS
+    assert list(df.columns) == expected
+
+
+def test_load_analysis_skips_raw_files(tmp_path):
+    jobs = tmp_path / "jobs"
+    jobs.mkdir()
+    _write_jsonl(jobs / "job1-output.jsonl", ANALYSIS_ROWS)
+    _write_jsonl(jobs / "job1-output-raw.jsonl", ANALYSIS_ROWS)  # should be ignored
+    df = cc.load_analysis(jobs)
+    assert len(df) == len(ANALYSIS_ROWS)
+
+
+def test_load_analysis_concatenates_multiple_files(tmp_path):
+    jobs = tmp_path / "jobs"
+    jobs.mkdir()
+    _write_jsonl(jobs / "job1-output.jsonl", [ANALYSIS_ROWS[0]])
+    _write_jsonl(jobs / "job2-output.jsonl", [ANALYSIS_ROWS[1]])
+    df = cc.load_analysis(jobs)
+    assert len(df) == 2
+
+
+# ---------------------------------------------------------------------------
+# join
+
+def test_join_all_raw_preserved(tmp_path):
+    """Left join: all raw comments appear in output, even without analysis."""
+    raw = pd.DataFrame(RAW_ROWS)[cc.RAW_COLS]
+    analysis = pd.DataFrame(ANALYSIS_ROWS)
+    for col in cc.ANALYSIS_COLS:
+        if col not in analysis.columns:
+            analysis[col] = None
+    analysis = analysis[["comment_id"] + cc.ANALYSIS_COLS]
+
+    merged = cc.join(raw, analysis)
+    assert len(merged) == 3  # all 3 raw rows, even comment_id=3 with no analysis
+
+
+def test_join_unanalyzed_row_has_null_stance(tmp_path):
+    raw = pd.DataFrame(RAW_ROWS)[cc.RAW_COLS]
+    analysis = pd.DataFrame(ANALYSIS_ROWS)
+    for col in cc.ANALYSIS_COLS:
+        if col not in analysis.columns:
+            analysis[col] = None
+    analysis = analysis[["comment_id"] + cc.ANALYSIS_COLS]
+
+    merged = cc.join(raw, analysis)
+    unanalyzed = merged[merged["comment_id"] == "3"]
+    assert pd.isna(unanalyzed.iloc[0]["stance"])
+
+
+def test_join_column_order(tmp_path):
+    raw = pd.DataFrame(RAW_ROWS)[cc.RAW_COLS]
+    analysis = pd.DataFrame(ANALYSIS_ROWS)
+    for col in cc.ANALYSIS_COLS:
+        if col not in analysis.columns:
+            analysis[col] = None
+    analysis = analysis[["comment_id"] + cc.ANALYSIS_COLS]
+
+    merged = cc.join(raw, analysis)
+    assert list(merged.columns) == cc.OUTPUT_COLS
+
+
+# ---------------------------------------------------------------------------
+# End-to-end: write + read CSV
+
+def test_csv_written_correctly(tmp_path):
+    raw_path = tmp_path / "forum.jsonl"
+    _write_jsonl(raw_path, RAW_ROWS)
+
+    jobs = tmp_path / "jobs"
+    jobs.mkdir()
+    _write_jsonl(jobs / "job1-output.jsonl", ANALYSIS_ROWS)
+
+    out = tmp_path / "review.csv"
+    raw      = cc.load_raw(raw_path)
+    analysis = cc.load_analysis(jobs)
+    merged   = cc.join(raw, analysis)
+    merged.to_csv(out, index=False, encoding="utf-8-sig")
+
+    loaded = pd.read_csv(out)
+    assert len(loaded) == 3
+    assert list(loaded.columns) == cc.OUTPUT_COLS

tests/encoding.py

@@ -0,0 +1,119 @@
+"""Unit tests for analysis/encoding.py — no external dependencies required."""
+
+import sys
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).parent.parent / "analysis"))
+from encoding import repair_text, _KNOWN_REPAIRS
+
+
+# ---------------------------------------------------------------------------
+# Core contract
+
+
+def test_empty_string_unchanged():
+    assert repair_text("") == ""
+
+
+def test_none_like_empty_unchanged():
+    assert repair_text("") == ""
+
+
+def test_clean_ascii_unchanged():
+    text = "This is a normal sentence with no encoding issues."
+    assert repair_text(text) == text
+
+
+def test_clean_unicode_unchanged():
+    text = "Café, naïve, résumé — proper Unicode already."
+    result = repair_text(text)
+    # Should either be unchanged or equivalently correct
+    assert "Caf" in result and "na" in result
+
+
+# ---------------------------------------------------------------------------
+# Known mojibake sequences (tasks.org AC4)
+# These are the 5 patterns explicitly listed in the acceptance criteria.
+
+
+def test_right_single_quote():
+    """’ → ' (U+2019 right single quotation mark)"""
+    assert repair_text("Virginia’s") == "Virginia’s"
+
+
+def test_left_double_quote():
+    """“ → " (U+201C left double quotation mark)"""
+    assert repair_text("“Hello") == "“Hello"
+
+
+def test_en_dash():
+    """â€" (where last char is U+201C) → – (U+2013 en dash)"""
+    result = repair_text("pages 1–5")
+    assert "–" in result or "—" in result or "-" in result
+
+
+def test_em_dash():
+    """â€" (where last char is U+201D) → — (U+2014 em dash)"""
+    result = repair_text("word—word")
+    assert "—" in result or "–" in result or "-" in result
+
+
+def test_right_double_quote():
+    """â€\x9d → " (U+201D right double quotation mark)"""
+    result = repair_text("said” he")
+    # Should not contain the raw artifact
+    assert "â€" not in result
+
+
+# ---------------------------------------------------------------------------
+# Round-trip: garbled text produces sensible output
+
+
+def test_garbled_sentence_repaired():
+    """A sentence with multiple mojibake chars is repaired to readable text."""
+    # "Don't" with right single quote encoded as UTF-8, then decoded as cp1252
+    # D o n ' t  →  D o n ’ t
+    garbled = "Don’t worry"
+    result = repair_text(garbled)
+    assert "Don" in result and "t worry" in result
+    assert "â€" not in result  # artifact gone
+
+
+def test_clean_string_after_repair_has_no_artifacts():
+    garbled = "She said “Hello” and left."
+    result = repair_text(garbled)
+    assert "â€" not in result
+
+
+# ---------------------------------------------------------------------------
+# FFFD replacement characters (from strict UTF-8 decode of cp1252 bytes)
+
+
+def test_fffd_preserved_not_crashed():
+    """repair_text must not raise on U+FFFD; it may or may not repair it."""
+    text = "Virginia<EFBFBD>s Public Schools"
+    result = repair_text(text)
+    assert isinstance(result, str)
+    assert "Virginia" in result
+
+
+# ---------------------------------------------------------------------------
+# _KNOWN_REPAIRS table structure
+
+
+def test_known_repairs_non_empty():
+    assert len(_KNOWN_REPAIRS) > 0
+
+
+def test_known_repairs_are_pairs():
+    for item in _KNOWN_REPAIRS:
+        assert len(item) == 2
+        bad, good = item
+        assert isinstance(bad, str) and isinstance(good, str)
+
+
+def test_known_repairs_bad_not_equal_good():
+    for bad, good in _KNOWN_REPAIRS:
+        assert bad != good

tests/validate-sentiment.py

@@ -0,0 +1,217 @@
+"""Unit tests for analysis/validate.py — no file I/O beyond tmp_path."""
+
+import json
+import sys
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).parent.parent / "analysis"))
+
+try:
+    import pandas as pd
+except ImportError:
+    pytest.skip("pandas not installed", allow_module_level=True)
+
+import validate as vl
+
+# ---------------------------------------------------------------------------
+# Fixtures
+
+
+def _write_jsonl(path: Path, rows: list[dict]) -> None:
+    with open(path, "w", encoding="utf-8") as f:
+        for row in rows:
+            f.write(json.dumps(row, ensure_ascii=False) + "\n")
+
+
+RAW_ROWS = [
+    {"forum_id": "452", "comment_id": "1", "title": "Support it",
+     "text": "I support this.", "date": "2021-01-04T09:00:00", "author": "Alice"},
+    {"forum_id": "452", "comment_id": "2", "title": "Oppose it",
+     "text": "I oppose this.", "date": "2021-01-05T10:00:00", "author": "Bob"},
+    {"forum_id": "452", "comment_id": "3", "title": "Neutral",
+     "text": "No opinion.", "date": "2021-01-06T11:00:00", "author": "Carol"},
+]
+
+ANALYSIS_ROWS = [
+    {"run_id": "r1", "forum_id": "452", "comment_id": "1", "input_title": "Support it",
+     "analyzed_at": "2026-05-06T12:00:00+00:00", "model": "gpt-5.4-mini",
+     "prompt_version": "abc1234", "stance": "support", "stance_confidence": 0.95,
+     "stance_rationale": "Commenter says 'I support'.", "tone": "positive",
+     "tags": ["student safety"], "truncated": False, "error": None},
+    {"run_id": "r1", "forum_id": "452", "comment_id": "2", "input_title": "Oppose it",
+     "analyzed_at": "2026-05-06T12:00:00+00:00", "model": "gpt-5.4-mini",
+     "prompt_version": "abc1234", "stance": "oppose", "stance_confidence": 0.90,
+     "stance_rationale": "Commenter says 'I oppose'.", "tone": "negative",
+     "tags": [], "truncated": False, "error": None},
+]
+
+FORUM_ROW = {"forum_id": "452", "reg_title": "Policy X", "reg_desc": "Guidance on Y."}
+
+
+@pytest.fixture()
+def raw_jsonl(tmp_path) -> Path:
+    p = tmp_path / "f452.jsonl"
+    _write_jsonl(p, [FORUM_ROW] + RAW_ROWS)
+    return p
+
+
+@pytest.fixture()
+def jobs_dir(tmp_path) -> Path:
+    d = tmp_path / "jobs" / "f452-1"
+    d.mkdir(parents=True)
+    _write_jsonl(d / "job1-output.jsonl", ANALYSIS_ROWS)
+    return d
+
+
+# ---------------------------------------------------------------------------
+# load_raw
+
+
+def test_load_raw_returns_only_comments(raw_jsonl):
+    df = vl.load_raw(raw_jsonl)
+    assert len(df) == 3
+    assert set(df.columns) == set(vl.RAW_COLS)
+
+
+def test_load_raw_correct_columns(raw_jsonl):
+    df = vl.load_raw(raw_jsonl)
+    for col in vl.RAW_COLS:
+        assert col in df.columns
+
+
+def test_load_raw_skips_forum_item(raw_jsonl):
+    df = vl.load_raw(raw_jsonl)
+    assert "reg_title" not in df.columns
+
+
+# ---------------------------------------------------------------------------
+# load_analysis
+
+
+def test_load_analysis_skips_raw_files(tmp_path):
+    d = tmp_path / "jobs" / "f452-1"
+    d.mkdir(parents=True)
+    _write_jsonl(d / "job1-output-raw.jsonl", ANALYSIS_ROWS)   # should be ignored
+    _write_jsonl(d / "job1-output.jsonl", ANALYSIS_ROWS)
+    df = vl.load_analysis(d)
+    assert len(df) == len(ANALYSIS_ROWS)
+
+
+def test_load_analysis_concatenates_multiple_files(tmp_path):
+    d = tmp_path / "jobs" / "f452-1"
+    d.mkdir(parents=True)
+    _write_jsonl(d / "job1-output.jsonl", [ANALYSIS_ROWS[0]])
+    _write_jsonl(d / "job2-output.jsonl", [ANALYSIS_ROWS[1]])
+    df = vl.load_analysis(d)
+    assert len(df) == 2
+
+
+def test_load_analysis_tags_serialized_as_json(jobs_dir):
+    df = vl.load_analysis(jobs_dir)
+    tags_val = df.loc[df["comment_id"] == "1", "tags"].iloc[0]
+    assert isinstance(tags_val, str)
+    assert json.loads(tags_val) == ["student safety"]
+
+
+def test_load_analysis_empty_tags_serialized(jobs_dir):
+    df = vl.load_analysis(jobs_dir)
+    tags_val = df.loc[df["comment_id"] == "2", "tags"].iloc[0]
+    assert json.loads(tags_val) == []
+
+
+# ---------------------------------------------------------------------------
+# join — by comment_id, not index
+
+
+def test_join_by_comment_id_not_index(raw_jsonl, jobs_dir):
+    raw      = vl.load_raw(raw_jsonl)
+    analysis = vl.load_analysis(jobs_dir)
+    # Shuffle raw order so comment_id ordering differs from index
+    raw = raw.sample(frac=1, random_state=42).reset_index(drop=True)
+    merged = vl.join(raw, analysis)
+    row_1 = merged[merged["comment_id"] == "1"].iloc[0]
+    assert row_1["stance"] == "support"
+    assert row_1["author"] == "Alice"
+
+
+def test_join_unanalyzed_comment_has_null_stance(raw_jsonl, jobs_dir):
+    """Comment 3 is in raw but not in analysis — stance should be NaN."""
+    raw      = vl.load_raw(raw_jsonl)
+    analysis = vl.load_analysis(jobs_dir)
+    merged   = vl.join(raw, analysis)
+    row_3 = merged[merged["comment_id"] == "3"].iloc[0]
+    assert pd.isna(row_3["stance"])
+
+
+def test_join_preserves_all_raw_comments(raw_jsonl, jobs_dir):
+    raw      = vl.load_raw(raw_jsonl)
+    analysis = vl.load_analysis(jobs_dir)
+    merged   = vl.join(raw, analysis)
+    assert len(merged) == len(raw)
+
+
+def test_join_output_columns_in_order(raw_jsonl, jobs_dir):
+    raw      = vl.load_raw(raw_jsonl)
+    analysis = vl.load_analysis(jobs_dir)
+    merged   = vl.join(raw, analysis)
+    assert list(merged.columns) == vl.OUTPUT_COLS
+
+
+# ---------------------------------------------------------------------------
+# Duplicate comment_id handling
+
+
+def test_duplicate_raw_id_flagged(raw_jsonl, jobs_dir):
+    raw      = vl.load_raw(raw_jsonl)
+    # Manually duplicate a row
+    raw = pd.concat([raw, raw.iloc[[0]]], ignore_index=True)
+    analysis = vl.load_analysis(jobs_dir)
+    merged   = vl.join(raw, analysis)
+    # join still produces a row for each raw row (left join)
+    assert len(merged) == len(raw)
+    assert raw["comment_id"].duplicated().sum() == 1
+
+
+def test_duplicate_analysis_id_produces_extra_rows(raw_jsonl, tmp_path):
+    """Two analysis records for the same comment_id create two joined rows."""
+    d = tmp_path / "jobs" / "f452-dup"
+    d.mkdir(parents=True)
+    dup_rows = [ANALYSIS_ROWS[0], {**ANALYSIS_ROWS[0], "stance": "oppose"}]
+    _write_jsonl(d / "job1-output.jsonl", dup_rows)
+    raw      = vl.load_raw(raw_jsonl)
+    analysis = vl.load_analysis(d)
+    merged   = vl.join(raw, analysis)
+    assert len(merged[merged["comment_id"] == "1"]) == 2
+
+
+# ---------------------------------------------------------------------------
+# Validation counts (smoke test — just confirm it runs without error)
+
+
+def test_print_validation_runs(raw_jsonl, jobs_dir, capsys):
+    raw      = vl.load_raw(raw_jsonl)
+    analysis = vl.load_analysis(jobs_dir)
+    merged   = vl.join(raw, analysis)
+    vl.print_validation(raw, analysis, merged)
+    out = capsys.readouterr().out
+    assert "Raw comments" in out
+    assert "Stance counts" in out
+    assert "Tone counts" in out
+
+
+# ---------------------------------------------------------------------------
+# CSV output
+
+
+def test_csv_written_to_jobs_dir(raw_jsonl, jobs_dir, tmp_path):
+    raw      = vl.load_raw(raw_jsonl)
+    analysis = vl.load_analysis(jobs_dir)
+    merged   = vl.join(raw, analysis)
+    out_path = jobs_dir / "review.csv"
+    merged.to_csv(out_path, index=False, encoding="utf-8-sig")
+    assert out_path.exists()
+    loaded = pd.read_csv(out_path, encoding="utf-8-sig")
+    assert list(loaded.columns) == vl.OUTPUT_COLS
+    assert len(loaded) == len(raw)