115 lines
5.9 KiB
Org Mode
115 lines
5.9 KiB
Org Mode
#+title: VA Townhall
|
|
#+date: [2026-05-05 Tue]
|
|
#+version: 1.1
|
|
|
|
** Project Goals
|
|
1. Document and analyze sentiment of public comments on Virginia law, to determine:
|
|
1. the utility of this forum as a mechanism for public comment, and
|
|
2. the impact of this forum on Virginia regulation.
|
|
2. Make data and insights broadly available.
|
|
3. Generalize to other public comment tools.
|
|
|
|
*** Research questions
|
|
1. What is the quality of the comments on the forum?
|
|
1. Are there duplicate entries?
|
|
2. Are there non-human-generated entries?
|
|
3. Are there entries intended to abuse the forum or drown out comment?
|
|
2. How do commenters feel about the proposed change?
|
|
1. What is the total number and percent supporting vs opposing, and how does this change over time?
|
|
2. What is the type of support, such as strong/weak, positive/negative?
|
|
3. What impact do the comments have on the proposed change?
|
|
(I anticipate this will not be measurable from currently available data)
|
|
|
|
** Architecture
|
|
1. Scrape/Parse: Scrapy
|
|
2. Sentiment analysis: gpt-5.4-mini
|
|
3. Display: streamlit
|
|
4. Storage: jsonl, csv, parquet
|
|
|
|
[[file://./pipeline-v1.2.3.svg]]
|
|
|
|
*** Scraper
|
|
Scrapy provides a simple mechanism for retrieving, parsing, and saving content form the forums.
|
|
1. Forums listing page: `Forums.cfm` - lists all open forums with agency, reg title, action type, brief description, closing date, comment count
|
|
2. Comment listing page: `comments.cfm?GDocForumID=X` or `comments.cfm?stageid=X` or `comments.cfm?petitionid=X` - lists comments with title, author, date
|
|
3. Individual comment page: `viewcomments.cfm?commentid=X` - shows regulation title + brief description at the top, plus the comment
|
|
|
|
*** Analysis
|
|
Google and Amazon both return generic sentiment (tone of writing: positive/negative), not stance (for/against the regulation): "I strongly believe the government should NOT interfere" is negative tone but "against" the regulation. We add the proposed change as context to the model.
|
|
|
|
Before sending the comments for sentiment analysis, `tokenizer.py` receives the forum to be processed and prompt as inputs, then generates a `report.json` estimating tokens (tiktoken), cost, and time to run for multiple models.
|
|
|
|
Then, the batch processing scripts uses the `report.json` to create multiple jobs, with subcommands to download and check their status.
|
|
|
|
We selected gpt-5.4-mini for a good balance of quality, cost, and time.
|
|
|
|
**** Prompt
|
|
```
|
|
You are an expert policy analyst classifying public comments submitted to the Virginia Town Hall
|
|
regulatory comment system. You will be given the text of a proposed regulation and a single
|
|
public comment. Return ONLY a JSON object — no other text.
|
|
|
|
Definitions:
|
|
- stance: the commenter's position on whether the regulation should be adopted.
|
|
"support" = wants it approved (as-is or with changes);
|
|
"oppose" = wants it rejected or substantially weakened;
|
|
"neutral" = takes no position, asks a question, or provides factual input only;
|
|
"unknown" = too vague, off-topic, or uninterpretable to classify.
|
|
- tone: the emotional register of the writing, independent of stance.
|
|
"positive" = affirming, hopeful, appreciative;
|
|
"negative" = angry, fearful, alarmed, or contemptuous;
|
|
"neutral" = matter-of-fact, procedural, or informational;
|
|
"mixed" = contains both positive and negative emotional content;
|
|
"unclear" = tone cannot be determined (e.g., a one-word comment).
|
|
- stance_confidence: float 0.0-1.0, your confidence in the stance label.
|
|
- stance_rationale: 1-3 sentences explaining the key evidence; quote specific phrases where possible.
|
|
- tags: up to 5 short topic labels relevant to the comment's specific concerns (e.g.
|
|
"parental rights", "student safety", "privacy", "religious freedom", "LGBTQ+ inclusion",
|
|
"bullying prevention", "school sports", "bathroom access"). Empty array if none apply.
|
|
|
|
Return exactly these keys: stance, stance_confidence, stance_rationale, tone, tags.
|
|
```
|
|
|
|
|
|
*** Storage
|
|
- Each scraped forum is saved to `output/<forum-id>.jsonl`
|
|
- Each report (forum + prompt) is saves to `reports/<forum-id-N>.json`
|
|
- Each job is saved to `analysis/jobs/<report-id>/:
|
|
└─`forum.jsonl` is a copy of the scraped forum for convenience
|
|
└─`prompt.txt` is a copy of the prompt used
|
|
└─`report.json` is a copy of the report used
|
|
└─`status.json` contains metadata about the job
|
|
For each batch in the job, four files are created:
|
|
└─`jobN-input.jsonl` contains the exact queries sent to the API, for troubleshooting
|
|
└─`jobN-output-raw.jsonl` contains the exact response from the API
|
|
└─`jobN-output.jsonl` contains the exact response from the API
|
|
└─`jobN-output-errors.jsonl` when errors are returned (this file may not exist)
|
|
- Once complete, the cleanup script saves `review.csv`, `review.pqt`, and `review.sqlite` in this folder.
|
|
|
|
** Instructions
|
|
1. Scrape the forum.
|
|
`python
|
|
2. Run model report.
|
|
`python analysis/tokenizer.py <input> --prompt <prompt>`
|
|
3. To run a realtime subset:
|
|
`python analysis/openai_realtime.py <input> --prompt <prompt> --model <model> --limit <N comments>`
|
|
`python analysis/openai_realtime.py output/f452.jsonl --prompt prompt-1.txt --model gpt-4o-mini --limit 10`
|
|
4. To create and run the whole thing in batches, first create the batch jobs from the report:
|
|
`python analysis/openai_batch.py create <report> --model <model>`
|
|
`python analysis/openai_batch.py create ./reports/f452-1.json --model gpt-5.4-mini`
|
|
5. Then, run the jobs sequentially. Don't submit more than one at a time, if the model fills up the batch will fail and resubmission is not implemented.
|
|
`python analysis/openai_batch.py submit`
|
|
# Check status
|
|
`python analysis/openai_batch.py status`
|
|
# When complete, download:
|
|
`python analysis/openai_batch.py download`
|
|
# Submit the next batch after the previous is complete:
|
|
`python analysis/openai_batch.py submit`
|
|
|
|
* Roadmap
|
|
1. Scrape one forum
|
|
2. Compare sentiment models
|
|
3. Display
|
|
4. Scrape all data
|
|
5. Scale?
|