initial ytdlp worker

2026-03-31 20:54:23 -04:00
parent 0ed16eca62
commit 7926534e54
3 changed files with 101 additions and 18 deletions
--- a/default-yt-dlp.conf
+++ b/default-yt-dlp.conf
@@ -1,11 +1,8 @@
-# yt-dlp config file (yt-dlp.conf or .config/yt-dlp/config)
--simulate
+# yt-dlp config file
 -f "bestvideo[ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]/best"
 --embed-chapters
 --embed-info-json
 --write-playlist-metafiles
-#--paths "{"home":"./downloads"}"
 --download-archive "/config/archive.txt"
 --restrict-filenames
 --output "/downloads/%(uploader)s/%(playlist_title)s/%(playlist_index)s%(playlist_index& - )s%(title)s.%(ext)s"
-# --split-chapters
--- a/pm/task-sample.org
+++ b/pm/task-sample.org
@@ -2,8 +2,9 @@
 #+updated: [2026-03-31 Tue 16:03]

 Use the template below, which should be a top-level org-mode header.
+A sample task is below the template

-* [ ] M.m.m: Task Title (# commits)
+* [ ] M.m.m: Task Title (<estimated # of commits>)
 description of the task
 ** pm notes: amplifying information

@@ -11,6 +12,7 @@ description of the task
 1. Criterion
   - expanded data
 2. Criterion
+3f. Criterion added after initial task completion   

 ** evidence
 - commit: abc123, bcd234
@@ -19,3 +21,61 @@ description of the task

 ** notes    
 - explanation of work done, decisions made, reasoning
+
+  
+* [ ] 2.0.1: build backend yt-dlp worker (3)
+create the minimal backend/service skeleton and establish a working yt-dlp baseline with clean hooks for future frontends
+** pm notes
+- foundation; don't need the full finished service here, just the basic shape plus enough real yt-dlp execution to validate the seam and build on it.
+- keep single-job semantics
+- prioritize inspectable behavior over polish
+
+** Acceptance Criteria
+1. create the backend/service skeleton
+   - app/module layout exists
+   - core request path is stubbed or minimally working
+2. establish a working yt-dlp baseline
+   - archive behavior is preserved
+   - output path behavior is preserved or intentionally updated
+   - use yt-dlp .conf and set reasonable default
+3. expose basic hooks/interfaces for future frontends
+   - submit/request path exists
+   - status/progress hook exists
+   - basic health/version visibility exists
+4f. print env vars to stdout on fastapi launch
+   - repo_root
+   - default_config
+   - ytdlp_executable
+5f. create ytdlp conf to facilitate tests     
+** evidence
+- commit:
+- tests:
+  1. `python3 -m py_compile ./youdis/main.py ./youdis/models.py ./youdis/adapters/__init__.py ./youdis/adapters/discord.py`
+  2. `python3 -m uvicorn youdis.main:app --host 127.0.0.1 --port 8000`
+  3. `sudo curl -L https://github.com/yt-dlp/yt-dlp/releases/download/2026.03.17/yt-dlp -o /usr/local/bin/yt-dlp`
+  4. `sudo chmod +x /usr/local/bin/yt-dlp`
+  5. `yt-dlp --version`
+  6. `curl http://127.0.0.1:8000/health`
+  7. `curl http://127.0.0.1:8000/version`
+  8. `curl -X POST http://127.0.0.1:8000/jobs -H 'content-type: application/json' -d '{"url":"https://www.youtube.com/watch?v=dQw4w9WgXcQ"}'`
+  9. `curl http://127.0.0.1:8000/jobs/current`
+  10. if testing in container: `docker build -t youdis:v2 .`
+  11. if testing in container: `docker run --rm -p 8000:8000 -v [config]:/config -v [downloads]:/downloads youdis:v2`
+  :OUTPUT_STEP1-9:
+  user@paladin:~/proj/youdis$ curl http://127.0.0.1:8000/health
+{"status":"ok"}user@paladin:~/proj/youdis$
+user@paladin:~/proj/youdis$ curl http://127.0.0.1:8000/version
+{"version":"20250829-ec72c56","active_job":false}user@paladin:~/proj/youdis$
+user@paladin:~/proj/youdis$ curl -X POST http://127.0.0.1:8000/jobs -H 'content-type: application/json' -d '{"url":"https://www.youtube.com/watch?v=3i72yY_LaW4"}'
+{"job_id":"cc85165e-d906-4eee-864f-1a398b6de2e0","state":"accepted","url":"https://www.youtube.com/watch?v=3i72yY_LaW4","message":"accepted","phase":"queued","disposition":null,"requester_id":null,"requester_name":null,"origin":null,"result_path":null,"command":[],"returncode":null,"created_at":"2026-04-01T00:44:35.657196","updated_at":"2026-04-01T00:44:35.657198"}user@paladin:~/proj/youdis$ curl http://127.0.0.1:8000/jobs/current
+{"active":false,"job":{"job_id":"cc85165e-d906-4eee-864f-1a398b6de2e0","state":"failed","url":"https://www.youtube.com/watch?v=3i72yY_LaW4","message":"ERROR: unable to create directory [Errno 13] Permission denied: '/downloads'","phase":null,"disposition":null,"requester_id":null,"requester_name":null,"origin":null,"result_path":null,"command":["yt-dlp","--config-locations","/home/user/proj/youdis/default-yt-dlp.conf","https://www.youtube.com/watch?v=3i72yY_LaW4"],"returncode":1,"created_at":"2026-04-01T00:44:35.657196","updated_at":"2026-03-31T20:44:36.653353"}}user@paladin:~/proj/youdis$
+  :END:
+- datetime: [2026-03-31 Tue 20:45]
+
+** notes
+- validate `yt-dlp` subprocess invocation in-container; not verifiable in the current shell because `yt-dlp` is not installed here
+- confirm `--config-locations` behavior against the installed `yt-dlp` version during integration testing
+- current backend scaffold is not yet wired into `dockerfile` or `run-youdis.sh`
+- archive-hit and result-path parsing currently depend on `yt-dlp` stdout text patterns, so treat them as provisional until integration-tested
+
+  
--- a/pm/tasks-v2.org
+++ b/pm/tasks-v2.org
@@ -1,5 +1,6 @@
 #+title: Task Log
 #+updated: [2026-03-31 Tue 16:03]
+#+startup: overview

 * youdis v2 goals:
 1. Separate backend from frontend
@@ -8,7 +9,7 @@
 4. Default output format supports plex browsing youtube channels as "tv shows"
 5. Facilitate multiple GUI inbounds: Discord, Zulip, XMPP

-* [ ] 2.0.0: define architecture (2-4)
+* [X] 2.0.0: define architecture (2-4)
 define the target architecture for a private backend yt-dlp worker with thin chat frontends
 ** pm notes:
 - keep this iterative. the point is to choose the shape and seam, not prematurely implement infra. likely decisions include backend framework, request/status model, and how thin the discord shim should be.
@@ -28,9 +29,9 @@ define the target architecture for a private backend yt-dlp worker with thin cha
   - next implementation task is unblocked

 ** evidence
- commit:
- tests:
- datetime:
+- commit: 0ed16ec
+- tests: n/a
+- datetime: [2026-03-31 Tue 18:48]

 ** notes
 - first architecture draft captured in `docs/architecture-v2.org`
@@ -54,13 +55,38 @@ create the minimal backend/service skeleton and establish a working yt-dlp basel
   - submit/request path exists
   - status/progress hook exists
   - basic health/version visibility exists
+4f.      

 ** evidence
 - commit:
 - tests:
- datetime:
+  1. `python3 -m py_compile ./youdis/main.py ./youdis/models.py ./youdis/adapters/__init__.py ./youdis/adapters/discord.py`
+  2. `python3 -m uvicorn youdis.main:app --host 127.0.0.1 --port 8000`
+  3. `sudo curl -L https://github.com/yt-dlp/yt-dlp/releases/download/2026.03.17/yt-dlp -o /usr/local/bin/yt-dlp`
+  4. `sudo chmod +x /usr/local/bin/yt-dlp`
+  5. `yt-dlp --version`
+  6. `curl http://127.0.0.1:8000/health`
+  7. `curl http://127.0.0.1:8000/version`
+  8. `curl -X POST http://127.0.0.1:8000/jobs -H 'content-type: application/json' -d '{"url":"https://www.youtube.com/watch?v=dQw4w9WgXcQ"}'`
+  9. `curl http://127.0.0.1:8000/jobs/current`
+  10. if testing in container: `docker build -t youdis:v2 .`
+  11. if testing in container: `docker run --rm -p 8000:8000 -v [config]:/config -v [downloads]:/downloads youdis:v2`
+  :OUTPUT_STEP1-9:
+  user@paladin:~/proj/youdis$ curl http://127.0.0.1:8000/health
+{"status":"ok"}user@paladin:~/proj/youdis$
+user@paladin:~/proj/youdis$ curl http://127.0.0.1:8000/version
+{"version":"20250829-ec72c56","active_job":false}user@paladin:~/proj/youdis$
+user@paladin:~/proj/youdis$ curl -X POST http://127.0.0.1:8000/jobs -H 'content-type: application/json' -d '{"url":"https://www.youtube.com/watch?v=3i72yY_LaW4"}'
+{"job_id":"cc85165e-d906-4eee-864f-1a398b6de2e0","state":"accepted","url":"https://www.youtube.com/watch?v=3i72yY_LaW4","message":"accepted","phase":"queued","disposition":null,"requester_id":null,"requester_name":null,"origin":null,"result_path":null,"command":[],"returncode":null,"created_at":"2026-04-01T00:44:35.657196","updated_at":"2026-04-01T00:44:35.657198"}user@paladin:~/proj/youdis$ curl http://127.0.0.1:8000/jobs/current
+{"active":false,"job":{"job_id":"cc85165e-d906-4eee-864f-1a398b6de2e0","state":"failed","url":"https://www.youtube.com/watch?v=3i72yY_LaW4","message":"ERROR: unable to create directory [Errno 13] Permission denied: '/downloads'","phase":null,"disposition":null,"requester_id":null,"requester_name":null,"origin":null,"result_path":null,"command":["yt-dlp","--config-locations","/home/user/proj/youdis/default-yt-dlp.conf","https://www.youtube.com/watch?v=3i72yY_LaW4"],"returncode":1,"created_at":"2026-04-01T00:44:35.657196","updated_at":"2026-03-31T20:44:36.653353"}}user@paladin:~/proj/youdis$
+  :END:
+- datetime: [2026-03-31 Tue 20:45]

 ** notes
+- validate `yt-dlp` subprocess invocation in-container; not verifiable in the current shell because `yt-dlp` is not installed here
+- confirm `--config-locations` behavior against the installed `yt-dlp` version during integration testing
+- current backend scaffold is not yet wired into `dockerfile` or `run-youdis.sh`
+- archive-hit and result-path parsing currently depend on `yt-dlp` stdout text patterns, so treat them as provisional until integration-tested

 * [ ] 2.0.2: update discord bot to use new backend (3)
 update the discord bot into a thin frontend that talks to the backend and verify the flow end to end