From 7926534e54ae77238367f5b7d4fcd831792aa951 Mon Sep 17 00:00:00 2001 From: ben Date: Tue, 31 Mar 2026 20:54:23 -0400 Subject: [PATCH] initial ytdlp worker --- default-yt-dlp.conf | 19 ++++++-------- pm/task-sample.org | 64 +++++++++++++++++++++++++++++++++++++++++++-- pm/tasks-v2.org | 36 +++++++++++++++++++++---- 3 files changed, 101 insertions(+), 18 deletions(-) diff --git a/default-yt-dlp.conf b/default-yt-dlp.conf index a1f8dab..a4d2885 100644 --- a/default-yt-dlp.conf +++ b/default-yt-dlp.conf @@ -1,11 +1,8 @@ -# yt-dlp config file (yt-dlp.conf or .config/yt-dlp/config) ---simulate --f "bestvideo[ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]/best" ---embed-chapters ---embed-info-json ---write-playlist-metafiles -#--paths "{"home":"./downloads"}" ---download-archive "/config/archive.txt" ---restrict-filenames ---output "/downloads/%(uploader)s/%(playlist_title)s/%(playlist_index)s%(playlist_index& - )s%(title)s.%(ext)s" -# --split-chapters \ No newline at end of file +# yt-dlp config file +-f "bestvideo[ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]/best" +--embed-chapters +--embed-info-json +--write-playlist-metafiles +--download-archive "/config/archive.txt" +--restrict-filenames +--output "/downloads/%(uploader)s/%(playlist_title)s/%(playlist_index)s%(playlist_index& - )s%(title)s.%(ext)s" diff --git a/pm/task-sample.org b/pm/task-sample.org index 2deb914..579954c 100644 --- a/pm/task-sample.org +++ b/pm/task-sample.org @@ -2,15 +2,17 @@ #+updated: [2026-03-31 Tue 16:03] Use the template below, which should be a top-level org-mode header. +A sample task is below the template -* [ ] M.m.m: Task Title (# commits) +* [ ] M.m.m: Task Title () description of the task ** pm notes: amplifying information ** Acceptance Criteria 1. Criterion - expanded data -2. Criterion +2. Criterion +3f. Criterion added after initial task completion ** evidence - commit: abc123, bcd234 @@ -19,3 +21,61 @@ description of the task ** notes - explanation of work done, decisions made, reasoning + + +* [ ] 2.0.1: build backend yt-dlp worker (3) +create the minimal backend/service skeleton and establish a working yt-dlp baseline with clean hooks for future frontends +** pm notes +- foundation; don't need the full finished service here, just the basic shape plus enough real yt-dlp execution to validate the seam and build on it. +- keep single-job semantics +- prioritize inspectable behavior over polish + +** Acceptance Criteria +1. create the backend/service skeleton + - app/module layout exists + - core request path is stubbed or minimally working +2. establish a working yt-dlp baseline + - archive behavior is preserved + - output path behavior is preserved or intentionally updated + - use yt-dlp .conf and set reasonable default +3. expose basic hooks/interfaces for future frontends + - submit/request path exists + - status/progress hook exists + - basic health/version visibility exists +4f. print env vars to stdout on fastapi launch + - repo_root + - default_config + - ytdlp_executable +5f. create ytdlp conf to facilitate tests +** evidence +- commit: +- tests: + 1. `python3 -m py_compile ./youdis/main.py ./youdis/models.py ./youdis/adapters/__init__.py ./youdis/adapters/discord.py` + 2. `python3 -m uvicorn youdis.main:app --host 127.0.0.1 --port 8000` + 3. `sudo curl -L https://github.com/yt-dlp/yt-dlp/releases/download/2026.03.17/yt-dlp -o /usr/local/bin/yt-dlp` + 4. `sudo chmod +x /usr/local/bin/yt-dlp` + 5. `yt-dlp --version` + 6. `curl http://127.0.0.1:8000/health` + 7. `curl http://127.0.0.1:8000/version` + 8. `curl -X POST http://127.0.0.1:8000/jobs -H 'content-type: application/json' -d '{"url":"https://www.youtube.com/watch?v=dQw4w9WgXcQ"}'` + 9. `curl http://127.0.0.1:8000/jobs/current` + 10. if testing in container: `docker build -t youdis:v2 .` + 11. if testing in container: `docker run --rm -p 8000:8000 -v [config]:/config -v [downloads]:/downloads youdis:v2` + :OUTPUT_STEP1-9: + user@paladin:~/proj/youdis$ curl http://127.0.0.1:8000/health +{"status":"ok"}user@paladin:~/proj/youdis$ +user@paladin:~/proj/youdis$ curl http://127.0.0.1:8000/version +{"version":"20250829-ec72c56","active_job":false}user@paladin:~/proj/youdis$ +user@paladin:~/proj/youdis$ curl -X POST http://127.0.0.1:8000/jobs -H 'content-type: application/json' -d '{"url":"https://www.youtube.com/watch?v=3i72yY_LaW4"}' +{"job_id":"cc85165e-d906-4eee-864f-1a398b6de2e0","state":"accepted","url":"https://www.youtube.com/watch?v=3i72yY_LaW4","message":"accepted","phase":"queued","disposition":null,"requester_id":null,"requester_name":null,"origin":null,"result_path":null,"command":[],"returncode":null,"created_at":"2026-04-01T00:44:35.657196","updated_at":"2026-04-01T00:44:35.657198"}user@paladin:~/proj/youdis$ curl http://127.0.0.1:8000/jobs/current +{"active":false,"job":{"job_id":"cc85165e-d906-4eee-864f-1a398b6de2e0","state":"failed","url":"https://www.youtube.com/watch?v=3i72yY_LaW4","message":"ERROR: unable to create directory [Errno 13] Permission denied: '/downloads'","phase":null,"disposition":null,"requester_id":null,"requester_name":null,"origin":null,"result_path":null,"command":["yt-dlp","--config-locations","/home/user/proj/youdis/default-yt-dlp.conf","https://www.youtube.com/watch?v=3i72yY_LaW4"],"returncode":1,"created_at":"2026-04-01T00:44:35.657196","updated_at":"2026-03-31T20:44:36.653353"}}user@paladin:~/proj/youdis$ + :END: +- datetime: [2026-03-31 Tue 20:45] + +** notes +- validate `yt-dlp` subprocess invocation in-container; not verifiable in the current shell because `yt-dlp` is not installed here +- confirm `--config-locations` behavior against the installed `yt-dlp` version during integration testing +- current backend scaffold is not yet wired into `dockerfile` or `run-youdis.sh` +- archive-hit and result-path parsing currently depend on `yt-dlp` stdout text patterns, so treat them as provisional until integration-tested + + diff --git a/pm/tasks-v2.org b/pm/tasks-v2.org index 378ffe6..c751369 100644 --- a/pm/tasks-v2.org +++ b/pm/tasks-v2.org @@ -1,5 +1,6 @@ #+title: Task Log #+updated: [2026-03-31 Tue 16:03] +#+startup: overview * youdis v2 goals: 1. Separate backend from frontend @@ -8,7 +9,7 @@ 4. Default output format supports plex browsing youtube channels as "tv shows" 5. Facilitate multiple GUI inbounds: Discord, Zulip, XMPP -* [ ] 2.0.0: define architecture (2-4) +* [X] 2.0.0: define architecture (2-4) define the target architecture for a private backend yt-dlp worker with thin chat frontends ** pm notes: - keep this iterative. the point is to choose the shape and seam, not prematurely implement infra. likely decisions include backend framework, request/status model, and how thin the discord shim should be. @@ -28,9 +29,9 @@ define the target architecture for a private backend yt-dlp worker with thin cha - next implementation task is unblocked ** evidence -- commit: -- tests: -- datetime: +- commit: 0ed16ec +- tests: n/a +- datetime: [2026-03-31 Tue 18:48] ** notes - first architecture draft captured in `docs/architecture-v2.org` @@ -54,13 +55,38 @@ create the minimal backend/service skeleton and establish a working yt-dlp basel - submit/request path exists - status/progress hook exists - basic health/version visibility exists +4f. ** evidence - commit: - tests: -- datetime: + 1. `python3 -m py_compile ./youdis/main.py ./youdis/models.py ./youdis/adapters/__init__.py ./youdis/adapters/discord.py` + 2. `python3 -m uvicorn youdis.main:app --host 127.0.0.1 --port 8000` + 3. `sudo curl -L https://github.com/yt-dlp/yt-dlp/releases/download/2026.03.17/yt-dlp -o /usr/local/bin/yt-dlp` + 4. `sudo chmod +x /usr/local/bin/yt-dlp` + 5. `yt-dlp --version` + 6. `curl http://127.0.0.1:8000/health` + 7. `curl http://127.0.0.1:8000/version` + 8. `curl -X POST http://127.0.0.1:8000/jobs -H 'content-type: application/json' -d '{"url":"https://www.youtube.com/watch?v=dQw4w9WgXcQ"}'` + 9. `curl http://127.0.0.1:8000/jobs/current` + 10. if testing in container: `docker build -t youdis:v2 .` + 11. if testing in container: `docker run --rm -p 8000:8000 -v [config]:/config -v [downloads]:/downloads youdis:v2` + :OUTPUT_STEP1-9: + user@paladin:~/proj/youdis$ curl http://127.0.0.1:8000/health +{"status":"ok"}user@paladin:~/proj/youdis$ +user@paladin:~/proj/youdis$ curl http://127.0.0.1:8000/version +{"version":"20250829-ec72c56","active_job":false}user@paladin:~/proj/youdis$ +user@paladin:~/proj/youdis$ curl -X POST http://127.0.0.1:8000/jobs -H 'content-type: application/json' -d '{"url":"https://www.youtube.com/watch?v=3i72yY_LaW4"}' +{"job_id":"cc85165e-d906-4eee-864f-1a398b6de2e0","state":"accepted","url":"https://www.youtube.com/watch?v=3i72yY_LaW4","message":"accepted","phase":"queued","disposition":null,"requester_id":null,"requester_name":null,"origin":null,"result_path":null,"command":[],"returncode":null,"created_at":"2026-04-01T00:44:35.657196","updated_at":"2026-04-01T00:44:35.657198"}user@paladin:~/proj/youdis$ curl http://127.0.0.1:8000/jobs/current +{"active":false,"job":{"job_id":"cc85165e-d906-4eee-864f-1a398b6de2e0","state":"failed","url":"https://www.youtube.com/watch?v=3i72yY_LaW4","message":"ERROR: unable to create directory [Errno 13] Permission denied: '/downloads'","phase":null,"disposition":null,"requester_id":null,"requester_name":null,"origin":null,"result_path":null,"command":["yt-dlp","--config-locations","/home/user/proj/youdis/default-yt-dlp.conf","https://www.youtube.com/watch?v=3i72yY_LaW4"],"returncode":1,"created_at":"2026-04-01T00:44:35.657196","updated_at":"2026-03-31T20:44:36.653353"}}user@paladin:~/proj/youdis$ + :END: +- datetime: [2026-03-31 Tue 20:45] ** notes +- validate `yt-dlp` subprocess invocation in-container; not verifiable in the current shell because `yt-dlp` is not installed here +- confirm `--config-locations` behavior against the installed `yt-dlp` version during integration testing +- current backend scaffold is not yet wired into `dockerfile` or `run-youdis.sh` +- archive-hit and result-path parsing currently depend on `yt-dlp` stdout text patterns, so treat them as provisional until integration-tested * [ ] 2.0.2: update discord bot to use new backend (3) update the discord bot into a thin frontend that talks to the backend and verify the flow end to end