ben/youdis
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
17573185236a50f99f070a7c65dc3b6f2b2987dd
youdis/pm/tasks-v2.org
ben 1757318523 consolidated env vars and .env
2026-04-02 17:14:05 -04:00

350 lines
17 KiB
Org Mode
Raw Blame History

#+title: Task Log
#+updated: [2026-03-31 Tue 16:03]
#+startup: overview
# Local Variables:
# org-babel-python-command: "/home/user/proj/youdis/.venv/bin/python"
# project-dir: "~/proj/youdis"
# End:
* youdis v2 goals:
1. Separate backend from frontend
2. Offload auth
3. Ensure auto nightly builds
4. Default output format supports plex browsing youtube channels as "tv shows"
5. Facilitate multiple GUI inbounds: Discord, Zulip, XMPP
* [X] 2.0.0: define architecture (2-4)
define the target architecture for a private backend yt-dlp worker with thin chat frontends
** pm notes:
- keep this iterative. the point is to choose the shape and seam, not prematurely implement infra. likely decisions include backend framework, request/status model, and how thin the discord shim should be.
- goal is simple, private, maintainable deployment
- avoid auth, queues, or persistence beyond clear & immediate needs
** Acceptance Criteria
1. document the target architecture at a high level
- backend owns yt-dlp execution and job state
- frontends own chat-specific UX
2. identify key decisions still open
- backend choice
- service seam/endpoints
- status/progress model
3. capture enough structure to begin implementation
- repo/component layout is sketched
- next implementation task is unblocked
** evidence
- commit: 0ed16ec
- tests: n/a
- datetime: [2026-03-31 Tue 18:48]
** notes
- first architecture draft captured in `docs/architecture-v2.org`
* [X] 2.0.1: build backend yt-dlp worker (3)
create the minimal backend/service skeleton and establish a working yt-dlp baseline with clean hooks for future frontends
** pm notes
- foundation; don't need the full finished service here, just the basic shape plus enough real yt-dlp execution to validate the seam and build on it.
- keep single-job semantics
- prioritize inspectable behavior over polish
** Acceptance Criteria
1. create the backend/service skeleton
- app/module layout exists
- core request path is stubbed or minimally working
2. establish a working yt-dlp baseline
- archive behavior is preserved
- output path behavior is preserved or intentionally updated
- use yt-dlp .conf and set reasonable default
3. expose basic hooks/interfaces for future frontends
- submit/request path exists
- status/progress hook exists
- basic health/version visibility exists
4. make local testing practical without breaking container defaults
- backend can run when `/config` or `/downloads` are not writable in local dev
- env vars may override executable/config/download paths
- status makes the effective runtime command/paths inspectable
** evidence
- commit: 7926534,2a56485
- tests:
1. `python3 -m py_compile ./youdis/main.py ./youdis/models.py ./youdis/adapters/__init__.py ./youdis/adapters/discord.py`
2. `python3 -m uvicorn youdis.main:app --host 127.0.0.1 --port 8000`
3. `sudo curl -L https://github.com/yt-dlp/yt-dlp/releases/download/2026.03.17/yt-dlp -o /usr/local/bin/yt-dlp`
4. `sudo chmod +x /usr/local/bin/yt-dlp`
5. `yt-dlp --version`
6. `curl http://127.0.0.1:8000/health`
7. `curl http://127.0.0.1:8000/version`
8. `curl -X POST http://127.0.0.1:8000/jobs -H 'content-type: application/json' -d '{"url":"https://www.youtube.com/watch?v=dQw4w9WgXcQ"}'`
9. `curl http://127.0.0.1:8000/jobs/current`
10. if testing in container: `docker build -t youdis:v2 .`
11. if testing in container: `docker run --rm -p 8000:8000 -v [config]:/config -v [downloads]:/downloads youdis:v2`
:OUTPUT_STEP1-9:
user@paladin:~/proj/youdis$ curl http://127.0.0.1:8000/health
{"status":"ok"}
user@paladin:~/proj/youdis$ curl http://127.0.0.1:8000/version
{"version":"20250829-ec72c56","active_job":false}
user@paladin:~/proj/youdis$ curl -X POST http://127.0.0.1:8000/jobs -H 'content-type: application/json' -d '{"url":"https://www.youtube.com/watch?v=3i72yY_LaW4"}'
{"job_id":"cc85165e-d906-4eee-864f-1a398b6de2e0","state":"accepted","url":"https://www.youtube.com/watch?v=3i72yY_LaW4","message":"accepted","phase":"queued","disposition":null,"requester_id":null,"requester_name":null,"origin":null,"result_path":null,"command":[],"returncode":null,"created_at":"2026-04-01T00:44:35.657196","updated_at":"2026-04-01T00:44:35.657198"}
user@paladin:~/proj/youdis$ curl http://127.0.0.1:8000/jobs/current
{"active":false,"job":{"job_id":"cc85165e-d906-4eee-864f-1a398b6de2e0","state":"failed","url":"https://www.youtube.com/watch?v=3i72yY_LaW4","message":"ERROR: unable to create directory [Errno 13] Permission denied: '/downloads'","phase":null,"disposition":null,"requester_id":null,"requester_name":null,"origin":null,"result_path":null,"command":["yt-dlp","--config-locations","/home/user/proj/youdis/default-yt-dlp.conf","https://www.youtube.com/watch?v=3i72yY_LaW4"],"returncode":1,"created_at":"2026-04-01T00:44:35.657196","updated_at":"2026-03-31T20:44:36.653353"}}user@paladin:~/proj/youdis$
:END:
- datetime: [2026-03-31 Tue 20:45]
** notes
- validate `yt-dlp` subprocess invocation in-container; not verifiable in the current shell because `yt-dlp` is not installed here
- confirm `--config-locations` behavior against the installed `yt-dlp` version during integration testing
- current backend scaffold is not yet wired into `dockerfile` or `run-youdis.sh`
- archive-hit and result-path parsing currently depend on `yt-dlp` stdout text patterns, so treat them as provisional until integration-tested
- local dev now falls back to repo-local `.runtime/{config,downloads}` when `/config` or `/downloads` are not writable
- had to uninstall yt-dlp python pkg from the venv, which resulted in a '403 Forbidden'
* [X] 2.0.2: update discord bot to use new backend (3)
update the discord bot into a thin frontend that talks to the backend and verify the flow end to end
** pm notes
- this is the first real frontend proof. once this works cleanly, zulip/xmpp should mostly be adapter work rather than downloader rewrites.
- keep discord logic thin; no auth
- do not duplicate yt-dlp behavior in the bot
** Acceptance Criteria
1. discord bot submits requests to backend
- command/input handling works
- acceptance/busy/failure responses are clear
2. discord bot relays useful backend status
- progress reporting works at a basic level
- completion/failure/skipped outcomes are surfaced
3. backend-discord flow is tested end to end
- valid request path tested
- busy or conflict behavior tested
- failure path tested
4. add dotenv support to ease dev
- os.getenv methods remain standard for prod/docker build
- populate .env with dev env defaults
** evidence
- commit: 5210d2c, 043cb4
- tests: https://youtu.be/20HxMMSqRyg?si=3v7mN2L88c_FxpQR 18m
1. start backend: `python3 -m uvicorn youdis.main:app --host 127.0.0.1 --port 8000`
2. create local env file: `cp .env.example .env`
3. add `api_token` to `.env`
4. start adapter: `python3 ./youdis.py`
5. in discord, run `/youtube url:https://www.youtube.com/watch?v=dQw4w9WgXcQ`
:out:
#+begin_src shell
(venv) user@paladin:~/proj/youdis$ python ./youdis.py
Task exception was never retrieved
future: <Task finished name='Task-43' coro=<Client._dispatch_interaction() done, defined at /home/user/proj/youdis/venv/lib/python3.10/site-packages/interactions/client/client.py:1773> exception=KeyError('youtube')>
Traceback (most recent call last):
File "/home/user/proj/youdis/venv/lib/python3.10/site-packages/interactions/client/client.py", line 1798, in _dispatch_interaction
if ctx.command:
File "/home/user/proj/youdis/venv/lib/python3.10/site-packages/interactions/models/internal/context.py", line 329, in command
return self.client._interaction_lookup[self._command_name]
KeyError: 'youtube'
#+end_src
:end:
6. confirm channel response says the job was submitted to backend
7. confirm requester receives DM updates for accepted/running/completed or failed
:out:
accepted job 4ef6fde5-57cc-478c-b0e2-18458c693fa4 for https://www.youtube.com/watch?v=dQw4w9WgXcQ
state=running | phase=running | [youtube] dQw4w9WgXcQ: Downloading webpage | path=/home/user/proj/youdis/downloads
state=running | phase=downloading | [download] Destination: /home/user/proj/youdis/downloads/Rick_Astley/NA/NANARickAstley-_Never_Gonna_Give_You_Up_Official_Video_4K_Remaster.f401.mp4 | path=/home/user/proj/youdis/downloads/Rick_Astley/NA/NANARickAstley-_Never_Gonna_Give_You_Up_Official_Video_4K_Remaster.f401.mp4
state=completed | [Metadata] There isn't any metadata to add | path=/home/user/proj/youdis/downloads/Rick_Astley/NA/NANARickAstley-_Never_Gonna_Give_You_Up_Official_Video_4K_Remaster.mp4
:end:
8. while first job is active, submit another `/youtube` and confirm busy behavior
:out:
#+begin_src shell
Error: AttributeError
Traceback (most recent call last):
File "/home/user/proj/youdis/venv/lib/python3.10/site-packages/interactions/client/client.py", line 1900, in __dispatch_interaction
response = await callback
File "/home/user/proj/youdis/venv/lib/python3.10/site-packages/interactions/client/client.py", line 1771, in _run_slash_command
return await command(ctx, **ctx.kwargs)
File "/home/user/proj/youdis/venv/lib/python3.10/site-packages/interactions/models/internal/command.py", line 132, in __call__
await self.call_callback(self.callback, context)
File "/home/user/proj/youdis/venv/lib/python3.10/site-packages/interactions/models/internal/application_commands.py", line 833, in call_callback
return await self.call_with_binding(callback, ctx, *new_args, **new_kwargs)
File "/home/user/proj/youdis/venv/lib/python3.10/site-packages/interactions/models/internal/callback.py", line 43, in call_with_binding
return await callback(*args, **kwargs)
File "/home/user/proj/youdis/youdis/adapters/discord.py", line 167, in youtube
await ctx.channel.send(f"Submitted <{url}> to the backend. Status updates via DM.")
AttributeError: 'NoneType' object has no attribute 'send'
#+end_src
busy: busy with https://youtu.be/20HxMMSqRyg?si=3v7mN2L88c_FxpQR
state=running | phase=running | cancel requested | path=/home/user/proj/youdis/downloads/James_Hoffmann/NA/NANAThe_Beginner_s_Guide_To_Latte_Art.f401.mp4
:end:
9. run `/status` and confirm it reflects current or last backend job
:out:
last job: state=completed | [Metadata] There isn't any metadata to add | path=/home/user/proj/youdis/downloads/Rick_Astley/NA/NANARickAstley-_Never_Gonna_Give_You_Up_Official_Video_4K_Remaster.mp4
:end:
10. run `/interrupt` as owner and confirm cancellation is surfaced via DM
:out:
last job: state=cancelled | cancelled | path=/home/user/proj/youdis/downloads/James_Hoffmann/NA/NANAThe_Beginner_s_Guide_To_Latte_Art.f401.mp4
:end:
- datetime:[2026-04-02 Thu 11:52]
*** org-block tests
#+begin_src shell :dir ~/proj/youdis :results output verbatim
source ./venv/bin/activate
python3 -m uvicorn youdis.main:app --host 127.0.0.1 --port 8000
#+end_src
#+begin_src shell :results output
echo ok
#+end_src
#+RESULTS:
: ok
#+begin_src python :dir ~/proj/youdis :results output verbatim
python ./youdis.py
#+end_src
#+RESULTS:
** notes
- discord adapter is now a thin HTTP client of the backend; it no longer imports or configures yt-dlp
- `YOUDIS_BACKEND_URL` controls which backend the adapter targets
- progress updates are currently implemented by polling `/jobs/current` and DMing only when the summary changes
- legacy auth/user-management commands were removed from the active adapter path
- `.env` is now supported for local/dev convenience, while real environment variables still override it in prod/docker
- command registration required explicit binding plus `@bot.listen()` listeners in this adapter structure
* [X] 2.0.3: remove deprecated discord-bot functionality (2)
delete or retire legacy bot behaviors that no longer fit once the backend split is in place
** pm notes
- only remove this after the new path works. this is cleanup, not pioneering work.
- favor deletion over compatibility shims
- keep operator controls only if still useful
** Acceptance Criteria
1. remove obsolete auth/user-management behavior
- old user persistence and commands are removed
- backend-facing flow no longer depends on them
2. remove obsolete downloader/runtime logic from bot
- bot no longer owns yt-dlp execution
- dead code paths are deleted
3. leave the bot in a coherent state
- remaining commands reflect actual supported behavior
- deprecated artifacts are clearly removed or marked
** evidence
- commit: 0aa9950
- tests:
1. `python3 -m py_compile ./youdis.py ./youdis/adapters/discord.py`
2. `rg -n "users.json|api_token" README.md unraid-ca-template.xml youdis.py youdis/adapters/discord.py`
3. start backend: `python3 -m uvicorn youdis.main:app --host 127.0.0.1 --port 8000`
4. start adapter: `python3 ./youdis.py`
5. in discord, run `/youtube`, `/status`, and `/interrupt` and confirm the adapter still works after cleanup
- datetime: [2026-04-02 Thu 12:09]
** notes
- active bot path no longer includes local auth or user-management behavior
- top-level `youdis.py` remains as a thin launcher so existing operator habits and scripts do not break during the refactor
- cleanup updated user-facing deployment artifacts to match the v2 naming and architecture, including `DISCORD_BOT_TOKEN` and removal of `users.json` references
- archived planning docs were intentionally left untouched: `pm/tasks.org` is historical and `pm/notes.org` is personal working notes
* [ ] 2.0.4: make youdis.py app runner for backend + adapters (3)
Build a simple Python orchestration layer in youdis.py so the standard app stack can be launched from one entrypoint while backend and adapters remain independently runnable for testing.
** pm notes
- keep components independently runnable
- youdis.py as app-level orchestrator; let Docker just run python3 /app/youdis.py
- no plugin discovery or hot-loading
- Docker should package and launch the app, not decide internal process topology
- preserve the ability to run backend or adapters directly for debugging
- optimize for one obvious default run path
** Acceptance Criteria
1. define the default app startup path
- youdis.py launches the standard stack for v2
- backend and Discord adapter startup order is explicit
- shutdown behavior is coherent enough for local/dev use
2. preserve modular run paths
- backend can still be run directly
- Discord adapter can still be run directly
- orchestration layer does not bury component-level testing
3. keep orchestration simple
- no dynamic adapter discovery
- no hot reloading framework
- configured components are started explicitly
4. document runtime ownership
- clarify what Python orchestrates vs what Docker orchestrates
- identify env vars or flags that control which components start
- leave room for future Zulip/XMPP adapters without redesigning the runner
** evidence
- commit: 309ce87
- tests:
1. `python3 -m py_compile ./youdis.py ./youdis/adapters/discord.py`
2. backend direct run still works: `python3 -m uvicorn youdis.main:app --host 127.0.0.1 --port 8000`
3. discord direct run still works: `python3 -m youdis.adapters.discord`
4. app runner backend-only smoke test: `ENABLE_DISCORD=0 timeout 5s python3 ./youdis.py`
5. app runner default path: `python3 ./youdis.py`
- date: [2026-04-02 Thu 14:13]
** notes
- `youdis.py` is now the default v2 app runner and starts the standard stack explicitly rather than dynamically discovering adapters
- backend starts first and must pass a health check before the Discord adapter is launched
- backend and Discord adapter remain directly runnable for debugging and tests
- Docker is intended to invoke `python3 /app/youdis.py`; Python owns app orchestration while Docker owns packaging and runtime environment
- adapter startup remains explicit in `youdis.py`, while each adapter gets its own `ENABLE_<ADAPTER>` flag
- adding an adapter means creating `youdis/adapters/<adapter>.py`, adding `start_<adapter>()` in `youdis.py`, wiring its `ENABLE_<ADAPTER>` flag, and documenting any supporting env vars in `.env.example`
- runner flags currently include `YOUDIS_RUN_BACKEND`, `YOUDIS_BACKEND_HOST`, `YOUDIS_BACKEND_PORT`, and `ENABLE_DISCORD`
* ==== BACKLOG ====
Tasks below this line are inactive and should not be touched.
* [ ] 2.0.X: fix automation and build pipeline (3)
repair and simplify the build/update/deploy path so it matches the new backend-plus-frontend structure
** pm notes
- this should come after architecture and discord integration stabilize. no point polishing the pipeline for the wrong shape.
- optimize for simple manual ops first
- stop here after pipeline is sane
** Acceptance Criteria
1. align build artifacts with the new structure
- docker/build scripts reflect current components
- runtime assumptions are consistent
2. review old automation artifacts
- stale runner/update/restart logic is removed or updated
- manual update/rebuild flow is clear
3. confirm deployment path works
- local or unraid deployment is validated
- pipeline is understandable enough to maintain
** evidence
- commit:
- tests:
- datetime:
** notes
* [ ] X.x.x: clean up discord adapter UI
** acceptance criteria
1. fix interaction pattern so it doesnt time out - prefer "command accepted" or somehting
2. remove all intermediate messages between "accepted/running" and "complete" - /status handles this!
- discord can also output a "busy" signal, research this
3. fix output syntax, we dont need to get crazy with discord cards
** evidence
- commit:
- tests:
- date:
** notes
* [ ] X.x.x: fix youtube -> plex default output
** acceptance criteria
1.
-
** evidence
- commit:
- tests:
- date:
** notes
* /
Reference in New Issue View Git Blame Copy Permalink