diff --git a/README.md b/README.md index 462dbe2..d830954 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,9 @@ -build and run the docker container -``` -api_token = [discord bot token] --v [downloads]:/downloads +v2 architecture draft: see `docs/architecture-v2.md` + +build and run the docker container +``` +api_token = [discord bot token] +-v [downloads]:/downloads -v [config]:/config ``` config contains data to persist across container updates, i.e., unraid appdata, diff --git a/docs/architecture-v2.org b/docs/architecture-v2.org new file mode 100644 index 0000000..c1aa4bf --- /dev/null +++ b/docs/architecture-v2.org @@ -0,0 +1,206 @@ +#+TITLE: youdis v2 architecture +#+VERSION: 1 +#+DATE: [2026-03-31 Tue] + +* Purpose +Youdis v2 splits the current monolithic Discord bot into: +- a private backend worker service that owns yt-dlp execution +- thin chat adapters that translate user actions into backend requests + +The goal is to make downloader behavior inspectable, reusable, and easier to debug across multiple frontends without introducing queueing infrastructure or complex deployment. + +** Goals +- Keep the backend transport-agnostic. +- Preserve archive-first duplicate prevention via `archive.txt`. +- Keep single-job semantics explicit. +- Treat requester and origin metadata as informational, not authorization. +- Make yt-dlp configuration understandable and debuggable. + +** Non-Goals +- No backend auth or user membership logic. +- No database, Redis, or external queue. +- No multi-worker scheduling in v2. +- No frontend-owned downloader logic. + +* Proposed Components +** Backend worker service +Owns: +- request validation +- single active job state +- yt-dlp configuration loading and merge rules +- download execution +- archive behavior +- output path behavior +- progress and final result events + +Does not own: +- Discord slash-command parsing +- chat formatting decisions +- access control policy + +** Frontend adapters +1. Discord +2. Zulip +3. XMPP + +Own: +- command parsing +- user-facing message formatting +- transport-specific reply behavior +- optional frontend-side gating if the deployment wants it + +Do not own: +- yt-dlp option construction +- archive checks +- output path logic +- job lifecycle rules + +* Initial Deployment Shape + +V2 should start as a single private process boundary with two layers: +1. backend service process +2. one Discord adapter process + +The backend is intended for private/local deployment. Trust comes from deployment topology and network placement, not backend auth. + +** Backend Contract +*** Request model +Minimal fields: +- `url` +- `requester_id` optional +- `requester_name` optional +- `origin` optional +- `requested_at` optional +Only `url` affects downloader behavior in v2. The rest is passthrough metadata for logs and frontend context. + +*** Result states +The backend should normalize job outcomes into a small stable vocabulary: +- `accepted` +- `busy` +- `validating` +- `downloading` +- `skipped` +- `completed` +- `failed` +- `cancelled` + +These states should be transport-agnostic and human-debuggable. + +*** Minimal endpoints +The first backend seam should stay small: +- `POST /jobs` +- `GET /jobs/current` +- `POST /jobs/current/cancel` +- `GET /health` +- `GET /version` + +Polling is the default integration model for v2. Streaming or push delivery is deferred unless a real frontend need appears. + +** Single-Job Semantics +V2 explicitly supports one active job at a time. +- If idle, the backend accepts a new job and marks it active. +- If busy, the backend rejects the new request with a `busy` result. +- Cancel is coarse and best-effort. +- The backend clears active state when the job reaches `skipped`, `completed`, `failed`, or `cancelled`. + +This is a feature, not a limitation, for the initial service. + +* yt-dlp Configuration Ownership + +This is the most important v2 cleanup. + +The backend owns all yt-dlp configuration behavior. Frontends must not build yt-dlp option dictionaries. + +** Config split + +Static settings belong in `default-yt-dlp.conf`: +- archive path +- output template +- embedding and metadata preferences +- stable format defaults +- retry defaults + +Runtime settings belong in backend code: +- target URL +- progress hooks +- cancellation hook +- request-scoped metadata +- any values that must vary per request + +*** Merge rule +The backend should load the default config first, then apply a small set of explicit runtime overrides. + +If a runtime override conflicts with file-backed config, the override wins and the backend should log the effective value used. + +*** Debuggability requirement +For each job, the backend should make it easy to inspect: + +- config file path used +- whether config loading succeeded +- effective key yt-dlp options after merge +- final normalized job result + +This exists specifically because the current config behavior has been difficult to reason about. + +*** Config hygiene expectations + +The default config should be safe for real downloads in production-like use. + +That means test-only settings such as `--simulate` should not remain enabled in the default runtime path unless the backend intentionally supports an explicit dry-run mode. + +** Recommended Repo Shape +One reasonable first cut: + +#+begin_quote +youdis/ + README.md + default-yt-dlp.conf + youdis/ + __init__.py + api.py + config.py + models.py + worker.py + ytdlp_backend.py + adapters/ + __init__.py + discord.py + docs/ + architecture-v2.md +#+end_quote + +Notes: + +- `ytdlp_backend.py` owns yt-dlp integration and result normalization. +- `worker.py` owns active-job state and lifecycle. +- `api.py` exposes the minimal HTTP seam. +- `adapters/discord.py` becomes a thin client of the backend. + +** Framework Recommendation + +FastAPI is the current recommendation for the backend seam. + +Why it fits: + +- typed request and response models help define the contract cleanly +- built-in docs make local inspection easier during early iterations +- it stays small enough for this service if we keep the surface area disciplined + +Why this is not a strong ideological choice: + +- Flask would also work if we decide the type/model ergonomics are not worth it +- the important decision is keeping the seam small, not choosing a fashionable framework + +** Immediate Next Task + +`2.0.1` should implement the backend skeleton with just enough real behavior to prove the seam: + +- package/module layout +- health and version endpoint +- single active-job state holder +- one submit-job path +- one current-job status path +- yt-dlp config loading from `default-yt-dlp.conf` +- explicit runtime overrides for request URL, progress hooks, and cancel behavior + +The Discord adapter should remain unchanged until that backend skeleton can accept a job and report status coherently. diff --git a/pm/arch-v2.org b/pm/arch-v2.org new file mode 100644 index 0000000..c1aa4bf --- /dev/null +++ b/pm/arch-v2.org @@ -0,0 +1,206 @@ +#+TITLE: youdis v2 architecture +#+VERSION: 1 +#+DATE: [2026-03-31 Tue] + +* Purpose +Youdis v2 splits the current monolithic Discord bot into: +- a private backend worker service that owns yt-dlp execution +- thin chat adapters that translate user actions into backend requests + +The goal is to make downloader behavior inspectable, reusable, and easier to debug across multiple frontends without introducing queueing infrastructure or complex deployment. + +** Goals +- Keep the backend transport-agnostic. +- Preserve archive-first duplicate prevention via `archive.txt`. +- Keep single-job semantics explicit. +- Treat requester and origin metadata as informational, not authorization. +- Make yt-dlp configuration understandable and debuggable. + +** Non-Goals +- No backend auth or user membership logic. +- No database, Redis, or external queue. +- No multi-worker scheduling in v2. +- No frontend-owned downloader logic. + +* Proposed Components +** Backend worker service +Owns: +- request validation +- single active job state +- yt-dlp configuration loading and merge rules +- download execution +- archive behavior +- output path behavior +- progress and final result events + +Does not own: +- Discord slash-command parsing +- chat formatting decisions +- access control policy + +** Frontend adapters +1. Discord +2. Zulip +3. XMPP + +Own: +- command parsing +- user-facing message formatting +- transport-specific reply behavior +- optional frontend-side gating if the deployment wants it + +Do not own: +- yt-dlp option construction +- archive checks +- output path logic +- job lifecycle rules + +* Initial Deployment Shape + +V2 should start as a single private process boundary with two layers: +1. backend service process +2. one Discord adapter process + +The backend is intended for private/local deployment. Trust comes from deployment topology and network placement, not backend auth. + +** Backend Contract +*** Request model +Minimal fields: +- `url` +- `requester_id` optional +- `requester_name` optional +- `origin` optional +- `requested_at` optional +Only `url` affects downloader behavior in v2. The rest is passthrough metadata for logs and frontend context. + +*** Result states +The backend should normalize job outcomes into a small stable vocabulary: +- `accepted` +- `busy` +- `validating` +- `downloading` +- `skipped` +- `completed` +- `failed` +- `cancelled` + +These states should be transport-agnostic and human-debuggable. + +*** Minimal endpoints +The first backend seam should stay small: +- `POST /jobs` +- `GET /jobs/current` +- `POST /jobs/current/cancel` +- `GET /health` +- `GET /version` + +Polling is the default integration model for v2. Streaming or push delivery is deferred unless a real frontend need appears. + +** Single-Job Semantics +V2 explicitly supports one active job at a time. +- If idle, the backend accepts a new job and marks it active. +- If busy, the backend rejects the new request with a `busy` result. +- Cancel is coarse and best-effort. +- The backend clears active state when the job reaches `skipped`, `completed`, `failed`, or `cancelled`. + +This is a feature, not a limitation, for the initial service. + +* yt-dlp Configuration Ownership + +This is the most important v2 cleanup. + +The backend owns all yt-dlp configuration behavior. Frontends must not build yt-dlp option dictionaries. + +** Config split + +Static settings belong in `default-yt-dlp.conf`: +- archive path +- output template +- embedding and metadata preferences +- stable format defaults +- retry defaults + +Runtime settings belong in backend code: +- target URL +- progress hooks +- cancellation hook +- request-scoped metadata +- any values that must vary per request + +*** Merge rule +The backend should load the default config first, then apply a small set of explicit runtime overrides. + +If a runtime override conflicts with file-backed config, the override wins and the backend should log the effective value used. + +*** Debuggability requirement +For each job, the backend should make it easy to inspect: + +- config file path used +- whether config loading succeeded +- effective key yt-dlp options after merge +- final normalized job result + +This exists specifically because the current config behavior has been difficult to reason about. + +*** Config hygiene expectations + +The default config should be safe for real downloads in production-like use. + +That means test-only settings such as `--simulate` should not remain enabled in the default runtime path unless the backend intentionally supports an explicit dry-run mode. + +** Recommended Repo Shape +One reasonable first cut: + +#+begin_quote +youdis/ + README.md + default-yt-dlp.conf + youdis/ + __init__.py + api.py + config.py + models.py + worker.py + ytdlp_backend.py + adapters/ + __init__.py + discord.py + docs/ + architecture-v2.md +#+end_quote + +Notes: + +- `ytdlp_backend.py` owns yt-dlp integration and result normalization. +- `worker.py` owns active-job state and lifecycle. +- `api.py` exposes the minimal HTTP seam. +- `adapters/discord.py` becomes a thin client of the backend. + +** Framework Recommendation + +FastAPI is the current recommendation for the backend seam. + +Why it fits: + +- typed request and response models help define the contract cleanly +- built-in docs make local inspection easier during early iterations +- it stays small enough for this service if we keep the surface area disciplined + +Why this is not a strong ideological choice: + +- Flask would also work if we decide the type/model ergonomics are not worth it +- the important decision is keeping the seam small, not choosing a fashionable framework + +** Immediate Next Task + +`2.0.1` should implement the backend skeleton with just enough real behavior to prove the seam: + +- package/module layout +- health and version endpoint +- single active-job state holder +- one submit-job path +- one current-job status path +- yt-dlp config loading from `default-yt-dlp.conf` +- explicit runtime overrides for request URL, progress hooks, and cancel behavior + +The Discord adapter should remain unchanged until that backend skeleton can accept a job and report status coherently. diff --git a/pm/tasks-v2.org b/pm/tasks-v2.org index 9d8a511..436ee49 100644 --- a/pm/tasks-v2.org +++ b/pm/tasks-v2.org @@ -1,138 +1,139 @@ -#+title: Task Log -#+updated: [2026-03-31 Tue 16:03] - -* youdis v2 goals: -1. Separate backend from frontend -2. Offload auth -3. Ensure auto nightly builds -4. Default output format supports plex browsing youtube channels as "tv shows" -5. Facilitate multiple GUI inbounds: Discord, Zulip, XMPP - -* [ ] 2.0.0: define architecture (2-4) -define the target architecture for a private backend yt-dlp worker with thin chat frontends -** pm notes: -- keep this iterative. the point is to choose the shape and seam, not prematurely implement infra. likely decisions include backend framework, request/status model, and how thin the discord shim should be. -- goal is simple, private, maintainable deployment -- avoid auth, queues, or persistence beyond clear & immediate needs - -** Acceptance Criteria -1. document the target architecture at a high level - - backend owns yt-dlp execution and job state - - frontends own chat-specific UX -2. identify key decisions still open - - backend choice - - service seam/endpoints - - status/progress model -3. capture enough structure to begin implementation - - repo/component layout is sketched - - next implementation task is unblocked - -** evidence -- commit: -- tests: -- datetime: - -** notes - -* [ ] 2.0.1: build backend yt-dlp worker (3) -create the minimal backend/service skeleton and establish a working yt-dlp baseline with clean hooks for future frontends -** pm notes -- foundation; don't need the full finished service here, just the basic shape plus enough real yt-dlp execution to validate the seam and build on it. -- keep single-job semantics -- prioritize inspectable behavior over polish - -** Acceptance Criteria -1. create the backend/service skeleton - - app/module layout exists - - core request path is stubbed or minimally working -2. establish a working yt-dlp baseline - - archive behavior is preserved - - output path behavior is preserved or intentionally updated - - use yt-dlp .conf and set reasonable default -3. expose basic hooks/interfaces for future frontends - - submit/request path exists - - status/progress hook exists - - basic health/version visibility exists - -** evidence -- commit: -- tests: -- datetime: - -** notes - -* [ ] 2.0.2: update discord bot to use new backend (3) -update the discord bot into a thin frontend that talks to the backend and verify the flow end to end -** pm notes -- this is the first real frontend proof. once this works cleanly, zulip/xmpp should mostly be adapter work rather than downloader rewrites. -- keep discord logic thin; no auth -- do not duplicate yt-dlp behavior in the bot - -** Acceptance Criteria -1. discord bot submits requests to backend - - command/input handling works - - acceptance/busy/failure responses are clear -2. discord bot relays useful backend status - - progress reporting works at a basic level - - completion/failure/skipped outcomes are surfaced -3. backend-discord flow is tested end to end - - valid request path tested - - busy or conflict behavior tested - - failure path tested - -** evidence -- commit: -- tests: -- datetime: - -** notes - -* [ ] 2.0.3: remove deprecated discord-bot functionality (2) -delete or retire legacy bot behaviors that no longer fit once the backend split is in place -** pm notes -- only remove this after the new path works. this is cleanup, not pioneering work. -- favor deletion over compatibility shims -- keep operator controls only if still useful - -** Acceptance Criteria -1. remove obsolete auth/user-management behavior - - old user persistence and commands are removed - - backend-facing flow no longer depends on them -2. remove obsolete downloader/runtime logic from bot - - bot no longer owns yt-dlp execution - - dead code paths are deleted -3. leave the bot in a coherent state - - remaining commands reflect actual supported behavior - - deprecated artifacts are clearly removed or marked - -** evidence -- commit: -- tests: -- datetime: - -** notes - -* [ ] 2.0.5: fix automation and build pipeline (3) -repair and simplify the build/update/deploy path so it matches the new backend-plus-frontend structure -** pm notes -- this should come after architecture and discord integration stabilize. no point polishing the pipeline for the wrong shape. -- optimize for simple manual ops first -- stop here after pipeline is sane - -** Acceptance Criteria -1. align build artifacts with the new structure - - docker/build scripts reflect current components - - runtime assumptions are consistent -2. review old automation artifacts - - stale runner/update/restart logic is removed or updated - - manual update/rebuild flow is clear -3. confirm deployment path works - - local or unraid deployment is validated - - pipeline is understandable enough to maintain - -** evidence -- commit: -- tests: -- datetime: - +#+title: Task Log +#+updated: [2026-03-31 Tue 16:03] + +* youdis v2 goals: +1. Separate backend from frontend +2. Offload auth +3. Ensure auto nightly builds +4. Default output format supports plex browsing youtube channels as "tv shows" +5. Facilitate multiple GUI inbounds: Discord, Zulip, XMPP + +* [ ] 2.0.0: define architecture (2-4) +define the target architecture for a private backend yt-dlp worker with thin chat frontends +** pm notes: +- keep this iterative. the point is to choose the shape and seam, not prematurely implement infra. likely decisions include backend framework, request/status model, and how thin the discord shim should be. +- goal is simple, private, maintainable deployment +- avoid auth, queues, or persistence beyond clear & immediate needs + +** Acceptance Criteria +1. document the target architecture at a high level + - backend owns yt-dlp execution and job state + - frontends own chat-specific UX +2. identify key decisions still open + - backend choice + - service seam/endpoints + - status/progress model +3. capture enough structure to begin implementation + - repo/component layout is sketched + - next implementation task is unblocked + +** evidence +- commit: +- tests: +- datetime: + ** notes +- first architecture draft captured in `docs/architecture-v2.md` + +* [ ] 2.0.1: build backend yt-dlp worker (3) +create the minimal backend/service skeleton and establish a working yt-dlp baseline with clean hooks for future frontends +** pm notes +- foundation; don't need the full finished service here, just the basic shape plus enough real yt-dlp execution to validate the seam and build on it. +- keep single-job semantics +- prioritize inspectable behavior over polish + +** Acceptance Criteria +1. create the backend/service skeleton + - app/module layout exists + - core request path is stubbed or minimally working +2. establish a working yt-dlp baseline + - archive behavior is preserved + - output path behavior is preserved or intentionally updated + - use yt-dlp .conf and set reasonable default +3. expose basic hooks/interfaces for future frontends + - submit/request path exists + - status/progress hook exists + - basic health/version visibility exists + +** evidence +- commit: +- tests: +- datetime: + +** notes + +* [ ] 2.0.2: update discord bot to use new backend (3) +update the discord bot into a thin frontend that talks to the backend and verify the flow end to end +** pm notes +- this is the first real frontend proof. once this works cleanly, zulip/xmpp should mostly be adapter work rather than downloader rewrites. +- keep discord logic thin; no auth +- do not duplicate yt-dlp behavior in the bot + +** Acceptance Criteria +1. discord bot submits requests to backend + - command/input handling works + - acceptance/busy/failure responses are clear +2. discord bot relays useful backend status + - progress reporting works at a basic level + - completion/failure/skipped outcomes are surfaced +3. backend-discord flow is tested end to end + - valid request path tested + - busy or conflict behavior tested + - failure path tested + +** evidence +- commit: +- tests: +- datetime: + +** notes + +* [ ] 2.0.3: remove deprecated discord-bot functionality (2) +delete or retire legacy bot behaviors that no longer fit once the backend split is in place +** pm notes +- only remove this after the new path works. this is cleanup, not pioneering work. +- favor deletion over compatibility shims +- keep operator controls only if still useful + +** Acceptance Criteria +1. remove obsolete auth/user-management behavior + - old user persistence and commands are removed + - backend-facing flow no longer depends on them +2. remove obsolete downloader/runtime logic from bot + - bot no longer owns yt-dlp execution + - dead code paths are deleted +3. leave the bot in a coherent state + - remaining commands reflect actual supported behavior + - deprecated artifacts are clearly removed or marked + +** evidence +- commit: +- tests: +- datetime: + +** notes + +* [ ] 2.0.5: fix automation and build pipeline (3) +repair and simplify the build/update/deploy path so it matches the new backend-plus-frontend structure +** pm notes +- this should come after architecture and discord integration stabilize. no point polishing the pipeline for the wrong shape. +- optimize for simple manual ops first +- stop here after pipeline is sane + +** Acceptance Criteria +1. align build artifacts with the new structure + - docker/build scripts reflect current components + - runtime assumptions are consistent +2. review old automation artifacts + - stale runner/update/restart logic is removed or updated + - manual update/rebuild flow is clear +3. confirm deployment path works + - local or unraid deployment is validated + - pipeline is understandable enough to maintain + +** evidence +- commit: +- tests: +- datetime: + +** notes