ben/youdis
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

draft arch-v2

Browse Source
This commit is contained in:
ben
2026-03-31 16:53:28 -04:00
parent 5f78ac6a1a
commit 35a2592dce
4 changed files with 556 additions and 141 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
README.md
View File

@@ -1,7 +1,9 @@
build and run the docker container v2 architecture draft: see `docs/architecture-v2.md`
```
api_token = [discord bot token] build and run the docker container
-v [downloads]:/downloads ```
api_token = [discord bot token]
-v [downloads]:/downloads
-v [config]:/config -v [config]:/config
``` ```
config contains data to persist across container updates, i.e., unraid appdata, config contains data to persist across container updates, i.e., unraid appdata,

206
docs/architecture-v2.org Normal file
View File

@@ -0,0 +1,206 @@
#+TITLE: youdis v2 architecture
#+VERSION: 1
#+DATE: [2026-03-31 Tue]
* Purpose
Youdis v2 splits the current monolithic Discord bot into:
- a private backend worker service that owns yt-dlp execution
- thin chat adapters that translate user actions into backend requests
The goal is to make downloader behavior inspectable, reusable, and easier to debug across multiple frontends without introducing queueing infrastructure or complex deployment.
** Goals
- Keep the backend transport-agnostic.
- Preserve archive-first duplicate prevention via `archive.txt`.
- Keep single-job semantics explicit.
- Treat requester and origin metadata as informational, not authorization.
- Make yt-dlp configuration understandable and debuggable.
** Non-Goals
- No backend auth or user membership logic.
- No database, Redis, or external queue.
- No multi-worker scheduling in v2.
- No frontend-owned downloader logic.
* Proposed Components
** Backend worker service
Owns:
- request validation
- single active job state
- yt-dlp configuration loading and merge rules
- download execution
- archive behavior
- output path behavior
- progress and final result events
Does not own:
- Discord slash-command parsing
- chat formatting decisions
- access control policy
** Frontend adapters
1. Discord
2. Zulip
3. XMPP
Own:
- command parsing
- user-facing message formatting
- transport-specific reply behavior
- optional frontend-side gating if the deployment wants it
Do not own:
- yt-dlp option construction
- archive checks
- output path logic
- job lifecycle rules
* Initial Deployment Shape
V2 should start as a single private process boundary with two layers:
1. backend service process
2. one Discord adapter process
The backend is intended for private/local deployment. Trust comes from deployment topology and network placement, not backend auth.
** Backend Contract
*** Request model
Minimal fields:
- `url`
- `requester_id` optional
- `requester_name` optional
- `origin` optional
- `requested_at` optional
Only `url` affects downloader behavior in v2. The rest is passthrough metadata for logs and frontend context.
*** Result states
The backend should normalize job outcomes into a small stable vocabulary:
- `accepted`
- `busy`
- `validating`
- `downloading`
- `skipped`
- `completed`
- `failed`
- `cancelled`
These states should be transport-agnostic and human-debuggable.
*** Minimal endpoints
The first backend seam should stay small:
- `POST /jobs`
- `GET /jobs/current`
- `POST /jobs/current/cancel`
- `GET /health`
- `GET /version`
Polling is the default integration model for v2. Streaming or push delivery is deferred unless a real frontend need appears.
** Single-Job Semantics
V2 explicitly supports one active job at a time.
- If idle, the backend accepts a new job and marks it active.
- If busy, the backend rejects the new request with a `busy` result.
- Cancel is coarse and best-effort.
- The backend clears active state when the job reaches `skipped`, `completed`, `failed`, or `cancelled`.
This is a feature, not a limitation, for the initial service.
* yt-dlp Configuration Ownership
This is the most important v2 cleanup.
The backend owns all yt-dlp configuration behavior. Frontends must not build yt-dlp option dictionaries.
** Config split
Static settings belong in `default-yt-dlp.conf`:
- archive path
- output template
- embedding and metadata preferences
- stable format defaults
- retry defaults
Runtime settings belong in backend code:
- target URL
- progress hooks
- cancellation hook
- request-scoped metadata
- any values that must vary per request
*** Merge rule
The backend should load the default config first, then apply a small set of explicit runtime overrides.
If a runtime override conflicts with file-backed config, the override wins and the backend should log the effective value used.
*** Debuggability requirement
For each job, the backend should make it easy to inspect:
- config file path used
- whether config loading succeeded
- effective key yt-dlp options after merge
- final normalized job result
This exists specifically because the current config behavior has been difficult to reason about.
*** Config hygiene expectations
The default config should be safe for real downloads in production-like use.
That means test-only settings such as `--simulate` should not remain enabled in the default runtime path unless the backend intentionally supports an explicit dry-run mode.
** Recommended Repo Shape
One reasonable first cut:
#+begin_quote
youdis/
README.md
default-yt-dlp.conf
youdis/
__init__.py
api.py
config.py
models.py
worker.py
ytdlp_backend.py
adapters/
__init__.py
discord.py
docs/
architecture-v2.md
#+end_quote
Notes:
- `ytdlp_backend.py` owns yt-dlp integration and result normalization.
- `worker.py` owns active-job state and lifecycle.
- `api.py` exposes the minimal HTTP seam.
- `adapters/discord.py` becomes a thin client of the backend.
** Framework Recommendation
FastAPI is the current recommendation for the backend seam.
Why it fits:
- typed request and response models help define the contract cleanly
- built-in docs make local inspection easier during early iterations
- it stays small enough for this service if we keep the surface area disciplined
Why this is not a strong ideological choice:
- Flask would also work if we decide the type/model ergonomics are not worth it
- the important decision is keeping the seam small, not choosing a fashionable framework
** Immediate Next Task
`2.0.1` should implement the backend skeleton with just enough real behavior to prove the seam:
- package/module layout
- health and version endpoint
- single active-job state holder
- one submit-job path
- one current-job status path
- yt-dlp config loading from `default-yt-dlp.conf`
- explicit runtime overrides for request URL, progress hooks, and cancel behavior
The Discord adapter should remain unchanged until that backend skeleton can accept a job and report status coherently.

206
pm/arch-v2.org Normal file
View File

@@ -0,0 +1,206 @@
#+TITLE: youdis v2 architecture
#+VERSION: 1
#+DATE: [2026-03-31 Tue]
* Purpose
Youdis v2 splits the current monolithic Discord bot into:
- a private backend worker service that owns yt-dlp execution
- thin chat adapters that translate user actions into backend requests
The goal is to make downloader behavior inspectable, reusable, and easier to debug across multiple frontends without introducing queueing infrastructure or complex deployment.
** Goals
- Keep the backend transport-agnostic.
- Preserve archive-first duplicate prevention via `archive.txt`.
- Keep single-job semantics explicit.
- Treat requester and origin metadata as informational, not authorization.
- Make yt-dlp configuration understandable and debuggable.
** Non-Goals
- No backend auth or user membership logic.
- No database, Redis, or external queue.
- No multi-worker scheduling in v2.
- No frontend-owned downloader logic.
* Proposed Components
** Backend worker service
Owns:
- request validation
- single active job state
- yt-dlp configuration loading and merge rules
- download execution
- archive behavior
- output path behavior
- progress and final result events
Does not own:
- Discord slash-command parsing
- chat formatting decisions
- access control policy
** Frontend adapters
1. Discord
2. Zulip
3. XMPP
Own:
- command parsing
- user-facing message formatting
- transport-specific reply behavior
- optional frontend-side gating if the deployment wants it
Do not own:
- yt-dlp option construction
- archive checks
- output path logic
- job lifecycle rules
* Initial Deployment Shape
V2 should start as a single private process boundary with two layers:
1. backend service process
2. one Discord adapter process
The backend is intended for private/local deployment. Trust comes from deployment topology and network placement, not backend auth.
** Backend Contract
*** Request model
Minimal fields:
- `url`
- `requester_id` optional
- `requester_name` optional
- `origin` optional
- `requested_at` optional
Only `url` affects downloader behavior in v2. The rest is passthrough metadata for logs and frontend context.
*** Result states
The backend should normalize job outcomes into a small stable vocabulary:
- `accepted`
- `busy`
- `validating`
- `downloading`
- `skipped`
- `completed`
- `failed`
- `cancelled`
These states should be transport-agnostic and human-debuggable.
*** Minimal endpoints
The first backend seam should stay small:
- `POST /jobs`
- `GET /jobs/current`
- `POST /jobs/current/cancel`
- `GET /health`
- `GET /version`
Polling is the default integration model for v2. Streaming or push delivery is deferred unless a real frontend need appears.
** Single-Job Semantics
V2 explicitly supports one active job at a time.
- If idle, the backend accepts a new job and marks it active.
- If busy, the backend rejects the new request with a `busy` result.
- Cancel is coarse and best-effort.
- The backend clears active state when the job reaches `skipped`, `completed`, `failed`, or `cancelled`.
This is a feature, not a limitation, for the initial service.
* yt-dlp Configuration Ownership
This is the most important v2 cleanup.
The backend owns all yt-dlp configuration behavior. Frontends must not build yt-dlp option dictionaries.
** Config split
Static settings belong in `default-yt-dlp.conf`:
- archive path
- output template
- embedding and metadata preferences
- stable format defaults
- retry defaults
Runtime settings belong in backend code:
- target URL
- progress hooks
- cancellation hook
- request-scoped metadata
- any values that must vary per request
*** Merge rule
The backend should load the default config first, then apply a small set of explicit runtime overrides.
If a runtime override conflicts with file-backed config, the override wins and the backend should log the effective value used.
*** Debuggability requirement
For each job, the backend should make it easy to inspect:
- config file path used
- whether config loading succeeded
- effective key yt-dlp options after merge
- final normalized job result
This exists specifically because the current config behavior has been difficult to reason about.
*** Config hygiene expectations
The default config should be safe for real downloads in production-like use.
That means test-only settings such as `--simulate` should not remain enabled in the default runtime path unless the backend intentionally supports an explicit dry-run mode.
** Recommended Repo Shape
One reasonable first cut:
#+begin_quote
youdis/
README.md
default-yt-dlp.conf
youdis/
__init__.py
api.py
config.py
models.py
worker.py
ytdlp_backend.py
adapters/
__init__.py
discord.py
docs/
architecture-v2.md
#+end_quote
Notes:
- `ytdlp_backend.py` owns yt-dlp integration and result normalization.
- `worker.py` owns active-job state and lifecycle.
- `api.py` exposes the minimal HTTP seam.
- `adapters/discord.py` becomes a thin client of the backend.
** Framework Recommendation
FastAPI is the current recommendation for the backend seam.
Why it fits:
- typed request and response models help define the contract cleanly
- built-in docs make local inspection easier during early iterations
- it stays small enough for this service if we keep the surface area disciplined
Why this is not a strong ideological choice:
- Flask would also work if we decide the type/model ergonomics are not worth it
- the important decision is keeping the seam small, not choosing a fashionable framework
** Immediate Next Task
`2.0.1` should implement the backend skeleton with just enough real behavior to prove the seam:
- package/module layout
- health and version endpoint
- single active-job state holder
- one submit-job path
- one current-job status path
- yt-dlp config loading from `default-yt-dlp.conf`
- explicit runtime overrides for request URL, progress hooks, and cancel behavior
The Discord adapter should remain unchanged until that backend skeleton can accept a job and report status coherently.

275
pm/tasks-v2.org
View File

@@ -1,138 +1,139 @@
#+title: Task Log #+title: Task Log
#+updated: [2026-03-31 Tue 16:03] #+updated: [2026-03-31 Tue 16:03]
* youdis v2 goals: * youdis v2 goals:
1. Separate backend from frontend 1. Separate backend from frontend
2. Offload auth 2. Offload auth
3. Ensure auto nightly builds 3. Ensure auto nightly builds
4. Default output format supports plex browsing youtube channels as "tv shows" 4. Default output format supports plex browsing youtube channels as "tv shows"
5. Facilitate multiple GUI inbounds: Discord, Zulip, XMPP 5. Facilitate multiple GUI inbounds: Discord, Zulip, XMPP
* [ ] 2.0.0: define architecture (2-4) * [ ] 2.0.0: define architecture (2-4)
define the target architecture for a private backend yt-dlp worker with thin chat frontends define the target architecture for a private backend yt-dlp worker with thin chat frontends
** pm notes: ** pm notes:
- keep this iterative. the point is to choose the shape and seam, not prematurely implement infra. likely decisions include backend framework, request/status model, and how thin the discord shim should be. - keep this iterative. the point is to choose the shape and seam, not prematurely implement infra. likely decisions include backend framework, request/status model, and how thin the discord shim should be.
- goal is simple, private, maintainable deployment - goal is simple, private, maintainable deployment
- avoid auth, queues, or persistence beyond clear & immediate needs - avoid auth, queues, or persistence beyond clear & immediate needs
** Acceptance Criteria ** Acceptance Criteria
1. document the target architecture at a high level 1. document the target architecture at a high level
- backend owns yt-dlp execution and job state - backend owns yt-dlp execution and job state
- frontends own chat-specific UX - frontends own chat-specific UX
2. identify key decisions still open 2. identify key decisions still open
- backend choice - backend choice
- service seam/endpoints - service seam/endpoints
- status/progress model - status/progress model
3. capture enough structure to begin implementation 3. capture enough structure to begin implementation
- repo/component layout is sketched - repo/component layout is sketched
- next implementation task is unblocked - next implementation task is unblocked
** evidence ** evidence
- commit: - commit:
- tests: - tests:
- datetime: - datetime:
** notes
* [ ] 2.0.1: build backend yt-dlp worker (3)
create the minimal backend/service skeleton and establish a working yt-dlp baseline with clean hooks for future frontends
** pm notes
- foundation; don't need the full finished service here, just the basic shape plus enough real yt-dlp execution to validate the seam and build on it.
- keep single-job semantics
- prioritize inspectable behavior over polish
** Acceptance Criteria
1. create the backend/service skeleton
- app/module layout exists
- core request path is stubbed or minimally working
2. establish a working yt-dlp baseline
- archive behavior is preserved
- output path behavior is preserved or intentionally updated
- use yt-dlp .conf and set reasonable default
3. expose basic hooks/interfaces for future frontends
- submit/request path exists
- status/progress hook exists
- basic health/version visibility exists
** evidence
- commit:
- tests:
- datetime:
** notes
* [ ] 2.0.2: update discord bot to use new backend (3)
update the discord bot into a thin frontend that talks to the backend and verify the flow end to end
** pm notes
- this is the first real frontend proof. once this works cleanly, zulip/xmpp should mostly be adapter work rather than downloader rewrites.
- keep discord logic thin; no auth
- do not duplicate yt-dlp behavior in the bot
** Acceptance Criteria
1. discord bot submits requests to backend
- command/input handling works
- acceptance/busy/failure responses are clear
2. discord bot relays useful backend status
- progress reporting works at a basic level
- completion/failure/skipped outcomes are surfaced
3. backend-discord flow is tested end to end
- valid request path tested
- busy or conflict behavior tested
- failure path tested
** evidence
- commit:
- tests:
- datetime:
** notes
* [ ] 2.0.3: remove deprecated discord-bot functionality (2)
delete or retire legacy bot behaviors that no longer fit once the backend split is in place
** pm notes
- only remove this after the new path works. this is cleanup, not pioneering work.
- favor deletion over compatibility shims
- keep operator controls only if still useful
** Acceptance Criteria
1. remove obsolete auth/user-management behavior
- old user persistence and commands are removed
- backend-facing flow no longer depends on them
2. remove obsolete downloader/runtime logic from bot
- bot no longer owns yt-dlp execution
- dead code paths are deleted
3. leave the bot in a coherent state
- remaining commands reflect actual supported behavior
- deprecated artifacts are clearly removed or marked
** evidence
- commit:
- tests:
- datetime:
** notes
* [ ] 2.0.5: fix automation and build pipeline (3)
repair and simplify the build/update/deploy path so it matches the new backend-plus-frontend structure
** pm notes
- this should come after architecture and discord integration stabilize. no point polishing the pipeline for the wrong shape.
- optimize for simple manual ops first
- stop here after pipeline is sane
** Acceptance Criteria
1. align build artifacts with the new structure
- docker/build scripts reflect current components
- runtime assumptions are consistent
2. review old automation artifacts
- stale runner/update/restart logic is removed or updated
- manual update/rebuild flow is clear
3. confirm deployment path works
- local or unraid deployment is validated
- pipeline is understandable enough to maintain
** evidence
- commit:
- tests:
- datetime:
** notes ** notes
- first architecture draft captured in `docs/architecture-v2.md`
* [ ] 2.0.1: build backend yt-dlp worker (3)
create the minimal backend/service skeleton and establish a working yt-dlp baseline with clean hooks for future frontends
** pm notes
- foundation; don't need the full finished service here, just the basic shape plus enough real yt-dlp execution to validate the seam and build on it.
- keep single-job semantics
- prioritize inspectable behavior over polish
** Acceptance Criteria
1. create the backend/service skeleton
- app/module layout exists
- core request path is stubbed or minimally working
2. establish a working yt-dlp baseline
- archive behavior is preserved
- output path behavior is preserved or intentionally updated
- use yt-dlp .conf and set reasonable default
3. expose basic hooks/interfaces for future frontends
- submit/request path exists
- status/progress hook exists
- basic health/version visibility exists
** evidence
- commit:
- tests:
- datetime:
** notes
* [ ] 2.0.2: update discord bot to use new backend (3)
update the discord bot into a thin frontend that talks to the backend and verify the flow end to end
** pm notes
- this is the first real frontend proof. once this works cleanly, zulip/xmpp should mostly be adapter work rather than downloader rewrites.
- keep discord logic thin; no auth
- do not duplicate yt-dlp behavior in the bot
** Acceptance Criteria
1. discord bot submits requests to backend
- command/input handling works
- acceptance/busy/failure responses are clear
2. discord bot relays useful backend status
- progress reporting works at a basic level
- completion/failure/skipped outcomes are surfaced
3. backend-discord flow is tested end to end
- valid request path tested
- busy or conflict behavior tested
- failure path tested
** evidence
- commit:
- tests:
- datetime:
** notes
* [ ] 2.0.3: remove deprecated discord-bot functionality (2)
delete or retire legacy bot behaviors that no longer fit once the backend split is in place
** pm notes
- only remove this after the new path works. this is cleanup, not pioneering work.
- favor deletion over compatibility shims
- keep operator controls only if still useful
** Acceptance Criteria
1. remove obsolete auth/user-management behavior
- old user persistence and commands are removed
- backend-facing flow no longer depends on them
2. remove obsolete downloader/runtime logic from bot
- bot no longer owns yt-dlp execution
- dead code paths are deleted
3. leave the bot in a coherent state
- remaining commands reflect actual supported behavior
- deprecated artifacts are clearly removed or marked
** evidence
- commit:
- tests:
- datetime:
** notes
* [ ] 2.0.5: fix automation and build pipeline (3)
repair and simplify the build/update/deploy path so it matches the new backend-plus-frontend structure
** pm notes
- this should come after architecture and discord integration stabilize. no point polishing the pipeline for the wrong shape.
- optimize for simple manual ops first
- stop here after pipeline is sane
** Acceptance Criteria
1. align build artifacts with the new structure
- docker/build scripts reflect current components
- runtime assumptions are consistent
2. review old automation artifacts
- stale runner/update/restart logic is removed or updated
- manual update/rebuild flow is clear
3. confirm deployment path works
- local or unraid deployment is validated
- pipeline is understandable enough to maintain
** evidence
- commit:
- tests:
- datetime:
** notes