finalized v2 architecture for 2.0.1
This commit is contained in:
206
pm/arch-v2.org
206
pm/arch-v2.org
@@ -1,206 +0,0 @@
|
||||
#+TITLE: youdis v2 architecture
|
||||
#+VERSION: 1
|
||||
#+DATE: [2026-03-31 Tue]
|
||||
|
||||
* Purpose
|
||||
Youdis v2 splits the current monolithic Discord bot into:
|
||||
- a private backend worker service that owns yt-dlp execution
|
||||
- thin chat adapters that translate user actions into backend requests
|
||||
|
||||
The goal is to make downloader behavior inspectable, reusable, and easier to debug across multiple frontends without introducing queueing infrastructure or complex deployment.
|
||||
|
||||
** Goals
|
||||
- Keep the backend transport-agnostic.
|
||||
- Preserve archive-first duplicate prevention via `archive.txt`.
|
||||
- Keep single-job semantics explicit.
|
||||
- Treat requester and origin metadata as informational, not authorization.
|
||||
- Make yt-dlp configuration understandable and debuggable.
|
||||
|
||||
** Non-Goals
|
||||
- No backend auth or user membership logic.
|
||||
- No database, Redis, or external queue.
|
||||
- No multi-worker scheduling in v2.
|
||||
- No frontend-owned downloader logic.
|
||||
|
||||
* Proposed Components
|
||||
** Backend worker service
|
||||
Owns:
|
||||
- request validation
|
||||
- single active job state
|
||||
- yt-dlp configuration loading and merge rules
|
||||
- download execution
|
||||
- archive behavior
|
||||
- output path behavior
|
||||
- progress and final result events
|
||||
|
||||
Does not own:
|
||||
- Discord slash-command parsing
|
||||
- chat formatting decisions
|
||||
- access control policy
|
||||
|
||||
** Frontend adapters
|
||||
1. Discord
|
||||
2. Zulip
|
||||
3. XMPP
|
||||
|
||||
Own:
|
||||
- command parsing
|
||||
- user-facing message formatting
|
||||
- transport-specific reply behavior
|
||||
- optional frontend-side gating if the deployment wants it
|
||||
|
||||
Do not own:
|
||||
- yt-dlp option construction
|
||||
- archive checks
|
||||
- output path logic
|
||||
- job lifecycle rules
|
||||
|
||||
* Initial Deployment Shape
|
||||
|
||||
V2 should start as a single private process boundary with two layers:
|
||||
1. backend service process
|
||||
2. one Discord adapter process
|
||||
|
||||
The backend is intended for private/local deployment. Trust comes from deployment topology and network placement, not backend auth.
|
||||
|
||||
** Backend Contract
|
||||
*** Request model
|
||||
Minimal fields:
|
||||
- `url`
|
||||
- `requester_id` optional
|
||||
- `requester_name` optional
|
||||
- `origin` optional
|
||||
- `requested_at` optional
|
||||
Only `url` affects downloader behavior in v2. The rest is passthrough metadata for logs and frontend context.
|
||||
|
||||
*** Result states
|
||||
The backend should normalize job outcomes into a small stable vocabulary:
|
||||
- `accepted`
|
||||
- `busy`
|
||||
- `validating`
|
||||
- `downloading`
|
||||
- `skipped`
|
||||
- `completed`
|
||||
- `failed`
|
||||
- `cancelled`
|
||||
|
||||
These states should be transport-agnostic and human-debuggable.
|
||||
|
||||
*** Minimal endpoints
|
||||
The first backend seam should stay small:
|
||||
- `POST /jobs`
|
||||
- `GET /jobs/current`
|
||||
- `POST /jobs/current/cancel`
|
||||
- `GET /health`
|
||||
- `GET /version`
|
||||
|
||||
Polling is the default integration model for v2. Streaming or push delivery is deferred unless a real frontend need appears.
|
||||
|
||||
** Single-Job Semantics
|
||||
V2 explicitly supports one active job at a time.
|
||||
- If idle, the backend accepts a new job and marks it active.
|
||||
- If busy, the backend rejects the new request with a `busy` result.
|
||||
- Cancel is coarse and best-effort.
|
||||
- The backend clears active state when the job reaches `skipped`, `completed`, `failed`, or `cancelled`.
|
||||
|
||||
This is a feature, not a limitation, for the initial service.
|
||||
|
||||
* yt-dlp Configuration Ownership
|
||||
|
||||
This is the most important v2 cleanup.
|
||||
|
||||
The backend owns all yt-dlp configuration behavior. Frontends must not build yt-dlp option dictionaries.
|
||||
|
||||
** Config split
|
||||
|
||||
Static settings belong in `default-yt-dlp.conf`:
|
||||
- archive path
|
||||
- output template
|
||||
- embedding and metadata preferences
|
||||
- stable format defaults
|
||||
- retry defaults
|
||||
|
||||
Runtime settings belong in backend code:
|
||||
- target URL
|
||||
- progress hooks
|
||||
- cancellation hook
|
||||
- request-scoped metadata
|
||||
- any values that must vary per request
|
||||
|
||||
*** Merge rule
|
||||
The backend should load the default config first, then apply a small set of explicit runtime overrides.
|
||||
|
||||
If a runtime override conflicts with file-backed config, the override wins and the backend should log the effective value used.
|
||||
|
||||
*** Debuggability requirement
|
||||
For each job, the backend should make it easy to inspect:
|
||||
|
||||
- config file path used
|
||||
- whether config loading succeeded
|
||||
- effective key yt-dlp options after merge
|
||||
- final normalized job result
|
||||
|
||||
This exists specifically because the current config behavior has been difficult to reason about.
|
||||
|
||||
*** Config hygiene expectations
|
||||
|
||||
The default config should be safe for real downloads in production-like use.
|
||||
|
||||
That means test-only settings such as `--simulate` should not remain enabled in the default runtime path unless the backend intentionally supports an explicit dry-run mode.
|
||||
|
||||
** Recommended Repo Shape
|
||||
One reasonable first cut:
|
||||
|
||||
#+begin_quote
|
||||
youdis/
|
||||
README.md
|
||||
default-yt-dlp.conf
|
||||
youdis/
|
||||
__init__.py
|
||||
api.py
|
||||
config.py
|
||||
models.py
|
||||
worker.py
|
||||
ytdlp_backend.py
|
||||
adapters/
|
||||
__init__.py
|
||||
discord.py
|
||||
docs/
|
||||
architecture-v2.md
|
||||
#+end_quote
|
||||
|
||||
Notes:
|
||||
|
||||
- `ytdlp_backend.py` owns yt-dlp integration and result normalization.
|
||||
- `worker.py` owns active-job state and lifecycle.
|
||||
- `api.py` exposes the minimal HTTP seam.
|
||||
- `adapters/discord.py` becomes a thin client of the backend.
|
||||
|
||||
** Framework Recommendation
|
||||
|
||||
FastAPI is the current recommendation for the backend seam.
|
||||
|
||||
Why it fits:
|
||||
|
||||
- typed request and response models help define the contract cleanly
|
||||
- built-in docs make local inspection easier during early iterations
|
||||
- it stays small enough for this service if we keep the surface area disciplined
|
||||
|
||||
Why this is not a strong ideological choice:
|
||||
|
||||
- Flask would also work if we decide the type/model ergonomics are not worth it
|
||||
- the important decision is keeping the seam small, not choosing a fashionable framework
|
||||
|
||||
** Immediate Next Task
|
||||
|
||||
`2.0.1` should implement the backend skeleton with just enough real behavior to prove the seam:
|
||||
|
||||
- package/module layout
|
||||
- health and version endpoint
|
||||
- single active-job state holder
|
||||
- one submit-job path
|
||||
- one current-job status path
|
||||
- yt-dlp config loading from `default-yt-dlp.conf`
|
||||
- explicit runtime overrides for request URL, progress hooks, and cancel behavior
|
||||
|
||||
The Discord adapter should remain unchanged until that backend skeleton can accept a job and report status coherently.
|
||||
276
pm/tasks-v2.org
276
pm/tasks-v2.org
@@ -1,139 +1,139 @@
|
||||
#+title: Task Log
|
||||
#+updated: [2026-03-31 Tue 16:03]
|
||||
|
||||
* youdis v2 goals:
|
||||
1. Separate backend from frontend
|
||||
2. Offload auth
|
||||
3. Ensure auto nightly builds
|
||||
4. Default output format supports plex browsing youtube channels as "tv shows"
|
||||
5. Facilitate multiple GUI inbounds: Discord, Zulip, XMPP
|
||||
|
||||
* [ ] 2.0.0: define architecture (2-4)
|
||||
define the target architecture for a private backend yt-dlp worker with thin chat frontends
|
||||
** pm notes:
|
||||
- keep this iterative. the point is to choose the shape and seam, not prematurely implement infra. likely decisions include backend framework, request/status model, and how thin the discord shim should be.
|
||||
- goal is simple, private, maintainable deployment
|
||||
- avoid auth, queues, or persistence beyond clear & immediate needs
|
||||
|
||||
** Acceptance Criteria
|
||||
1. document the target architecture at a high level
|
||||
- backend owns yt-dlp execution and job state
|
||||
- frontends own chat-specific UX
|
||||
2. identify key decisions still open
|
||||
- backend choice
|
||||
- service seam/endpoints
|
||||
- status/progress model
|
||||
3. capture enough structure to begin implementation
|
||||
- repo/component layout is sketched
|
||||
- next implementation task is unblocked
|
||||
|
||||
** evidence
|
||||
- commit:
|
||||
- tests:
|
||||
- datetime:
|
||||
|
||||
#+title: Task Log
|
||||
#+updated: [2026-03-31 Tue 16:03]
|
||||
|
||||
* youdis v2 goals:
|
||||
1. Separate backend from frontend
|
||||
2. Offload auth
|
||||
3. Ensure auto nightly builds
|
||||
4. Default output format supports plex browsing youtube channels as "tv shows"
|
||||
5. Facilitate multiple GUI inbounds: Discord, Zulip, XMPP
|
||||
|
||||
* [ ] 2.0.0: define architecture (2-4)
|
||||
define the target architecture for a private backend yt-dlp worker with thin chat frontends
|
||||
** pm notes:
|
||||
- keep this iterative. the point is to choose the shape and seam, not prematurely implement infra. likely decisions include backend framework, request/status model, and how thin the discord shim should be.
|
||||
- goal is simple, private, maintainable deployment
|
||||
- avoid auth, queues, or persistence beyond clear & immediate needs
|
||||
|
||||
** Acceptance Criteria
|
||||
1. document the target architecture at a high level
|
||||
- backend owns yt-dlp execution and job state
|
||||
- frontends own chat-specific UX
|
||||
2. identify key decisions still open
|
||||
- backend choice
|
||||
- service seam/endpoints
|
||||
- status/progress model
|
||||
3. capture enough structure to begin implementation
|
||||
- repo/component layout is sketched
|
||||
- next implementation task is unblocked
|
||||
|
||||
** evidence
|
||||
- commit:
|
||||
- tests:
|
||||
- datetime:
|
||||
|
||||
** notes
|
||||
- first architecture draft captured in `docs/architecture-v2.org`
|
||||
|
||||
* [ ] 2.0.1: build backend yt-dlp worker (3)
|
||||
create the minimal backend/service skeleton and establish a working yt-dlp baseline with clean hooks for future frontends
|
||||
** pm notes
|
||||
- foundation; don't need the full finished service here, just the basic shape plus enough real yt-dlp execution to validate the seam and build on it.
|
||||
- keep single-job semantics
|
||||
- prioritize inspectable behavior over polish
|
||||
|
||||
** Acceptance Criteria
|
||||
1. create the backend/service skeleton
|
||||
- app/module layout exists
|
||||
- core request path is stubbed or minimally working
|
||||
2. establish a working yt-dlp baseline
|
||||
- archive behavior is preserved
|
||||
- output path behavior is preserved or intentionally updated
|
||||
- use yt-dlp .conf and set reasonable default
|
||||
3. expose basic hooks/interfaces for future frontends
|
||||
- submit/request path exists
|
||||
- status/progress hook exists
|
||||
- basic health/version visibility exists
|
||||
|
||||
** evidence
|
||||
- commit:
|
||||
- tests:
|
||||
- datetime:
|
||||
|
||||
** notes
|
||||
|
||||
* [ ] 2.0.2: update discord bot to use new backend (3)
|
||||
update the discord bot into a thin frontend that talks to the backend and verify the flow end to end
|
||||
** pm notes
|
||||
- this is the first real frontend proof. once this works cleanly, zulip/xmpp should mostly be adapter work rather than downloader rewrites.
|
||||
- keep discord logic thin; no auth
|
||||
- do not duplicate yt-dlp behavior in the bot
|
||||
|
||||
** Acceptance Criteria
|
||||
1. discord bot submits requests to backend
|
||||
- command/input handling works
|
||||
- acceptance/busy/failure responses are clear
|
||||
2. discord bot relays useful backend status
|
||||
- progress reporting works at a basic level
|
||||
- completion/failure/skipped outcomes are surfaced
|
||||
3. backend-discord flow is tested end to end
|
||||
- valid request path tested
|
||||
- busy or conflict behavior tested
|
||||
- failure path tested
|
||||
|
||||
** evidence
|
||||
- commit:
|
||||
- tests:
|
||||
- datetime:
|
||||
|
||||
** notes
|
||||
|
||||
* [ ] 2.0.3: remove deprecated discord-bot functionality (2)
|
||||
delete or retire legacy bot behaviors that no longer fit once the backend split is in place
|
||||
** pm notes
|
||||
- only remove this after the new path works. this is cleanup, not pioneering work.
|
||||
- favor deletion over compatibility shims
|
||||
- keep operator controls only if still useful
|
||||
|
||||
** Acceptance Criteria
|
||||
1. remove obsolete auth/user-management behavior
|
||||
- old user persistence and commands are removed
|
||||
- backend-facing flow no longer depends on them
|
||||
2. remove obsolete downloader/runtime logic from bot
|
||||
- bot no longer owns yt-dlp execution
|
||||
- dead code paths are deleted
|
||||
3. leave the bot in a coherent state
|
||||
- remaining commands reflect actual supported behavior
|
||||
- deprecated artifacts are clearly removed or marked
|
||||
|
||||
** evidence
|
||||
- commit:
|
||||
- tests:
|
||||
- datetime:
|
||||
|
||||
** notes
|
||||
|
||||
* [ ] 2.0.5: fix automation and build pipeline (3)
|
||||
repair and simplify the build/update/deploy path so it matches the new backend-plus-frontend structure
|
||||
** pm notes
|
||||
- this should come after architecture and discord integration stabilize. no point polishing the pipeline for the wrong shape.
|
||||
- optimize for simple manual ops first
|
||||
- stop here after pipeline is sane
|
||||
|
||||
** Acceptance Criteria
|
||||
1. align build artifacts with the new structure
|
||||
- docker/build scripts reflect current components
|
||||
- runtime assumptions are consistent
|
||||
2. review old automation artifacts
|
||||
- stale runner/update/restart logic is removed or updated
|
||||
- manual update/rebuild flow is clear
|
||||
3. confirm deployment path works
|
||||
- local or unraid deployment is validated
|
||||
- pipeline is understandable enough to maintain
|
||||
|
||||
** evidence
|
||||
- commit:
|
||||
- tests:
|
||||
- datetime:
|
||||
|
||||
** notes
|
||||
- first architecture draft captured in `docs/architecture-v2.md`
|
||||
|
||||
* [ ] 2.0.1: build backend yt-dlp worker (3)
|
||||
create the minimal backend/service skeleton and establish a working yt-dlp baseline with clean hooks for future frontends
|
||||
** pm notes
|
||||
- foundation; don't need the full finished service here, just the basic shape plus enough real yt-dlp execution to validate the seam and build on it.
|
||||
- keep single-job semantics
|
||||
- prioritize inspectable behavior over polish
|
||||
|
||||
** Acceptance Criteria
|
||||
1. create the backend/service skeleton
|
||||
- app/module layout exists
|
||||
- core request path is stubbed or minimally working
|
||||
2. establish a working yt-dlp baseline
|
||||
- archive behavior is preserved
|
||||
- output path behavior is preserved or intentionally updated
|
||||
- use yt-dlp .conf and set reasonable default
|
||||
3. expose basic hooks/interfaces for future frontends
|
||||
- submit/request path exists
|
||||
- status/progress hook exists
|
||||
- basic health/version visibility exists
|
||||
|
||||
** evidence
|
||||
- commit:
|
||||
- tests:
|
||||
- datetime:
|
||||
|
||||
** notes
|
||||
|
||||
* [ ] 2.0.2: update discord bot to use new backend (3)
|
||||
update the discord bot into a thin frontend that talks to the backend and verify the flow end to end
|
||||
** pm notes
|
||||
- this is the first real frontend proof. once this works cleanly, zulip/xmpp should mostly be adapter work rather than downloader rewrites.
|
||||
- keep discord logic thin; no auth
|
||||
- do not duplicate yt-dlp behavior in the bot
|
||||
|
||||
** Acceptance Criteria
|
||||
1. discord bot submits requests to backend
|
||||
- command/input handling works
|
||||
- acceptance/busy/failure responses are clear
|
||||
2. discord bot relays useful backend status
|
||||
- progress reporting works at a basic level
|
||||
- completion/failure/skipped outcomes are surfaced
|
||||
3. backend-discord flow is tested end to end
|
||||
- valid request path tested
|
||||
- busy or conflict behavior tested
|
||||
- failure path tested
|
||||
|
||||
** evidence
|
||||
- commit:
|
||||
- tests:
|
||||
- datetime:
|
||||
|
||||
** notes
|
||||
|
||||
* [ ] 2.0.3: remove deprecated discord-bot functionality (2)
|
||||
delete or retire legacy bot behaviors that no longer fit once the backend split is in place
|
||||
** pm notes
|
||||
- only remove this after the new path works. this is cleanup, not pioneering work.
|
||||
- favor deletion over compatibility shims
|
||||
- keep operator controls only if still useful
|
||||
|
||||
** Acceptance Criteria
|
||||
1. remove obsolete auth/user-management behavior
|
||||
- old user persistence and commands are removed
|
||||
- backend-facing flow no longer depends on them
|
||||
2. remove obsolete downloader/runtime logic from bot
|
||||
- bot no longer owns yt-dlp execution
|
||||
- dead code paths are deleted
|
||||
3. leave the bot in a coherent state
|
||||
- remaining commands reflect actual supported behavior
|
||||
- deprecated artifacts are clearly removed or marked
|
||||
|
||||
** evidence
|
||||
- commit:
|
||||
- tests:
|
||||
- datetime:
|
||||
|
||||
** notes
|
||||
|
||||
* [ ] 2.0.5: fix automation and build pipeline (3)
|
||||
repair and simplify the build/update/deploy path so it matches the new backend-plus-frontend structure
|
||||
** pm notes
|
||||
- this should come after architecture and discord integration stabilize. no point polishing the pipeline for the wrong shape.
|
||||
- optimize for simple manual ops first
|
||||
- stop here after pipeline is sane
|
||||
|
||||
** Acceptance Criteria
|
||||
1. align build artifacts with the new structure
|
||||
- docker/build scripts reflect current components
|
||||
- runtime assumptions are consistent
|
||||
2. review old automation artifacts
|
||||
- stale runner/update/restart logic is removed or updated
|
||||
- manual update/rebuild flow is clear
|
||||
3. confirm deployment path works
|
||||
- local or unraid deployment is validated
|
||||
- pipeline is understandable enough to maintain
|
||||
|
||||
** evidence
|
||||
- commit:
|
||||
- tests:
|
||||
- datetime:
|
||||
|
||||
** notes
|
||||
|
||||
Reference in New Issue
Block a user