Compare commits

..

43 Commits

Author SHA1 Message Date
ben
18537b9de9 removed duplicate command registration 2026-04-02 17:54:45 -04:00
ben
1757318523 consolidated env vars and .env 2026-04-02 17:14:05 -04:00
ben
90b9dad59c Add configurable adapter selection to app runner 2026-04-02 14:47:03 -04:00
ben
309ce87abb Add app runner for backend and discord stack 2026-04-02 12:41:53 -04:00
ben
4abd39f884 finished discord > adapter functionality 2026-04-02 12:27:31 -04:00
ben
266ed86217 added docstring 2026-04-02 12:27:22 -04:00
ben
0aa9950e18 cleanup discord adapter artifacts 2026-04-02 12:19:56 -04:00
ben
043c8b4df3 implemented discord adapter 2026-04-02 11:57:54 -04:00
ben
5210d2c032 moved discord functionality into adapter 2026-04-02 10:53:48 -04:00
ben
8b21034d78 configured initial yt-dlp worker fallback download location 2026-03-31 21:20:46 -04:00
ben
2a5648506e initial ytdlp worker 2026-03-31 20:54:56 -04:00
ben
7926534e54 initial ytdlp worker 2026-03-31 20:54:23 -04:00
ben
0ed16eca62 finalized v2 architecture for 2.0.1 2026-03-31 18:48:10 -04:00
ben
35a2592dce draft arch-v2 2026-03-31 16:53:28 -04:00
5f78ac6a1a added v2 tasks 2026-03-31 16:23:37 -04:00
2beb54f219 udpated pm note format 2026-03-31 16:22:46 -04:00
f546c9fbb7 update 1.1.2 2026-03-31 14:01:25 -04:00
667b06fe4a enforce single-job blocking 2026-03-31 13:45:17 -04:00
033d9dd167 load authorized_users.json robustly 2026-03-31 13:24:25 -04:00
7a97e1f23d updated gitignore with template 2026-03-31 10:44:06 -04:00
a399a91ab0 fixed typo
All checks were successful
push new builds to gitea, dockerhub / build (push) Successful in 1m27s
2025-08-29 20:22:34 -04:00
ec72c5657b reindex files to force unix line endings in .sh files
All checks were successful
push new builds to gitea, dockerhub / build (push) Successful in 1m26s
2025-08-29 16:24:33 -04:00
f8f625f0bf fix typo
All checks were successful
push new builds to gitea, dockerhub / build (push) Successful in 1m6s
2025-08-29 16:11:29 -04:00
22fb495e0d fix docker syntax
Some checks failed
push new builds to gitea, dockerhub / build (push) Failing after 2m40s
2025-08-29 15:29:03 -04:00
33d5de6a77 update ytdlp on container start; cleanup gitea workflow
Some checks failed
push new builds to gitea, dockerhub / build (push) Failing after 38s
2025-08-29 15:22:10 -04:00
d539915486 cleanup and remove github workflow
Some checks failed
weekly, if new yt-dlp, build + push container to docker/gitea / build (push) Failing after 1m6s
2025-08-24 22:42:15 -04:00
f173c64b70 Revert "removed github workflow"
Some checks failed
weekly, if new yt-dlp, build + push container to docker/gitea / build (push) Failing after 26s
This reverts commit 750dc265ca.
2025-08-24 22:40:20 -04:00
750dc265ca removed github workflow
Some checks failed
weekly, if new yt-dlp version, build + push containers to docker / build (push) Failing after 27s
2025-08-24 22:29:45 -04:00
846d07f099 fixed gitea api url
Some checks failed
weekly, if new yt-dlp, build + push container to docker/gitea / build (push) Failing after 1m7s
2025-08-24 22:25:19 -04:00
adca108376 chagnged to link using api
Some checks failed
weekly, if new yt-dlp, build + push container to docker/gitea / build (push) Failing after 1m45s
2025-08-24 22:16:05 -04:00
748e3fb22a gitea vars redux
Some checks failed
weekly, if new yt-dlp, build + push container to docker/gitea / build (push) Failing after 58s
2025-08-24 19:00:35 -04:00
0968aa84b8 fixed username
Some checks failed
weekly, if new yt-dlp, build + push container to docker/gitea / build (push) Failing after 59s
2025-08-24 17:59:28 -04:00
b74f70c42e added gitea repo variables
Some checks failed
weekly, if new yt-dlp, build + push container to docker/gitea / build (push) Failing after 49s
2025-08-24 17:56:31 -04:00
07cdae823a gitea registry /youdis, tags ben/youdis
Some checks failed
weekly, if new yt-dlp, build + push container to docker/gitea / build (push) Failing after 2m32s
2025-08-24 17:20:05 -04:00
81619074d7 fixed youdis shortsha syntax
All checks were successful
weekly, if new yt-dlp, build + push container to docker/gitea / build (push) Successful in 2m29s
2025-08-24 15:35:21 -04:00
690f97ed82 consol. dev build tag; push directly to youdis container repo
Some checks failed
build + push containers to gitea (every) and docker (daily) / build (push) Failing after 56s
2025-08-24 15:28:44 -04:00
b772383d14 collapsed to single job
All checks were successful
build + push containers to gitea (every) and docker (daily) / build (push) Successful in 3m2s
2025-08-24 14:57:32 -04:00
53fbcc5a8a fix gitea user 2025-08-24 14:28:34 -04:00
713f316679 fixed: secrets cannot begin with "GITEA_" or "GITHUB_" 2025-08-24 14:24:14 -04:00
08bcc3d3bd split jobs into versioining, dockerhub, gitea 2025-08-23 14:43:48 -04:00
0abf3a5adb added ytdlp/youdis version tags, check youdis:latest before build
Some checks failed
build + push containers to gitea (every) and docker (daily) / docker (push) Has been cancelled
2025-08-23 14:36:30 -04:00
32ed9bc81b update version to 20250823-b37e943
Some checks failed
build + push containers to gitea (every) and docker (daily) / dockerhub (push) Failing after 25s
build + push containers to gitea (every) and docker (daily) / gitea (push) Failing after 15s
2025-08-23 12:15:11 -04:00
b37e943a8a added workflow.txt 2025-08-23 12:15:05 -04:00
30 changed files with 1870 additions and 365 deletions

12
.env.example Normal file
View File

@@ -0,0 +1,12 @@
YOUDIS_ENABLE_BACKEND=1
YOUDIS_BACKEND_HOST=127.0.0.1
YOUDIS_BACKEND_PORT=8000
YOUDIS_BACKEND_HEALTH_TIMEOUT=20
YOUDIS_BACKEND_HEALTH_INTERVAL=0.5
YOUDIS_POLL_INTERVAL_SECONDS=2
YOUDIS_YTDLP_EXECUTABLE=yt-dlp
YOUDIS_CONFIG_DIR=/home/user/proj/youdis/test/
YOUDIS_DOWNLOAD_DIR=/home/user/proj/youdis/downloads
ENABLE_DISCORD=1
DISCORD_BOT_TOKEN=
DISCORD_BOT_SCOPE=2147491904

1
.gitattributes vendored Normal file
View File

@@ -0,0 +1 @@
*.sh text eol=lf

View File

@@ -1,70 +1,72 @@
name: build + push containers to gitea (every) and docker (daily)
on:
push:
branches: [ "master" ]
schedule:
- cron: '0 0 * * 0' # weekly check
workflow_dispatch:
jobs:
dockerhub:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Get current version
id: current
run: |
CURRENT=$(cat version.txt || echo "none")
echo "version=$CURRENT" >> $GITHUB_OUTPUT
- name: check latest yt-dlp release
id: latest
run: |
LATEST=$(curl -s https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest | jq -r .tag_name || echo "error")
if [ "$LATEST" == "error" ]; then
echo "failed to fetch latest version" >&2
exit 1
fi
echo "yt-dlp version=$LATEST" >> $GITHUB_OUTPUT
- name: Update version file if changed
run: echo ${{ steps.latest.outputs.version }} > version.txt
if: ${{ steps.current.outputs.version != steps.latest.outputs.version }}
- name: Login to Dockerhub
uses: docker/login-action@v2
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Build and push to Dockerhub
uses: docker/build-push-action@v5
with:
push: true
tags: |
eulaly/youdis:latest
eulaly/youdis:${{ steps.latest.outputs.version }}
if: ${{ steps.current.outputs.version != steps.latest.outputs.version }}
gitea:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Login to Gitea (local)
uses: docker/login-action@v2
with:
registry: git.hgsky.me
username: ${{ secrets.GITEA_USERNAME }}
password: ${{ secrets.GITEA_TOKEN }}
- name: Build and push to Gitea (all pushes)
uses: docker/build-push-action@v5
with:
push: true
tags: |
git.hgsky.me/eulaly/youdis:latest
git.hgsky.me/eulaly/youdis:dev-${{ github.sha }}
name: push new builds to gitea, dockerhub
on:
push:
branches: ["master"]
workflow_dispatch:
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Get youdis version
id: youdis
run: |
YOUDIS_VER=$(cat youdis-version.txt)
if [ -z "$YOUDIS_VER" ]; then
echo "youdis version empty" >&2
exit 1
fi
echo "youdis-version=$YOUDIS_VER" >> $GITHUB_OUTPUT
echo "shortsha=${GITHUB_SHA::5}" >> $GITHUB_OUTPUT
- name: Decide if build needed
id: check_build
run: |
SHOULD_BUILD=false
if [ "${{ steps.youdis.outputs.youdis-version }}" != "$(docker inspect --format='{{ index .Config.Labels "YOUDIS_VERSION" }}' eulaly/youdis:latest || echo none)" ]; then
SHOULD_BUILD=true
fi
echo "should_build=$SHOULD_BUILD" >> $GITHUB_OUTPUT
- name: Login to Dockerhub
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Build and push to Dockerhub
if: ${{ steps.check_build.outputs.should_build == 'true' }}
uses: docker/build-push-action@v5
with:
push: true
tags: |
eulaly/youdis:latest
eulaly/youdis:${{ steps.youdis.outputs.shortsha }}
labels: |
YOUDIS_VERSION=${{ steps.youdis.outputs.youdis-version }}
- name: Login to Gitea
uses: docker/login-action@v3
with:
registry: git.hgsky.me
username: ${{ secrets.USERNAME }}
password: ${{ secrets.YOUDIS_PASSWORD }}
- name: Build and push to Gitea
if: ${{ steps.check_build.outputs.should_build == 'true' }}
uses: docker/build-push-action@v5
with:
push: true
tags: |
git.hgsky.me/ben/youdis:latest
git.hgsky.me/ben/youdis:${{ steps.youdis.outputs.shortsha }}
labels: |
YOUDIS_VERSION=${{ steps.youdis.outputs.youdis-version }}
- name: link container in gitea
run: |
curl -X POST -H "Authorization: token ${{ secrets.SUPER }}" https://git.hgsky.me/api/v1/packages/ben/container/youdis/-/link/youdis

View File

@@ -1,44 +0,0 @@
name: Check yt-dlp and rebuild if new
on:
schedule:
- cron: '0 0 * * 0'
workflow_dispatch:
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Get current version
id: current
run: |
CURRENT=$(cat version.txt || echo "none")
echo "version=$CURRENT" >> $GITHUB_OUTPUT
- name: Check latest version
id: latest
run: |
LATEST=$(curl -s https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest | jq -r .tag_name || echo "error")
if [ "$LATEST" == "error" ]; then
echo "failed to fetch latest version" >&2
exit 1
fi
echo "version=$LATEST" >> $GITHUB_OUTPUT
- name: Update version file if changed
run: echo ${{ steps.latest.outputs.version }} > version.txt
if: ${{ steps.current.outputs.version != steps.latest.outputs.version }}
- name: Login to Dockerhub
uses: docker/login-action@v2
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Build and push docker image
uses: docker/build-push-action@v5
with:
push: true
tags: eulaly/youdis:latest,eulaly/youdis:${{ steps.latest.outputs.version }}
if: ${{ steps.current.outputs.version != steps.latest.outputs.version }}

39
.gitignore vendored
View File

@@ -1,13 +1,36 @@
*.pyc
*.org
*.ps1
.env
#*
.#*
# --- python bytecode ---
__pycache__/
*.py[cod]
*$py.class
# --- virtual environments ---
.venv/
venv/
env/
# --- environment files ---
.env
.env.*
!.env.example
*.local
# --- emacs ---
*~
\#*\#
.\#*
*.elc
# --- project private data ---
/private/
archive/
config/
downloads/
test/
data.json
# --- django ---
db.sqlite3
staticfiles/
media/
# --- misc ---
.DS_Store

48
LICENSE
View File

@@ -1,24 +1,24 @@
This is free and unencumbered software released into the public domain.
Anyone is free to copy, modify, publish, use, compile, sell, or
distribute this software, either in source code form or as a compiled
binary, for any purpose, commercial or non-commercial, and by any
means.
In jurisdictions that recognize copyright laws, the author or authors
of this software dedicate any and all copyright interest in the
software to the public domain. We make this dedication for the benefit
of the public at large and to the detriment of our heirs and
successors. We intend this dedication to be an overt act of
relinquishment in perpetuity of all present and future rights to this
software under copyright law.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR
OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
OTHER DEALINGS IN THE SOFTWARE.
For more information, please refer to <https://unlicense.org>
This is free and unencumbered software released into the public domain.
Anyone is free to copy, modify, publish, use, compile, sell, or
distribute this software, either in source code form or as a compiled
binary, for any purpose, commercial or non-commercial, and by any
means.
In jurisdictions that recognize copyright laws, the author or authors
of this software dedicate any and all copyright interest in the
software to the public domain. We make this dedication for the benefit
of the public at large and to the detriment of our heirs and
successors. We intend this dedication to be an overt act of
relinquishment in perpetuity of all present and future rights to this
software under copyright law.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR
OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
OTHER DEALINGS IN THE SOFTWARE.
For more information, please refer to <https://unlicense.org>

View File

@@ -1,8 +1,60 @@
# Running youdis
build and run the docker container
```
api_token = [discord bot token]
-v [downloads]:/downloads
-v [config]:/config
-v [host_downloads]:/downloads
-v [host_config]:/config
-e ENABLE_DISCORD=1
-e DISCORD_BOT_TOKEN=<token>
-e DISCORD_BOT_SCOPE=<scope-int>
```
config contains data to persist across container updates, i.e., unraid appdata,
such as users.json (authorized users) and yt-dlp's archive.txt
such as yt-dlp's archive.txt
# Development
v2 architecture draft: `docs/architecture-v2.org`
The app runs with `youdis.py`.
This starts the backend first, waits for health, then starts each enabled adapter explicitly.
Test components directly with uvicorn:
```
python3 -m uvicorn youdis.main:app --host 127.0.0.1 --port 8000
python3 -m youdis.adapters.discord
```
Key runner/config vars:
```
YOUDIS_ENABLE_BACKEND=1 #default enabled, dont disable unless testing
YOUDIS_BACKEND_HOST=127.0.0.1
YOUDIS_BACKEND_PORT=8000
YOUDIS_BACKEND_HEALTH_TIMEOUT=20
YOUDIS_BACKEND_HEALTH_INTERVAL=0.5
YOUDIS_POLL_INTERVAL_SECONDS=2
YOUDIS_YTDLP_EXECUTABLE=yt-dlp
YOUDIS_CONFIG_DIR=/path/to/config/
YOUDIS_DOWNLOAD_DIR=/path/to/downloads/
ENABLE_DISCORD=1
DISCORD_BOT_TOKEN=<token-string>
DISCORD_BOT_SCOPE=123456789
```
## Add Adapter
Configure adapter selection in .env, like so:
```
YOUDIS_RUN_BACKEND=1
YOUDIS_BACKEND_HOST=127.0.0.1
YOUDIS_BACKEND_PORT=8000
ENABLE_DISCORD=1
DISCORD_BOT_TOKEN=<api_key>
DISCORD_BOT_SCOPE=123456789
```
to add a new adapter:
1. add `youdis/adapters/<adapter>.py`
2. make it independently runnable
3. add `start_<adapter>()` to `youdis.py`
4. register it in the explicit adapter starter map in `youdis.py`
5. add `ENABLE_<adapter>` and supporting env vars to `.env.example`

24
agents.md Normal file
View File

@@ -0,0 +1,24 @@
# agent rules
## priorities
- optimize for simplicity, boringness, and long-term maintainability
- prefer minimal diffs; avoid refactors unless required for the active task
## tech stack
- python
- file storage: json and csv, no sqlite or databases
- assume local virtual env is available and accessible
- do not add new dependencies unless explicitly approved; if unavoidable, document justification in the active task notes
## workflow
- work on ONE task at a time unless explicitly instructed otherwise
- at the start of work, state the task id you are executing
- do not start work unless a task id is specified; if missing, choose the earliest unchecked task and say so
- propose incremental steps
- always include basic tests for core logic
- add assumptions and questions along-the-way to the ** notes section under the active task
- when you complete a task:
- mark it [X] in pm/tasks.org
- fill in evidence with commit hash + commands run
- never mark complete unless acceptance criteria are met
- include date and time (HH:MM)

File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 24 KiB

After

Width:  |  Height:  |  Size: 24 KiB

View File

@@ -1,8 +0,0 @@
#!/bin/sh
#copy yt-dlp.conf if missing
mkdir -p /config
if [ ! -f /config/yt-dlp.conf ]; then
cp /app/default-yt-dlp.conf /config/yt-dlp.conf
fi
exec "$@"

View File

@@ -1,11 +1,8 @@
# yt-dlp config file (yt-dlp.conf or .config/yt-dlp/config)
--simulate
# yt-dlp config file
-f "bestvideo[ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]/best"
--embed-chapters
--embed-info-json
--write-playlist-metafiles
#--paths "{"home":"./downloads"}"
--download-archive "/config/archive.txt"
--restrict-filenames
--output "/downloads/%(uploader)s/%(playlist_title)s/%(playlist_index)s%(playlist_index& - )s%(title)s.%(ext)s"
# --split-chapters

View File

@@ -1,14 +1,16 @@
# syntax=docker/dockerfile:1
FROM python:3.12.1-alpine3.18
WORKDIR /app
RUN apk update && \
apk add --no-cache build-base ffmpeg && \
rm -rf /var/cache/apk/*
COPY requirements.txt requirements.txt
RUN python3 -m pip install --no-cache-dir -r requirements.txt
COPY youdis.py youdis.py
COPY default-yt-dlp.conf /app/default-yt-dlp.conf
COPY create-ytdlp-config.sh /app/create-ytdlp-config.sh
RUN chmod 755 /app/create-ytdlp-config.sh
ENTRYPOINT ["/app/create-ytdlp-config.sh"]
CMD ["python3", "youdis.py"]
# syntax=docker/dockerfile:1
FROM python:3.12.1-alpine3.18
WORKDIR /app
RUN apk update && \
apk add --no-cache build-base ffmpeg && \
rm -rf /var/cache/apk/*
COPY requirements.txt .
RUN python3 -m pip install --no-cache-dir -r requirements.txt
COPY youdis.py default-yt-dlp.conf update-ytdlp.sh run-youdis.sh /app/
RUN chmod +x /app/update-ytdlp.sh /app/run-youdis.sh
COPY weekly-restart /etc/cron.d/
RUN chmod 0644 /etc/cron.d/weekly-restart
ENTRYPOINT ["/app/run-youdis.sh"]

248
docs/architecture-v2.org Normal file
View File

@@ -0,0 +1,248 @@
#+TITLE: youdis v2 architecture
#+DATE: [2026-03-31 Tue]
#+VERSION: 3
* Purpose
Youdis v2 splits the current monolithic Discord bot into:
- a private backend worker service that owns yt-dlp execution
- thin chat adapters that translate user actions into backend requests
The goal is to make downloader behavior inspectable, reusable, and easier to debug across multiple frontends without introducing queueing infrastructure or complex deployment.
** Goals
- Keep the backend transport-agnostic.
- Preserve archive-first duplicate prevention via `archive.txt`.
- Keep single-job semantics explicit.
- Treat requester and origin metadata as informational, not authorization.
- Make yt-dlp configuration understandable and debuggable.
** Non-Goals
- No backend auth or user membership logic.
- No database, Redis, or external queue.
- No multi-worker scheduling in v2.
- No frontend-owned downloader logic.
* Proposed Components
** Backend worker service
Owns:
- request validation
- single active job state
- yt-dlp executable invocation
- yt-dlp configuration selection and runtime argument construction
- archive behavior
- output path behavior
- progress and final result events
Does not own:
- Discord slash-command parsing
- chat formatting decisions
- access control policy
** Frontend adapters
1. Discord
2. Zulip
3. XMPP
Own:
- command parsing
- user-facing message formatting
- transport-specific reply behavior
- optional frontend-side gating if the deployment wants it
Do not own:
- yt-dlp option construction
- archive checks
- output path logic
- job lifecycle rules
* Initial Deployment Shape
V2 should start as a single private process boundary with two layers:
1. backend service process
2. one Discord adapter process
The backend is intended for private/local deployment. Trust comes from deployment topology and network placement, not backend auth.
** Execution model
The backend is a thin FastAPI wrapper around the `yt-dlp` executable, not a deep Python integration, unless proven necessary.
Why this is the default:
- keeps behavior closer to native `yt-dlp`
- makes container debugging easier because commands can be reproduced manually
- avoids reimplementing `yt-dlp` option parsing unless we truly need tighter integration
This means the backend is primarily responsible for:
- deciding when `yt-dlp` may run
- constructing a safe effective command
- supervising the subprocess while it is active
- translating process outcomes into a small API contract
** Backend Contract
*** Request model
Minimal fields:
- `url`
- `requester_id` optional
- `requester_name` optional
- `origin` optional
- `requested_at` optional
Only `url` affects downloader behavior in v2. The rest is passthrough metadata for logs and frontend context.
*** Result states
The backend should keep its external state model deliberately small:
- `accepted`
- `busy`
- `running`
- `completed`
- `failed`
- `cancelled`
These states should be transport-agnostic, human-debuggable, and only distinct when a frontend would behave differently.
Internal phases and details may be richer, but they should usually be carried as metadata rather than promoted to top-level states.
Examples of detail fields that can travel alongside the small state model:
- `phase=validating`
- `phase=downloading`
- `disposition=archive_hit`
- `message=already in archive`
- `result_path=/downloads/...`
This keeps the API simple while still leaving room for useful operator-facing detail.
*** Minimal endpoints
The first backend seam should stay small:
- `POST /jobs`
- `GET /jobs/current`
- `POST /jobs/current/cancel`
- `GET /health`
- `GET /version`
Polling is the default integration model for v2. Streaming or push delivery is deferred unless a real frontend need appears.
** Single-Job Semantics
V2 explicitly supports one active job at a time.
- If idle, the backend accepts a new job and marks it active.
- If busy, the backend rejects the new request with a `busy` result.
- Cancel is coarse and best-effort.
- The backend clears active state when the job reaches `completed`, `failed`, or `cancelled`.
This is a feature, not a limitation, for the initial service.
* yt-dlp Configuration Ownership
This is the most important v2 cleanup.
The backend owns all yt-dlp invocation behavior. Frontends must not build `yt-dlp` commands or option dictionaries.
** Config split
Static settings belong in `default-yt-dlp.conf`:
- archive path
- output template
- embedding and metadata preferences
- stable format defaults
- retry defaults
Runtime settings belong in backend code:
- target URL
- config-file path, if explicitly set at launch
- request-scoped flags or overrides
- subprocess lifecycle and cancellation behavior
- any values that must vary per request
*** Merge rule
The backend should invoke `yt-dlp` with the default config first, then apply a small set of explicit runtime overrides.
If a runtime override conflicts with file-backed config, the override wins and the backend should log the effective value used.
*** Debuggability requirement
For each job, the backend should make it easy to inspect:
- config file path used
- effective `yt-dlp` command or key override arguments
- final normalized job result
This exists specifically because the current config behavior has been difficult to reason about.
*** Config hygiene expectations
The default config should be safe for real downloads in production-like use.
That means test-only settings such as `--simulate` should not remain enabled in the default runtime path unless the backend intentionally supports an explicit dry-run mode.
** Recommended Repo Shape
One reasonable first cut:
#+begin_quote
youdis/
README.md
default-yt-dlp.conf
youdis/
__init__.py
main.py
models.py
adapters/
__init__.py
discord.py
docs/
architecture-v2.org
#+end_quote
Notes:
- `main.py` owns the FastAPI app, active-job state, and `yt-dlp` subprocess lifecycle.
- `models.py` owns request and response models.
- `adapters/discord.py` becomes a thin client of the backend.
** Example response shape
One likely response shape for v1:
#+begin_src json
{
"state": "completed",
"disposition": "archive_hit",
"message": "already in archive",
"job_id": "abc123"
}
#+end_src
And for an active job:
#+begin_src json
{
"state": "running",
"phase": "downloading",
"message": "downloading 3 of 9",
"job_id": "abc123"
}
#+end_src
** Framework Recommendation
FastAPI is the current recommendation for the backend seam.
Why it fits:
- typed request and response models help define the contract cleanly
- built-in docs make local inspection easier during early iterations
- it stays small enough for this service if we keep the surface area disciplined
Why this is not a strong ideological choice:
- Flask would also work if we decide the type/model ergonomics are not worth it
- the important decision is keeping the seam small, not choosing a fashionable framework
** Immediate Next Task
`2.0.1` should implement the backend skeleton with just enough real behavior to prove the seam:
- package/module layout
- health and version endpoint
- single active-job state holder
- one submit-job path
- one current-job status path
- `yt-dlp` subprocess invocation using `default-yt-dlp.conf`
- explicit runtime overrides for request URL and cancel behavior
The Discord adapter should remain unchanged until that backend skeleton can accept a job and report status coherently.

81
pm/task-sample.org Normal file
View File

@@ -0,0 +1,81 @@
#+title: Task Log
#+updated: [2026-03-31 Tue 16:03]
Use the template below, which should be a top-level org-mode header.
A sample task is below the template
* [ ] M.m.m: Task Title (<estimated # of commits>)
description of the task
** pm notes: amplifying information
** Acceptance Criteria
1. Criterion
- expanded data
2. Criterion
3f. Criterion added after initial task completion
** evidence
- commit: abc123, bcd234
- tests:
- datetime: [2026-03-18 Wed 14:15]
** notes
- explanation of work done, decisions made, reasoning
* [ ] 2.0.1: build backend yt-dlp worker (3)
create the minimal backend/service skeleton and establish a working yt-dlp baseline with clean hooks for future frontends
** pm notes
- foundation; don't need the full finished service here, just the basic shape plus enough real yt-dlp execution to validate the seam and build on it.
- keep single-job semantics
- prioritize inspectable behavior over polish
** Acceptance Criteria
1. create the backend/service skeleton
- app/module layout exists
- core request path is stubbed or minimally working
2. establish a working yt-dlp baseline
- archive behavior is preserved
- output path behavior is preserved or intentionally updated
- use yt-dlp .conf and set reasonable default
3. expose basic hooks/interfaces for future frontends
- submit/request path exists
- status/progress hook exists
- basic health/version visibility exists
4f. print env vars to stdout on fastapi launch
- repo_root
- default_config
- ytdlp_executable
5f. create ytdlp conf to facilitate tests
** evidence
- commit:
- tests:
1. `python3 -m py_compile ./youdis/main.py ./youdis/models.py ./youdis/adapters/__init__.py ./youdis/adapters/discord.py`
2. `python3 -m uvicorn youdis.main:app --host 127.0.0.1 --port 8000`
3. `sudo curl -L https://github.com/yt-dlp/yt-dlp/releases/download/2026.03.17/yt-dlp -o /usr/local/bin/yt-dlp`
4. `sudo chmod +x /usr/local/bin/yt-dlp`
5. `yt-dlp --version`
6. `curl http://127.0.0.1:8000/health`
7. `curl http://127.0.0.1:8000/version`
8. `curl -X POST http://127.0.0.1:8000/jobs -H 'content-type: application/json' -d '{"url":"https://www.youtube.com/watch?v=dQw4w9WgXcQ"}'`
9. `curl http://127.0.0.1:8000/jobs/current`
10. if testing in container: `docker build -t youdis:v2 .`
11. if testing in container: `docker run --rm -p 8000:8000 -v [config]:/config -v [downloads]:/downloads youdis:v2`
:OUTPUT_STEP1-9:
user@paladin:~/proj/youdis$ curl http://127.0.0.1:8000/health
{"status":"ok"}user@paladin:~/proj/youdis$
user@paladin:~/proj/youdis$ curl http://127.0.0.1:8000/version
{"version":"20250829-ec72c56","active_job":false}user@paladin:~/proj/youdis$
user@paladin:~/proj/youdis$ curl -X POST http://127.0.0.1:8000/jobs -H 'content-type: application/json' -d '{"url":"https://www.youtube.com/watch?v=3i72yY_LaW4"}'
{"job_id":"cc85165e-d906-4eee-864f-1a398b6de2e0","state":"accepted","url":"https://www.youtube.com/watch?v=3i72yY_LaW4","message":"accepted","phase":"queued","disposition":null,"requester_id":null,"requester_name":null,"origin":null,"result_path":null,"command":[],"returncode":null,"created_at":"2026-04-01T00:44:35.657196","updated_at":"2026-04-01T00:44:35.657198"}user@paladin:~/proj/youdis$ curl http://127.0.0.1:8000/jobs/current
{"active":false,"job":{"job_id":"cc85165e-d906-4eee-864f-1a398b6de2e0","state":"failed","url":"https://www.youtube.com/watch?v=3i72yY_LaW4","message":"ERROR: unable to create directory [Errno 13] Permission denied: '/downloads'","phase":null,"disposition":null,"requester_id":null,"requester_name":null,"origin":null,"result_path":null,"command":["yt-dlp","--config-locations","/home/user/proj/youdis/default-yt-dlp.conf","https://www.youtube.com/watch?v=3i72yY_LaW4"],"returncode":1,"created_at":"2026-04-01T00:44:35.657196","updated_at":"2026-03-31T20:44:36.653353"}}user@paladin:~/proj/youdis$
:END:
- datetime: [2026-03-31 Tue 20:45]
** notes
- validate `yt-dlp` subprocess invocation in-container; not verifiable in the current shell because `yt-dlp` is not installed here
- confirm `--config-locations` behavior against the installed `yt-dlp` version during integration testing
- current backend scaffold is not yet wired into `dockerfile` or `run-youdis.sh`
- archive-hit and result-path parsing currently depend on `yt-dlp` stdout text patterns, so treat them as provisional until integration-tested

349
pm/tasks-v2.org Normal file
View File

@@ -0,0 +1,349 @@
#+title: Task Log
#+updated: [2026-03-31 Tue 16:03]
#+startup: overview
# Local Variables:
# org-babel-python-command: "/home/user/proj/youdis/.venv/bin/python"
# project-dir: "~/proj/youdis"
# End:
* youdis v2 goals:
1. Separate backend from frontend
2. Offload auth
3. Ensure auto nightly builds
4. Default output format supports plex browsing youtube channels as "tv shows"
5. Facilitate multiple GUI inbounds: Discord, Zulip, XMPP
* [X] 2.0.0: define architecture (2-4)
define the target architecture for a private backend yt-dlp worker with thin chat frontends
** pm notes:
- keep this iterative. the point is to choose the shape and seam, not prematurely implement infra. likely decisions include backend framework, request/status model, and how thin the discord shim should be.
- goal is simple, private, maintainable deployment
- avoid auth, queues, or persistence beyond clear & immediate needs
** Acceptance Criteria
1. document the target architecture at a high level
- backend owns yt-dlp execution and job state
- frontends own chat-specific UX
2. identify key decisions still open
- backend choice
- service seam/endpoints
- status/progress model
3. capture enough structure to begin implementation
- repo/component layout is sketched
- next implementation task is unblocked
** evidence
- commit: 0ed16ec
- tests: n/a
- datetime: [2026-03-31 Tue 18:48]
** notes
- first architecture draft captured in `docs/architecture-v2.org`
* [X] 2.0.1: build backend yt-dlp worker (3)
create the minimal backend/service skeleton and establish a working yt-dlp baseline with clean hooks for future frontends
** pm notes
- foundation; don't need the full finished service here, just the basic shape plus enough real yt-dlp execution to validate the seam and build on it.
- keep single-job semantics
- prioritize inspectable behavior over polish
** Acceptance Criteria
1. create the backend/service skeleton
- app/module layout exists
- core request path is stubbed or minimally working
2. establish a working yt-dlp baseline
- archive behavior is preserved
- output path behavior is preserved or intentionally updated
- use yt-dlp .conf and set reasonable default
3. expose basic hooks/interfaces for future frontends
- submit/request path exists
- status/progress hook exists
- basic health/version visibility exists
4. make local testing practical without breaking container defaults
- backend can run when `/config` or `/downloads` are not writable in local dev
- env vars may override executable/config/download paths
- status makes the effective runtime command/paths inspectable
** evidence
- commit: 7926534,2a56485
- tests:
1. `python3 -m py_compile ./youdis/main.py ./youdis/models.py ./youdis/adapters/__init__.py ./youdis/adapters/discord.py`
2. `python3 -m uvicorn youdis.main:app --host 127.0.0.1 --port 8000`
3. `sudo curl -L https://github.com/yt-dlp/yt-dlp/releases/download/2026.03.17/yt-dlp -o /usr/local/bin/yt-dlp`
4. `sudo chmod +x /usr/local/bin/yt-dlp`
5. `yt-dlp --version`
6. `curl http://127.0.0.1:8000/health`
7. `curl http://127.0.0.1:8000/version`
8. `curl -X POST http://127.0.0.1:8000/jobs -H 'content-type: application/json' -d '{"url":"https://www.youtube.com/watch?v=dQw4w9WgXcQ"}'`
9. `curl http://127.0.0.1:8000/jobs/current`
10. if testing in container: `docker build -t youdis:v2 .`
11. if testing in container: `docker run --rm -p 8000:8000 -v [config]:/config -v [downloads]:/downloads youdis:v2`
:OUTPUT_STEP1-9:
user@paladin:~/proj/youdis$ curl http://127.0.0.1:8000/health
{"status":"ok"}
user@paladin:~/proj/youdis$ curl http://127.0.0.1:8000/version
{"version":"20250829-ec72c56","active_job":false}
user@paladin:~/proj/youdis$ curl -X POST http://127.0.0.1:8000/jobs -H 'content-type: application/json' -d '{"url":"https://www.youtube.com/watch?v=3i72yY_LaW4"}'
{"job_id":"cc85165e-d906-4eee-864f-1a398b6de2e0","state":"accepted","url":"https://www.youtube.com/watch?v=3i72yY_LaW4","message":"accepted","phase":"queued","disposition":null,"requester_id":null,"requester_name":null,"origin":null,"result_path":null,"command":[],"returncode":null,"created_at":"2026-04-01T00:44:35.657196","updated_at":"2026-04-01T00:44:35.657198"}
user@paladin:~/proj/youdis$ curl http://127.0.0.1:8000/jobs/current
{"active":false,"job":{"job_id":"cc85165e-d906-4eee-864f-1a398b6de2e0","state":"failed","url":"https://www.youtube.com/watch?v=3i72yY_LaW4","message":"ERROR: unable to create directory [Errno 13] Permission denied: '/downloads'","phase":null,"disposition":null,"requester_id":null,"requester_name":null,"origin":null,"result_path":null,"command":["yt-dlp","--config-locations","/home/user/proj/youdis/default-yt-dlp.conf","https://www.youtube.com/watch?v=3i72yY_LaW4"],"returncode":1,"created_at":"2026-04-01T00:44:35.657196","updated_at":"2026-03-31T20:44:36.653353"}}user@paladin:~/proj/youdis$
:END:
- datetime: [2026-03-31 Tue 20:45]
** notes
- validate `yt-dlp` subprocess invocation in-container; not verifiable in the current shell because `yt-dlp` is not installed here
- confirm `--config-locations` behavior against the installed `yt-dlp` version during integration testing
- current backend scaffold is not yet wired into `dockerfile` or `run-youdis.sh`
- archive-hit and result-path parsing currently depend on `yt-dlp` stdout text patterns, so treat them as provisional until integration-tested
- local dev now falls back to repo-local `.runtime/{config,downloads}` when `/config` or `/downloads` are not writable
- had to uninstall yt-dlp python pkg from the venv, which resulted in a '403 Forbidden'
* [X] 2.0.2: update discord bot to use new backend (3)
update the discord bot into a thin frontend that talks to the backend and verify the flow end to end
** pm notes
- this is the first real frontend proof. once this works cleanly, zulip/xmpp should mostly be adapter work rather than downloader rewrites.
- keep discord logic thin; no auth
- do not duplicate yt-dlp behavior in the bot
** Acceptance Criteria
1. discord bot submits requests to backend
- command/input handling works
- acceptance/busy/failure responses are clear
2. discord bot relays useful backend status
- progress reporting works at a basic level
- completion/failure/skipped outcomes are surfaced
3. backend-discord flow is tested end to end
- valid request path tested
- busy or conflict behavior tested
- failure path tested
4. add dotenv support to ease dev
- os.getenv methods remain standard for prod/docker build
- populate .env with dev env defaults
** evidence
- commit: 5210d2c, 043cb4
- tests: https://youtu.be/20HxMMSqRyg?si=3v7mN2L88c_FxpQR 18m
1. start backend: `python3 -m uvicorn youdis.main:app --host 127.0.0.1 --port 8000`
2. create local env file: `cp .env.example .env`
3. add `api_token` to `.env`
4. start adapter: `python3 ./youdis.py`
5. in discord, run `/youtube url:https://www.youtube.com/watch?v=dQw4w9WgXcQ`
:out:
#+begin_src shell
(venv) user@paladin:~/proj/youdis$ python ./youdis.py
Task exception was never retrieved
future: <Task finished name='Task-43' coro=<Client._dispatch_interaction() done, defined at /home/user/proj/youdis/venv/lib/python3.10/site-packages/interactions/client/client.py:1773> exception=KeyError('youtube')>
Traceback (most recent call last):
File "/home/user/proj/youdis/venv/lib/python3.10/site-packages/interactions/client/client.py", line 1798, in _dispatch_interaction
if ctx.command:
File "/home/user/proj/youdis/venv/lib/python3.10/site-packages/interactions/models/internal/context.py", line 329, in command
return self.client._interaction_lookup[self._command_name]
KeyError: 'youtube'
#+end_src
:end:
6. confirm channel response says the job was submitted to backend
7. confirm requester receives DM updates for accepted/running/completed or failed
:out:
accepted job 4ef6fde5-57cc-478c-b0e2-18458c693fa4 for https://www.youtube.com/watch?v=dQw4w9WgXcQ
state=running | phase=running | [youtube] dQw4w9WgXcQ: Downloading webpage | path=/home/user/proj/youdis/downloads
state=running | phase=downloading | [download] Destination: /home/user/proj/youdis/downloads/Rick_Astley/NA/NANARickAstley-_Never_Gonna_Give_You_Up_Official_Video_4K_Remaster.f401.mp4 | path=/home/user/proj/youdis/downloads/Rick_Astley/NA/NANARickAstley-_Never_Gonna_Give_You_Up_Official_Video_4K_Remaster.f401.mp4
state=completed | [Metadata] There isn't any metadata to add | path=/home/user/proj/youdis/downloads/Rick_Astley/NA/NANARickAstley-_Never_Gonna_Give_You_Up_Official_Video_4K_Remaster.mp4
:end:
8. while first job is active, submit another `/youtube` and confirm busy behavior
:out:
#+begin_src shell
Error: AttributeError
Traceback (most recent call last):
File "/home/user/proj/youdis/venv/lib/python3.10/site-packages/interactions/client/client.py", line 1900, in __dispatch_interaction
response = await callback
File "/home/user/proj/youdis/venv/lib/python3.10/site-packages/interactions/client/client.py", line 1771, in _run_slash_command
return await command(ctx, **ctx.kwargs)
File "/home/user/proj/youdis/venv/lib/python3.10/site-packages/interactions/models/internal/command.py", line 132, in __call__
await self.call_callback(self.callback, context)
File "/home/user/proj/youdis/venv/lib/python3.10/site-packages/interactions/models/internal/application_commands.py", line 833, in call_callback
return await self.call_with_binding(callback, ctx, *new_args, **new_kwargs)
File "/home/user/proj/youdis/venv/lib/python3.10/site-packages/interactions/models/internal/callback.py", line 43, in call_with_binding
return await callback(*args, **kwargs)
File "/home/user/proj/youdis/youdis/adapters/discord.py", line 167, in youtube
await ctx.channel.send(f"Submitted <{url}> to the backend. Status updates via DM.")
AttributeError: 'NoneType' object has no attribute 'send'
#+end_src
busy: busy with https://youtu.be/20HxMMSqRyg?si=3v7mN2L88c_FxpQR
state=running | phase=running | cancel requested | path=/home/user/proj/youdis/downloads/James_Hoffmann/NA/NANAThe_Beginner_s_Guide_To_Latte_Art.f401.mp4
:end:
9. run `/status` and confirm it reflects current or last backend job
:out:
last job: state=completed | [Metadata] There isn't any metadata to add | path=/home/user/proj/youdis/downloads/Rick_Astley/NA/NANARickAstley-_Never_Gonna_Give_You_Up_Official_Video_4K_Remaster.mp4
:end:
10. run `/interrupt` as owner and confirm cancellation is surfaced via DM
:out:
last job: state=cancelled | cancelled | path=/home/user/proj/youdis/downloads/James_Hoffmann/NA/NANAThe_Beginner_s_Guide_To_Latte_Art.f401.mp4
:end:
- datetime:[2026-04-02 Thu 11:52]
*** org-block tests
#+begin_src shell :dir ~/proj/youdis :results output verbatim
source ./venv/bin/activate
python3 -m uvicorn youdis.main:app --host 127.0.0.1 --port 8000
#+end_src
#+begin_src shell :results output
echo ok
#+end_src
#+RESULTS:
: ok
#+begin_src python :dir ~/proj/youdis :results output verbatim
python ./youdis.py
#+end_src
#+RESULTS:
** notes
- discord adapter is now a thin HTTP client of the backend; it no longer imports or configures yt-dlp
- `YOUDIS_BACKEND_URL` controls which backend the adapter targets
- progress updates are currently implemented by polling `/jobs/current` and DMing only when the summary changes
- legacy auth/user-management commands were removed from the active adapter path
- `.env` is now supported for local/dev convenience, while real environment variables still override it in prod/docker
- command registration required explicit binding plus `@bot.listen()` listeners in this adapter structure
* [X] 2.0.3: remove deprecated discord-bot functionality (2)
delete or retire legacy bot behaviors that no longer fit once the backend split is in place
** pm notes
- only remove this after the new path works. this is cleanup, not pioneering work.
- favor deletion over compatibility shims
- keep operator controls only if still useful
** Acceptance Criteria
1. remove obsolete auth/user-management behavior
- old user persistence and commands are removed
- backend-facing flow no longer depends on them
2. remove obsolete downloader/runtime logic from bot
- bot no longer owns yt-dlp execution
- dead code paths are deleted
3. leave the bot in a coherent state
- remaining commands reflect actual supported behavior
- deprecated artifacts are clearly removed or marked
** evidence
- commit: 0aa9950
- tests:
1. `python3 -m py_compile ./youdis.py ./youdis/adapters/discord.py`
2. `rg -n "users.json|api_token" README.md unraid-ca-template.xml youdis.py youdis/adapters/discord.py`
3. start backend: `python3 -m uvicorn youdis.main:app --host 127.0.0.1 --port 8000`
4. start adapter: `python3 ./youdis.py`
5. in discord, run `/youtube`, `/status`, and `/interrupt` and confirm the adapter still works after cleanup
- datetime: [2026-04-02 Thu 12:09]
** notes
- active bot path no longer includes local auth or user-management behavior
- top-level `youdis.py` remains as a thin launcher so existing operator habits and scripts do not break during the refactor
- cleanup updated user-facing deployment artifacts to match the v2 naming and architecture, including `DISCORD_BOT_TOKEN` and removal of `users.json` references
- archived planning docs were intentionally left untouched: `pm/tasks.org` is historical and `pm/notes.org` is personal working notes
* [ ] 2.0.4: make youdis.py app runner for backend + adapters (3)
Build a simple Python orchestration layer in youdis.py so the standard app stack can be launched from one entrypoint while backend and adapters remain independently runnable for testing.
** pm notes
- keep components independently runnable
- youdis.py as app-level orchestrator; let Docker just run python3 /app/youdis.py
- no plugin discovery or hot-loading
- Docker should package and launch the app, not decide internal process topology
- preserve the ability to run backend or adapters directly for debugging
- optimize for one obvious default run path
** Acceptance Criteria
1. define the default app startup path
- youdis.py launches the standard stack for v2
- backend and Discord adapter startup order is explicit
- shutdown behavior is coherent enough for local/dev use
2. preserve modular run paths
- backend can still be run directly
- Discord adapter can still be run directly
- orchestration layer does not bury component-level testing
3. keep orchestration simple
- no dynamic adapter discovery
- no hot reloading framework
- configured components are started explicitly
4. document runtime ownership
- clarify what Python orchestrates vs what Docker orchestrates
- identify env vars or flags that control which components start
- leave room for future Zulip/XMPP adapters without redesigning the runner
** evidence
- commit: 309ce87
- tests:
1. `python3 -m py_compile ./youdis.py ./youdis/adapters/discord.py`
2. backend direct run still works: `python3 -m uvicorn youdis.main:app --host 127.0.0.1 --port 8000`
3. discord direct run still works: `python3 -m youdis.adapters.discord`
4. app runner backend-only smoke test: `ENABLE_DISCORD=0 timeout 5s python3 ./youdis.py`
5. app runner default path: `python3 ./youdis.py`
- date: [2026-04-02 Thu 14:13]
** notes
- `youdis.py` is now the default v2 app runner and starts the standard stack explicitly rather than dynamically discovering adapters
- backend starts first and must pass a health check before the Discord adapter is launched
- backend and Discord adapter remain directly runnable for debugging and tests
- Docker is intended to invoke `python3 /app/youdis.py`; Python owns app orchestration while Docker owns packaging and runtime environment
- adapter startup remains explicit in `youdis.py`, while each adapter gets its own `ENABLE_<ADAPTER>` flag
- adding an adapter means creating `youdis/adapters/<adapter>.py`, adding `start_<adapter>()` in `youdis.py`, wiring its `ENABLE_<ADAPTER>` flag, and documenting any supporting env vars in `.env.example`
- runner flags currently include `YOUDIS_RUN_BACKEND`, `YOUDIS_BACKEND_HOST`, `YOUDIS_BACKEND_PORT`, and `ENABLE_DISCORD`
* ==== BACKLOG ====
Tasks below this line are inactive and should not be touched.
* [ ] 2.0.X: fix automation and build pipeline (3)
repair and simplify the build/update/deploy path so it matches the new backend-plus-frontend structure
** pm notes
- this should come after architecture and discord integration stabilize. no point polishing the pipeline for the wrong shape.
- optimize for simple manual ops first
- stop here after pipeline is sane
** Acceptance Criteria
1. align build artifacts with the new structure
- docker/build scripts reflect current components
- runtime assumptions are consistent
2. review old automation artifacts
- stale runner/update/restart logic is removed or updated
- manual update/rebuild flow is clear
3. confirm deployment path works
- local or unraid deployment is validated
- pipeline is understandable enough to maintain
** evidence
- commit:
- tests:
- datetime:
** notes
* [ ] X.x.x: clean up discord adapter UI
** acceptance criteria
1. fix interaction pattern so it doesnt time out - prefer "command accepted" or somehting
2. remove all intermediate messages between "accepted/running" and "complete" - /status handles this!
- discord can also output a "busy" signal, research this
3. fix output syntax, we dont need to get crazy with discord cards
** evidence
- commit:
- tests:
- date:
** notes
* [ ] X.x.x: fix youtube -> plex default output
** acceptance criteria
1.
-
** evidence
- commit:
- tests:
- date:
** notes
* /

123
pm/tasks.org Normal file
View File

@@ -0,0 +1,123 @@
#+title: Youdis Task Log
#+updated: [2026-03-31 Tue 08:00]
* [X] 1.1.1: stabilize youdis core bot behavior (estimate 3 commits)
refactor the current `youdis.py` flow so authorization, download execution, and user feedback are correct and predictable without changing the product shape. keep this narrowly scoped to correctness and maintainability; do not redesign into a queueing platform yet. preserve archive-first behavior and dm status updates; do not add new infrastructure dependencies and prefer boring explicit state over clever concurrency.
** acceptance criteria
1. initialize and load `/config/users.json` safely in all cases
- create parent dirs before touch/open
- ensure `authorized_users` always has a valid default
- normalize stored ids to a single type
2. fix command-path correctness for `/youtube`, `/adduser`, and `/removeuser`
- authorized users can successfully invoke downloads
- add/remove user commands persist changes correctly
- remove broken/incomplete code paths
3. duplicate prevention relies on archive.txt
** pm notes
** evidence
- commit: 033d9dd
- tests: ~python3 -m py_compile ./youdis.py~
- datetime: [2026-03-31 Tue 13:28]
** notes
- store Discord user ids as strings in `users.json`
- duplicate prevention should continue to rely on `archive.txt`, not inferred hook errors
* [X] 1.1.2: remove global mutable download state and define single-job semantics (estimate 2 commits)
eliminate shared mutable hook state and make concurrent behavior explicit, even if the initial policy is just "one active job at a time." don't build a scheduler; ok if simplest outcome is single active job with clear busy message. cancellation can be coarse if yt-dlp/process boundaries make graceful stop annoying
** acceptance criteria
1. improve runtime handling for downloads
- replace brittle thread/join pattern with a simpler async-safe execution path
- catch and report real yt-dlp failures
- avoid misleading "already exists" error assumptions
2. progress reporting is isolated per request
- no module-level mutable title state shared across jobs
- hooks derive state from request-local context
3. active-job behavior is explicit
- either reject a second request while busy or implement a minimal tracked active job
- user-facing response explains current behavior
4. `/interrupt` is either implemented minimally or downgraded honestly
- no fake command implying cancellation works when it does not
- command behavior matches implementation
** evidence
- commit: 667b06f
- tests: ~python3 -m py_compile /home/user/proj/youdis/youdis.py~
- datetime: [2026-03-31 Tue 14:00]
** notes
- verify slash-command response patterns against the `interactions` library while touching runtime flow
* [ ] 1.1.3: move static yt-dlp behavior into config and shrink python surface area (estimate 2 commits)
shift stable downloader options into `default-yt-dlp.conf` so the bot code only handles dynamic inputs and orchestration. optimize for inspectability and low-friction manual ops. keep output naming durable enough for plex/plain-file use. avoid duplicating config values across code and conf.
** acceptance criteria
1. separate static vs dynamic yt-dlp options cleanly
- stable defaults live in `default-yt-dlp.conf`
- python injects only request-specific/runtime values
2. preserve archive and output behavior
- `archive.txt` remains the duplicate-prevention mechanism
- output paths remain stable and browseable
3. document config ownership
- clarify which settings belong in config vs code
- make future yt-dlp tuning possible without major python edits
** evidence
- commit:
- tests:
- datetime:
** notes
* [ ] 1.1.4: simplify image/build/update workflow around manual ops (estimate 3 commits)
reduce repo cruft from the gitea-runner/nightly-update experiment and replace it with explicit manual update/rebuild mechanics.
** acceptance criteria
1. define a manual update path for yt-dlp and app image lifecycle
- document or script manual `git pull`, rebuild, and redeploy
- remove or quarantine brittle auto-update assumptions
2. review and simplify `update-ytdlp.sh`, workflow yaml, and weekly restart artifacts
- keep only artifacts that serve the current manual-ops model
- delete or mark deprecated anything tied to abandoned automation paths
3. retain unraid deployment viability
- container can still be rebuilt and redeployed cleanly on jeeves
- resulting flow is understandable without rereading old ci experiments
- pm note: weekly restart is presumed suspect until proven necessary
** evidence
- commit:
- tests:
- datetime:
** notes
- do not let runner/workflow complexity dominate a small bot
- prefer explicit version pinning or manual binary refresh over magical nightlies
* [ ] 1.1.5: clean up packaging/deployment artifacts for unraid consumption (estimate 2 commits)
make the dockerfile, run script, and unraid-ca template consistent with the refactored app so deployment is less of a ritual ordeal.
** acceptance criteria
1. align docker/runtime assumptions
- paths like `/config` and `/downloads` are consistent across code, scripts, and container metadata
- env vars are documented and validated
2. review deployment artifacts for drift
- `dockerfile`, `run-youdis.sh`, and `unraid-ca-template.xml` reflect current behavior
- remove stale references and dead assumptions
3. make fresh deployment understandable
- a new deploy on unraid is possible without reconstructing tribal knowledge from old files
- pm note: this is packaging polish after core correctness, not before
** evidence
- commit:
- tests:
- datetime:
** notes
- keep container surface area small
- optimize for “future me can redeploy this without cursing past me too hard”

Binary file not shown.

21
run-youdis.sh Normal file
View File

@@ -0,0 +1,21 @@
#!/bin/sh
set -e
#start crond in bg
crond -l 2
echo "checking for /config/yt-dlp.conf"
#copy yt-dlp.conf if missing
mkdir -p /config
if [ ! -f /config/yt-dlp.conf ]; then
echo "yt-dlp.conf not found, setting default"
cp /app/default-yt-dlp.conf /config/yt-dlp.conf
echo "created yt-dlp.conf"
fi
python3 -m pip install --no-cache-dir --upgrade --pre "yt-dlp[default]"
VERSION=$(python3 -m pip show yt-dlp 2>/dev/null | awk '/Version:/ {print $2}')
echo "updated yt-dlp to $VERSION"
echo "starting youdis"
exec python3 /app/youdis.py

View File

@@ -1,18 +1,18 @@
<?xml version="1.0"?>
<Container version="2">
<Name>youdis</Name>
<Repository>eulaly/youdis</Repository>
<Registry>https://hub.docker.com/r/eulaly/youdis/</Registry>
<Network>bridge</Network>
<Shell>sh</Shell>
<Privileged>false</Privileged>
<Support>[Unraid Support Thread]</Support>
<Project>https://github.com/eulaly/youdis</Project>
<Overview>Discord bot-based wrapper for yt-dlp. Let your friends download videos to your server! Supports playlists, requires a configured Discord bot.</Overview>
<Category>Downloaders: Tools:</Category>
<TemplateURL>https://raw.githubusercontent.com/eulaly/unraid-templates/refs/heads/master/unraid-ca-template.xml</TemplateURL>
<Icon>https://github.com/eulaly/youdis/blob/c978a2326984efa9670678687ed1a1473478d753/yt_dlp.png</Icon>
<Config Name="api_token" Target="api_token" Default="" Mode="" Description="Discord bot token" Type="Variable" Display="always" Required="true" Mask="true"/>
<Config Name="Downloads" Target="/downloads" Default="" Mode="rw" Description="Video download location" Type="Path" Display="always" Required="false"/>
<Config Name="Config" Target="/config" Default="/mnt/user/appdata/youdis/config" Mode="rw" Description="Config location (archive.txt, users.json)" Type="Path" Display="always" Required="false"/>
</Container>
<?xml version="1.0"?>
<Container version="2">
<Name>youdis</Name>
<Repository>eulaly/youdis</Repository>
<Registry>https://hub.docker.com/r/eulaly/youdis/</Registry>
<Network>bridge</Network>
<Shell>sh</Shell>
<Privileged>false</Privileged>
<Support>[Unraid Support Thread]</Support>
<Project>https://github.com/eulaly/youdis</Project>
<Overview>Private yt-dlp worker with a Discord adapter. Submit downloads through Discord while the backend owns yt-dlp execution and job state.</Overview>
<Category>Downloaders: Tools:</Category>
<TemplateURL>https://raw.githubusercontent.com/eulaly/unraid-templates/refs/heads/master/unraid-ca-template.xml</TemplateURL>
<Icon>https://github.com/eulaly/youdis/blob/c978a2326984efa9670678687ed1a1473478d753/yt_dlp.png</Icon>
<Config Name="DISCORD_BOT_TOKEN" Target="DISCORD_BOT_TOKEN" Default="" Mode="" Description="Discord bot token" Type="Variable" Display="always" Required="true" Mask="true"/>
<Config Name="Downloads" Target="/downloads" Default="" Mode="rw" Description="Video download location" Type="Path" Display="always" Required="false"/>
<Config Name="Config" Target="/config" Default="/mnt/user/appdata/youdis/config" Mode="rw" Description="Config location (archive.txt and backend runtime data)" Type="Path" Display="always" Required="false"/>
</Container>

12
update-ytdlp.sh Normal file
View File

@@ -0,0 +1,12 @@
#!/bin/sh
set -e
echo "updating yt-dlp"
echo "killing youdis"
pkill -f youdis.py || true
python3 -m pip install --no-cache-dir --upgrade --pre "yt-dlp[default]"
VERSION=$(python3 -m pip show yt-dlp 2>/dev/null | awk '/Version:/ {print $2}')
echo "updated yt-dlp to $VERSION"
echo "restarting youdis"
python3 /app/youdis.py &

View File

@@ -1 +0,0 @@
20250823-d707074

1
weekly-restart Normal file
View File

@@ -0,0 +1 @@
0 3 * * 0 root /app/update-ytdlp.sh

1
youdis-version.txt Normal file
View File

@@ -0,0 +1 @@
20250829-ec72c56

284
youdis.py
View File

@@ -1,157 +1,161 @@
#!/usr/bin/env python3
'''
youdis v1.1
bot for downloading youtube videos using yt-dlp
discord-py-interactions 5.11.0 has new option
requires python>=3.9
'''
# match_filter: info_dict -> Raise utils.DownloadCancelled(msg) ? interrupt
"""App runner for FastAPI backend and discord adapter."""
from __future__ import annotations
import interactions
from os import getenv
from pathlib import Path
import yt_dlp
import json
import asyncio
import threading
import os
import sys
from dataclasses import dataclass
from pathlib import Path
from urllib.error import URLError
from urllib.request import urlopen
userFile = Path('/config/users.json')
userFile.touch(exist_ok=True)
from youdis.env import load_project_dotenv
bot = interactions.Client(intents=interactions.Intents.DEFAULT,default_scope=2147491904)
userFile.parent.mkdir(exist_ok=True, parents=True)
try:
with open(userFile, 'x') as f:
print(f'users.json not found; saving to {userFile}')
except FileExistsError:
with open(userFile, 'r') as f:
authorized_users = json.load(f).get('authorized_users')
print(f'authorized_users:{authorized_users}')
title = ''
load_project_dotenv()
async def send_message(ctx, message):
await ctx.author.send(message)
def download_video(url, options):
with yt_dlp.YoutubeDL(options) as ydl:
ydl.download(url)
def create_hook(ctx,loop):
def hook(d):
global title
status = d.get('status')
if status == 'error':
msg = f'error; video probably already exists, have you checked archive.txt'
asyncio.run_coroutine_threadsafe(send_message(ctx,msg),loop)
elif d.get('info_dict').get('title') != title:
title = d.get('info_dict').get('title')
playlist_index = d.get('info_dict').get('playlist_index')
playlist_count = d.get('info_dict').get('playlist_count')
filename = d.get('filename')
url = d.get('info_dict').get('webpage_url')
msg = f'{status} {playlist_index} of {playlist_count}: {filename} <{url}>'
asyncio.run_coroutine_threadsafe(send_message(ctx,msg),loop)
return hook
REPO_ROOT = Path(__file__).resolve().parent
DEFAULT_BACKEND_HOST = os.getenv("YOUDIS_BACKEND_HOST", "127.0.0.1")
DEFAULT_BACKEND_PORT = int(os.getenv("YOUDIS_BACKEND_PORT", "8000"))
ENABLE_BACKEND = os.getenv("YOUDIS_RUN_BACKEND", "1") not in {"0", "false", "False"}
ENABLE_DISCORD = os.getenv("ENABLE_DISCORD", "1") not in {"0", "false", "False"}
BACKEND_HEALTH_TIMEOUT = float(os.getenv("YOUDIS_BACKEND_HEALTH_TIMEOUT", "20"))
BACKEND_HEALTH_INTERVAL = float(os.getenv("YOUDIS_BACKEND_HEALTH_INTERVAL", "0.5"))
@interactions.slash_command(name="youtube",description="download video from youtube to server")
@interactions.slash_option(
name='url',
opt_type=interactions.OptionType.STRING,
required=True,
description='url target'
)
async def youtube(ctx: interactions.SlashContext, url:str):
print(f'{ctx.author.id} requested {url}')
loop = asyncio.get_running_loop()
hook = create_hook(ctx,loop)
msg = ''
# use api_to_cli and paste cli options to get the output you need
yoptions = {
'format':'bestvideo[ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]/best',
'fragment_tries': 10,
'restrictfilenames':True,
'paths': {'home':'/downloads'},
'retries':10,
'writeinfojson':False,
'allow_playlist_files':True,
'noplaylist':True,
'download_archive':'/config/archive.txt',
'progress_hooks':[hook],
'outtmpl': '%(uploader)s/%(playlist_title)s/%(playlist_index)s%(playlist_index& - )s%(title)s.%(ext)s',
'outtmpl_na_placeholder':'',
}
# check that user is authorized
if ctx.author.id not in authorized_users:
if ctx.author.id == 127831327012683776:
await ctx.author.send('potato stop')
await ctx.author.send('you are not authorized to use this command. message my owner to be added.')
@dataclass
class ManagedProcess:
name: str
process: asyncio.subprocess.Process
def backend_url() -> str:
return f"http://{DEFAULT_BACKEND_HOST}:{DEFAULT_BACKEND_PORT}"
def build_child_env() -> dict[str, str]:
env = os.environ.copy()
env.setdefault("YOUDIS_BACKEND_URL", backend_url())
return env
async def start_backend(env: dict[str, str]) -> ManagedProcess:
process = await asyncio.create_subprocess_exec(
sys.executable,
"-m",
"uvicorn",
"youdis.main:app",
"--host",
DEFAULT_BACKEND_HOST,
"--port",
str(DEFAULT_BACKEND_PORT),
cwd=str(REPO_ROOT),
env=env,
)
return ManagedProcess(name="backend", process=process)
async def start_discord(env: dict[str, str]) -> ManagedProcess:
process = await asyncio.create_subprocess_exec(
sys.executable,
"-m",
"youdis.adapters.discord",
cwd=str(REPO_ROOT),
env=env,
)
return ManagedProcess(name="discord", process=process)
ADAPTER_STARTERS = {
"discord": start_discord,
}
def selected_adapters() -> list[str]:
requested: list[str] = []
if ENABLE_DISCORD:
requested.append("discord")
return requested
async def wait_for_backend_health(timeout_seconds: float) -> None:
deadline = asyncio.get_running_loop().time() + timeout_seconds
health_url = f"{backend_url()}/health"
while True:
try:
with urlopen(health_url, timeout=2) as response:
if response.status == 200:
return
except URLError:
pass
if asyncio.get_running_loop().time() >= deadline:
raise TimeoutError(f"backend did not become healthy at {health_url}")
await asyncio.sleep(BACKEND_HEALTH_INTERVAL)
async def stop_process(proc: ManagedProcess) -> None:
if proc.process.returncode is not None:
return
else:
await ctx.channel.send(f'Downloading from <{url}>. Status updates via DM.')
#await ctx.defer() #if you need up to 15m to respond
# 1/2 - download in separate thread, else progress_hook blocks downstream async ctx.send
download_thread = threading.Thread(target=download_video, args=(url,yoptions))
download_thread.start()
await asyncio.to_thread(download_thread.join)
# 2/2 - replace the above with this next try:
#try:
# await asyncio.to_thread(download_video, url, yoptions)
#except Exception as e:
# print(f"download failed: {e}")
# await ctx.author.send(f"download failed: {str(e)}")
proc.process.terminate()
try:
await asyncio.wait_for(proc.process.wait(), timeout=10)
except asyncio.TimeoutError:
proc.process.kill()
await proc.process.wait()
@interactions.slash_command(name="interrupt",description="cancel current job")
@interactions.check(interactions.is_owner())
async def _interrupt(ctx):
# interrupt here
print('interrupting current job - not implemented')
await ctx.author.send('interrupting current job - not implemented')
@interactions.slash_command(name="adduser",description="authorize target user")
@interactions.slash_option(
name="user",
opt_type=interactions.OptionType.USER,
required=True,
description='enable this bot for target user',
)
@interactions.check(interactions.is_owner())
async def _adduser(ctx: interactions.SlashContext, user:interactions.OptionType.USER):
if str(user.id) not in authorized_users:
authorized_users.append(str(user.id))
with open(userFile,'w') as f: #overwrite file - fix later if other params come up
json.dump({'authorized_users':authorized_users})
print('react:checkmark')
await ctx.message.add_reaction('')
async def run() -> int:
managed: list[ManagedProcess] = []
env = build_child_env()
adapters = selected_adapters()
@interactions.slash_command(name="removeuser",description="deauthorize target user")
@interactions.slash_option(
name="user",
opt_type=interactions.OptionType.USER,
required=True,
description='disable this bot for target user',
)
@interactions.check(interactions.is_owner())
async def _removeuser(ctx: interactions.SlashContext, user:interactions.OptionType.USER):
if str(user.id) in authorized_users:
# ? ? ? fix pls
i = index(authorized_users(str(user.id)))
if not ENABLE_BACKEND and not adapters:
print("nothing to start: backend is disabled and no adapters are enabled")
return 1
# update list, rewrite json
print('react:checkmark')
await ctx.message.add_reaction('')
try:
if ENABLE_BACKEND:
backend = await start_backend(env)
managed.append(backend)
print(f"started backend on {backend_url()}")
await wait_for_backend_health(BACKEND_HEALTH_TIMEOUT)
print("backend is healthy")
async def dl_hook(d):
msg = f'{d["status"]} {d["filename"]}'
print(msg)
await ctx.author.send(msg)
for adapter_name in adapters:
process = await ADAPTER_STARTERS[adapter_name](env)
managed.append(process)
print(f"started {adapter_name} adapter")
api_token = getenv('api_token')
if not api_token:
raise ValueError('API token not set. Retrieve from your Discord bot.')
bot.start(api_token)
wait_tasks = [asyncio.create_task(proc.process.wait()) for proc in managed]
done, pending = await asyncio.wait(wait_tasks, return_when=asyncio.FIRST_COMPLETED)
for task in pending:
task.cancel()
first_returncode = 0
for task in done:
first_returncode = task.result()
break
return first_returncode
finally:
for proc in reversed(managed):
await stop_process(proc)
def main() -> None:
try:
raise SystemExit(asyncio.run(run()))
except KeyboardInterrupt:
raise SystemExit(130)
if __name__ == "__main__":
main()

1
youdis/__init__.py Normal file
View File

@@ -0,0 +1 @@
"""Youdis v2 backend package."""

View File

@@ -0,0 +1 @@
"""Frontend adapters for the youdis backend."""

231
youdis/adapters/discord.py Normal file
View File

@@ -0,0 +1,231 @@
"""Discord adapter for interacting with FastAPI backend worker."""
import asyncio
from os import getenv
import aiohttp
import interactions
from ..env import load_project_dotenv
load_project_dotenv()
BACKEND_HOST = getenv("YOUDIS_BACKEND_HOST", "127.0.0.1")
BACKEND_PORT = int(getenv("YOUDIS_BACKEND_PORT", "8000"))
BACKEND_URL = f"http://{BACKEND_HOST}:{BACKEND_PORT}".rstrip("/")
POLL_INTERVAL_SECONDS = float(getenv("YOUDIS_POLL_INTERVAL_SECONDS", "2"))
DEFAULT_SCOPE = int(getenv("DISCORD_BOT_SCOPE", "2147491904"))
bot = interactions.Client(
intents=interactions.Intents.DEFAULT,
default_scope=DEFAULT_SCOPE,
)
http_session: aiohttp.ClientSession | None = None
poll_tasks: dict[str, asyncio.Task] = {}
def backend_url(path: str) -> str:
return f"{BACKEND_URL}{path}"
async def get_session() -> aiohttp.ClientSession:
global http_session
if http_session is None or http_session.closed:
http_session = aiohttp.ClientSession()
return http_session
async def request_json(method: str, path: str, **kwargs):
session = await get_session()
async with session.request(method, backend_url(path), **kwargs) as response:
data = await response.json()
return response.status, data
def format_status_message(job: dict) -> str:
state = job.get("state")
phase = job.get("phase")
disposition = job.get("disposition")
message = job.get("message")
result_path = job.get("result_path")
parts = [f"state={state}"]
if phase:
parts.append(f"phase={phase}")
if disposition:
parts.append(f"disposition={disposition}")
if message:
parts.append(message)
if result_path:
parts.append(f"path={result_path}")
return " | ".join(parts)
async def dm(ctx: interactions.SlashContext, message: str) -> None:
await ctx.author.send(message)
async def respond(ctx: interactions.SlashContext, message: str) -> None:
if ctx.channel is not None:
await ctx.channel.send(message)
return
await dm(ctx, message)
async def poll_job_updates(ctx: interactions.SlashContext, job_id: str) -> None:
last_sent = None
try:
while True:
status_code, payload = await request_json("GET", "/jobs/current")
if status_code != 200:
await dm(ctx, f"backend status check failed: HTTP {status_code}")
return
job = payload.get("job")
if not job:
await dm(ctx, f"job {job_id} is no longer visible from the backend")
return
if job.get("job_id") != job_id:
await dm(ctx, f"job {job_id} is no longer the current backend job")
return
summary = format_status_message(job)
if summary != last_sent:
await dm(ctx, summary)
last_sent = summary
if job.get("state") in {"completed", "failed", "cancelled"}:
return
await asyncio.sleep(POLL_INTERVAL_SECONDS)
except asyncio.CancelledError:
raise
except aiohttp.ClientError as exc:
await dm(ctx, f"backend poll failed: {exc}")
finally:
poll_tasks.pop(job_id, None)
def ensure_poll_task(ctx: interactions.SlashContext, job_id: str) -> None:
existing = poll_tasks.get(job_id)
if existing and not existing.done():
return
poll_tasks[job_id] = asyncio.create_task(poll_job_updates(ctx, job_id))
@interactions.slash_command(name="youtube", description="submit a youtube download to the backend")
@interactions.slash_option(
name="url",
opt_type=interactions.OptionType.STRING,
required=True,
description="url target",
)
async def youtube(ctx: interactions.SlashContext, url: str):
payload = {
"url": url,
"requester_id": str(ctx.author.id),
"requester_name": ctx.author.username,
"origin": f"discord:{ctx.guild_id or 'dm'}:{ctx.channel_id}",
}
try:
status_code, job = await request_json("POST", "/jobs", json=payload)
except aiohttp.ClientError as exc:
await dm(ctx, f"backend request failed: {exc}")
return
if status_code != 200:
await dm(ctx, f"backend request failed: HTTP {status_code}")
return
state = job.get("state")
job_id = job.get("job_id", "unknown")
if state == "busy":
await respond(ctx, "Backend is busy with another job. Details via DM.")
await dm(ctx, f"busy: {job.get('message')}")
return
if state != "accepted":
await respond(ctx, "Backend rejected the request. Details via DM.")
await dm(ctx, format_status_message(job))
return
await respond(ctx, f"Submitted <{url}> to the backend. Status updates via DM.")
await dm(ctx, f"accepted job {job_id} for <{url}>")
ensure_poll_task(ctx, job_id)
@interactions.slash_command(name="interrupt", description="cancel the current backend job")
@interactions.check(interactions.is_owner())
async def interrupt(ctx: interactions.SlashContext):
try:
status_code, payload = await request_json("POST", "/jobs/current/cancel")
except aiohttp.ClientError as exc:
await dm(ctx, f"backend cancel failed: {exc}")
return
if status_code == 404:
await dm(ctx, "no active backend job to interrupt")
return
if status_code != 200:
await dm(ctx, f"backend cancel failed: HTTP {status_code}")
return
await dm(ctx, format_status_message(payload))
@interactions.slash_command(name="status", description="show the current backend job status")
async def status(ctx: interactions.SlashContext):
try:
status_code, payload = await request_json("GET", "/jobs/current")
except aiohttp.ClientError as exc:
await dm(ctx, f"backend status failed: {exc}")
return
if status_code != 200:
await dm(ctx, f"backend status failed: HTTP {status_code}")
return
job = payload.get("job")
if not job:
await dm(ctx, "backend has no active or recent job")
return
active = payload.get("active")
prefix = "active" if active else "last"
await dm(ctx, f"{prefix} job: {format_status_message(job)}")
@bot.listen()
async def on_startup():
await get_session()
print(f"discord adapter configured for backend {BACKEND_URL}")
print(f"{bot.application_commands.count} registered commands:")
for i, x in enumerate(bot.application_commands,start=1):
print(f" {i}. {x.name}")
@bot.listen()
async def on_shutdown():
global http_session
for task in list(poll_tasks.values()):
task.cancel()
poll_tasks.clear()
if http_session is not None and not http_session.closed:
await http_session.close()
http_session = None
def main() -> None:
api_token = getenv("DISCORD_BOT_TOKEN")
if not api_token:
raise ValueError("API token not set. Retrieve from your Discord bot.")
# bot.add_command(youtube)
# bot.add_command(status)
# bot.add_command(interrupt)
bot.start(api_token)
if __name__ == "__main__":
main()

13
youdis/env.py Normal file
View File

@@ -0,0 +1,13 @@
from pathlib import Path
try:
from dotenv import load_dotenv
except ModuleNotFoundError: # pragma: no cover - optional local convenience dependency
load_dotenv = None
def load_project_dotenv() -> None:
if load_dotenv is None:
return
repo_root = Path(__file__).resolve().parent.parent
load_dotenv(repo_root / ".env", override=False)

311
youdis/main.py Normal file
View File

@@ -0,0 +1,311 @@
"""FastAPI backend worker managing yt-dlp subprocess jobs and exposing job state."""
import asyncio
from asyncio.subprocess import PIPE, STDOUT
from collections import deque
from dataclasses import dataclass, field
from datetime import datetime
from os import getenv
from pathlib import Path
import tempfile
from uuid import uuid4
from fastapi import FastAPI, HTTPException
from .env import load_project_dotenv
from .models import CurrentJobResponse, HealthResponse, JobRequest, JobStatus, VersionResponse
load_project_dotenv()
REPO_ROOT = Path(__file__).resolve().parent.parent
DEFAULT_CONFIG = REPO_ROOT / "default-yt-dlp.conf"
VERSION_FILE = REPO_ROOT / "youdis-version.txt"
YTDLP_EXECUTABLE = getenv("YOUDIS_YTDLP_EXECUTABLE", "yt-dlp")
PREFERRED_CONFIG_ROOT = Path(getenv("YOUDIS_CONFIG_DIR", "/config"))
PREFERRED_DOWNLOAD_ROOT = Path(getenv("YOUDIS_DOWNLOAD_DIR", "/downloads"))
LOCAL_RUNTIME_ROOT = REPO_ROOT / ".runtime"
FALLBACK_CONFIG_ROOT = LOCAL_RUNTIME_ROOT / "config"
FALLBACK_DOWNLOAD_ROOT = LOCAL_RUNTIME_ROOT / "downloads"
OUTPUT_TEMPLATE = "%(uploader)s/%(playlist_title)s/%(playlist_index)s%(playlist_index& - )s%(title)s.%(ext)s"
@dataclass
class ManagedJob:
status: JobStatus
process: asyncio.subprocess.Process | None = None
task: asyncio.Task | None = None
cancel_requested: bool = False
output_lines: deque[str] = field(default_factory=lambda: deque(maxlen=25))
app = FastAPI(title="youdis", version="2")
job_lock = asyncio.Lock()
active_job: ManagedJob | None = None
last_job: JobStatus | None = None
def now_utc() -> datetime:
return datetime.now()
def read_version() -> str:
if VERSION_FILE.exists():
return VERSION_FILE.read_text().strip()
return "unknown"
def ensure_writable_directory(preferred: Path, fallback: Path) -> Path:
for candidate in (preferred, fallback):
try:
candidate.mkdir(parents=True, exist_ok=True)
with tempfile.NamedTemporaryFile(dir=candidate, prefix=".youdis-write-", delete=True):
pass
return candidate
except OSError:
continue
raise OSError(f"no writable runtime directory available for {preferred} or {fallback}")
def resolve_runtime_paths() -> tuple[Path, Path]:
config_root = ensure_writable_directory(PREFERRED_CONFIG_ROOT, FALLBACK_CONFIG_ROOT)
download_root = ensure_writable_directory(PREFERRED_DOWNLOAD_ROOT, FALLBACK_DOWNLOAD_ROOT)
return config_root, download_root
def build_ytdlp_command(request: JobRequest, config_root: Path, download_root: Path) -> list[str]:
return [
YTDLP_EXECUTABLE,
"--config-locations",
str(DEFAULT_CONFIG),
"--download-archive",
str(config_root / "archive.txt"),
"--output",
str(download_root / OUTPUT_TEMPLATE),
request.url,
]
def clone_status(status: JobStatus) -> JobStatus:
return JobStatus(**status.model_dump())
def update_status(job: ManagedJob, **changes: object) -> None:
for key, value in changes.items():
setattr(job.status, key, value)
job.status.updated_at = now_utc()
def classify_output_line(job: ManagedJob, line: str) -> None:
if not line:
return
job.output_lines.append(line)
message = line.strip()
if not message:
return
lowered = message.lower()
if "has already been recorded in the archive" in lowered:
update_status(
job,
disposition="archive_hit",
phase="downloading",
message="already in archive",
)
return
if "[download]" in lowered:
update_status(job, phase="downloading", message=message)
if "destination:" in lowered:
result_path = message.split(":", 1)[1].strip()
update_status(job, result_path=result_path)
return
if "merging formats into" in lowered:
result_path = message.split("into", 1)[1].strip().strip('"')
update_status(job, phase="postprocessing", message=message, result_path=result_path)
return
update_status(job, message=message)
async def finalize_job(job: ManagedJob, returncode: int) -> None:
disposition = job.status.disposition
if job.cancel_requested:
state = "cancelled"
message = "cancelled"
elif returncode == 0 and disposition == "archive_hit":
state = "completed"
message = "already in archive"
elif returncode == 0:
state = "completed"
message = job.status.message or "completed"
else:
state = "failed"
message = job.status.message or f"yt-dlp exited with code {returncode}"
update_status(job, state=state, phase=None, returncode=returncode, message=message)
global active_job, last_job
async with job_lock:
if active_job is job:
active_job = None
last_job = clone_status(job.status)
async def run_job(job: ManagedJob, request: JobRequest) -> None:
update_status(
job,
state="running",
phase="starting",
message="starting yt-dlp",
)
try:
if not DEFAULT_CONFIG.exists():
update_status(
job,
state="failed",
phase=None,
message=f"default config not found: {DEFAULT_CONFIG}",
returncode=78,
)
await finalize_job(job, 78)
return
config_root, download_root = resolve_runtime_paths()
command = build_ytdlp_command(request, config_root, download_root)
update_status(
job,
command=command,
archive_path=str(config_root / "archive.txt"),
result_path=str(download_root),
)
process = await asyncio.create_subprocess_exec(
*command,
stdout=PIPE,
stderr=STDOUT,
)
except FileNotFoundError:
update_status(
job,
state="failed",
phase=None,
message="yt-dlp executable not found",
returncode=127,
)
await finalize_job(job, 127)
return
except OSError as exc:
update_status(
job,
state="failed",
phase=None,
message=f"runtime path setup failed: {exc}",
returncode=73,
)
await finalize_job(job, 73)
return
try:
job.process = process
update_status(job, phase="running", message="yt-dlp running")
assert process.stdout is not None
while True:
line = await process.stdout.readline()
if not line:
break
classify_output_line(job, line.decode(errors="replace").strip())
returncode = await process.wait()
await finalize_job(job, returncode)
except Exception as exc:
update_status(
job,
state="failed",
phase=None,
message=f"backend runner error: {exc}",
returncode=1,
)
await finalize_job(job, 1)
async def create_job(request: JobRequest) -> JobStatus:
global active_job
async with job_lock:
if active_job is not None:
busy_job = active_job.status
return JobStatus(
job_id=busy_job.job_id,
state="busy",
url=request.url,
message=f"busy with {busy_job.url}",
requester_id=request.requester_id,
requester_name=request.requester_name,
origin=request.origin,
)
status = JobStatus(
job_id=str(uuid4()),
state="accepted",
url=request.url,
message="accepted",
phase="queued",
requester_id=request.requester_id,
requester_name=request.requester_name,
origin=request.origin,
)
managed_job = ManagedJob(status=status)
managed_job.task = asyncio.create_task(run_job(managed_job, request))
active_job = managed_job
return clone_status(status)
@app.get("/health", response_model=HealthResponse)
async def health() -> HealthResponse:
return HealthResponse(status="ok")
@app.get("/version", response_model=VersionResponse)
async def version() -> VersionResponse:
return VersionResponse(
version=read_version(),
active_job=active_job is not None,
ytdlp_executable=YTDLP_EXECUTABLE,
)
@app.get("/jobs/current", response_model=CurrentJobResponse)
async def current_job() -> CurrentJobResponse:
async with job_lock:
if active_job is not None:
return CurrentJobResponse(active=True, job=clone_status(active_job.status))
if last_job is not None:
return CurrentJobResponse(active=False, job=clone_status(last_job))
return CurrentJobResponse(active=False, job=None)
@app.post("/jobs", response_model=JobStatus)
async def submit_job(request: JobRequest) -> JobStatus:
return await create_job(request)
@app.post("/jobs/current/cancel", response_model=JobStatus)
async def cancel_current_job() -> JobStatus:
async with job_lock:
if active_job is None:
raise HTTPException(status_code=404, detail="no active job")
job = active_job
if job.process is None:
update_status(job, message="cancel requested before yt-dlp started")
job.cancel_requested = True
return clone_status(job.status)
job.cancel_requested = True
update_status(job, message="cancel requested", phase="running")
job.process.terminate()
return clone_status(job.status)

48
youdis/models.py Normal file
View File

@@ -0,0 +1,48 @@
from datetime import datetime
from typing import Literal
from pydantic import BaseModel, Field
JobState = Literal["accepted", "busy", "running", "completed", "failed", "cancelled"]
class JobRequest(BaseModel):
url: str
requester_id: str | None = None
requester_name: str | None = None
origin: str | None = None
requested_at: datetime | None = None
class JobStatus(BaseModel):
job_id: str
state: JobState
url: str
message: str | None = None
phase: str | None = None
disposition: str | None = None
requester_id: str | None = None
requester_name: str | None = None
origin: str | None = None
result_path: str | None = None
archive_path: str | None = None
command: list[str] = Field(default_factory=list)
returncode: int | None = None
created_at: datetime = Field(default_factory=datetime.utcnow)
updated_at: datetime = Field(default_factory=datetime.utcnow)
class CurrentJobResponse(BaseModel):
active: bool
job: JobStatus | None = None
class HealthResponse(BaseModel):
status: Literal["ok"]
class VersionResponse(BaseModel):
version: str
active_job: bool
ytdlp_executable: str