Compare commits
4 Commits
4abd39f884
...
v2
| Author | SHA1 | Date | |
|---|---|---|---|
| 18537b9de9 | |||
| 1757318523 | |||
| 90b9dad59c | |||
| 309ce87abb |
12
.env.example
Normal file
12
.env.example
Normal file
@@ -0,0 +1,12 @@
|
||||
YOUDIS_ENABLE_BACKEND=1
|
||||
YOUDIS_BACKEND_HOST=127.0.0.1
|
||||
YOUDIS_BACKEND_PORT=8000
|
||||
YOUDIS_BACKEND_HEALTH_TIMEOUT=20
|
||||
YOUDIS_BACKEND_HEALTH_INTERVAL=0.5
|
||||
YOUDIS_POLL_INTERVAL_SECONDS=2
|
||||
YOUDIS_YTDLP_EXECUTABLE=yt-dlp
|
||||
YOUDIS_CONFIG_DIR=/home/user/proj/youdis/test/
|
||||
YOUDIS_DOWNLOAD_DIR=/home/user/proj/youdis/downloads
|
||||
ENABLE_DISCORD=1
|
||||
DISCORD_BOT_TOKEN=
|
||||
DISCORD_BOT_SCOPE=2147491904
|
||||
1
.gitignore
vendored
1
.gitignore
vendored
@@ -11,6 +11,7 @@ env/
|
||||
# --- environment files ---
|
||||
.env
|
||||
.env.*
|
||||
!.env.example
|
||||
*.local
|
||||
|
||||
# --- emacs ---
|
||||
|
||||
58
README.md
58
README.md
@@ -1,10 +1,60 @@
|
||||
v2 architecture draft: see `docs/architecture-v2.org`
|
||||
# Running youdis
|
||||
|
||||
build and run the docker container
|
||||
```
|
||||
DISCORD_BOT_TOKEN = [discord bot token]
|
||||
-v [downloads]:/downloads
|
||||
-v [config]:/config
|
||||
-v [host_downloads]:/downloads
|
||||
-v [host_config]:/config
|
||||
-e ENABLE_DISCORD=1
|
||||
-e DISCORD_BOT_TOKEN=<token>
|
||||
-e DISCORD_BOT_SCOPE=<scope-int>
|
||||
```
|
||||
config contains data to persist across container updates, i.e., unraid appdata,
|
||||
such as yt-dlp's archive.txt
|
||||
|
||||
# Development
|
||||
|
||||
v2 architecture draft: `docs/architecture-v2.org`
|
||||
|
||||
The app runs with `youdis.py`.
|
||||
This starts the backend first, waits for health, then starts each enabled adapter explicitly.
|
||||
|
||||
Test components directly with uvicorn:
|
||||
```
|
||||
python3 -m uvicorn youdis.main:app --host 127.0.0.1 --port 8000
|
||||
python3 -m youdis.adapters.discord
|
||||
```
|
||||
|
||||
Key runner/config vars:
|
||||
```
|
||||
YOUDIS_ENABLE_BACKEND=1 #default enabled, dont disable unless testing
|
||||
YOUDIS_BACKEND_HOST=127.0.0.1
|
||||
YOUDIS_BACKEND_PORT=8000
|
||||
YOUDIS_BACKEND_HEALTH_TIMEOUT=20
|
||||
YOUDIS_BACKEND_HEALTH_INTERVAL=0.5
|
||||
YOUDIS_POLL_INTERVAL_SECONDS=2
|
||||
YOUDIS_YTDLP_EXECUTABLE=yt-dlp
|
||||
YOUDIS_CONFIG_DIR=/path/to/config/
|
||||
YOUDIS_DOWNLOAD_DIR=/path/to/downloads/
|
||||
ENABLE_DISCORD=1
|
||||
DISCORD_BOT_TOKEN=<token-string>
|
||||
DISCORD_BOT_SCOPE=123456789
|
||||
```
|
||||
|
||||
|
||||
## Add Adapter
|
||||
Configure adapter selection in .env, like so:
|
||||
```
|
||||
YOUDIS_RUN_BACKEND=1
|
||||
YOUDIS_BACKEND_HOST=127.0.0.1
|
||||
YOUDIS_BACKEND_PORT=8000
|
||||
ENABLE_DISCORD=1
|
||||
DISCORD_BOT_TOKEN=<api_key>
|
||||
DISCORD_BOT_SCOPE=123456789
|
||||
```
|
||||
|
||||
to add a new adapter:
|
||||
1. add `youdis/adapters/<adapter>.py`
|
||||
2. make it independently runnable
|
||||
3. add `start_<adapter>()` to `youdis.py`
|
||||
4. register it in the explicit adapter starter map in `youdis.py`
|
||||
5. add `ENABLE_<adapter>` and supporting env vars to `.env.example`
|
||||
|
||||
@@ -242,7 +242,58 @@ delete or retire legacy bot behaviors that no longer fit once the backend split
|
||||
- cleanup updated user-facing deployment artifacts to match the v2 naming and architecture, including `DISCORD_BOT_TOKEN` and removal of `users.json` references
|
||||
- archived planning docs were intentionally left untouched: `pm/tasks.org` is historical and `pm/notes.org` is personal working notes
|
||||
|
||||
* [ ] 2.0.4: fix automation and build pipeline (3)
|
||||
* [ ] 2.0.4: make youdis.py app runner for backend + adapters (3)
|
||||
Build a simple Python orchestration layer in youdis.py so the standard app stack can be launched from one entrypoint while backend and adapters remain independently runnable for testing.
|
||||
|
||||
** pm notes
|
||||
- keep components independently runnable
|
||||
- youdis.py as app-level orchestrator; let Docker just run python3 /app/youdis.py
|
||||
- no plugin discovery or hot-loading
|
||||
- Docker should package and launch the app, not decide internal process topology
|
||||
- preserve the ability to run backend or adapters directly for debugging
|
||||
- optimize for one obvious default run path
|
||||
|
||||
** Acceptance Criteria
|
||||
1. define the default app startup path
|
||||
- youdis.py launches the standard stack for v2
|
||||
- backend and Discord adapter startup order is explicit
|
||||
- shutdown behavior is coherent enough for local/dev use
|
||||
2. preserve modular run paths
|
||||
- backend can still be run directly
|
||||
- Discord adapter can still be run directly
|
||||
- orchestration layer does not bury component-level testing
|
||||
3. keep orchestration simple
|
||||
- no dynamic adapter discovery
|
||||
- no hot reloading framework
|
||||
- configured components are started explicitly
|
||||
4. document runtime ownership
|
||||
- clarify what Python orchestrates vs what Docker orchestrates
|
||||
- identify env vars or flags that control which components start
|
||||
- leave room for future Zulip/XMPP adapters without redesigning the runner
|
||||
|
||||
** evidence
|
||||
- commit: 309ce87
|
||||
- tests:
|
||||
1. `python3 -m py_compile ./youdis.py ./youdis/adapters/discord.py`
|
||||
2. backend direct run still works: `python3 -m uvicorn youdis.main:app --host 127.0.0.1 --port 8000`
|
||||
3. discord direct run still works: `python3 -m youdis.adapters.discord`
|
||||
4. app runner backend-only smoke test: `ENABLE_DISCORD=0 timeout 5s python3 ./youdis.py`
|
||||
5. app runner default path: `python3 ./youdis.py`
|
||||
- date: [2026-04-02 Thu 14:13]
|
||||
|
||||
** notes
|
||||
- `youdis.py` is now the default v2 app runner and starts the standard stack explicitly rather than dynamically discovering adapters
|
||||
- backend starts first and must pass a health check before the Discord adapter is launched
|
||||
- backend and Discord adapter remain directly runnable for debugging and tests
|
||||
- Docker is intended to invoke `python3 /app/youdis.py`; Python owns app orchestration while Docker owns packaging and runtime environment
|
||||
- adapter startup remains explicit in `youdis.py`, while each adapter gets its own `ENABLE_<ADAPTER>` flag
|
||||
- adding an adapter means creating `youdis/adapters/<adapter>.py`, adding `start_<adapter>()` in `youdis.py`, wiring its `ENABLE_<ADAPTER>` flag, and documenting any supporting env vars in `.env.example`
|
||||
- runner flags currently include `YOUDIS_RUN_BACKEND`, `YOUDIS_BACKEND_HOST`, `YOUDIS_BACKEND_PORT`, and `ENABLE_DISCORD`
|
||||
|
||||
|
||||
* ==== BACKLOG ====
|
||||
Tasks below this line are inactive and should not be touched.
|
||||
* [ ] 2.0.X: fix automation and build pipeline (3)
|
||||
repair and simplify the build/update/deploy path so it matches the new backend-plus-frontend structure
|
||||
** pm notes
|
||||
- this should come after architecture and discord integration stabilize. no point polishing the pipeline for the wrong shape.
|
||||
@@ -267,8 +318,6 @@ repair and simplify the build/update/deploy path so it matches the new backend-p
|
||||
|
||||
** notes
|
||||
|
||||
* ==== BACKLOG ====
|
||||
Tasks below this line are inactive and should not be touched.
|
||||
* [ ] X.x.x: clean up discord adapter UI
|
||||
|
||||
** acceptance criteria
|
||||
|
||||
157
youdis.py
157
youdis.py
@@ -1,7 +1,160 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Launcher shim for the Discord adapter."""
|
||||
"""App runner for FastAPI backend and discord adapter."""
|
||||
|
||||
from youdis.adapters.discord import main
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
import sys
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from urllib.error import URLError
|
||||
from urllib.request import urlopen
|
||||
|
||||
from youdis.env import load_project_dotenv
|
||||
|
||||
|
||||
load_project_dotenv()
|
||||
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parent
|
||||
DEFAULT_BACKEND_HOST = os.getenv("YOUDIS_BACKEND_HOST", "127.0.0.1")
|
||||
DEFAULT_BACKEND_PORT = int(os.getenv("YOUDIS_BACKEND_PORT", "8000"))
|
||||
ENABLE_BACKEND = os.getenv("YOUDIS_RUN_BACKEND", "1") not in {"0", "false", "False"}
|
||||
ENABLE_DISCORD = os.getenv("ENABLE_DISCORD", "1") not in {"0", "false", "False"}
|
||||
BACKEND_HEALTH_TIMEOUT = float(os.getenv("YOUDIS_BACKEND_HEALTH_TIMEOUT", "20"))
|
||||
BACKEND_HEALTH_INTERVAL = float(os.getenv("YOUDIS_BACKEND_HEALTH_INTERVAL", "0.5"))
|
||||
|
||||
|
||||
@dataclass
|
||||
class ManagedProcess:
|
||||
name: str
|
||||
process: asyncio.subprocess.Process
|
||||
|
||||
|
||||
def backend_url() -> str:
|
||||
return f"http://{DEFAULT_BACKEND_HOST}:{DEFAULT_BACKEND_PORT}"
|
||||
|
||||
|
||||
def build_child_env() -> dict[str, str]:
|
||||
env = os.environ.copy()
|
||||
env.setdefault("YOUDIS_BACKEND_URL", backend_url())
|
||||
return env
|
||||
|
||||
|
||||
async def start_backend(env: dict[str, str]) -> ManagedProcess:
|
||||
process = await asyncio.create_subprocess_exec(
|
||||
sys.executable,
|
||||
"-m",
|
||||
"uvicorn",
|
||||
"youdis.main:app",
|
||||
"--host",
|
||||
DEFAULT_BACKEND_HOST,
|
||||
"--port",
|
||||
str(DEFAULT_BACKEND_PORT),
|
||||
cwd=str(REPO_ROOT),
|
||||
env=env,
|
||||
)
|
||||
return ManagedProcess(name="backend", process=process)
|
||||
|
||||
|
||||
async def start_discord(env: dict[str, str]) -> ManagedProcess:
|
||||
process = await asyncio.create_subprocess_exec(
|
||||
sys.executable,
|
||||
"-m",
|
||||
"youdis.adapters.discord",
|
||||
cwd=str(REPO_ROOT),
|
||||
env=env,
|
||||
)
|
||||
return ManagedProcess(name="discord", process=process)
|
||||
|
||||
|
||||
ADAPTER_STARTERS = {
|
||||
"discord": start_discord,
|
||||
}
|
||||
|
||||
|
||||
def selected_adapters() -> list[str]:
|
||||
requested: list[str] = []
|
||||
if ENABLE_DISCORD:
|
||||
requested.append("discord")
|
||||
return requested
|
||||
|
||||
|
||||
async def wait_for_backend_health(timeout_seconds: float) -> None:
|
||||
deadline = asyncio.get_running_loop().time() + timeout_seconds
|
||||
health_url = f"{backend_url()}/health"
|
||||
|
||||
while True:
|
||||
try:
|
||||
with urlopen(health_url, timeout=2) as response:
|
||||
if response.status == 200:
|
||||
return
|
||||
except URLError:
|
||||
pass
|
||||
|
||||
if asyncio.get_running_loop().time() >= deadline:
|
||||
raise TimeoutError(f"backend did not become healthy at {health_url}")
|
||||
|
||||
await asyncio.sleep(BACKEND_HEALTH_INTERVAL)
|
||||
|
||||
|
||||
async def stop_process(proc: ManagedProcess) -> None:
|
||||
if proc.process.returncode is not None:
|
||||
return
|
||||
|
||||
proc.process.terminate()
|
||||
try:
|
||||
await asyncio.wait_for(proc.process.wait(), timeout=10)
|
||||
except asyncio.TimeoutError:
|
||||
proc.process.kill()
|
||||
await proc.process.wait()
|
||||
|
||||
|
||||
async def run() -> int:
|
||||
managed: list[ManagedProcess] = []
|
||||
env = build_child_env()
|
||||
adapters = selected_adapters()
|
||||
|
||||
if not ENABLE_BACKEND and not adapters:
|
||||
print("nothing to start: backend is disabled and no adapters are enabled")
|
||||
return 1
|
||||
|
||||
try:
|
||||
if ENABLE_BACKEND:
|
||||
backend = await start_backend(env)
|
||||
managed.append(backend)
|
||||
print(f"started backend on {backend_url()}")
|
||||
await wait_for_backend_health(BACKEND_HEALTH_TIMEOUT)
|
||||
print("backend is healthy")
|
||||
|
||||
for adapter_name in adapters:
|
||||
process = await ADAPTER_STARTERS[adapter_name](env)
|
||||
managed.append(process)
|
||||
print(f"started {adapter_name} adapter")
|
||||
|
||||
wait_tasks = [asyncio.create_task(proc.process.wait()) for proc in managed]
|
||||
done, pending = await asyncio.wait(wait_tasks, return_when=asyncio.FIRST_COMPLETED)
|
||||
|
||||
for task in pending:
|
||||
task.cancel()
|
||||
|
||||
first_returncode = 0
|
||||
for task in done:
|
||||
first_returncode = task.result()
|
||||
break
|
||||
|
||||
return first_returncode
|
||||
finally:
|
||||
for proc in reversed(managed):
|
||||
await stop_process(proc)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
try:
|
||||
raise SystemExit(asyncio.run(run()))
|
||||
except KeyboardInterrupt:
|
||||
raise SystemExit(130)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -12,7 +12,9 @@ from ..env import load_project_dotenv
|
||||
load_project_dotenv()
|
||||
|
||||
|
||||
BACKEND_URL = getenv("YOUDIS_BACKEND_URL", "http://127.0.0.1:8000").rstrip("/")
|
||||
BACKEND_HOST = getenv("YOUDIS_BACKEND_HOST", "127.0.0.1")
|
||||
BACKEND_PORT = int(getenv("YOUDIS_BACKEND_PORT", "8000"))
|
||||
BACKEND_URL = f"http://{BACKEND_HOST}:{BACKEND_PORT}".rstrip("/")
|
||||
POLL_INTERVAL_SECONDS = float(getenv("YOUDIS_POLL_INTERVAL_SECONDS", "2"))
|
||||
DEFAULT_SCOPE = int(getenv("DISCORD_BOT_SCOPE", "2147491904"))
|
||||
|
||||
@@ -114,22 +116,6 @@ def ensure_poll_task(ctx: interactions.SlashContext, job_id: str) -> None:
|
||||
return
|
||||
poll_tasks[job_id] = asyncio.create_task(poll_job_updates(ctx, job_id))
|
||||
|
||||
@bot.listen()
|
||||
async def on_startup():
|
||||
await get_session()
|
||||
print(f"discord adapter configured for backend {BACKEND_URL}")
|
||||
|
||||
@bot.listen()
|
||||
async def on_shutdown():
|
||||
global http_session
|
||||
for task in list(poll_tasks.values()):
|
||||
task.cancel()
|
||||
poll_tasks.clear()
|
||||
if http_session is not None and not http_session.closed:
|
||||
await http_session.close()
|
||||
http_session = None
|
||||
|
||||
|
||||
@interactions.slash_command(name="youtube", description="submit a youtube download to the backend")
|
||||
@interactions.slash_option(
|
||||
name="url",
|
||||
@@ -213,12 +199,33 @@ async def status(ctx: interactions.SlashContext):
|
||||
prefix = "active" if active else "last"
|
||||
await dm(ctx, f"{prefix} job: {format_status_message(job)}")
|
||||
|
||||
@bot.listen()
|
||||
async def on_startup():
|
||||
await get_session()
|
||||
print(f"discord adapter configured for backend {BACKEND_URL}")
|
||||
print(f"{bot.application_commands.count} registered commands:")
|
||||
for i, x in enumerate(bot.application_commands,start=1):
|
||||
print(f" {i}. {x.name}")
|
||||
|
||||
@bot.listen()
|
||||
async def on_shutdown():
|
||||
global http_session
|
||||
for task in list(poll_tasks.values()):
|
||||
task.cancel()
|
||||
poll_tasks.clear()
|
||||
if http_session is not None and not http_session.closed:
|
||||
await http_session.close()
|
||||
http_session = None
|
||||
|
||||
def main() -> None:
|
||||
api_token = getenv("DISCORD_BOT_TOKEN")
|
||||
if not api_token:
|
||||
raise ValueError("API token not set. Retrieve from your Discord bot.")
|
||||
bot.add_command(youtube)
|
||||
bot.add_command(status)
|
||||
bot.add_command(interrupt)
|
||||
# bot.add_command(youtube)
|
||||
# bot.add_command(status)
|
||||
# bot.add_command(interrupt)
|
||||
bot.start(api_token)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
||||
@@ -1,8 +1,13 @@
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
from dotenv import load_dotenv
|
||||
except ModuleNotFoundError: # pragma: no cover - optional local convenience dependency
|
||||
load_dotenv = None
|
||||
|
||||
|
||||
def load_project_dotenv() -> None:
|
||||
if load_dotenv is None:
|
||||
return
|
||||
repo_root = Path(__file__).resolve().parent.parent
|
||||
load_dotenv(repo_root / ".env", override=False)
|
||||
|
||||
Reference in New Issue
Block a user