Background processes / automated jobs (e.g. "subscribing" to a playlist to be constantly queried for new videos

[GlassedSilver]

Would love if I could have a docker that's both a manual downloader for one-off videos as well as defining criteria for what to watch for new content on a constant basis.

This would be especially helpful in one single yt-dl instance because then the archive.txt that stores already downloaded videos would skip duplicates. :)

[manbearwiz]

I do this now using flexget combined with youtube-dl. You can have flexget subscribe and filter to content and then post the URL to the API exposed by youtube-dl.

[GlassedSilver]

Does youtube-dl within this docker keep an archive.txt?

And what are its default options? I would love for metadata to be embedded and put into a separate file for example amongst other things like customizing the filename and path logic to automate importing into Plex and generally just having clean metadata independent of Plex' database...

Overall the documentation about the default behavior is a bit lukewarm...

[JunNineKim]

I want to try to do this one which you mentioned "I do this now using flexget combined with youtube-dl. You can have flexget subscribe and filter to content and then post the URL to the API exposed by youtube-dl."
Could you describe that one more detail?, I will try to do it.

[manbearwiz]

Here is a flexget task that subscribes to the rss feed for the official YouTube Music channel and sends new entries to the youtube-dl server for download. You should just have to change the curl url to point to your youtube-dl instance, and change the playlist_id to any channel you want.

tasks:
  get-youtubemusic:
    priority: 4
    rss:
      url: https://www.youtube.com/feeds/videos.xml?playlist_id=PLFgquLnL59alCl_2TQvOiD5Vgm1hCaGSI
    accept_all: yes
    exec:
      auto_escape: yes
      on_output:
        for_accepted: 'curl -X POST --data-urlencode "url={{url}}" http://web/youtube-dl/q'

[JunNineKim]

Great!
I have been digging much time for how to connect between flexget and youtube-dl.
I will try it later.
It's great for me.

[GlassedSilver]

Here is a flexget task that subscribes to the rss feed for the official YouTube Music channel and sends new entries to the youtube-dl server for download. You should just have to change the curl url to point to your youtube-dl instance, and change the playlist_id to any channel you want.

tasks:
  get-youtubemusic:
    priority: 4
    rss:
      url: https://www.youtube.com/feeds/videos.xml?playlist_id=PLFgquLnL59alCl_2TQvOiD5Vgm1hCaGSI
    accept_all: yes
    exec:
      auto_escape: yes
      on_output:
        for_accepted: 'curl -X POST --data-urlencode "url={{url}}" http://web/youtube-dl/q'

I'll try that out, would still love if you could provide some insight into the questions I asked in my second post. :)

Cheers

[JunNineKim]

I saw the this url.
This one is very helpful.
https://github.com/ytdl-org/youtube-dl/blob/master/README.md#embedding-youtube-dl

How do I download only new videos from a playlist?

Use download-archive feature. With this feature you should initially download the complete playlist with --download-archive /path/to/download/archive/file.txt that will record identifiers of all the videos in a special file. Each subsequent run with the same --download-archive will download only new videos and skip all videos that have been downloaded before. Note that only successful downloads are recorded in the file.

For example, at first,

youtube-dl --download-archive archive.txt "https://www.youtube.com/playlist?list=PLwiyx1dc3P2JR9N8gQaQN_BCvlSlap7re"
will download the complete PLwiyx1dc3P2JR9N8gQaQN_BCvlSlap7re playlist and create a file archive.txt. Each subsequent run will only download new videos if any:

youtube-dl --download-archive archive.txt "https://www.youtube.com/playlist?list=PLwiyx1dc3P2JR9N8gQaQN_BCvlSlap7re"

I tried this one in command line.

/usr/bin/youtube-dl -f 'bestvideo[ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]/best' --merge-output-format mp4 -o '/downloads/complete/%(title)s - %(upload_date)s.%(width)sx%(height)s.%(ext)s' --download-archive /cron_jobs/archive.txt https://www.youtube.com/results?search_query=_+김어준의+뉴스공장 --playlist-end 2 -R 5

remarks

download folder : /downloads/complete/
Achive folder and file : /cron_jobs/archive.txt
search word : https://www.youtube.com/results?search_query=_+김어준의+뉴스공장
get video (--playlist-end 2) : end 2 EA
repeat 5 times : -R 5

Cheers too.

[GlassedSilver]

I saw the this url.
This one is very helpful.
https://github.com/ytdl-org/youtube-dl/blob/master/README.md#embedding-youtube-dl

How do I download only new videos from a playlist?

Use download-archive feature. With this feature you should initially download the complete playlist with --download-archive /path/to/download/archive/file.txt that will record identifiers of all the videos in a special file. Each subsequent run with the same --download-archive will download only new videos and skip all videos that have been downloaded before. Note that only successful downloads are recorded in the file.

For example, at first,

youtube-dl --download-archive archive.txt "https://www.youtube.com/playlist?list=PLwiyx1dc3P2JR9N8gQaQN_BCvlSlap7re"
will download the complete PLwiyx1dc3P2JR9N8gQaQN_BCvlSlap7re playlist and create a file archive.txt. Each subsequent run will only download new videos if any:

youtube-dl --download-archive archive.txt "https://www.youtube.com/playlist?list=PLwiyx1dc3P2JR9N8gQaQN_BCvlSlap7re"

I tried this one in command line.

/usr/bin/youtube-dl -f 'bestvideo[ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]/best' --merge-output-format mp4 -o '/downloads/complete/%(title)s - %(upload_date)s.%(width)sx%(height)s.%(ext)s' --download-archive /cron_jobs/archive.txt https://www.youtube.com/results?search_query=_+김어준의+뉴스공장 --playlist-end 2 -R 5

remarks

download folder : /downloads/complete/
Achive folder and file : /cron_jobs/archive.txt
search word : https://www.youtube.com/results?search_query=_+김어준의+뉴스공장
get video (--playlist-end 2) : end 2 EA
repeat 5 times : -R 5

Cheers too.

I know about those parameters, that's why I bring this up to begin with. :)

[PDSCi996]

Excuse me, I set it up according to the configuration given by the blogger, but I don't know why the setting has been unsuccessful. Can you help me find the suitable reason? The following is my configuration.

192.168.1.234: 8080 is my youtube-dl container address

web_server:
bind: 0.0.0.0
port: 5050
web_ui: yes

tasks:
get-youtubemusic:
priority: 4
rss:
url: https://www.youtube.com/feeds/videos.xml?channel_id=UCUXi5mmqbvIithAs9AaxEtw
accept_all: yes
exec:
auto_escape: yes
on_output:
for_accepted: 'curl -X POST --data-urlencode "url={{url}}" http://192.168.1.234:8080/youtube-dl/q'