V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
推荐学习书目
Learn Python the Hard Way
Python Sites
PyPI - Python Package Index
http://diveintopython.org/toc/index.html
Pocoo
值得关注的项目
PyPy
Celery
Jinja2
Read the Docs
gevent
pyenv
virtualenv
Stackless Python
Beautiful Soup
结巴中文分词
Green Unicorn
Sentry
Shovel
Pyflakes
pytest
Python 编程
pep8 Checker
Styles
PEP 8
Google Python Style Guide
Code Style from The Hitchhiker's Guide
CSM
V2EX  ›  Python

让你在线看视频也能达到多线程下载的速度

  •  1
     
  •   CSM ·
    cshuaimin · 2018-01-05 22:10:49 +08:00 · 9823 次点击
    这是一个创建于 2498 天前的主题,其中的信息可能已经有所发展或是发生改变。

    Video Funnel - 让你在线看视频也能达到多线程下载的速度


    马上使用:

    1. PyPI 安装:
    $ pip(3) install --user video_funnel
    # or
    $ sudo pip(3) install video_funnel
    
    1. 启动 video_funnel 的服务器:
    $ vf http://tulip.ink/test.mp4
    ======== Running on http://0.0.0.0:8080 ========
    (Press CTRL+C to quit)
    
    1. mpv 播放:
    $ mpv http://localhost:8080
    

    动机:

    众所周知,百度网盘之类产品的视频在线播放非常模糊,下载吧又限速,于是我写了 aiodl 这个下载器,通过 EX-百度云盘 获取的直链来“多线程”下载。可是每次都要下载完才能看又十分不爽,直接用 mpv 之类的播放器播放直链又因为限速的原因根本没法看,遂有了本项目。

    实现思路:

    1. 先将视频按照一定大小分块。块的大小根据视频的清晰度而异,以下载完一个块后视频可以播放为准。可通过命令行参数 --block-size/-b 来指定,默认为 8MB。
    2. 对于上一步中的一个块,再次分块——为区别改叫切片,启动多个协程来下载这些切片,以实现“多线程”提速的目的。块和切片大小一起决定了有多少个连接在同时下载。切片的大小通过 --piece-size/-p 来指定,默认为 1MB。
    3. 一个块中的切片全部下载完后,就可以将数据传给播放器了。当播放器播放这一块的时候,回到第 2 步下载下一块数据。为节省内存,设置了在内存中最多存在 2 个下载完而又没有传给播放器的块。

    一些细节:

    1. 该如何把数据传给播放器呢?我最初的设想是通过标准输出,这样简单好写。但 stdio 是无法 seek 的,这就意味着你只能从视频的开头看起,无法快进 :P 如你所见,现在的解决方案是用 HTTP 协议与播放器传输数据。需要快进的时候播放器发送 HTTP Range 请求,video_funnel 将请求中的范围经过分块、切片后“多线程”下载。但这样就又带来了两个问题:

      1. 需要播放器支持从 URL 播放。mplayer、mpv 之类的命令行播放器大多都支持,但一些 Windows 的播放器就不得而知了 :P 不过可以使用 HTML 的 video 标签在浏览器播放。
      2. 怎么就没有处理 Range 请求的包啊,自己处理很麻烦的好吗~
    2. 由于下载的部分是用异步 IO 写的,与播放器交互的服务器部分就不能使用 Flask 之类阻塞的框架了,幸好 aiohttp 居然同时支持客户端和服务端。

    3. 说起来简单,实际写起来处处是坑啊 :(

    参加 https://www.v2ex.com/t/405569

    GitHub: https://github.com/cshuaimin/video-funnel

    第 1 条附言  ·  2018-01-08 17:19:10 +08:00

    刚刚把输出调整了以下,用上了进度条,大家快来更新啊 ;-D

    $ pip install video_funnel --upgrade
    
    $ vf https://tulip.ink/test.mp4 &
    * Listening at port 8080 ...
    
    $ mpv http://localhost:8080 --no-ytdl --quiet
    Playing: http://localhost:8080
    Block #0:  15%|████▏                        | 600k/4.00M [00:02<03:30, 17.0kB/s]
    
    28 条回复    2019-10-12 11:26:20 +08:00
    nazor
        1
    nazor  
       2018-01-05 23:40:45 +08:00 via iPhone   ❤️ 2
    挺好的,要是能用 js 实现就更好了。
    davy1995
        2
    davy1995  
       2018-01-05 23:47:01 +08:00 via Android   ❤️ 1
    可以可以,很实用啊感觉
    mingyun
        3
    mingyun  
       2018-01-06 00:08:17 +08:00   ❤️ 1
    $ pip install video_funnel
    Collecting video_funnel
    Could not find a version that satisfies the requirement video_funnel (from ver
    sions: )
    No matching distribution found for video_funnel
    CSM
        4
    CSM  
    OP
       2018-01-06 00:11:31 +08:00 via Android
    @nazor 对啊,要是 js 实现的就可以直接在浏览器播放了

    @mingyun 你是不是没有用官方源?可能是还没有同步过去
    zingl
        5
    zingl  
       2018-01-06 00:24:10 +08:00   ❤️ 1
    youtube 就是这个套路
    swulling
        6
    swulling  
       2018-01-06 00:29:52 +08:00 via iPhone   ❤️ 1
    @mingyun 应该是 pip 版本太老升级下
    diggerdu
        7
    diggerdu  
       2018-01-06 00:46:31 +08:00 via iPad   ❤️ 1
    mo 一波 dalao 的 aiodl 真不错
    madfloyd
        8
    madfloyd  
       2018-01-06 01:10:00 +08:00 via Android   ❤️ 1
    大有用处。。。搞个视频网站
    netsail
        9
    netsail  
       2018-01-06 01:37:27 +08:00 via iPad   ❤️ 1
    前排围观支持!!!
    ryd994
        10
    ryd994  
       2018-01-06 01:52:38 +08:00 via Android   ❤️ 6
    aria2c -x10 --stream-piece-selector=inorder
    CSM
        11
    CSM  
    OP
       2018-01-06 02:04:42 +08:00
    @ryd994 唉你看我这轮子造的,早知道有 aira2 这等神器还自己折腾什么。。。
    CSM
        12
    CSM  
    OP
       2018-01-06 02:13:54 +08:00   ❤️ 1
    不过 aira2 貌似不能快进?容我加上”边看边下“的功能 :)
    linap
        13
    linap  
       2018-01-06 07:59:56 +08:00 via Android
    mark.摩拜大佬
    wzhndd2
        14
    wzhndd2  
       2018-01-06 09:36:21 +08:00
    mark.摩拜大佬
    chen2016
        15
    chen2016  
       2018-01-06 10:22:44 +08:00 via Android
    收藏多,回复少,一群大佬准备大干
    honkew
        16
    honkew  
       2018-01-06 10:33:43 +08:00   ❤️ 1
    这个叫做 串流播放
    gabrielsong
        17
    gabrielsong  
       2018-01-06 10:58:01 +08:00 via Android
    膜拜大佬…
    CSM
        18
    CSM  
    OP
       2018-01-06 11:45:10 +08:00 via Android
    @chen2016 我也很奇怪为什么这么多收藏😂
    zqqian
        19
    zqqian  
       2018-01-06 12:22:56 +08:00
    mark.摩拜大佬
    wcsjtu
        20
    wcsjtu  
       2018-01-07 12:05:11 +08:00 via Android
    TS 流怎么搞……
    CSM
        21
    CSM  
    OP
       2018-01-07 17:54:24 +08:00 via Android
    @wcsjtu m3u8 吗?那就没办法了……
    cnaol
        22
    cnaol  
       2018-01-07 21:56:07 +08:00
    Command "python setup.py egg_info" failed with error code 1 in C:\Users\UrAir\AppData\Local\Temp\pip-build-4koh515g\video-funnel\

    怎么破
    CSM
        23
    CSM  
    OP
       2018-01-07 22:11:19 +08:00 via Android
    @cnaol 具体错误是什么呢?需要 Python 版本为 3.6,是不是这个原因?
    cnaol
        24
    cnaol  
       2018-01-07 22:55:57 +08:00
    @CSM
    > PS F:\myproject> pip install --user video_funnel
    Collecting video_funnel
    Using cached video_funnel-0.0.3.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "C:\Users\UrAir\AppData\Local\Temp\pip-build-v519sou0\video-funnel\setup.py", line 29, in <module>
    'vf = video_funnel.__main__:main'
    File "c:\users\urair\appdata\local\programs\python\python36\lib\distutils\core.py", line 108, in setup
    _setup_distribution = dist = klass(attrs)
    File "c:\users\urair\appdata\local\programs\python\python36\lib\site-packages\setuptools\dist.py", line 315, in __init__
    self.fetch_build_eggs(attrs['setup_requires'])
    File "c:\users\urair\appdata\local\programs\python\python36\lib\site-packages\setuptools\dist.py", line 361, in fetch_build_eggs
    replace_conflicting=True,
    File "c:\users\urair\appdata\local\programs\python\python36\lib\site-packages\pkg_resources\__init__.py", line 850, in resolve
    dist = best[req.key] = env.best_match(req, ws, installer)
    File "c:\users\urair\appdata\local\programs\python\python36\lib\site-packages\pkg_resources\__init__.py", line 1122, in best_match
    return self.obtain(req, installer)
    File "c:\users\urair\appdata\local\programs\python\python36\lib\site-packages\pkg_resources\__init__.py", line 1134, in obtain
    return installer(requirement)
    File "c:\users\urair\appdata\local\programs\python\python36\lib\site-packages\setuptools\dist.py", line 429, in fetch_build_egg
    return cmd.easy_install(req)
    File "c:\users\urair\appdata\local\programs\python\python36\lib\site-packages\setuptools\command\easy_install.py", line 653, in easy_install
    not self.always_copy, self.local_index
    File "c:\users\urair\appdata\local\programs\python\python36\lib\site-packages\setuptools\package_index.py", line 636, in fetch_distribution
    dist = find(requirement)
    File "c:\users\urair\appdata\local\programs\python\python36\lib\site-packages\setuptools\package_index.py", line 617, in find
    dist.download_location = self.download(dist.location, tmpdir)
    File "c:\users\urair\appdata\local\programs\python\python36\lib\site-packages\setuptools\package_index.py", line 566, in download
    found = self._download_url(scheme.group(1), spec, tmpdir)
    File "c:\users\urair\appdata\local\programs\python\python36\lib\site-packages\setuptools\package_index.py", line 805, in _download_url
    return self._attempt_download(url, filename)
    File "c:\users\urair\appdata\local\programs\python\python36\lib\site-packages\setuptools\package_index.py", line 811, in _attempt_download
    headers = self._download_to(url, filename)
    File "c:\users\urair\appdata\local\programs\python\python36\lib\site-packages\setuptools\package_index.py", line 710, in _download_to
    fp = self.open_url(strip_fragment(url))
    File "c:\users\urair\appdata\local\programs\python\python36\lib\site-packages\setuptools\package_index.py", line 747, in open_url
    return open_with_auth(url, self.opener)
    File "c:\users\urair\appdata\local\programs\python\python36\lib\site-packages\setuptools\package_index.py", line 948, in _socket_timeout
    return func(*args, **kwargs)
    File "c:\users\urair\appdata\local\programs\python\python36\lib\site-packages\setuptools\package_index.py", line 1067, in open_with_auth
    fp = opener(request)
    File "c:\users\urair\appdata\local\programs\python\python36\lib\urllib\request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
    File "c:\users\urair\appdata\local\programs\python\python36\lib\urllib\request.py", line 526, in open
    response = self._open(req, data)
    File "c:\users\urair\appdata\local\programs\python\python36\lib\urllib\request.py", line 544, in _open
    '_open', req)
    File "c:\users\urair\appdata\local\programs\python\python36\lib\urllib\request.py", line 504, in _call_chain
    result = func(*args)
    File "c:\users\urair\appdata\local\programs\python\python36\lib\urllib\request.py", line 1361, in https_open
    context=self._context, check_hostname=self._check_hostname)
    File "c:\users\urair\appdata\local\programs\python\python36\lib\urllib\request.py", line 1321, in do_open
    r = h.getresponse()
    File "c:\users\urair\appdata\local\programs\python\python36\lib\http\client.py", line 1331, in getresponse
    response.begin()
    File "c:\users\urair\appdata\local\programs\python\python36\lib\http\client.py", line 297, in begin
    version, status, reason = self._read_status()
    File "c:\users\urair\appdata\local\programs\python\python36\lib\http\client.py", line 258, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
    File "c:\users\urair\appdata\local\programs\python\python36\lib\socket.py", line 586, in readinto
    return self._sock.recv_into(b)
    File "c:\users\urair\appdata\local\programs\python\python36\lib\ssl.py", line 1009, in recv_into
    return self.read(nbytes, buffer)
    File "c:\users\urair\appdata\local\programs\python\python36\lib\ssl.py", line 871, in read
    return self._sslobj.read(len, buffer)
    File "c:\users\urair\appdata\local\programs\python\python36\lib\ssl.py", line 631, in read
    v = self._sslobj.read(len, buffer)
    socket.timeout: The read operation timed out

    ----------------------------------------
    Command "python setup.py egg_info" failed with error code 1 in C:\Users\UrAir\AppData\Local\Temp\pip-build-v519sou0\video-funnel\
    CSM
        25
    CSM  
    OP
       2018-01-07 23:49:31 +08:00 via Android
    @cnaol 看最后一行,socket.timeout: The read operation timed out 应该是网络原因,换个 pip 源或挂代理试试
    learningman
        26
    learningman  
       2019-02-25 19:11:12 +08:00 via iPad
    * Listening at port 8080 ...
    Error handling request
    Traceback (most recent call last):
    File "/root/.local/lib/python3.6/site-packages/aiohttp/web_protocol.py", line 418, in start
    resp = await task
    File "/root/.local/lib/python3.6/site-packages/aiohttp/web_app.py", line 458, in _handle
    resp = await handler(request)
    File "/root/.local/lib/python3.6/site-packages/video_funnel/__init__.py", line 85, in handler
    del request.headers['Host']
    TypeError: 'multidict._multidict.CIMultiDictProxy' object does not support item deletion
    貌似库改了
    vcheckzen
        27
    vcheckzen  
       2019-03-01 17:53:44 +08:00 via Android
    windows 下安装了找不到 vf 命令
    gIrl1990
        28
    gIrl1990  
       2019-10-12 11:26:20 +08:00
    @ryd994 --stream-piece-selector= inorder 和 geom 有什么区别?
    关于   ·   帮助文档   ·   博客   ·   API   ·   FAQ   ·   实用小工具   ·   2786 人在线   最高记录 6679   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 27ms · UTC 13:02 · PVG 21:02 · LAX 05:02 · JFK 08:02
    Developed with CodeLauncher
    ♥ Do have faith in what you're doing.