V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
推荐学习书目
Learn Python the Hard Way
Python Sites
PyPI - Python Package Index
http://diveintopython.org/toc/index.html
Pocoo
值得关注的项目
PyPy
Celery
Jinja2
Read the Docs
gevent
pyenv
virtualenv
Stackless Python
Beautiful Soup
结巴中文分词
Green Unicorn
Sentry
Shovel
Pyflakes
pytest
Python 编程
pep8 Checker
Styles
PEP 8
Google Python Style Guide
Code Style from The Hitchhiker's Guide
hanssx
V2EX  ›  Python

flask 程序使用 websocket,部署使用 gunicorn, gunicorn 导致 websocket 运行有问题

  •  
  •   hanssx · 2018-10-11 12:07:09 +08:00 · 8679 次点击
    这是一个创建于 2235 天前的主题,其中的信息可能已经有所发展或是发生改变。

    #1 本地开发 flask 应用程序,里面使用了 websocket,异步模式使用得是 gevent,并在程序刚开始打了 gevent 猴子补丁,s 端主要监听 connect disconnect 两个事件用来做一些前端页面的初始化和离开动作。 启动代码:

    socketio.run(app=app, host='0.0.0.0', port=5000, debug=True)
    

    #2 正常在本地 debug 或 run 都没问题,部署到测试环境遇到问题,测试环境采用得是 Nginx+supervisor+gunicorn+flask, nginx 配置:

        server {
            listen       8000;
            listen       [::]:8000;
            server_name  x.x.x.x;
            root /root/python/asset;
            location / {
                	proxy_pass http://127.0.0.1:5000;
    	    	proxy_redirect off;
    	    	proxy_set_header Host $host;
    	    	proxy_set_header X-Real-IP $remote_addr;
    	    	proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    	    	access_log /var/log/asset/access.log;
    	    	error_log /var/log/asset/error.log;
    	    	proxy_set_header Upgrade $http_upgrade;
                	proxy_set_header Connection "Upgrade";
            }
        }
    

    supervisor 配置:

    [program:xx]
    directory = /root/python/xx
    command = /root/.local/xxx/bin/gunicorn -b 127.0.0.1:5000 -k geventwebsocket.gunicorn.workers.GeventWebSocketWorker -w 1 module:app
    user = root
    autostart = true
    autorestart = true
    stopasgroup = true
    killasgroup = true
    startsecs = 5
    startretries = 3
    redirect_stderr = true
    stdout_logfile_maxbytes = 20MB
    stdout_logfile_backups = 20
    stdout_logfile = /var/log/xx/stdout.log
    stderr_logfile = /var/log/xx/stderr.log
    

    其中启动 gunicorn 的命令参考自
    https://stackoverflow.com/questions/38624447/websockets-proxied-by-nginx-to-gunicorn-over-https-giving-400-bad-request
    https://segmentfault.com/q/1010000007495163
    https://flask-socketio.readthedocs.io/en/latest/#gunicorn-web-server
    gunicorn 也在 pipenv 环境下安装

    #3 不使用 gunicorn 的情况下没任何问题,如图: image 使用了 gunicorn 的情况,
    首先会导致 402 错误, image
    402 详情, image
    其次,会一直 pending,一直在发请求, image
    最后是 402 详情在控制台的信息, image
    请各位帮帮忙,尝试了好长时间没办法解决,感谢各位。

    30 条回复    2018-10-11 17:56:05 +08:00
    greyli
        1
    greyli  
       2018-10-11 12:18:59 +08:00
    按照 Flask-SocketIO 的文档,使用 Gunicorn 的时候要使用 eventlet 或 gevent worker,所以启动命令应该是:
    ```
    gunicorn --worker-class eventlet -w 1 module:app
    ```
    或:
    ```
    gunicorn -k gevent -w 1 module:app
    ```
    greyli
        2
    greyli  
       2018-10-11 12:19:31 +08:00
    原来回复不支持 Mardown ……
    hanssx
        3
    hanssx  
    OP
       2018-10-11 12:22:16 +08:00
    对了,supervisor 有错误日志,一直在尝试连接,如下图:
    ![image]( http://pgf1p2i0h.bkt.clouddn.com/supervisor%E9%94%99%E8%AF%AF%E6%97%A5%E5%BF%9755.png)
    hanssx
        4
    hanssx  
    OP
       2018-10-11 12:23:38 +08:00
    @greyli 辉哥,我改了启动命令了,参考的文档里面
    gunicorn -k geventwebsocket.gunicorn.workers.GeventWebSocketWorker -w 1 module:app
    hanssx
        5
    hanssx  
    OP
       2018-10-11 12:25:19 +08:00
    这是 flask-socketio 官网给的例子,说了一大堆英文,反正就是要用 gunicorn 就要那么写,如果搭配 gevent 的话,
    https://flask-socketio.readthedocs.io/en/latest/#gunicorn-web-server
    最后命令是这么写的,
    gunicorn -b 127.0.0.1:5000 -k geventwebsocket.gunicorn.workers.GeventWebSocketWorker -w 1 module:app
    itertools
        6
    itertools  
       2018-10-11 13:41:05 +08:00
    gunicorn 版本问题
    itertools
        7
    itertools  
       2018-10-11 13:42:19 +08:00
    使用 gunicorn==19.9.0 这个版本试试。
    FiveDDD
        8
    FiveDDD  
       2018-10-11 13:47:20 +08:00
    启动命令:gunicorn -b :5000 -w 1 module:app -k eventlet

    版本是 19.9.0 没问题
    hanssx
        9
    hanssx  
    OP
       2018-10-11 13:48:22 +08:00   ❤️ 1
    @itertools 感谢关注,但是测试机上面的 gunicorn 版本就是你说的这个,
    (asset-ar0OxIPP) [root@VM asset]# pipenv graph | grep gunicorn
    gunicorn==19.9.0
    hanssx
        10
    hanssx  
    OP
       2018-10-11 13:49:47 +08:00
    @FiveDDD 感谢回复,我用的 gevent,命令和你这个不一样
    itertools
        11
    itertools  
       2018-10-11 13:52:32 +08:00
    @hanssx 看日记,可能是之前启动的进程还在导致。
    hanssx
        12
    hanssx  
    OP
       2018-10-11 13:57:57 +08:00
    @itertools
    supervisorctl stop all
    find / -name supervisor.sock 后 unlink 掉
    再重新启动 supervisord -c /etc/supervisord.conf
    还是不行。。。
    supervisorctl status,显示
    asset STARTING
    没 RUNNING 起来。。。
    hanssx
        13
    hanssx  
    OP
       2018-10-11 13:59:40 +08:00
    看日志,还是那些东西,我去 gg 一下日志相关的东西,
    [2018-10-11 13:58:33 +0800] [19707] [INFO] Starting gunicorn 19.9.0
    [2018-10-11 13:58:33 +0800] [19707] [ERROR] Connection in use: ('127.0.0.1', 5000)
    [2018-10-11 13:58:33 +0800] [19707] [ERROR] Retrying in 1 second.
    [2018-10-11 13:58:34 +0800] [19702] [ERROR] Connection in use: ('127.0.0.1', 5000)
    [2018-10-11 13:58:34 +0800] [19702] [ERROR] Retrying in 1 second.
    [2018-10-11 13:58:34 +0800] [19707] [ERROR] Connection in use: ('127.0.0.1', 5000)
    [2018-10-11 13:58:34 +0800] [19707] [ERROR] Retrying in 1 second.
    [2018-10-11 13:58:35 +0800] [19702] [ERROR] Can't connect to ('127.0.0.1', 5000)
    [2018-10-11 13:58:35 +0800] [19707] [ERROR] Connection in use: ('127.0.0.1', 5000)
    [2018-10-11 13:58:35 +0800] [19707] [ERROR] Retrying in 1 second.
    [2018-10-11 13:58:36 +0800] [19707] [ERROR] Connection in use: ('127.0.0.1', 5000)
    [2018-10-11 13:58:36 +0800] [19707] [ERROR] Retrying in 1 second.
    [2018-10-11 13:58:37 +0800] [19716] [INFO] Starting gunicorn 19.9.0
    [2018-10-11 13:58:37 +0800] [19716] [ERROR] Connection in use: ('127.0.0.1', 5000)
    [2018-10-11 13:58:37 +0800] [19716] [ERROR] Retrying in 1 second.
    hanssx
        14
    hanssx  
    OP
       2018-10-11 14:06:19 +08:00
    @itertools
    我根据 https://stackoverflow.com/questions/16756624/gunicorn-connection-in-use-0-0-0-0-5000/40894414
    [root@VM asset]# sudo fuser -k 5000/tcp
    5000/tcp: 18762 18765
    [root@VM asset]# supervisorctl status
    asset RUNNING pid 20781, uptime 0:00:32
    [root@VM asset]# netstat -antlp | grep 5000
    tcp 0 0 127.0.0.1:5000 0.0.0.0:* LISTEN 20781/python3.7m
    tcp 0 0 127.0.0.1:49678 127.0.0.1:5000 TIME_WAIT -
    tcp 0 0 127.0.0.1:5000 127.0.0.1:49708 ESTABLISHED 20787/python3.7m
    tcp 0 0 127.0.0.1:49650 127.0.0.1:5000 TIME_WAIT -
    tcp 0 0 127.0.0.1:5000 127.0.0.1:49706 TIME_WAIT -
    tcp 0 0 127.0.0.1:49708 127.0.0.1:5000 ESTABLISHED 25344/nginx: worker
    tcp 0 0 127.0.0.1:5000 127.0.0.1:49680 TIME_WAIT -
    so1n
        15
    so1n  
       2018-10-11 14:15:33 +08:00
    有没有内存不够的可能呢?
    hanssx
        16
    hanssx  
    OP
       2018-10-11 14:31:45 +08:00
    @itertools
    @so1n
    @greyli 大概率是我 gunicorn 进程的问题,现在正在学习咋清除干净。。。
    hanssx
        17
    hanssx  
    OP
       2018-10-11 14:52:21 +08:00
    @itertools 打扰,我已经重新清理了之后使用 sudo kill -9,确认网站访问不了,然后重新 supervisord -c /etc/supervisord.conf
    问题依然还在,日志中是这样,请问这是 work 吗?这和我在 supervior conf 中配置的不一样啊,我用了参数-w 1 呀。
    [2018-10-11 14:28:36 +0800] [25625] [ERROR] Connection in use: ('127.0.0.1', 5000)
    [2018-10-11 14:28:36 +0800] [25625] [ERROR] Retrying in 1 second.
    [2018-10-11 14:28:37 +0800] [25625] [INFO] Listening at: http://127.0.0.1:5000 (25625)
    [2018-10-11 14:28:37 +0800] [25625] [INFO] Using worker: geventwebsocket.gunicorn.workers.GeventWebSocketWorker
    [2018-10-11 14:28:37 +0800] [25635] [INFO] Booting worker with pid: 25635
    [2018-10-11 14:32:15 +0800] [26148] [INFO] Starting gunicorn 19.9.0
    [2018-10-11 14:32:15 +0800] [26148] [INFO] Listening at: http://127.0.0.1:5000 (26148)
    [2018-10-11 14:32:15 +0800] [26148] [INFO] Using worker: geventwebsocket.gunicorn.workers.GeventWebSocketWorker
    [2018-10-11 14:32:15 +0800] [26151] [INFO] Booting worker with pid: 26151
    [2018-10-11 14:33:32 +0800] [26345] [INFO] Starting gunicorn 19.9.0
    [2018-10-11 14:33:32 +0800] [26345] [INFO] Listening at: http://127.0.0.1:5000 (26345)
    [2018-10-11 14:33:32 +0800] [26345] [INFO] Using worker: geventwebsocket.gunicorn.workers.GeventWebSocketWorker
    [2018-10-11 14:33:32 +0800] [26348] [INFO] Booting worker with pid: 26348
    [2018-10-11 14:37:35 +0800] [26931] [INFO] Starting gunicorn 19.9.0
    [2018-10-11 14:37:35 +0800] [26931] [INFO] Listening at: http://127.0.0.1:5000 (26931)
    [2018-10-11 14:37:35 +0800] [26931] [INFO] Using worker: geventwebsocket.gunicorn.workers.GeventWebSocketWorker
    [2018-10-11 14:37:35 +0800] [26934] [INFO] Booting worker with pid: 26934
    [2018-10-11 14:39:18 +0800] [27180] [INFO] Starting gunicorn 19.9.0
    [2018-10-11 14:39:18 +0800] [27180] [INFO] Listening at: http://127.0.0.1:5000 (27180)
    [2018-10-11 14:39:18 +0800] [27180] [INFO] Using worker: geventwebsocket.gunicorn.workers.GeventWebSocketWorker
    [2018-10-11 14:39:18 +0800] [27183] [INFO] Booting worker with pid: 27183
    hanssx
        18
    hanssx  
    OP
       2018-10-11 15:29:56 +08:00
    开启了 flask-socketio 的日志,现在重新启动,ps -ef | grep guni 出现 2 行,其中 1 行是 work,work 数应该没错,日志现在报错,继续搜索解决方案,
    [2018-10-11 15:19:04 +0800] [697] [INFO] Starting gunicorn 19.9.0
    [2018-10-11 15:19:04 +0800] [697] [INFO] Listening at: http://127.0.0.1:5000 (697)
    [2018-10-11 15:19:04 +0800] [697] [INFO] Using worker: geventwebsocket.gunicorn.workers.GeventWebSocketWorker
    [2018-10-11 15:19:04 +0800] [700] [INFO] Booting worker with pid: 700
    Server initialized for gevent.
    bae5c5f54be848fcabd9e5cfe8ce59fa: Sending packet OPEN data {'sid': 'bae5c5f54be848fcabd9e5cfe8ce59fa', 'upgrades': ['websocket'], 'pingTimeout': 60000, 'pingInterval': 25000}
    bae5c5f54be848fcabd9e5cfe8ce59fa: Sending packet MESSAGE data 0
    bae5c5f54be848fcabd9e5cfe8ce59fa: Received packet MESSAGE data 0/process_upload_csv
    bae5c5f54be848fcabd9e5cfe8ce59fa: Sending packet MESSAGE data 0/process_upload_csv
    Traceback (most recent call last):
    File "/root/.local/share/virtualenvs/asset-ar0OxIPP/lib/python3.7/site-packages/gevent/pywsgi.py", line 975, in handle_one_response
    self.run_application()
    File "/root/.local/share/virtualenvs/asset-ar0OxIPP/lib/python3.7/site-packages/geventwebsocket/handler.py", line 82, in run_application
    self.process_result()
    File "/root/.local/share/virtualenvs/asset-ar0OxIPP/lib/python3.7/site-packages/gevent/pywsgi.py", line 909, in process_result
    self.write(data)
    File "/root/.local/share/virtualenvs/asset-ar0OxIPP/lib/python3.7/site-packages/gevent/pywsgi.py", line 756, in write
    self._write_with_headers(data)
    File "/root/.local/share/virtualenvs/asset-ar0OxIPP/lib/python3.7/site-packages/gevent/pywsgi.py", line 777, in _write_with_headers
    self._write(data)
    File "/root/.local/share/virtualenvs/asset-ar0OxIPP/lib/python3.7/site-packages/gevent/pywsgi.py", line 741, in _write
    self._sendall(data)
    File "/root/.local/share/virtualenvs/asset-ar0OxIPP/lib/python3.7/site-packages/gevent/pywsgi.py", line 703, in _sendall
    self.socket.sendall(data)
    File "/root/.local/share/virtualenvs/asset-ar0OxIPP/lib/python3.7/site-packages/gevent/_socket3.py", line 457, in sendall
    data_memory = _get_memory(data)
    File "/root/.local/share/virtualenvs/asset-ar0OxIPP/lib/python3.7/site-packages/gevent/_socket3.py", line 45, in _get_memory
    mv = memoryview(data)
    TypeError: memoryview: a bytes-like object is required, not 'str'
    2018-10-11T07:20:48Z {'REMOTE_ADDR': '127.0.0.1', 'REMOTE_PORT': '54310', 'HTTP_HOST': '10.26.15.222', (hidden keys: 31)} failed with TypeError

    bae5c5f54be848fcabd9e5cfe8ce59fa: Client is gone, closing socket
    bae5c5f54be848fcabd9e5cfe8ce59fa: Client is gone, closing socket
    ca996671e95d4e54834a745c9129caa8: Sending packet OPEN data {'sid': 'ca996671e95d4e54834a745c9129caa8', 'upgrades': ['websocket'], 'pingTimeout': 60000, 'pingInterval': 25000}
    ca996671e95d4e54834a745c9129caa8: Sending packet MESSAGE data 0
    ca996671e95d4e54834a745c9129caa8: Received packet MESSAGE data 0/process_upload_csv
    ca996671e95d4e54834a745c9129caa8: Sending packet MESSAGE data 0/process_upload_csv
    Traceback (most recent call last):
    File "/root/.local/share/virtualenvs/asset-ar0OxIPP/lib/python3.7/site-packages/gevent/pywsgi.py", line 975, in handle_one_response
    self.run_application()
    File "/root/.local/share/virtualenvs/asset-ar0OxIPP/lib/python3.7/site-packages/geventwebsocket/handler.py", line 82, in run_application
    self.process_result()
    File "/root/.local/share/virtualenvs/asset-ar0OxIPP/lib/python3.7/site-packages/gevent/pywsgi.py", line 909, in process_result
    self.write(data)
    File "/root/.local/share/virtualenvs/asset-ar0OxIPP/lib/python3.7/site-packages/gevent/pywsgi.py", line 756, in write
    itertools
        20
    itertools  
       2018-10-11 16:21:27 +08:00
    hanssx
        21
    hanssx  
    OP
       2018-10-11 16:44:18 +08:00
    清理了 gunicorn 进程,抛弃了 websocket-gevent,使用了 gevent,gunicorn -b 127.0.0.1:5000 -k gevent -w 1 module:app,不再报 402 错误了,但是程序还是会一直 pending,一直发请求。。。
    hanssx
        22
    hanssx  
    OP
       2018-10-11 16:47:12 +08:00
    @itertools 是的,不支持 py3,真是我的锅,第一次部署花了很多时间,使用了 gevent 之后没有 402 错误了,但是会一直 pending,gunicorn 日志显示好像是心跳包,正在找办法关了它。。。
    [2018-10-11 16:40:27 +0800] [12128] [INFO] Shutting down: Master
    [2018-10-11 16:41:47 +0800] [13014] [INFO] Starting gunicorn 19.9.0
    [2018-10-11 16:41:47 +0800] [13014] [INFO] Listening at: http://127.0.0.1:5000 (13014)
    [2018-10-11 16:41:47 +0800] [13014] [INFO] Using worker: gevent
    [2018-10-11 16:41:47 +0800] [13017] [INFO] Booting worker with pid: 13017
    Server initialized for gevent.
    94207eaf37114c118b05480ba12679ef: Sending packet OPEN data {'sid': '94207eaf37114c118b05480ba12679ef', 'upgrades': [], 'pingTimeout': 60000, 'pingInterval': 25000}
    94207eaf37114c118b05480ba12679ef: Sending packet MESSAGE data 0
    94207eaf37114c118b05480ba12679ef: Received packet MESSAGE data 0/process_upload_csv
    94207eaf37114c118b05480ba12679ef: Sending packet MESSAGE data 0/process_upload_csv
    94207eaf37114c118b05480ba12679ef: Received packet PING data None
    94207eaf37114c118b05480ba12679ef: Sending packet PONG data None
    94207eaf37114c118b05480ba12679ef: Received packet PING data None
    94207eaf37114c118b05480ba12679ef: Sending packet PONG data None
    94207eaf37114c118b05480ba12679ef: Received packet PING data None
    hanssx
        23
    hanssx  
    OP
       2018-10-11 16:50:01 +08:00
    hanssx
        24
    hanssx  
    OP
       2018-10-11 16:53:37 +08:00
    @itertools 我 flask 应用程序的启动代码:socketio.run(app=app, host='0.0.0.0', port=5000, debug=True)
    其中 socketio 初始化时,指定了 async_mode='gevent',是这个原因吗?然后 gunicorn 也使用了 gevent。
    hanssx
        25
    hanssx  
    OP
       2018-10-11 17:26:37 +08:00
    @itertools 我晕,我本地不加 gunicorn,socketio.run(app=app, host='0.0.0.0', port=5000, debug=True),指定 async_mode='gevent',也会一直 pending,一直请求。。。
    itertools
        26
    itertools  
       2018-10-11 17:43:29 +08:00   ❤️ 1
    @hanssx 目前最解决问题的办法是使用 gunicorn + eventlet 组合跑,卸载 gevent 安装 eventlet 然后把 async_mode="gevent"改成 async_mode="eventlet " 启动命令为 gunicorn --worker-class eventlet -w 1 module:app。
    如果一定要用 gevent 请尝试使用 uWSGI + gevent
    hanssx
        27
    hanssx  
    OP
       2018-10-11 17:46:27 +08:00
    @itertools 确实,老哥,能加您个 QQ 吗?想感谢您一下,gevent+flask-socketio 就会一直发请求并且 pending,eventlet 则不会,怎么会这样我也不明白,我的扣扣 9614 六 2392,期待老哥,哈哈。
    hanssx
        28
    hanssx  
    OP
       2018-10-11 17:48:46 +08:00
    已解决,感谢各位的帮助,特别是 itertools。
    daya
        29
    daya  
       2018-10-11 17:53:42 +08:00 via iPhone
    怎么解决的?
    hanssx
        30
    hanssx  
    OP
       2018-10-11 17:56:05 +08:00
    @daya 首先清除 gunicorn 多余进程,使用 netstat -antlp | grep port 或者 ps -ef | grep guni,然后 sudo kill -9 pid
    然后再把 gevent-websocket 换成 gevetn,因为前者不支持 py3
    最后把 flask-socketio 的 async_mode 换成 eventlet 而不是 gevent,gevent 会一直发送类似心跳包的东西,不知道有没有选项可以清除。
    关于   ·   帮助文档   ·   博客   ·   API   ·   FAQ   ·   实用小工具   ·   2823 人在线   最高记录 6679   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 88ms · UTC 13:47 · PVG 21:47 · LAX 05:47 · JFK 08:47
    Developed with CodeLauncher
    ♥ Do have faith in what you're doing.