V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
推荐学习书目
Learn Python the Hard Way
Python Sites
PyPI - Python Package Index
http://diveintopython.org/toc/index.html
Pocoo
值得关注的项目
PyPy
Celery
Jinja2
Read the Docs
gevent
pyenv
virtualenv
Stackless Python
Beautiful Soup
结巴中文分词
Green Unicorn
Sentry
Shovel
Pyflakes
pytest
Python 编程
pep8 Checker
Styles
PEP 8
Google Python Style Guide
Code Style from The Hitchhiker's Guide
wsgzao
V2EX  ›  Python

使用 pypiserver 快速搭建内网离线 pypi 仓库实践

  •  
  •   wsgzao ·
    wsgzao · 2018-04-13 15:47:33 +08:00 · 4803 次点击
    这是一个创建于 2416 天前的主题,其中的信息可能已经有所发展或是发生改变。

    前言

    本文介绍了如何快速搭建一个 pypiserver,通过自建 pypiserver,我们可以解决网络环境不好,或者离线无法安装 python 包的问题。如果结合最新的 GitLab CI/CD 和 pipenv 我相信各位还可以玩出更多的花样。

    pypiserver - minimal PyPI server for use with pip/easy_install

    更新记录

    2018 年 04 月 12 日 - 初稿

    阅读原文 - https://wsgzao.github.io/post/pypiserver/

    扩展阅读

    pypiserver - https://github.com/pypiserver/pypiserver

    pypiserver 简介

    pypiserver is a minimal PyPI compatible server for pip or easy_install. It is based on bottle and serves packages from regular directories. Wheels, bdists, eggs and accompanying PGP-signatures can be uploaded either with pip, setuptools, twine, pypi-uploader, or simply copied with scp.

    pypiserver 服务端配置

    如果你的 Linux 环境缺少 Python 2.7 可以参考我的文章直接离线升级至最新版本

    Python 2.6 升级至 Python 2.7 的实践心得 https://wsgzao.github.io/post/python-2-6-update-to-2-7/

    pypiserver > 1.2.x works with python 2.7 and 3.3+ or pypy. Older python-versions may still work, but they are not tested. For legacy python versions, use pypiserver-1.1.x series.

    # 替换 pip 为阿里云,感概豆瓣的时代已经过去
    tee ~/.pip/pip.conf <<-'EOF'
    [global]
    index-url = https://mirrors.aliyun.com/pypi/simple/
    [install]
    trusted-host= mirrors.aliyun.com
    EOF
    
    # 直接在线安装 pypiserver
    pip install pypiserver
    # 离线下载 pypiserver
    mkdir /tmp/pypiserver
    pip install -d /tmp/pypiserver/ pypiserver
    
    # Copy packages into this directory.
    mkdir ~/packages
    # Copy some packages into your ~/packages folder and then get your pypiserver up and running:
    pypi-server -p 8080 ~/packages &
    
    

    pypiserver 客户端配置

    ## Download and Install hosted packages.
    pip install  --extra-index-url http://localhost:8080/simple/ ...
    
    # or
    pip install --extra-index-url http://localhost:8080
    
    ## Search hosted packages
    pip search --index http://localhost:8080 ...
    
    # 个人推荐的配置
    tee ~/.pip/pip.conf <<-'EOF'
    [global]
    index-url = http://172.28.70.126/simple
    extra-index-url = https://mirrors.aliyun.com/pypi/simple/
    [install]
    trusted-host = 172.28.70.126
    EOF
    
    

    pypiserver 进阶配置

    pypiserver Running as a systemd service

    https://github.com/pypiserver/pypiserver#running-as-a-systemd-service

    Adjusting the paths and adding this file as pypiserver.service into your systemd/system directory will allow management of the pypiserver process with systemctl, e.g. systemctl start pypiserver.

    More useful information about systemd can be found at https://www.digitalocean.com/community/tutorials/how-to-use-systemctl-to-manage-systemd-services-and-units

    # 安装需要的包
    yum install nginx -y
    pip install passlib pypiserver gunicorn
    
    # 创建 pypiserver 服务方便服务启停管理
    tee /usr/lib/systemd/system/pypiserver.service <<-'EOF'
    [Unit]
    Description=gunicorn daemon
    After=network.target
    
    [Service]
    PIDFile=/run/pypiserver.pid
    ExecStart=/usr/local/bin/gunicorn -w16 \
        --pid /run/pypiserver.pid \
        -b :10012 \
        'pypiserver:app(root="/var/www/pypi")'
    ExecReload=/bin/kill -s HUP $MAINPID
    ExecStop=/bin/kill -s TERM $MAINPID
    
    [Install]
    WantedBy=multi-user.target 
    EOF
    
    # Warning: pypiserver.service changed on disk. Run 'systemctl daemon-reload' to reload units.
    systemctl daemon-reload
    
    # 启动 pypiserver 服务
    systemctl enable pypiserver.service
    systemctl start pypiserver.service
    systemctl status pypiserver.service
    
    # 停止 pypiserver 服务
    systemctl disable pypiserver.service
    systemctl stop pypiserver.service
    systemctl status pypiserver.service
    
    [root@centos7 run]# systemctl status pypiserver.service
    ● pypiserver.service - gunicorn daemon
       Loaded: loaded (/usr/lib/systemd/system/pypiserver.service; disabled; vendor preset: disabled)
       Active: active (running) since Fri 2018-04-13 15:14:08 CST; 859ms ago
     Main PID: 22524 (gunicorn)
       CGroup: /system.slice/pypiserver.service
               ├─22524 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
               ├─22530 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
               ├─22531 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
               ├─22532 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
               ├─22533 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
               ├─22534 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
               ├─22535 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
               ├─22536 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
               ├─22537 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
               ├─22538 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
               ├─22539 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
               └─22540 /usr/local/bin/python /usr/local/bin/gunicorn -w16 --pid /run/pypiserver.pid -b :10012 pypiserver:app(root="/var/www/pypi")
    
    Apr 13 15:14:08 centos7 gunicorn[22524]: [2018-04-13 15:14:08 +0000] [22531] [INFO] Booting worker with pid: 22531
    Apr 13 15:14:08 centos7 gunicorn[22524]: [2018-04-13 15:14:08 +0000] [22532] [INFO] Booting worker with pid: 22532
    Apr 13 15:14:08 centos7 gunicorn[22524]: [2018-04-13 15:14:08 +0000] [22533] [INFO] Booting worker with pid: 22533
    Apr 13 15:14:08 centos7 gunicorn[22524]: [2018-04-13 15:14:08 +0000] [22534] [INFO] Booting worker with pid: 22534
    Apr 13 15:14:08 centos7 gunicorn[22524]: [2018-04-13 15:14:08 +0000] [22535] [INFO] Booting worker with pid: 22535
    Apr 13 15:14:08 centos7 gunicorn[22524]: [2018-04-13 15:14:08 +0000] [22536] [INFO] Booting worker with pid: 22536
    Apr 13 15:14:08 centos7 gunicorn[22524]: [2018-04-13 15:14:08 +0000] [22537] [INFO] Booting worker with pid: 22537
    Apr 13 15:14:08 centos7 gunicorn[22524]: [2018-04-13 15:14:08 +0000] [22538] [INFO] Booting worker with pid: 22538
    Apr 13 15:14:08 centos7 gunicorn[22524]: [2018-04-13 15:14:08 +0000] [22539] [INFO] Booting worker with pid: 22539
    Apr 13 15:14:08 centos7 gunicorn[22524]: [2018-04-13 15:14:08 +0000] [22540] [INFO] Booting worker with pid: 22540
    [root@centos7 run]# netstat -tnlp
    Active Internet connections (only servers)
    Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
    tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      1/systemd           
    tcp        0      0 192.168.122.1:53        0.0.0.0:*               LISTEN      1517/dnsmasq        
    tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      977/sshd            
    tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      978/cupsd           
    tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1383/master         
    tcp        0      0 127.0.0.1:6011          0.0.0.0:*               LISTEN      19378/sshd: root@pt 
    tcp        0      0 0.0.0.0:10012           0.0.0.0:*               LISTEN      22524/python        
    tcp6       0      0 :::111                  :::*                    LISTEN      1/systemd           
    tcp6       0      0 :::22                   :::*                    LISTEN      977/sshd            
    tcp6       0      0 ::1:631                 :::*                    LISTEN      978/cupsd           
    tcp6       0      0 ::1:25                  :::*                    LISTEN      1383/master         
    tcp6       0      0 ::1:6011                :::*                    LISTEN      19378/sshd: root@pt 
    
    # 检查 pypiserver 服务
    cd /var/www/pypi
    # 向仓库中添加 python package
    [root@centos7 pypi]# pip download pypiserver
    Collecting pypiserver
      Downloading https://mirrors.aliyun.com/pypi/packages/d7/78/5772432dad2b9e754ab92f4d301fa507069b9decc8c943c1b18c2043ff4f/pypiserver-1.2.1-py2.py3-none-any.whl (83kB)
        100% |████████████████████████████████| 92kB 643kB/s 
      Saved ./pypiserver-1.2.1-py2.py3-none-any.whl
    Successfully downloaded pypiserver
    
    [root@centos7 pypi]# ll
    total 84
    -rw-r--r-- 1 root root 83529 Apr 13 14:55 pypiserver-1.2.1-py2.py3-none-any.whl
    
    # 搜索刚才下载的 package
    [root@centos7 pypi]# pip search -i http://127.0.0.1:10012 pypiserver
    pypiserver (1.2.1)  - 1.2.1
      INSTALLED: 1.2.1 (latest)
    
    # 配置 nginx 做反向代理
    tee /etc/nginx/conf.d/pypi.conf <<-'EOF'
    upstream pypiserver {
      server 127.0.0.1:10012;
    }
    
    server {
      listen 10087;
    
      # disable any limits to avoid HTTP 413 for large package uploads
      client_max_body_size 0;
    
      location / {
        proxy_pass http://pypiserver/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    
        # When setting up pypiserver behind other proxy, such as an Nginx instance, remove the below line if the proxy already has similar settings.
        proxy_set_header X-Forwarded-Proto $scheme;
    
        proxy_buffering off;
        proxy_request_buffering off;
      }
    
      location /packages/ {
        alias /var/www/pypi; # static file
      }
    }
    EOF
    
    # 启动 nginx
    systemctl enable nginx
    systemctl start nginx
    systemctl status nginx
    
    # 检查 nginx 服务
    pip search -i http://172.28.79.126:10087 pypiserver
    pypiserver (1.2.1)  - 1.2.1
      INSTALLED: 1.2.1 (latest)
    
    
    11 条回复    2018-04-18 11:09:56 +08:00
    lfzyx
        1
    lfzyx  
       2018-04-13 15:55:14 +08:00
    2018 年 04 月 12 日
    Python 2.7
    wellsc
        2
    wellsc  
       2018-04-13 15:57:09 +08:00 via iPhone
    Docker container 也很好用
    z550665887
        3
    z550665887  
       2018-04-13 18:12:02 +08:00
    我记得当初测试 pypiserver 对于本地没有缓存的 pip 包的 pip install 请求会返回一个官网(默认)的源下载链接。对于无法访问公网的服务器来说这个 pip install 请求不会生效。。。
    后来采用 docker 启的 devpi 服务完美解决的。
    julyclyde
        4
    julyclyde  
       2018-04-13 20:06:47 +08:00
    2.6 升级到 2.7 居然还能有心得??
    python 的安装做的有那么差吗?
    python 还会完善 yum 的设置?越俎代庖还是你没搞清楚这俩东西没关联?

    为什么那么爱好“离线包”、“上传到网盘”?
    你制作的离线包,你会继续投入精力去持续更新吗?已经放出去的旧版本链接,你能召回吗?
    wsgzao
        5
    wsgzao  
    OP
       2018-04-13 21:03:59 +08:00
    @julyclyde #4 Blog 本身记录的是一种方法和解决问题的思路,很少有人可以持续更新技术类笔记,只是作为一个参考,我本来也觉得不是很复杂的东西但有时候就是会采坑,所以记录一点简单实用的,很复杂的技术自己放在笔记里和公司内部团队共享就好了
    wsgzao
        6
    wsgzao  
    OP
       2018-04-13 21:06:53 +08:00
    @z550665887 #3 pip search --index http://localhost:8080,我们内部用 GitLab+Jenkins 做团队开发的时候是通过 fabfile 写在脚本里执行的,等后续研究透彻 GitLab CI/CD + pipenv 的新方法后再做分享
    julyclyde
        7
    julyclyde  
       2018-04-13 21:52:40 +08:00   ❤️ 1
    @wsgzao 问题是,你的“因为 Python 2.7.13 以后版本会自动完善 yum 配置,所以不必参考以前的网上文章去修改其他地方”既没有正确的说明(你以为的) python 和 yum 的关系,也没说“以前网上文章”哪些内容是不再需要的,你这样的写法并不能起到记录的作用
    wsgzao
        8
    wsgzao  
    OP
       2018-04-15 12:40:54 +08:00
    @julyclyde #7 Sorry,我点我确实没有做好,因为在 Python 官方安装的方法中没有看到有 FQ 提示,而 Google 搜索到的大部分信息都是通过修改软连接修复,将#!/usr/bin/python 修改为 #!/usr/bin/python2.6,我也没想太多,就以实际情况为主,没有去思考深沉次的原因
    julyclyde
        9
    julyclyde  
       2018-04-15 22:59:45 +08:00
    @wsgzao 现在,新装的 2.7 可以在以前 2.6 的 site-packages 目录里成功找到 yum 的源码了吗?
    wsgzao
        10
    wsgzao  
    OP
       2018-04-16 08:50:42 +08:00
    @julyclyde #9 从路径上来看应该是独立的,官方倒是也没有说明优化了哪些安装的方式,我实际测试从 python2.6 升级至 python2.7.13 没有问题,然后最近又升级到 python2.7.14 也是直接编译安装,相当顺利,后续估计要开始测试 python2.7 和 python3.6 使用 pipenv 共存的问题,估计又要折腾好久了

    ``` bash
    test101@JQ/root#whereis python
    python: /usr/bin/python2.6 /usr/bin/python /usr/lib/python2.6 /usr/lib64/python2.6 /usr/local/bin/python2.7 /usr/local/bin/python2.7-config /usr/local/bin/python /usr/local/lib/python2.7 /usr/include/python2.6 /usr/share/man/man1/python.1.gz
    test101@JQ/root#cat /usr/bin/yum
    #!/usr/bin/python

    test101@JQ/root#find / -name python
    /usr/share/doc/m2crypto-0.20.2/demo/ZopeX3/install_dir/lib/python
    /usr/share/doc/m2crypto-0.20.2/demo/Zope27/install_dir/lib/python
    /usr/share/doc/m2crypto-0.20.2/demo/Zope/lib/python
    /usr/share/gdb/python
    /usr/local/bin/python
    /usr/bin/python
    /root/.local/share/virtualenvs/pipenv-qb5OO9ES/bin/python
    /root/Python-2.7.14-pip/Python-2.7.14/python
    test101@JQ/root#find / -name site-packages
    /usr/lib/python2.6/site-packages
    /usr/local/lib/python2.7/site-packages
    /usr/lib64/python2.6/site-packages
    /root/.local/share/virtualenvs/pipenv-qb5OO9ES/lib/python2.7/site-packages
    /root/Python-2.7.14-pip/Python-2.7.14/Lib/site-packages

    ```
    julyclyde
        11
    julyclyde  
       2018-04-18 11:09:56 +08:00
    @wsgzao 所以你的文章是既没有提到原有的问题,也没有论证新版没有这个问题,就直接武断的下结论说没问题?然后还要作为经验推广?
    从你用 whereis 和 find 的做法来看,我猜你大概不知道 python 的 import 是从什么路径里找文件的?
    关于   ·   帮助文档   ·   博客   ·   API   ·   FAQ   ·   实用小工具   ·   1413 人在线   最高记录 6679   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 24ms · UTC 17:29 · PVG 01:29 · LAX 09:29 · JFK 12:29
    Developed with CodeLauncher
    ♥ Do have faith in what you're doing.