不懂就问: django 不是一个请求就是一个生命周期吗? 为什么还会有 Too many open files 的错误

[firejoke]

一个从远程获取安装日志, 以文件流的形式返回前端的逻辑
先贴代码:

class LogFile(LoginRequiredMixin, View):
    """获取日志文件
    """

    def get(self, request, *args, **kwargs):
        data = json.loads(request.GET.get('data', '{}'))
        host = data.get('host', None)
        user = data.get('user', None)
        pwd = data.get('password', None)
        if host and user and pwd:
            c = Connection(host, user, connect_kwargs={'password': pwd})
            try:
                logs = c.run(
                        'ls -t ' + DEPLOYMENT_LOG_PATH +
                        ' | grep install', pty=True)
                logs = logs.stdout.split('\n')
                return JsonResponse({'logs': logs})
            except (AuthenticationException, NoValidConnectionsError,
                    UnexpectedExit) as e:
                return JsonResponse({'error': e.__str__()})
        else:
            return HttpResponse(0)

    def post(self, request, *args, **kwargs):
        data = json.loads(request.POST.get('data', '{}'))
        host = data.get('host', None)
        user = data.get('user', None)
        pwd = data.get('password', None)
        seek = data.get('seek', None)
        log_name = data.get('logfile', None)
        log_name = log_name[:-1] if log_name else None
        if host and log_name and (isinstance(seek, int) or seek.isdigit()):
            if not (user and pwd) and (user or pwd):
                return JsonResponse({'error': 'args error'})
            elif user and pwd:
                c = Connection(host, user, connect_kwargs={'password': pwd})
                try:
                    c.get(DEPLOYMENT_LOG_PATH + log_name, '%s/log/%s' % (
                        settings.BASE_DIR, log_name))
                    log_file = '%s/log/%s' % (settings.BASE_DIR, log_name)
                except (IOError, AuthenticationException,
                        NoValidConnectionsError, UnexpectedExit) as e:
                    return JsonResponse({'error': e.__str__()})
            else:
                log_file = '/var/log/' + log_name

            def read_log(log_file):
                with open(log_file, 'rb') as f:
                    f.seek(0, 2)
                    if f.tell() == int(seek):
                        yield '\nseek:' + seek
                    else:
                        f.seek(int(seek))
                        while 1:
                            log = f.read(512)
                            if log:
                                yield log
                            else:
                                break
                        yield '\nseek:' + str(f.tell())
                os.remove(log_file)
            response = StreamingHttpResponse(read_log(log_file))
            return response
        else:
            return JsonResponse({'error': 'args error'})

前端在迭代完之后,会等待 2 秒再发送下一个请求, 在匹配到安装完成的关键字后就不再请求
上周五走的时候,放着测试
21:13:22 开始查看日志
21:53:18 返回的错误信息
错误信息在这一段

c = Connection(host, user, connect_kwargs={'password': pwd})
try:
    c.get(DEPLOYMENT_LOG_PATH + log_name, '%s/log/%s' % (
        settings.BASE_DIR, log_name))
    log_file = '%s/log/%s' % (settings.BASE_DIR, log_name)
except (IOError, AuthenticationException,
        NoValidConnectionsError, UnexpectedExit) as e:
    return JsonResponse({'error': e.__str__()})

查了一下说是达到了系统限制的打开文件句柄数量
但奇怪的是,我的逻辑是每次获取请求后,
拉取文件,读文件,关闭文件,删除文件,
按说,一个请求结束,文件也就没了,打开的文件句柄也就没了呀,
怎么会报这个错呢?
难道说, 要整个 django 进程都关闭了才算?

午休的时候又跑了一遍, 又出这个问题

Traceback (most recent call last):
  File "/usr/lib64/python2.7/wsgiref/handlers.py", line 85, in run
  File "/usr/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 157, in __call__
  File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 124, in get_response
  File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 43, in inner
  File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 93, in response_for_exception
  File "/usr/lib/python2.7/site-packages/django/core/handlers/exception.py", line 143, in handle_uncaught_exception
  File "/usr/lib/python2.7/site-packages/django/utils/decorators.py", line 149, in _wrapped_view
  File "/usr/lib/python2.7/site-packages/django/views/defaults.py", line 68, in server_error
  File "/usr/lib/python2.7/site-packages/django/template/loader.py", line 21, in get_template
  File "/usr/lib/python2.7/site-packages/django/template/backends/django.py", line 39, in get_template
  File "/usr/lib/python2.7/site-packages/django/template/engine.py", line 162, in get_template
  File "/usr/lib/python2.7/site-packages/django/template/engine.py", line 136, in find_template
  File "/usr/lib/python2.7/site-packages/django/template/loaders/cached.py", line 60, in get_template
  File "/usr/lib/python2.7/site-packages/django/template/loaders/base.py", line 38, in get_template
  File "/usr/lib/python2.7/site-packages/django/template/loaders/cached.py", line 28, in get_contents
  File "/usr/lib/python2.7/site-packages/django/template/loaders/filesystem.py", line 28, in get_contents
IOError: [Errno 24] Too many open files: '/root/setupUI/templates/500.html'

[CEBBCAT]

traceback 贴一下？

[firejoke]

@CEBBCAT
贴了, 看起来就是整个 django 进程开了太多文件, 以至于连 html 文件都不能打开,
必须把整个服务重启一遍

[PureWhiteWu]

是不是你哪里资源泄露了？
你代码中有打开文件的地方么？

[firejoke]

@PureWhiteWu
不会, 其他的地方涉及到文件操作的,基本都是用的 with 语法,
而且, 中午再次测的时候, 从头到尾, 就只是访问部署接口, put 了以个压缩包到目标服务器,
然后就访问这个接口看日志

[julyclyde]

出故障的时候别重启，检查一下 /proc/XXXX/fd/ 目录

[firejoke]

@julyclyde 看了, 有 1024 个

[katsusan]

lsof 看一下打开了哪些文件

[firejoke]

@PureWhiteWu
@julyclyde

找到问题了, Fabric 的连接实例没关......
大意了

[firejoke]

@katsusan 找到原因了

[julyclyde]

@firejoke 是指 ssh 那个 Fabric 吗？似乎你上面的代码没提到这个

[firejoke]

@julyclyde 是的, 一开始看到 files error, 完全没往这个方面想, 反复检查的时候才发现我没关连接,
现在我只要能用 with 的都用 with 了

[firejoke]

@julyclyde 奇怪的是, 我发现就算使用了 connection.close(), connection 实例竟然还是可以用, .run 方法也可以用

[firejoke]

@julyclyde
嗯, 看了源码, 每次调用 run 方法, 都被 open 方法装饰过......

[julyclyde]

@firejoke 没太能理解呢，run 和 open 到底啥关系？

[firejoke]

@julyclyde
run 被加了一个装饰器, 该装饰器的作用是在调用之前, 先调用 open 方法
@decorator
def opens(method, self, *args, **kwargs):
self.open()
return method(self, *args, **kwargs)


@opens
def run(self, command, **kwargs):
return self._run(self._remote_runner(), command, **kwargs)