分布式环境下使用 MongoDB 作为数据库异常的卡

[Emptycyx]

这个问题困扰了我许久，我在写一个分布式项目时，使用 MongoDB 作为数据库，单独架设在一台 4H8G 的服务器上，并供另外两台 worker 服务器作数据存储使用，然而不知道是什么原因，这种情况下我的 MongoDB 服务器异常卡顿，经常会在执行一些诸如 find_one 、insert_one 方法时 timeout 。实在是不清楚什么原因了，望解答。

下面是 MongoDB 操作的 Util ，不太清除是不是 Util 部分出了问题...

from pymongo import MongoClient


class mongo_util:
    def __init__(self, host, port, username, password, db_name):
        try:
            self.client = MongoClient(host=host, port=port, serverSelectionTimeoutMS=10000, socketTimeoutMS=10000)
            if username != '' and password != '':
                self.client['admin'].authenticate(name=username, password=password)
            self.db = self.client[db_name]
        except Exception as e:
            print('[-] 连接 MongoDB 服务器失败：{}'.format(e))
            exit(0)

    def find(self, col_name, conditions, keys=None):
        i = 0
        results = None
        while True:
            try:
                col = self.db[col_name]
                results = col.find(conditions)
                break
            except Exception as e:
                i += 1
                print(f'[-] Fail to find {conditions} --> {e} , current retry time is {i}.')
                continue

        if keys is None:
            return results
        else:
            new_results = []
            for result in results:
                if len(keys) == 1:
                    if keys[0] not in result.keys(): break
                    new_results.append(result[keys[0]])
                else:
                    temp = []
                    for key in keys:
                        if key in result.keys():
                            temp.append(result[key])
                    new_results.append(temp)
            return new_results

    def find_one(self, col_name, conditions, keys=None):
        i = 0
        result = None
        while True:
            try:
                col = self.db[col_name]
                result = col.find_one(conditions)
                break
            except Exception as e:
                i += 1
                print(f'[-] Fail to find_one {conditions} --> {e} , current retry time is {i}.')
                continue

        if not result: return
        if keys is None:
            return result
        elif len(keys) == 1 and keys[0] in result.keys():
            return result[keys[0]]
        else:
            temp = {}
            for key in keys:
                if key in result.keys():
                    temp[key] = result[key]
            return result

    def insert(self, col_name, data_list):
        i = 0
        while True:
            try:
                col = self.db[col_name]
                return col.insert_many(data_list, bypass_document_validation=True).inserted_ids
            except Exception as e:
                if 'E11000 duplicate key' in str(e):
                    print(f'[-] Fail to insert_one {data_list} --> {e}')
                    return True
                i += 1
                print(f'[-] Fail to insert {data_list} --> {e} , current retry time is {i}.')
                continue
        return False

    def insert_one(self, col_name, data):
        i = 0
        while True:
            try:
                col = self.db[col_name]
                return col.insert_one(data, bypass_document_validation=True).inserted_id
            except Exception as e:
                if 'E11000 duplicate key' in str(e):
                    print(f'[-] Fail to insert_one {data} --> {e}')
                    return True
                i += 1
                print(f'[-] Fail to insert_one {data} --> {e} , current retry time is {i}.')
                continue
        return False

    def update(self, col_name, conditions, update_data):
        i = 0
        while True:
            try:
                col = self.db[col_name]
                return col.update_many(conditions, {'$set': update_data}).modified_count
            except Exception as e:
                i += 1
                print(f'[-] Fail to update {conditions} --> {e} , current retry time is {i}.')
                continue
        return False

    def origin_update(self, col_name, conditions, update_data):
        i = 0
        while True:
            try:
                col = self.db[col_name]
                return col.update_many(conditions, update_data).modified_count
            except Exception as e:
                i += 1
                print(f'[-] Fail to origin_update {conditions} --> {e} , current retry time is {i}.')
                continue
        return False

    def delete(self, col_name, conditions):
        i = 0
        while True:
            try:
                col = self.db[col_name]
                return col.delete_many(conditions).deleted_count
            except Exception as e:
                i += 1
                print(f'[-] Fail to delete {conditions} --> {e} , current retry time is {i}.')
                continue

    def count(self, col_name, conditions):
        i = 0
        while True:
            try:
                col = self.db[col_name]
                return col.count(conditions)
            except Exception as e:
                i += 1
                print(f'[-] Fail to count {conditions} --> {e} , current retry time is {i}.')
                continue
        return False

    def __del__(self):
        self.client.close()


if __name__ == '__main__':
    from configs.db_config import mongo_host
    from configs.db_config import mongo_port
    from configs.db_config import mongo_user
    from configs.db_config import mongo_pass
    from configs.db_config import mongo_db_name

    mongo_instance = mongo_util(host=mongo_host, port=mongo_port, username=mongo_user, password=mongo_pass,
                                db_name=mongo_db_name)

[liprais]

秘诀就是没事别用 mongodb....

[hunk]

@liprais 类似 json 数据的存储用啥？ mysql 可以，但确实不如 mongodb 粗暴

[qW7bo2FbzbC0]

1.看起来像是英文机翻出来的中文
2.无法理解 while true 的使用方式

[hotcool100]

MongoDB 支持集群

[hidemyself]

@hunk postgresql

[liprais]

@hunk 用你会用的

[starsky007]

@liprais 我最近的一个项目就用的 MongoDB ，语言也是 Python ，数据量的话，有几个表近百万，不算大，并发也不高，没遇见啥问题。

[Emptycyx]

@hjahgdthab750 while true 的原因就是因为某些时候会 timeout 所以不断重试，本人英文较差，不过是自己写的而非机翻哈哈

@hotcool100 个人小项目啊，这用得上集群吗，就是比较卡，不太清除到底什么原因导致的，难道 4H8G 做服务器真的支撑不了我一个人使用的分布式小项目嘛

[dream4ever]

搜索一下性能方面的资料，排查一下是哪个环节出了问题。我之前解决过同事弄出来的 CPU 100% 的问题，最后发现是 MySQL 所有表都没加索引，加了索引之后立刻解决问题。