Mitogen for Ansible 性能优化插件推荐，有效提升 1.25x - 7x 倍执行效率

[wsgzao]

众所周知，Ansible 是基于 ssh(当然还有 telnet，winrm 等连接插件)的自动化配置管理工具，其简单易用，无 agent 式的工作方式在很多场景中都有不少优势，不过也是由于这种工作方式导致了它没有其他 c/s 类的工具执行效率高，饱受其他 C/S 类工具使用者的讥讽，对此，Ansible 官方也对 Ansible 的速度效率做了不少优化手段：

| 参数名 /优化类别 | 说明 |

---|---

| fact cache | 将 facts 信息第一次收集后缓存到memory或者redis或者文件中。 |

| gather_subset | 可选择性的收集network,hardware等信息，而不是全部 |

| control_path | 开启ssh socket持久化，复用 ssh 连接 |

| pipelinling | 开启ssh pipelining,客户端从管道中读取执行渲染后的脚本，而不是在客户端创建临时文件 |

| fork | 提高并行执行主机的数量 |

| serial | 将play_hosts``①中主机再分批执行 |

| strategy | 默认linear,每个主机的单个 task 执行完成会等待其他都完成后再执行下个任务，设置free可不等待其他主机，继续往下执行(看起来会比较乱)，还有一个选项host_pinned，我也不知道干嘛的 |

Overview

无意发现了一个 Mitogen 的 Ansible plugin （ strategy plugin ），当前已迭代到 0.29 版本，看介绍说能提升 1.2x ~ 7x 以上的执行效率，着实惊人！

它通过高效的远程过程调用来取代 ansible 默认的嵌入式与纯 python shell 调用，它不会优化模块本身的执行效率，只会尽可能快的②去执行模块获取返回(执行模块前也是有一系列连接，发送数据，传输渲染脚本等操作的)来提高整体的效率，特性如下

Expect a 1.25x - 7x speedup and a CPU usage reduction of at least 2x, depending on network conditions, modules executed, and time already spent by targets on useful work. Mitogen cannot improve a module once it is executing, it can only ensure the module executes as quickly as possible.

• One connection is used per target, in addition to one sudo invocation per user account. This is much better than SSH multiplexing combined with pipelining, as significant state can be maintained in RAM between steps, and system logs aren’t spammed with repeat authentication events.

• A single network roundtrip is used to execute a step whose code already exists in RAM on the target. Eliminating multiplexed SSH channel creation saves 4 ms runtime per 1 ms of network latency for every playbook step.

• Processes are aggressively reused, avoiding the cost of invoking Python and recompiling imports, saving 300-800 ms for every playbook step.

• Code is ephemerally cached in RAM, reducing bandwidth usage by an order of magnitude compared to SSH pipelining, with around 5x fewer frames traversing the network in a typical run.

• Fewer writes to the target filesystem occur. In typical configurations, Ansible repeatedly rewrites and extracts ZIP files to multiple temporary directories on the target. Security issues relating to temporary files in cross-account scenarios are entirely avoided.

The effect is most potent on playbooks that execute many short-lived actions, where Ansible’s overhead dominates the cost of the operation, for example when executing large with_items loops to run simple commands or write files.

大体就是执行过程中主机使用一个连接(默认每执行一个task或者loop循环都会重新打开一次连接的)；渲染的执行代码暂存于内存中；减少多路复用ssh隧道的时间消耗；减少临时文件传输的带宽；代码重用，避免代码的重新编译成本等

实现原理的话，可以去看下官网解释

① . play_hosts为内置参数，指当前正在执行的 playbook 中的主机列表

②. 尽可能快的 指到运行模块前的阶段

Installation

1. Download and extract mitogen-0.2.9.tar.gz.

2. Modify ansible.cfg

[defaults]
strategy_plugins = /path/to/mitogen-0.2.9/ansible_mitogen/plugins/strategy
strategy = mitogen_linear

The strategy key is optional. If omitted, the ANSIBLE_STRATEGY=mitogen_linear environment variable can be set on a per-run basis. Like mitogen_linear, the mitogen_free and mitogen_host_pinned strategies exists to mimic the free and host_pinned strategies.

Reference

https://networkgenomics.com/ansible/

https://mitogen.networkgenomics.com/ansible_detailed.html#overview