大家有没有分析过红帽系 Linux 内核崩溃日志？

[zhoudaiyu]

日志在 /var/crash/foo/bar_vmcore，是内核崩溃时系统运行时的 snapshot，文章讲的一般都是如何 bar_vmcore.txt ，而讲如何分析这个文件的的非常少，而且听说是比较难的，好像还得懂点 C 和汇编吧。不知道大家有啥经验没？

[liuxu]

https://termbin.com/n303

这是我 ubuntu 的 crash 文件，teamviewerd 崩过，可以用 apport-retrace 分析
正常的话编译的二进制文件如果带有符号文件，得到的输出可以从源码级分析，但是由于 apt 安装的 teamviewerd 没有带符号文件，所以用 gdb 只能输出汇编和内存地址

liuxu@liuxu-Inspiron-7559:/tmp$ apport-retrace -g log.crash
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04) 9.2
...
Type "apropos word" to search for commands related to "word".
Reading symbols from /opt/teamviewer/tv_bin/teamviewerd...
(No debugging symbols found in /opt/teamviewer/tv_bin/teamviewerd)
[New LWP 1649]
[New LWP 1663]
...
warning: Error reading shared library list entry at 0x448b4800081b92e8
warning: Error reading shared library list entry at 0xab868358d48fb89
Core was generated by `/opt/teamviewer/tv_bin/teamviewerd -d'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007ff64976b18b in ?? ()
[Current thread is 1 (LWP 1649)]


如果你是用 gcc 自己编译的程序，可以用-g 选项带上符号文件
liuxu@liuxu-Inspiron-7559:/tmp$ emacs helloworld.c
liuxu@liuxu-Inspiron-7559:/tmp$ cat helloworld.c
#include <stdio.h>

int main()
{
printf("helloworld");
return 0;
}


liuxu@liuxu-Inspiron-7559:/tmp$ gcc -g -o helloworld helloworld.c
liuxu@liuxu-Inspiron-7559:/tmp$ ls -l helloworld
-rwxrwxr-x 1 liuxu liuxu 19160 4 月 17 17:43 helloworld


liuxu@liuxu-Inspiron-7559:/tmp$ readelf -h ./helloworld
ELF Header:
Magic: 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
Class: ELF64
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - System V
ABI Version: 0
Type: DYN (Shared object file)
Machine: Advanced Micro Devices X86-64
Version: 0x1
Entry point address: 0x1060
Start of program headers: 64 (bytes into file)
Start of section headers: 16856 (bytes into file)
Flags: 0x0
Size of this header: 64 (bytes)
Size of program headers: 56 (bytes)
Number of program headers: 13
Size of section headers: 64 (bytes)
Number of section headers: 36
Section header string table index: 35


liuxu@liuxu-Inspiron-7559:/tmp$ gdb ./helloworld
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./helloworld...
(gdb) list
1 #include <stdio.h>
2
3 int main()
4 {
5 printf("helloworld");
6 return 0;
7 }
(gdb) quit


这个时候如果 printf 崩了的话，apport-retrace 通过 gdb 可以直接输出代码了，不过一般也还需要内核的符号文件，https://wiki.ubuntu.com/Debug%20Symbol%20Packages

[feather12315]

1.分析 demesg 的 call trace，能解决大部分问题
2. 其他的 crash 加载 vmlinux 与 vmcore，看执行的指令+源码寻找蛛丝马迹