一個linux內核模塊移植到低版本時發生的異常

在3.10的內核版本下，有一個運行穩定的內核模塊，移植到suse11的時候，編譯正常，運行則直接出現crash：

<4>[  503.347297] CPU 0
<4>[  503.347300] Modules linked in: caq_sendmsg(EN) mysendmsg(EN) witdriver(EN) datalink(EN) w83627dhg(EN) tipc(EX) ossmod(EN) pagecachelimit(EN) xfs ip6table_filter ip6_tables iptable_filter ip_tables ebtable_nat ebtables x_tables ipmi_devintf ipmi_si ipmi_msghandler edd cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf af_packet bonding fuse loop dm_mod vhost_net macvtap ipv6 ipv6_lib macvlan tun kvm_intel kvm pcspkr ses enclosure usbhid hid i40e(EX) sg igb i2c_i801 iTCO_wdt iTCO_vendor_support mei dca mptctl ptp pps_core mptbase rtc_cmos acpi_power_meter container button ext3 jbd mbcache ttm drm_kms_helper drm i2c_algo_bit sysimgblt sysfillrect i2c_core syscopyarea ehci_hcd usbcore usb_common sd_mod crc_t10dif processor thermal_sys hwmon scsi_dh_hp_sw scsi_dh_alua scsi_dh_rdac scsi_dh_emc scsi_dh mpt3sas(EX) configfs scsi_transport_sas raid_class scsi_mod
<4>[  503.347404] Supported: No, Unsupported modules are loaded
<4>[  503.347408]
<4>[  503.347413] Pid: 30269, comm: 00-IFileSender Tainted: G           ENX 3.0.101-0.47.90-default #1 ZTE Grantley/S1008
<4>[  503.347422] RIP: 0010:[<ffffffff813e9830>]  [<ffffffff813e9830>] get_page+0x0/0x30
<4>[  503.347434] RSP: 0018:ffff88334a88b4e0  EFLAGS: 00010246
<4>[  503.347438] RAX: ffffffff81a77be0 RBX: ffff88198c904b80 RCX: ffff881f76b3b3f0
<4>[  503.347443] RDX: 0000000000001000 RSI: 0000000000003ebc RDI: 0020000000000000
<4>[  503.347449] RBP: ffff881f76b3b300 R08: 0000000000000000 R09: 0020000000000000
<4>[  503.347453] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
<4>[  503.347458] R13: 0000000000000000 R14: 0000000000000004 R15: 0000000000000000
<4>[  503.347464] FS:  00007f3678a69700(0000) GS:ffff88207fc00000(0000) knlGS:0000000000000000
<4>[  503.347470] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
<4>[  503.347474] CR2: 00007f3659bcceac CR3: 00000034bfc1e000 CR4: 00000000001407f0
<4>[  503.347479] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<4>[  503.347484] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<4>[  503.347490] Process 00-IFileSender (pid: 30269, threadinfo ffff88334a88a000, task ffff88334a888540)
<0>[  503.347495] Stack:
<4>[  503.347498]  ffffffff813eb098 000000004a88b5d8 ffff882000000000 0020000000000000
<4>[  503.347510]  ffff883300000000 000000007ffd9e00 0000000000000000 0000000000000004
<4>[  503.347519]  000000004a888e50 ffffea005c719dd8 000005b4000000d0 ffff88198c904c88
<0>[  503.347528] Call Trace:
<4>[  503.347541]  [<ffffffff813eb098>] do_tcp_sendpages+0x438/0x530
<4>[  503.347556]  [<ffffffffa06ed4cd>] caq_sendmsg_for_file+0x20d/0x480 [caq_sendmsg]
<4>[  503.347583]  [<ffffffffa06ee574>] sendmsg_for_file+0x154/0x220 [caq_sendmsg]
<4>[  503.347594]  [<ffffffffa06ee801>] my_sys_sendmsg+0x1c1/0x208 [caq_sendmsg]
<4>[  503.347608]  [<ffffffff8146f5f2>] system_call_fastpath+0x16/0x1b
<4>[  503.347621]  [<00007f36ad754e4d>] 0x7f36ad754e4c

咱們修改了一個函數：

tcp_sendpage，這個函數會在lock sock的狀況下，再調用do_tcp_sendpage,suse11的源代碼以下：

int tcp_sendpage(struct sock *sk, struct page *page, int offset,
         size_t size, int flags)
{
    ssize_t res;

    if (!(sk->sk_route_caps & NETIF_F_SG) ||
        !(sk->sk_route_caps & NETIF_F_ALL_CSUM))
        return sock_no_sendpage(sk->sk_socket, page, offset, size,
                    flags);

    lock_sock(sk);
    res = do_tcp_sendpages(sk, &page, offset, size, flags);
    release_sock(sk);
    return res;
}

而對應的3.10版本，則是以下：

int tcp_sendpage(struct sock *sk, struct page *page, int offset,
         size_t size, int flags)
{
    ssize_t res;

    if (!(sk->sk_route_caps & NETIF_F_SG) ||
        !(sk->sk_route_caps & NETIF_F_CSUM_MASK))
        return sock_no_sendpage(sk->sk_socket, page, offset, size,
                    flags);

    lock_sock(sk);
    res = do_tcp_sendpages(sk, page, offset, size, flags);
    release_sock(sk);
    return res;
}

一開始從crash排查，發現傳入的page是NULL指針，而後開始在入參以前檢查，發現bug_on沒有觸發。

最後經過merge代碼，發現suse11的do_tcp_sendpages 和cgslv5的 do_tcp_sendpages 的定義不同，第二個參數，高版本的不須要取地址，低版本須要取page的地址。

爲了這麼低級的bug查了幾個小時，哎。

總結：

 不一樣內核版本移植模塊的時候，對於修改的代碼，最好先對比下函數，防止犯低級錯誤。