OSX master crash in cancel_call

[zevweiss]

(Report received via email, opening an issue here for tracking/note-taking purposes...)

An OSX (10.10.5) master crashed with the following (abridged) crash dump:

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       EXC_I386_GPFLT

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   com.apple.CoreFoundation            0x00007fff95b5f702 CFRunLoopContainsTimer + 50
1   enthrall                            0x0000000101dcce3d cancel_call + 45
2   enthrall                            0x0000000101dc9780 focus_node + 112
3   enthrall                            0x0000000101dc9df5 action_cb + 565
4   enthrall                            0x0000000101dcd48a do_hotkey + 90
5   enthrall                            0x0000000101dcd047 evtap_callback + 103
6   com.apple.CoreGraphics              0x00007fff97ad04a0 processEventTapData + 714
7   com.apple.CoreGraphics              0x00007fff97ad00b3 _CGYPostEventTapData + 221
8   com.apple.CoreGraphics              0x00007fff97acb040 _XPostEventTapData + 89
9   com.apple.CoreGraphics              0x00007fff97acb129 CGYEventTap_server + 86
10  com.apple.CoreGraphics              0x00007fff97ad019b eventTapMessageHandler + 46
11  com.apple.CoreFoundation            0x00007fff95ab6d87 __CFMachPortPerform + 247
12  com.apple.CoreFoundation            0x00007fff95ab6c79 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE1_PERFORM_FUNCTION__ + 41
13  com.apple.CoreFoundation            0x00007fff95ab6beb __CFRunLoopDoSource1 + 475
14  com.apple.CoreFoundation            0x00007fff95aa8767 __CFRunLoopRun + 2375
15  com.apple.CoreFoundation            0x00007fff95aa7bd8 CFRunLoopRunSpecific + 296
16  com.apple.CoreFoundation            0x00007fff95b5f671 CFRunLoopRun + 97
17  enthrall                            0x0000000101dccf63 run_event_loop + 227
18  enthrall                            0x0000000101dc7e5b main + 2235
19  libdyld.dylib                       0x00007fff8b9065c9 start + 1

Thread 1:: Dispatch queue: com.apple.libdispatch-manager
0   libsystem_kernel.dylib              0x00007fff95df0232 kevent64 + 10
1   libdispatch.dylib                   0x00007fff91cb926e _dispatch_mgr_thread + 52

Thread 2:
0   libsystem_kernel.dylib              0x00007fff95def94a __workq_kernreturn + 10
1   libsystem_pthread.dylib             0x00007fff89b683dd start_wqthread + 13

Thread 3:
0   libsystem_kernel.dylib              0x00007fff95def94a __workq_kernreturn + 10
1   libsystem_pthread.dylib             0x00007fff89b683dd start_wqthread + 13

Thread 4:
0   libsystem_kernel.dylib              0x00007fff95def94a __workq_kernreturn + 10
1   libsystem_pthread.dylib             0x00007fff89b683dd start_wqthread + 13

Thread 0 crashed with X86 Thread State (64-bit):
  rax: 0x00007fff7a8c8575  rbx: 0x00007fc7d8700cf0  rcx: 0x0000000000000000  rdx: 0x00007fff7a898060
  rdi: 0x00007fc7d8700cf0  rsi: 0x200007fc7d86029e  rbp: 0x00007fff5de2eed0  rsp: 0x00007fff5de2eea0
   r8: 0x00007fff5de2ecb0   r9: 0x00007fff78961300  r10: 0x000000000000002b  r11: 0x0000000000000246
  r12: 0x00007fff7a898060  r13: 0x00007fc7da9000c0  r14: 0x00007fff7a898060  r15: 0x200007fc7d86029e
  rip: 0x00007fff95b5f702  rfl: 0x0000000000010246  cr2: 0x00007fff7890507c

This occurred on attempting to focus a remote, shortly after opening the clamshell.

[zevweiss]

Disassembly of CFRunLoopContainsTimer up to the crash point:

_CFRunLoopContainsTimer:
    +0 00000000001296d0 pushq   %rbp
    +1 00000000001296d1 movq    %rsp, %rbp
    +4 00000000001296d4 pushq   %r15
    +6 00000000001296d6 pushq   %r14
    +8 00000000001296d8 pushq   %r13
   +10 00000000001296da pushq   %r12
   +12 00000000001296dc pushq   %rbx
   +13 00000000001296dd pushq   %rax
   +14 00000000001296de movq    %rdx, %r12
   +17 00000000001296e1 movq    %rsi, %r15
   +20 00000000001296e4 movq    %rdi, %rbx
   +23 00000000001296e7 leaq    ___CF120290(%rip), %rax
   +30 00000000001296ee movb    $0x1, (%rax)
   +33 00000000001296f1 leaq    ___CF120293(%rip), %rax
   +40 00000000001296f8 cmpb    $0x0, (%rax)
   +43 00000000001296fb je      0x129702
   +45 00000000001296fd callq   ___THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__
   +50 0000000000129702 movq    0x58(%r15), %rax

%r15 being 0x200007fc7d86029e in the crash dump is clearly bogus; it looks to have come from %rsi -- the second argument to CFRunLoopContainsTimer, or ti->timer in osx.c's cancel_call.

Best guess based on that: ti already got freed somewhere, and this is a use-after-free...but where/how did the free occur?

[zevweiss]

Another instance:

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       EXC_I386_GPFLT

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   com.apple.CoreFoundation            0x00007fff95b5f702 CFRunLoopContainsTimer + 50
1   enthrall                            0x0000000109e5ce3d cancel_call + 45
2   enthrall                            0x0000000109e59780 focus_node + 112
3   enthrall                            0x0000000109e59df5 action_cb + 565
4   enthrall                            0x0000000109e5d48a do_hotkey + 90
5   enthrall                            0x0000000109e5d047 evtap_callback + 103
6   com.apple.CoreGraphics              0x00007fff97ad04a0 processEventTapData + 714
7   com.apple.CoreGraphics              0x00007fff97ad00b3 _CGYPostEventTapData + 221
8   com.apple.CoreGraphics              0x00007fff97acb040 _XPostEventTapData + 89
9   com.apple.CoreGraphics              0x00007fff97acb129 CGYEventTap_server + 86
10  com.apple.CoreGraphics              0x00007fff97ad019b eventTapMessageHandler + 46
11  com.apple.CoreFoundation            0x00007fff95ab6d87 __CFMachPortPerform + 247
12  com.apple.CoreFoundation            0x00007fff95ab6c79 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE1_PERFORM_FUNCTION__ + 41
13  com.apple.CoreFoundation            0x00007fff95ab6beb __CFRunLoopDoSource1 + 475
14  com.apple.CoreFoundation            0x00007fff95aa8767 __CFRunLoopRun + 2375
15  com.apple.CoreFoundation            0x00007fff95aa7bd8 CFRunLoopRunSpecific + 296
16  com.apple.CoreFoundation            0x00007fff95b5f671 CFRunLoopRun + 97
17  enthrall                            0x0000000109e5cf63 run_event_loop + 227
18  enthrall                            0x0000000109e57e5b main + 2235
19  libdyld.dylib                       0x00007fff8b9065c9 start + 1

Thread 1:: Dispatch queue: com.apple.libdispatch-manager
0   libsystem_kernel.dylib              0x00007fff95df0232 kevent64 + 10
1   libdispatch.dylib                   0x00007fff91cb926e _dispatch_mgr_thread + 52

Thread 2:
0   libsystem_kernel.dylib              0x00007fff95def94a __workq_kernreturn + 10
1   libsystem_pthread.dylib             0x00007fff89b683dd start_wqthread + 13

Thread 3:
0   libsystem_kernel.dylib              0x00007fff95def94a __workq_kernreturn + 10
1   libsystem_pthread.dylib             0x00007fff89b683dd start_wqthread + 13

Thread 0 crashed with X86 Thread State (64-bit):
  rax: 0x00007fff7a8c8575  rbx: 0x00007fbf55a00670  rcx: 0x0000000000000000  rdx: 0x00007fff7a898060
  rdi: 0x00007fbf55a00670  rsi: 0x5000000000000000  rbp: 0x00007fff55d9eed0  rsp: 0x00007fff55d9eea0
   r8: 0x00007fff55d9ecb0   r9: 0x00007fff78961300  r10: 0x000000000000002b  r11: 0x0000000000000246
  r12: 0x00007fff7a898060  r13: 0x00007fbf53600260  r14: 0x00007fff7a898060  r15: 0x5000000000000000
  rip: 0x00007fff95b5f702  rfl: 0x0000000000010246  cr2: 0x00007fff7890507c

Same stack trace, different register contents; use-after-free still seems likely, though the cause is no more obvious...