This Question is Possibly Answered

1 "correct" answer available (5 pts) 2 "helpful" answers available (1 pts)
41 Replies Last post: Jan 20, 2010 9:11 PM by joegen   1 2 3 Previous Next
Click to view mparadis's profile   89 posts since
Oct 2, 2009

Dec 26, 2009 5:14 PM

Crashes out of the blue


Everything has been running fine for some time now. I have two identical servers, one is running ok, the other is now crashing but not sure why.

I do a ps after restarting and it is running, the prompt comes back, then it crashes.


root 5374 21.0 0.1 32892 4188 pts/0 Sl 16:08 0:23 /usr/local/bin/opensbc -d -p /var/run/opensbc.pid -H 65536 -C 1024000
root 5479 0.0 0.0 2304 916 pts/0 R+ 16:10 0:00 ps -aux
sbc5:~# RTNETLINK answers: Invalid argument

      • glibc detected *** /usr/local/bin/opensbc: free(): invalid pointer: 0xb7b3287c ***
======= Backtrace: =========
/lib/libc.so.6[0xb7db9845]
/lib/libc.so.6(cfree+0x9c)[0xb7dbb6ec]
/usr/local/bin/opensbc[0x81fb314]
/usr/local/bin/opensbc[0x805cc26]
/usr/local/bin/opensbc[0x81aaa99]
/usr/local/bin/opensbc[0x82390ad]
/usr/local/bin/opensbc[0x80b0b71]
/usr/local/bin/opensbc[0x80a9083]
/usr/local/bin/opensbc[0x822029d]
/usr/local/bin/opensbc[0x82204e6]
/usr/local/bin/opensbc[0x819cf56]
/usr/local/bin/opensbc[0x81a43ae]
/usr/local/bin/opensbc[0x82d73d2]
/lib/libpthread.so.0[0xb7ff1f3b]
/lib/libc.so.6(clone+0x5e)[0xb7e19bee]
======= Memory map: ========
08048000-0845e000 r-xp 00000000 68:01 1645593 /usr/local/bin/opensbc
0845e000-0846e000 rw-p 00415000 68:01 1645593 /usr/local/bin/opensbc
0846e000-08479000 rw-p 0846e000 00:00 0
0853d000-085c7000 rw-p 0853d000 00:00 0 [heap]
b6480000-b6481000 ---p b6480000 00:00 0
b6481000-b64c0000 rw-p b6481000 00:00 0
b64c0000-b64c1000 ---p b64c0000 00:00 0
b64c1000-b6500000 rw-p b64c1000 00:00 0
b6500000-b6501000 ---p b6500000 00:00 0
b6501000-b6540000 rw-p b6501000 00:00 0
b6540000-b6541000 ---p b6540000 00:00 0
b6541000-b6580000 rw-p b6541000 00:00 0
b6580000-b6581000 ---p b6580000 00:00 0
b6581000-b65c0000 rw-p b6581000 00:00 0
b65c0000-b65c1000 ---p b65c0000 00:00 0
b65c1000-b6600000 rw-p b65c1000 00:00 0
b6600000-b6601000 ---p b6600000 00:00 0
b6601000-b6640000 rw-p b6601000 00:00 0
b6640000-b6641000 ---p b6640000 00:00 0
b6641000-b6680000 rw-p b6641000 00:00 0
b6680000-b6681000 ---p b6680000 00:00 0
b6681000-b66c0000 rw-p b6681000 00:00 0
b66c0000-b66c1000 ---p b66c0000 00:00 0
b66c1000-b6700000 rw-p b66c1000 00:00 0
b6700000-b6701000 ---p b6700000 00:00 0
b6701000-b6740000 rw-p b6701000 00:00 0
b6740000-b6741000 ---p b6740000 00:00 0
b6741000-b6780000 rw-p b6741000 00:00 0
b6780000-b6781000 ---p b6780000 00:00 0
b6781000-b67c0000 rw-p b6781000 00:00 0
b67c0000-b67c1000 ---p b67c0000 00:00 0
b67c1000-b6800000 rw-p b67c1000 00:00 0
b6800000-b6801000 ---p b6800000 00:00 0
b6801000-b6840000 rw-p b6801000 00:00 0
b6840000-b6841000 ---p b6840000 00:00 0
b6841000-b6880000 rw-p b6841000 00:00 0
b6880000-b6881000 ---p b6880000 00:00 0
b6881000-b68c0000 rw-p b6881000 00:00 0
b68c0000-b68c1000 ---p b68c0000 00:00 0
b68c1000-b6900000 rw-p b68c1000 00:00 0
b6900000-b6901000 ---p b6900000 00:00 0
b6901000-b6940000 rw-p b6901000 00:00 0
b6940000-b6941000 ---p b6940000 00:00 0
b6941000-b6980000 rw-p b6941000 00:00 0
b6980000-b6981000 ---p b6980000 00:00 0
b6981000-b69c0000 rw-p b6981000 00:00 0
b69c0000-b69c1000 ---p b69c0000 00:00 0
b69c1000-b6a00000 rw-p b69c1000 00:00 0
b6a00000-b6a01000 ---p b6a00000 00:00 0
b6a01000-b6a40000 rw-p b6a01000 00:00 0
b6a40000-b6a41000 ---p b6a40000 00:00 0
b6a41000-b6a80000 rw-p b6a41000 00:00 0
b6a80000-b6a81000 ---p b6a80000 00:00 0
b6a81000-b6ac0000 rw-p b6a81000 00:00 0
b6ac0000-b6ac1000 ---p b6ac0000 00:00 0
b6ac1000-b6b00000 rw-p b6ac1000 00:00 0
b6b00000-b6b01000 ---p b6b00000 00:00 0
b6b01000-b6b40000 rw-p b6b01000 00:00 0
b6b40000-b6b41000 ---p b6b40000 00:00 0
b6b41000-b6b80000 rw-p b6b41000 00:00 0
b6b80000-b6b81000 ---p b6b80000 00:00 0
b6b81000-b6bc0000 rw-p b6b81000 00:00 0
b6bc0000-b6bc1000 ---p b6bc0000 00:00 0
b6bc1000-b6c00000 rw-p b6bc1000 00:00 0
b6c00000-b6c01000 ---p b6c00000 00:00 0
b6c01000-b6c40000 rw-p b6c01000 00:00 0
b6c40000-b6c41000 ---p b6c40000 00:00 0
b6c41000-b6c80000 rw-p b6c41000 00:00 0
b6c80000-b6c81000 ---p b6c80000 00:00 0
b6c81000-b6cc0000 rw-p b6c81000 00:00 0
b6cc0000-b6cc1000 ---p b6cc0000 00:00 0
b6cc1000-b6d00000 rw-p b6cc1000 00:00 0
b6d00000-b6d01000 ---p b6d00000 00:00 0
b6d01000-b6d40000 rw-p b6d01000 00:00 0
b6d40000-b6d41000 ---p b6d40000 00:00 0
b6d41000-b6d80000 rw-p b6d41000 00:00 0
b6d80000-b6d81000 ---p b6d80000 00:00 0
b6d81000-b6dc0000 rw-p b6d81000 00:00 0
b6dc0000-b6dc1000 ---p b6dc0000 00:00 0
b6dc1000-b6e00000 rw-p b6dc1000 00:00 0
b6e00000-b6e01000 ---p b6e00000 00:00 0
b6e01000-b6e40000 rw-p b6e01000 00:00 0
b6e40000-b6e41000 ---p b6e40000 00:00 0
b6e41000-b6e80000 rw-p b6e41000 00:00 0
b6e80000-b6e81000 ---p b6e80000 00:00 0
b6e81000-b6ec0000 rw-p b6e81000 00:00 0
b6ec0000-b6ec1000 ---p b6ec0000 00:00 0
b6ec1000-b6f00000 rw-p b6ec1000 00:00 0
b6f00000-b6f01000 ---p b6f00000 00:00 0
b6f01000-b6f40000 rw-p b6f01000 00:00 0
b6f40000-b6f41000 ---p b6f40000 00:00 0
b6f41000-b6f80000 rw-p b6f41000 00:00 0
b6f80000-b6f81000 ---p b6f80000 00:00 0
b6f81000-b6fc0000 rw-p b6f81000 00:00 0
b6fc0000-b6fc1000 ---p b6fc0000 00:00 0
b6fc1000-b7000000 rw-p b6fc1000 00:00 0
b7000000-b7001000 ---p b7000000 00:00 0
b7001000-b7040000 rw-p b7001000 00:00 0
b7040000-b7041000 ---p b7040000 00:00 0
b7041000-b7080000 rw-p b7041000 00:00 0
b7080000-b7081000 ---p b7080000 00:00 0
b7081000-b70c0000 rw-p b7081000 00:00 0
b70c0000-b70c1000 ---p b70c0000 00:00 0
b70c1000-b7100000 rw-p b70c1000 00:00 0
b7100000-b7101000 ---p b7100000 00:00 0
b7101000-b7140000 rw-p b7101000 00:00 0
b7140000-b7141000 ---p b7140000 00:00 0
b7141000-b7180000 rw-p b7141000 00:00 0
b7180000-b7181000 ---p b7180000 00:00 0
b7181000-b71c0000 rw-p b7181000 00:00 0
b71c0000-b71c1000 ---p b71c0000 00:00 0
b71c1000-b7200000 rw-p b71c1000 00:00 0
b7200000-b7201000 ---p b7200000 00:00 0
b7201000-b7240000 rw-p b7201000 00:00 0
b7240000-b7241000 ---p b7240000 00:00 0
b7241000-b7280000 rw-p b7241000 00:00 0
b7280000-b7281000 ---p b7280000 00:00 0
b7281000-b72c0000 rw-p b7281000 00:00 0
b72c0000-b72c1000 ---p b72c0000 00:00 0
b72c1000-b7300000 rw-p b72c1000 00:00 0
b7300000-b7301000 ---p b7300000 00:00 0
b7301000-b7340000 rw-p b7301000 00:00 0
b7340000-b7341000 ---p b7340000 00:00 0
b7341000-b7380000 rw-p b7341000 00:00 0
b7380000-b7381000 ---p b7380000 00:00 0
b7381000-b73c0000 rw-p b7381000 00:00 0
b73c0000-b73c1000 ---p b73c0000 00:00 0
b73c1000-b7400000 rw-p b73c1000 00:00 0
b7400000-b7401000 ---p b7400000 00:00 0
b7401000-b7440000 rw-p b7401000 00:00 0
b7440000-b7441000 ---p b7440000 00:00 0
b7441000-b7480000 rw-p b7441000 00:00 0
b7480000-b7481000 ---p b7480000 00:00 0
b7481000-b74c0000 rw-p b7481000 00:00 0
b74c0000-b74c1000 ---p b74c0000 00:00 0
b74c1000-b7500000 rw-p b74c1000 00:00 0
b7500000-b7501000 ---p b7500000 00:00 0
b7501000-b7540000 rw-p b7501000 00:00 0
b7540000-b7541000 ---p b7540000 00:00 0
b7541000-b7580000 rw-p b7541000 00:00 0
b7580000-b7581000 ---p b7580000 00:00 0
b7581000-b75c0000 rw-p b7581000 00:00 0
b75c0000-b75c1000 ---p b75c0000 00:00 0
b75c1000-b7600000 rw-p b75c1000 00:00 0
b7600000-b7601000 ---p b7600000 00:00 0
b7601000-b7640000 rw-p b7601000 00:00 0
b7640000-b7641000 ---p b7640000 00:00 0
b7641000-b7680000 rw-p b7641000 00:00 0
b7680000-b7681000 ---p b7680000 00:00 0
b7681000-b76c0000 rw-p b7681000 00:00 0
b76c0000-b76c1000 ---p b76c0000 00:00 0
b76c1000-b7700000 rw-p b76c1000 00:00 0
b7700000-b7701000 ---p b7700000 00:00 0
b7701000-b7740000 rw-p b7701000 00:00 0
b7740000-b7741000 ---p b7740000 00:00 0
b7741000-b7780000 rw-p b7741000 00:00 0
b7780000-b7781000 ---p b7780000 00:00 0
b7781000-b77c0000 rw-p b7781000 00:00 0
b77c0000-b77c1000 ---p b77c0000 00:00 0
b77c1000-b7800000 rw-p b77c1000 00:00 0
b7800000-b7801000 ---p b7800000 00:00 0
b7801000-b7840000 rw-p b7801000 00:00 0
b7840000-b7841000 ---p b7840000 00:00 0
b7841000-b7880000 rw-p b7841000 00:00 0
b7880000-b7881000 ---p b7880000 00:00 0
b7881000-b78c0000 rw-p b7881000 00:00 0
b78c0000-b78c1000 ---p b78c0000 00:00 0
b78c1000-b7900000 rw-p b78c1000 00:00 0
b7900000-b7901000 ---p b7900000 00:00 0
b7901000-b7940000 rw-p b7901000 00:00 0
b7940000-b7941000 ---p b7940000 00:00 0
b7941000-b7980000 rw-p b7941000 00:00 0
b7980000-b7981000 ---p b7980000 00:00 0
b7981000-b79c0000 rw-p b7981000 00:00 0
b79c0000-b79c1000 ---p b79c0000 00:00 0
b79c1000-b7a00000 rw-p b79c1000 00:00 0
b7a00000-b7a01000 ---p b7a00000 00:00 0
b7a01000-b7a40000 rw-p b7a01000 00:00 0
b7a40000-b7a41000 ---p b7a40000 00:00 0
b7a41000-b7a80000 rw-p b7a41000 00:00 0
b7a80000-b7a81000 ---p b7a80000 00:00 0
b7a81000-b7ac0000 rw-p b7a81000 00:00 0
b7ac0000-b7ac1000 ---p b7ac0000 00:00 0
b7ac1000-b7b36000 rw-p b7ac1000 00:00 0
b7b36000-b7c00000 ---p b7b36000 00:00 0
b7c18000-b7c19000 ---p b7c18000 00:00 0
b7c19000-b7c58000 rw-p b7c19000 00:00 0
b7c58000-b7c59000 ---p b7c58000 00:00 0
b7c59000-b7c98000 rw-p b7c59000 00:00 0
b7c98000-b7c99000 ---p b7c98000 00:00 0
b7c99000-b7cd8000 rw-p b7c99000 00:00 0
b7cd8000-b7cd9000 ---p b7cd8000 00:00 0
b7cd9000-b7d18000 rw-p b7cd9000 00:00 0
b7d18000-b7d21000 r-xp 00000000 68:01 901446 /lib/libnss_files-2.7.so
b7d21000-b7d23000 rw-p 00008000 68:01 901446 /lib/libnss_files-2.7.so
b7d23000-b7d2b000 r-xp 00000000 68:01 901448 /lib/libnss_nis-2.7.so
b7d2b000-b7d2d000 rw-p 00007000 68:01 901448 /lib/libnss_nis-2.7.so
b7d2d000-b7d40000 r-xp 00000000 68:01 901443 /lib/libnsl-2.7.so
b7d40000-b7d42000 rw-p 00012000 68:01 901443 /lib/libnsl-2.7.so
b7d42000-b7d44000 rw-p b7d42000 00:00 0
b7d44000-b7d4b000 r-xp 00000000 68:01 901444 /lib/libnss_compat-2.7.so
b7d4b000-b7d4d000 rw-p 00006000 68:01 901444 /lib/libnss_compat-2.7.so
b7d4d000-b7d4f000 rw-p b7d4d000 00:00 0
b7d4f000-b7e87000 r-xp 00000000 68:01 901437 /lib/libc-2.7.so
b7e87000-b7e88000 r--p 00138000 68:01 901437 /lib/libc-2.7.so
b7e88000-b7e8a000 rw-p 00139000 68:01 901437 /lib/libc-2.7.so
b7e8a000-b7e8d000 rw-p b7e8a000 00:00 0
b7e8d000-b7e99000 r-xp 00000000 68:01 901433 /lib/libgcc_s.so.1
b7e99000-b7e9a000 rw-p 0000b000 68:01 901433 /lib/libgcc_s.so.1
b7e9a000-b7ebe000 r-xp 00000000 68:01 901441 /lib/libm-2.7.so
b7ebe000-b7ec0000 rw-p 00023000 68:01 901441 /lib/libm-2.7.so
b7ec0000-b7fa3000 r-xp 00000000 68:01 1629152 /usr/lib/libstdc++.so.6.0.10
b7fa3000-b7fa6000 r--p 000e2000 68:01 1629152 /usr/lib/libstdc++.so.6.0.10
b7fa6000-b7fa8000 rw-p 000e5000 68:01 1629152 /usr/lib/libstdc++.so.6.0.10
b7fa8000-b7fae000 rw-p b7fa8000 00:00 0
b7fae000-b7fb0000 r-xp 00000000 68:01 901440 /lib/libdl-2.7.so
b7fb0000-b7fb2000 rw-p 00001000 68:01 901440 /lib/libdl-2.7.so
b7fb2000-b7fc1000 r-xp 00000000 68:01 901452 /lib/libresolv-2.7.so
b7fc1000-b7fc3000 rw-p 0000e000 68:01 901452 /lib/libresolv-2.7.so
b7fc3000-b7fc6000 rw-p b7fc3000 00:00 0
b7fc6000-b7fea000 r-xp 00000000 68:01 1629441 /usr/lib/libexpat.so.1.5.2
b7fea000-b7fec000 rw-p 00023000 68:01 1629441 /usr/lib/libexpat.so.1.5.2
b7fec000-b8000000 r-xp 00000000 68:01 901451 /lib/libpthread-2.7.so
b8000000-b8002000 rw-p 00013000 68:01 901451 /lib/libpthread-2.7.so
b8002000-b8005000 rw-p b8002000 00:00 0
b8005000-b800a000 rw-p b8005000 00:00 0
b800a000-b800b000 r-xp b800a000 00:00 0 [vdso]
b800b000-b8025000 r-xp 00000000 68:01 901434 /lib/ld-2.7.so
b8025000-b8027000 rw-p 0001a000 68:01 901434 /lib/ld-2.7.so
bff11000-bff26000 rw-p bffeb000 00:00 0 [stack]

Click to view joegen's profile   469 posts since
Apr 28, 2007
6. Dec 27, 2009 10:47 PM in response to: mparadis
Re: Crashes out of the blue

hi empradis,

A crash is ALWAYS a bug :-). If opensbc crashes, it must have generated core.* files. For us to have a better idea which portion of the code causes opensbc to crash, you must provide a GDB backtrace of the core file generated. Make sure you are using the latest CVS for bug reports.

Click to view joegen's profile   469 posts since
Apr 28, 2007
9. Dec 28, 2009 6:35 PM in response to: mparadis
Re: Crashes out of the blue
The log is not a problem since the log and the core file generated by the crash will have the same modified date property. Please repost the corefile back trace. After you see the screen below, simply type 'bt' inside GDB. IT should display the callstack. paste that instead.
Click to view joegen's profile   469 posts since
Apr 28, 2007
10. Dec 28, 2009 7:05 PM in response to: joegen
Re: Crashes out of the blue
I think I have an Idea where it crashes. I have commited changes to both opensipstack and opensbc. Can you try the latest from CVS and see if the registration from this rogue user still results toa crash?
Click to view joegen's profile   469 posts since
Apr 28, 2007
12. Dec 28, 2009 11:10 PM in response to: mparadis
Re: Crashes out of the blue
See http://www.opensipstack.org/cvs.html
Click to view joegen's profile   469 posts since
Apr 28, 2007
14. Dec 28, 2009 11:36 PM in response to: mparadis
Re: Crashes out of the blue

  1. cd opensipstack

  1. cvs update

#cd ../opensbc

#cvs update