Unix Programming - segmentation fault mystery..

Author

segmentation fault mystery..

ryantimmins@gmail.com

2006-03-21, 3:17 am

maybe someone here can help me find what is causing this
when i run buildmoz (from dmoz.org) , i get this error:

Program received signal SIGSEGV, Segmentation fault.
0x080518c0 in process_string (p=0xbfe5d848 "NPR Topics: Movies",
count=0, start=4096, total=4693, flags=536870912, data=0xbfe3c340,
list=0xbfeff8a8) at buildmoz.c:5290

When i look at that line in the code it's line 5290:

5274 static int process_string(const unsigned char *p,int count,int
start,int total,
5275 int flags,WORDLIST **data,WORDLIST **list)
5276 {
5277 int j=0;
5278 int l=0;
5279 int m=0;
5280 int current=0;
5281 int delta=0;
5282 unsigned int v=0;
5283 unsigned int w=0;
5284 unsigned char *q=0;
5285 WORDLIST *r=0;
5286 unsigned char word[MAXWORD*6+1];
5287 unsigned int wrk[BUFSIZEH];
5288
5289 if(!p)return count;
5290 l=strlen(p);

When i break in the debugger strlen(p) gives 18 and l gives 0 and i can
assign l = strlen(p) using set var l = strlen(p) with no problems....

This method process_string gets called hundreds of times elsewhere with
no problems...p points to a null terminated string "NPR: Movies etc.
etc.. \0"

Any suggestions how to debug this further ?

Paul Pluzhnikov

2006-03-21, 3:17 am

ryantimmins@gmail.com writes:

> maybe someone here can help me find what is causing this
> when i run buildmoz (from dmoz.org) , i get this error:

When I connect to dmoz.org, it provides no clues what "buildmoz"
is or what it is you are actually building.

Please read this:
http://www.catb.org/~esr/faqs/smart-questions.html
and try to be more precise next time.

You may also wish to specify your OS (some variant of Linux/x86?) and
the compiler -- that way someone might be able to actually replicate
your problem and show you how one debugs such things.

> Program received signal SIGSEGV, Segmentation fault.
> 0x080518c0 in process_string (p=0xbfe5d848 "NPR Topics: Movies",
> count=0, start=4096, total=4693, flags=536870912, data=0xbfe3c340,
> list=0xbfeff8a8) at buildmoz.c:5290
>
>
> When i look at that line in the code it's line 5290:
>
> 5274 static int process_string(const unsigned char *p,int count,int start,int total,
> 5275 int flags,WORDLIST **data,WORDLIST **list)
> 5276 {
....
> 5289 if(!p)return count;
> 5290 l=strlen(p);
>
> When i break in the debugger

What do you mean "break into debugger"?
The debugger should have already stopped when SIGSEGV was delivered
to the debuggee.

> strlen(p) gives 18 and l gives 0 and i can
> assign l = strlen(p) using set var l = strlen(p) with no problems....
>
> This method process_string gets called hundreds of times elsewhere with
> no problems...p points to a null terminated string "NPR: Movies etc.
> etc.. \0"
>
> Any suggestions how to debug this further ?

What is the output from the gdb commands below at the crash point:

x/8i $pc
x/20i &process_string
where
info reg
info shared

Cheers,
--
In order to understand recursion you must first understand recursion.
Remove /-nsp/ for email.

ryantimmins@gmail.com

2006-03-21, 3:17 am

thanks alot for your response.

Buildmoz is a tool that is a part of this application:
http://developer.berlios.de/projects/minimoz/

Buildmoz basically takes the rdf files from here:
http://rdf.dmoz.org/rdf/
and puts the data into a mysql database

The system i am running this on is Debian Linux x86 , the compiler is
gcc

re: "break into debugger" - when SIGSEGV was delivered the debugger
stopped

more info:
(gdb) x/8i $pc
0x80518c0 <process_string+138>: mov %eax,0x4(%esp)
0x80518c4 <process_string+142>: movl $0x8076191,(%esp)
0x80518cb <process_string+149>: call 0x8048b60 <_init+120>
0x80518d0 <process_string+154>: test %eax,%eax
0x80518d2 <process_string+156>: jne 0x80518e1 <process_string+171>
0x80518d4 <process_string+158>: movl $0x12,0x40148(%esp)
0x80518df <process_string+169>: jmp 0x80518f7 <process_string+193>
0x80518e1 <process_string+171>: mov 0x40160(%esp),%eax

(gdb) x/20i &process_string
0x8051836 <process_string>: sub $0x4015c,%esp
0x805183c <process_string+6>: movl $0x0,0x4014c(%esp)
0x8051847 <process_string+17>: movl $0x0,0x40148(%esp)
0x8051852 <process_string+28>: movl $0x0,0x40144(%esp)
0x805185d <process_string+39>: movl $0x0,0x40140(%esp)
0x8051868 <process_string+50>: movl $0x0,0x4013c(%esp)
0x8051873 <process_string+61>: movl $0x0,0x40138(%esp)
0x805187e <process_string+72>: movl $0x0,0x40134(%esp)
0x8051889 <process_string+83>: movl $0x0,0x40130(%esp)
0x8051894 <process_string+94>: movl $0x0,0x4012c(%esp)
0x805189f <process_string+105>: cmpl $0x0,0x40160(%esp)
0x80518a7 <process_string+113>: jne 0x80518b9 <process_string+131>
0x80518a9 <process_string+115>: mov 0x40164(%esp),%eax
0x80518b0 <process_string+122>: mov %eax,0x1c(%esp)
0x80518b4 <process_string+126>: jmp 0x8051f6c <process_string+1846>
0x80518b9 <process_string+131>: mov 0x40160(%esp),%eax
0x80518c0 <process_string+138>: mov %eax,0x4(%esp)
0x80518c4 <process_string+142>: movl $0x8076191,(%esp)
0x80518cb <process_string+149>: call 0x8048b60 <_init+120>
0x80518d0 <process_string+154>: test %eax,%eax

#0 0x080518c0 in process_string (p=0xbfe5d848 "NPR Topics: Movies",
count=0, start=4096, total=4693, flags=536870912, data=0xbfe3c340,
list=0xbfeff8a8) at buildmoz.c:5289
#1 0x08051fe8 in update_pagesearch (u=0xbfe4d760) at buildmoz.c:5417
#2 0x0805f62a in contentend (name=0xbfe3d926 "ExternalPage",
_u=0xbfe4d760) at buildmoz.c:8156
#3 0x0804fb64 in parse (data=0xbfe3d660 "<?xml version='1.0'
encoding='UTF-8' ?>\n<RDF", len=65536, line=0xbfe4d754, final=0,
checkutf=1, userdata=0xbfe4d760, datahandler=0x80607dd <contentdata>,
starthandler=0x805e9b6 <contentstart>, endhandler=0x805f110
<contentend> ) at buildmoz.c:4655
#4 0x08060ca2 in get_content (contentfile=0xbffffc5c
"/root/dmoz.org/content.rdf.u8.gz", database=0xbffffc2e "dmoz",
user=0xbffffcc7 "root", password=0xbffffccf "******") at
buildmoz.c:8561
#5 0x080644a8 in builder (database=0xbffffc2e "dmoz", user=0xbffffcc7
"root", password=0xbffffccf "******", sname=0xbffffc36
"/root/dmoz.org/structure.rdf.u8.gz", cname=0xbffffc5c
"/root/dmoz.org/content.rdf.u8.gz", rname=0xbffffca2
"/root/dmoz.org/redirect.rdf.u8.gz", tname=0xbffffc80
"/root/dmoz.org/terms.rdf.u8.gz") at buildmoz.c:9191
#6 0x080648cf in main (argc=16, argv=0xbffffab4) at buildmoz.c:9308

(gdb) info reg
eax 0xbfe5d848 -1075455928
ecx 0x40bef04b 1086255179
edx 0x0 0
ebx 0x40294880 1076447360
esp 0xbfdfc1b0 0xbfdfc1b0
ebp 0xbffffa58 0xbffffa58
esi 0x40016540 1073833280
edi 0xbffffab4 -1073743180
eip 0x80518c0 0x80518c0
eflags 0x10286 66182
cs 0x23 35
ss 0x2b 43
ds 0x2b 43
es 0x2b 43
fs 0x0 0
gs 0x0 0

>From To Syms Read Shared Object Library
0x40027780 0x40031f90 Yes /usr/lib/libz.so.1
0x40045140 0x40073840 Yes /usr/lib/libmysqlclient_r.so.14
0x40146540 0x4015e000 Yes /lib/libm.so.6
0x4017ac50 0x40270968 Yes /lib/libc.so.6
0x4029c1e0 0x402a4d50 Yes /lib/libpthread.so.0
0x402e9a00 0x402ec330 Yes /lib/libcrypt.so.1
0x40319c30 0x40325ac0 Yes /lib/libnsl.so.1
0x40000c00 0x40011eaf Yes /lib/ld-linux.so.2
0x4001a380 0x4001cbe0 Yes /lib/libnss_db.so.2
0x4033bd90 0x40341e50 Yes /lib/libnss_files.so.2
0x4034f9e0 0x403dfa40 Yes /usr/lib/libdb3.so.3

and these are the arguments passed into build moz

-d dmoz -s /root/dmoz.org/structure.rdf.u8.gz -c
/root/dmoz.org/content.rdf.u8.gz -t /root/dmoz.org/terms.rdf.u8.gz -r
/root/dmoz.org/redirect.rdf.u8.gz -u root -p ****** -a

Paul Pluzhnikov

2006-03-21, 3:17 am

ryantimmins@gmail.com writes:

> The system i am running this on is Debian Linux x86 , the compiler is
> gcc

This isn't precise enough. It's roughly equivalent to answering
"a Ford, with a gasoline engine" when asked "what car do you want
to have a $30K mile scheduled service performed on".

> (gdb) x/8i $pc
> 0x80518c0 <process_string+138>: mov %eax,0x4(%esp)

This is the instruction that crashed ...

> (gdb) info reg
> esp 0xbfdfc1b0 0xbfdfc1b0

Stack on Linux usually begins at 0xC0000000 and grows down.
The current function (at entry) reserved 0x4015c == 260K of
stack space:

0x8051836 <process_string>: sub $0x4015c,%esp

pushing total stack usage to just over 2M (0xC0000000 - 0xbfdfc1b0)
and that exceeded your stack limit (probably 2M, do "ulimit -s"
to find out what your stack limit is).

Attempt to access %esp+4 == 0xbfdfc1b4 caused access to a new
stack page; the kernel refused to extend the stack beyond its
current limit, and delivered SIGSEGV instead.

The fix: increase your stack limit ("ulimit -s 10240", or
"ulimit -s unlimited")

Cheers,
--
In order to understand recursion you must first understand recursion.
Remove /-nsp/ for email.

ryantimmins@gmail.com

2006-03-21, 3:17 am

The ulimit is already unlimited. I am running the debian linux on a
virtual machine with 1gb ram on the physical machine and 635MB
allocated to the virtual machine.(2.8ghz intel processor). The files
buildmoz is processing are very large - 2gb and 600mb, maybe the system
is running into some other memory limit ? I have been digging around
and attached some more info that maybe could help. Any more
suggestions ? Thanks for the assistance.

debian1:~/dmoz.org# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
stack size (kbytes, -s) unlimited
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited

debian1:~/dmoz.org/minimoz/src# cat /proc/1184/maps
08048000-08081000 r-xp 00000000 03:01 2327229
/root/dmoz.org/minimoz/src/buildmoz
08081000-08087000 rw-p 00039000 03:01 2327229
/root/dmoz.org/minimoz/src/buildmoz
08087000-081b7000 rwxp 00000000 00:00 0
40000000-40016000 r-xp 00000000 03:01 720913 /lib/ld-2.3.2.so
40016000-40017000 rw-p 00015000 03:01 720913 /lib/ld-2.3.2.so
40017000-40019000 rw-p 00000000 00:00 0
40019000-4001d000 r-xp 00000000 03:01 720992 /lib/libnss_db-2.2.so
4001d000-4001e000 rw-p 00004000 03:01 720992 /lib/libnss_db-2.2.so
4001e000-4001f000 rw-p 00000000 00:00 0
40026000-40037000 r-xp 00000000 03:01 573489 /usr/lib/libz.so.1.2.2
40037000-40038000 rw-p 00010000 03:01 573489 /usr/lib/libz.so.1.2.2
40038000-40079000 r-xp 00000000 03:01 575137
/usr/lib/libmysqlclient_r.so.14.0.0
40079000-4013f000 rw-p 00040000 03:01 575137
/usr/lib/libmysqlclient_r.so.14.0.0
4013f000-40143000 rw-p 00000000 00:00 0
40143000-40164000 r-xp 00000000 03:01 720922 /lib/libm-2.3.2.so
40164000-40165000 rw-p 00020000 03:01 720922 /lib/libm-2.3.2.so
40165000-4028d000 r-xp 00000000 03:01 720919 /lib/libc-2.3.2.so
4028d000-40295000 rw-p 00127000 03:01 720919 /lib/libc-2.3.2.so
40295000-40298000 rw-p 00000000 00:00 0
40298000-402a5000 r-xp 00000000 03:01 720932
/lib/libpthread-0.10.so
402a5000-402a7000 rw-p 0000c000 03:01 720932
/lib/libpthread-0.10.so
402a7000-402e9000 rw-p 00000000 00:00 0
402e9000-402ee000 r-xp 00000000 03:01 720920 /lib/libcrypt-2.3.2.so
402ee000-402ef000 rw-p 00004000 03:01 720920 /lib/libcrypt-2.3.2.so
402ef000-40316000 rw-p 00000000 00:00 0
40316000-40328000 r-xp 00000000 03:01 720924 /lib/libnsl-2.3.2.so
40328000-40329000 rw-p 00011000 03:01 720924 /lib/libnsl-2.3.2.so
40329000-4032c000 rw-p 00000000 00:00 0
4033a000-40342000 r-xp 00000000 03:01 720927
/lib/libnss_files-2.3.2.so
40342000-40343000 rw-p 00008000 03:01 720927
/lib/libnss_files-2.3.2.so
40343000-403ec000 r-xp 00000000 03:01 573459
/usr/lib/libdb3.so.3.0.2
403ec000-403ed000 rw-p 000a8000 03:01 573459
/usr/lib/libdb3.so.3.0.2
403ed000-413f1000 rw-p 00000000 00:00 0
417f2000-41ff4000 rw-p 01405000 00:00 0
423f5000-427f6000 rw-p 02008000 00:00 0
42bf7000-42ff8000 rw-p 0280a000 00:00 0
433f9000-43bfb000 rw-p 0300c000 00:00 0
43ffc000-447fe000 rw-p 03c0f000 00:00 0
44bff000-45000000 rw-p 04812000 00:00 0
45401000-45c03000 rw-p 05014000 00:00 0
46004000-46806000 rw-p 05c17000 00:00 0
46c07000-47008000 rw-p 0681a000 00:00 0
47409000-4780a000 rw-p 0701c000 00:00 0
4c579000-4cad6000 rw-p 0c18c000 00:00 0
bfe37000-c0000000 rwxp ffe38000 00:00 0

Paul Pluzhnikov

2006-03-21, 3:17 am

ryantimmins@gmail.com writes:

> The ulimit is already unlimited. I am running the debian linux on a
> virtual machine with 1gb ram on the physical machine and 635MB
> allocated to the virtual machine.(2.8ghz intel processor).

You *still* don't get the concept of specifying your environment
precisely :-(

> The files
> buildmoz is processing are very large - 2gb and 600mb, maybe the system
> is running into some other memory limit ?

It's probably the virtual machine that runs out of some resource,
and looking *inside* this virtual machine will not give you any
clues.

Since you didn't even tell us what virtual machine you are using,
nor what is the "host OS", nor what is the "guest OS" (you did say
"debian linux", but that's as meaningful as "it's a Ford"); it's
pretty hard to give you any further clues.

One thing you might try is run this on a *real* Linux machine,
and see if you still have the same problem.

Cheers,
--
In order to understand recursion you must first understand recursion.
Remove /-nsp/ for email.

ryantimmins@gmail.com

2006-03-21, 3:17 am

I resolved this issue by rewriting the app a little bit so that it
doesn't use so much stack space. Even though ulimit -s unlimited was
set, the stack size could go no higher than 2M. I tried increasing
it using pthread_attr_setstacksize but it threw an error. I am a
newbie to linux - Thankyou very much for your time and help.

Tux

2006-03-21, 3:17 am

Hi Ryan,

I would appreciate if you could send me the working copy of minimoz. I
am also struggling with segfaults. Additionally, were you able to
generate a directory using the data? Did you have to write the search
function yourself or minimoz generates all of it for you?

Thanks.