Hello,
I have a Sun Blade 1500 which is choking in UFS code and I wonder what
on earth it is. It is a shame that we don't have a support contract
for this one. But I see the following:
panic[cpu0]/thread=2a100057cc0:
sync initiated
sched:
software trap 0x7f
pid=0, pc=0xf004763c, sp=0x2a100056ec1, tstate=0x4400001405, context=0x0
g1-g7: 1047b74, 0, 183d400, 0, 1074c00, 16, 2a100057cc0
00000000fff07cd0 unix:sync_handler+138 (feddefc8, 1000000, 1, 1075c00, 1, 181400
0)
%l0-3: 000000000184c5d0 000000000184c400 000000000000017f 0000000001843c00
%l4-7: 0000000000000000 000000000183d400 000000000002008e 000000000180fc00
00000000fff07da0 unix:vx_handler+80 (feddefc8, 181c858, 180c000, 2a100057d98, 18
1c960, f0059899)
%l0-3: 000000000181c960 0000000000000000 0000000000000001 0000000000000001
%l4-7: 0000000001810400 00000000f0000000 0000000001000000 0000000001017e04
00000000fff07e50 unix:callback_handler+20 (feddefc8, fff68280, 0, 0, 0, 0)
%l0-3: 0000000000000016 00000000fff07701 0000004480001600 000000000000080e
%l4-7: 0000000001853b38 0000000000000000 0000000000000000 000002a1001578b0
syncing file systems...
113
93
92
92
92
92
92
92
92
92
92
92
92
92
92
92
92
92
92
92
92
92
92
done (not all i/o completed)
dumping to /dev/dsk/c0t0d0s1, offset 838926336, content: kernel
> ::status
debugging crash dump vmcore.1 (64-bit) from schweiz
operating system: 5.10 Generic_118833-36 (sun4u)
panic message: sync initiated
dump content: kernel pages only
> $c
0xf004763c(f0059b78, 2a100057828, 0, 181c850, 1848400, f0059b78)
prom_enter_mon+0x24(0, 0, 183b400, 1, 1812140, 181ae60)
debug_enter+0x118(0, a, a, 180fc00, 0, 183d400)
abort_seq_softintr+0x94(180fc00, 18a9400, 180c000, 2a100057d98, 2, 1857c00)
intr_thread+0x170(30002a5f510, 0, 11efc00, 2, 2a100157cc4, 0)
ufs_idle_some+0x15c(110b, 30002a5f430, 940, 0, 18beb80, 18bcc78)
ufs_thread_idle+0x1dc(0, 2, 18be800, 18be800, 100, 0)
thread_start+4(0, 0, 2c2912000001, 122ff04, 10c, 211e1200fff1)
>
As you can see the machine was running in the kernel debugger. I wonder
what "software trap 0x7f" means and where I can read about that stuff?
I did a "smpatch analyze" on that machine and came back with the
following (and 98 others which don't match "grep -i ufs"):
124258-07 SunOS 5.10: ufs and nfs driver patch
I installed that patch and rebooted. Hopefully that fixed it. But we
already changed everything (including the whole machine) and it is the
only one (out of ~ 70) who has that problem. We also
ufsdumped/ufsrestored the filesystems on _fresh_ disks. I ran fsck in
force modus on all involved filesystems. This machines crashes at least
once a day.
Thomas
|
|
0
|
|
|
|
Reply
|
 Thomas
|
8/2/2007 7:14:51 AM |
|
Thomas Glanzmann <sithglan@stud.uni-erlangen.de> writes:
>Hello,
>I have a Sun Blade 1500 which is choking in UFS code and I wonder what
>on earth it is. It is a shame that we don't have a support contract
>for this one. But I see the following:
> panic[cpu0]/thread=2a100057cc0:
> sync initiated
Someone typed Stop-A and then "sync".
It is possible it hung but the information you provided only tells that
it was interrupted with Stop-A and that a core dump was then generated using
"sync".
Casper
--
Expressed in this posting are my opinions. They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.
|
|
|
0
|
|
|
|
Reply
|
Casper
|
8/2/2007 10:24:29 AM
|
|
Casper H.S. Dik wrote:
> Thomas Glanzmann <sithglan@stud.uni-erlangen.de> writes:
>
>> Hello,
>> I have a Sun Blade 1500 which is choking in UFS code and I wonder what
>> on earth it is. It is a shame that we don't have a support contract
>> for this one. But I see the following:
>
>> panic[cpu0]/thread=2a100057cc0:
>> sync initiated
>
> Someone typed Stop-A and then "sync".
>
> It is possible it hung but the information you provided only tells that
> it was interrupted with Stop-A and that a core dump was then generated using
> "sync".
>
> Casper
I have seen broken keyboards that frequently do internal resets and produce
a Stop-A
--
Michael Tosch @ hp : com
|
|
|
0
|
|
|
|
Reply
|
Michael
|
8/2/2007 10:42:46 AM
|
|
Hello,
> I have seen broken keyboards that frequently do internal resets and
> produce a Stop-A
we already switch the keyboard, too. And even the mouse on this system.
Thomas
|
|
|
0
|
|
|
|
Reply
|
Thomas
|
8/2/2007 10:48:42 AM
|
|
Hello Casper,
>> panic[cpu0]/thread=2a100057cc0:
>> sync initiated
> Someone typed Stop-A and then "sync".
> It is possible it hung but the information you provided only tells
> that it was interrupted with Stop-A and that a core dump was then
> generated using "sync".
that is absolut correct. But Stop-A was pressed because the system hang
in a lock aquire and was dead (it still answered pings becaus I guess
ICMP replies are handled in a interrupt handler or function called by
the interrupt handler). I could find out as much as that:
As you can see ufs_idle_some+0x15c is a call to "rw_enter" and that is
the last thing you can see. I have another crash dump with the exact
same offset. That is why we pressed Stop-A.
There seems to be a Bug Reports which match that offset:
bug 6299287 and bug 4667688
State of the latter is "State: 10-Fix Delivered". That useally means
that it is fixed but not released. So I open a call to get the patch for
this one.
Can someone tell me how I get the inode number of file name or the file
that causes this out of the vnode or inode struct?
Thomas
(schweiz) [/var/crash/schweiz] mdb -k unix.1 vmcore.1
Loading modules: [ unix krtld genunix specfs dtrace ufs pcisch ip sctp usba s1394 fctl lofs zfs random nfs audiosup crypto sd ptm cpc fcip ipc ]
> ::stack
0xf004763c(f0059b78, 2a100057828, 0, 181c850, 1848400, f0059b78)
prom_enter_mon+0x24(0, 0, 183b400, 1, 1812140, 181ae60)
debug_enter+0x118(0, a, a, 180fc00, 0, 183d400)
abort_seq_softintr+0x94(180fc00, 18a9400, 180c000, 2a100057d98, 2, 1857c00)
intr_thread+0x170(30002a5f510, 0, 11efc00, 2, 2a100157cc4, 0)
ufs_idle_some+0x15c(110b, 30002a5f430, 940, 0, 18beb80, 18bcc78)
ufs_thread_idle+0x1dc(0, 2, 18be800, 18be800, 100, 0)
thread_start+4(0, 0, 2c2912000001, 122ff04, 10c, 211e1200fff1)
> ufs_idle_some::dis
ufs_idle_some: save %sp, -0xb0, %sp
ufs_idle_some+4: cmp %i0, 0
ufs_idle_some+8: ble,pn %icc, +0x1bc <ufs_idle_some+0x1c4>
ufs_idle_some+0xc: clr %i3
ufs_idle_some+0x10: sethi %hi(0x18be800), %i1
ufs_idle_some+0x14: sethi %hi(0x18bcc00), %i4
ufs_idle_some+0x18: sethi %hi(0x18bcc00), %g1
ufs_idle_some+0x1c: sethi %hi(0x18be800), %i2
ufs_idle_some+0x20: add %i1, 0x3c0, %l1
ufs_idle_some+0x24: add %i4, 0x70, %l0
ufs_idle_some+0x28: add %g1, 0x78, %i5
ufs_idle_some+0x2c: add %i2, 0x380, %i4
ufs_idle_some+0x30: clr %l2
ufs_idle_some+0x34: mov %l1, %o0
ufs_idle_some+0x38: call -0x1f0690 <mutex_enter>
ufs_idle_some+0x3c: nop
ufs_idle_some+0x40: ld [%l0], %l4
ufs_idle_some+0x44: cmp %l4, 0
ufs_idle_some+0x48: be,pn %icc, +0x70 <ufs_idle_some+0xb8>
ufs_idle_some+0x4c: sethi %hi(0x18be800), %g2
ufs_idle_some+0x50: ld [%i5], %g4
ufs_idle_some+0x54: ldx [%g2 + 0x2c0], %g1
ufs_idle_some+0x58: sra %g4, 0, %g3
ufs_idle_some+0x5c: sllx %g3, 5, %i2
ufs_idle_some+0x60: add %g1, 0x10, %l6
ufs_idle_some+0x64: ldx [%i2 + %l6], %i1
ufs_idle_some+0x68: add %g1, %i2, %l5
ufs_idle_some+0x6c: cmp %i1, %l5
ufs_idle_some+0x70: bne,pn %xcc, +0x3c <ufs_idle_some+0xac>
ufs_idle_some+0x74: add %g2, 0x2c0, %l7
ufs_idle_some+0x78: ld [%i5], %o7
ufs_idle_some+0x7c: ld [%i4], %o5
ufs_idle_some+0x80: add %o7, 1, %o4
ufs_idle_some+0x84: ldx [%l7], %o1
ufs_idle_some+0x88: and %o4, %o5, %o3
ufs_idle_some+0x8c: st %o3, [%i5]
ufs_idle_some+0x90: sra %o3, 0, %o2
ufs_idle_some+0x94: sllx %o2, 5, %o0
ufs_idle_some+0x98: ldx [%o0 + %l6], %g5
ufs_idle_some+0x9c: add %o1, %o0, %l5
ufs_idle_some+0xa0: cmp %g5, %l5
ufs_idle_some+0xa4: be,a,pt %xcc, -0x28 <ufs_idle_some+0x7c>
ufs_idle_some+0xa8: ld [%i5], %o7
ufs_idle_some+0xac: ldx [%l5 + 0x10], %l3
ufs_idle_some+0xb0: ba +0x80 <ufs_idle_some+0x130>
ufs_idle_some+0xb4: ldx [%l3 + 0xa0], %l4
ufs_idle_some+0xb8: sethi %hi(0x18bcc00), %l5
ufs_idle_some+0xbc: ld [%l5 + 0x74], %l4
ufs_idle_some+0xc0: cmp %l4, 0
ufs_idle_some+0xc4: be,pn %icc, +0xf4 <ufs_idle_some+0x1b8>
ufs_idle_some+0xc8: sethi %hi(0x18be800), %l3
ufs_idle_some+0xcc: ld [%i5 + 4], %g4
ufs_idle_some+0xd0: ldx [%l3 + 0x288], %g2
ufs_idle_some+0xd4: sra %g4, 0, %g3
ufs_idle_some+0xd8: sllx %g3, 5, %g1
ufs_idle_some+0xdc: add %g2, 0x10, %l7
ufs_idle_some+0xe0: ldx [%g1 + %l7], %i2
ufs_idle_some+0xe4: add %g2, %g1, %l6
ufs_idle_some+0xe8: cmp %i2, %l6
ufs_idle_some+0xec: bne,pn %xcc, +0x3c <ufs_idle_some+0x128>
ufs_idle_some+0xf0: add %l3, 0x288, %i1
ufs_idle_some+0xf4: ld [%i5 + 4], %o7
ufs_idle_some+0xf8: ld [%i4], %o5
ufs_idle_some+0xfc: add %o7, 1, %o4
ufs_idle_some+0x100: ldx [%i1], %o1
ufs_idle_some+0x104: and %o4, %o5, %o3
ufs_idle_some+0x108: st %o3, [%i5 + 4]
ufs_idle_some+0x10c: sra %o3, 0, %o2
ufs_idle_some+0x110: sllx %o2, 5, %o0
ufs_idle_some+0x114: ldx [%o0 + %l7], %g5
ufs_idle_some+0x118: add %o1, %o0, %l6
ufs_idle_some+0x11c: cmp %g5, %l6
ufs_idle_some+0x120: be,a,pt %xcc, -0x28 <ufs_idle_some+0xf8>
ufs_idle_some+0x124: ld [%i5 + 4], %o7
ufs_idle_some+0x128: ldx [%l6 + 0x10], %l3
ufs_idle_some+0x12c: ldx [%l3 + 0xa0], %l4
ufs_idle_some+0x130: call -0x1f0788 <mutex_enter>
ufs_idle_some+0x134: mov %l4, %o0
ufs_idle_some+0x138: ld [%l4 + 0xc], %l7
ufs_idle_some+0x13c: add %l3, 0xe0, %l5
ufs_idle_some+0x140: add %l7, 1, %l6
ufs_idle_some+0x144: st %l6, [%l4 + 0xc]
ufs_idle_some+0x148: call -0x1f0720 <mutex_exit>
ufs_idle_some+0x14c: mov %l4, %o0
ufs_idle_some+0x150: call -0x1f0728 <mutex_exit>
ufs_idle_some+0x154: mov %l1, %o0
ufs_idle_some+0x158: mov %l5, %o0
ufs_idle_some+0x15c: call -0x1f0714 <rw_enter>
ufs_idle_some+0x160: mov %i3, %o1
ufs_idle_some+0x164: call -0x147a0 <ufs_rmidle>
ufs_idle_some+0x168: mov %l3, %o0
ufs_idle_some+0x16c: tst %o0
ufs_idle_some+0x170: be,pn %icc, +0x20 <ufs_idle_some+0x190>
ufs_idle_some+0x174: nop
ufs_idle_some+0x178: call -0x1f06c0 <rw_exit>
ufs_idle_some+0x17c: mov %l5, %o0
ufs_idle_some+0x180: call -0x620 <ufs_idle_free>
ufs_idle_some+0x184: mov %l3, %o0
ufs_idle_some+0x188: ba +0x1c <ufs_idle_some+0x1a4>
ufs_idle_some+0x18c: add %l2, 1, %l2
ufs_idle_some+0x190: call -0x1f06d8 <rw_exit>
ufs_idle_some+0x194: mov %l5, %o0
ufs_idle_some+0x198: call -0x69ed0 <vn_rele>
ufs_idle_some+0x19c: mov %l4, %o0
ufs_idle_some+0x1a0: add %l2, 1, %l2
ufs_idle_some+0x1a4: cmp %l2, %i0
ufs_idle_some+0x1a8: bl,pt %icc, -0x170 <ufs_idle_some+0x38>
ufs_idle_some+0x1ac: mov %l1, %o0
ufs_idle_some+0x1b0: ret
ufs_idle_some+0x1b4: restore
ufs_idle_some+0x1b8: mov %l1, %i0
ufs_idle_some+0x1bc: call -0x1f0794 <mutex_exit>
ufs_idle_some+0x1c0: restore
ufs_idle_some+0x1c4: ret
ufs_idle_some+0x1c8: restore
> rw_enter::dis
rw_enter: cmp %o1, 0
rw_enter+4: be,a,pn %icc, +0x4c <rw_enter+0x50>
rw_enter+8: or %g7, 4, %o5
rw_enter+0xc: ld [%g7 + 0x1d0], %o3
rw_enter+0x10: ldx [%o0], %o4
rw_enter+0x14: add %o3, 1, %o3
rw_enter+0x18: st %o3, [%g7 + 0x1d0]
rw_enter+0x1c: btst 6, %o4
rw_enter+0x20: be,pt %xcc, +0x14 <rw_enter+0x34>
rw_enter+0x24: add %o4, 8, %o5
rw_enter+0x28: sethi %hi(0x105fc00), %o2
rw_enter+0x2c: jmp %o2 + 0x158
rw_enter+0x30: nop
rw_enter+0x34: casx [%o0] , %o4, %o5
rw_enter+0x38: cmp %o4, %o5
rw_enter+0x3c: bne,pn %xcc, -0x20 <rw_enter+0x1c>
rw_enter+0x40: mov %o5, %o4
rw_enter+0x44: membar #LoadLoad
rw_enter+0x48: retl
rw_enter+0x4c: nop
rw_enter+0x50: casx [%o0] , %g0, %o5
rw_enter+0x54: brz,pt %o5, +0x14 <rw_enter+0x68>
rw_enter+0x58: membar #LoadLoad
rw_enter+0x5c: sethi %hi(0x105fc00), %o2
rw_enter+0x60: jmp %o2 + 0x158
rw_enter+0x64: nop
rw_enter+0x68: retl
rw_enter+0x6c: nop
> 0xf004763c::dis
mdb: failed to read instruction at f004763c: no mapping for address
> $r
%g0 = 0x0000000000000000 %l0 = 0x000000000183d7d0 panicstr
%g1 = 0x0000000001047b74 prom_enter_mon+0x24 %l1 = 0x0000000000000001
%g2 = 0x0000000000000000 %l2 = 0x0000000001810400
%g3 = 0x000000000183d400 cpunodes+0x1140 %l3 = 0x0000000000000001
%g4 = 0x0000000000000000 %l4 = 0x0000000000000016
%g5 = 0x0000000001074c00 unix`kssl_hold_ctx+0x28 %l5 = 0x000000000000000e
%g6 = 0x0000000000000016 %l6 = 0x0000000000000016
%g7 = 0x000002a100057cc0 %l7 = 0x000000000180c000 cpu0
%o0 = 0x0000000001000000 scb %i0 = 0x00000000f0059b78
%o1 = 0x0000000000000016 %i1 = 0x000002a100057828
%o2 = 0x00000000f0000000 %i2 = 0x0000000000000000
%o3 = 0x0000000000000000 %i3 = 0x000000000181c850 prom_cpu
%o4 = 0x0000000000000000 %i4 = 0x0000000001848400 swapfile+0x18
%o5 = 0x0000000000000001 %i5 = 0x00000000f0059b78
%o6 = 0x000002a100056ec1 %i6 = 0x000002a100056f71
%o7 = 0x0000000001007c38 client_handler+0x70 %i7 = 0x0000000001047b74 prom_enter_mon+0x24
%ccr = 0x44 xcc=nZvc icc=nZvc
%fprs = 0x00 fef=0 du=0 dl=0
%asi = 0x00
%y = 0x0000000000000000
%pc = 0x00000000f004763c
%npc = 0x00000000f0047640
%sp = 0x000002a100056ec1 unbiased=0x000002a1000576c0
%fp = 0x000002a100056f71
%tick = 0x0000000000000000
%tba = 0x0000000000000000
%tt = 0x17f
%tl = 0x0
%pil = 0xf
%pstate = 0x014 cle=0 tle=0 mm=TSO red=0 pef=1 am=0 priv=1 ie=0 ag=0
%cwp = 0x05 %cansave = 0x00
%canrestore = 0x00 %otherwin = 0x00
%wstate = 0x00 %cleanwin = 0x00
> 0x000000000181c850::dis
promplat_last_size: illtrap 0
promplat_last_size+4: illtrap 0
prom_cv: illtrap 0
0x181c834: illtrap 0
prom_mutex: illtrap 0
prom_mutex+4: illtrap 0
prom_holdcnt: illtrap 1
0x181c844: illtrap 0
prom_thread: illtrap 0x2a1
prom_thread+4: illtrap 0x57cc0
prom_cpu: illtrap 0
prom_cpu+4: fbn +0x30000 <dynseg+4>
vx_cmd: illtrap 0
vx_cmd+4: fbn +0x67080 <tpi_provinfo_table+4>
vx_cmd+8: illtrap 1
vx_cmd+0xc: illtrap 0
vx_cmd+0x10: illtrap 0
vx_cmd+0x14: sethi %hi(0x5f81000), %g0
vx_cmd+0x18: illtrap 0
vx_cmd+0x1c: illtrap 0
vx_cmd+0x20: illtrap 0
>
Thomas
|
|
|
0
|
|
|
|
Reply
|
Thomas
|
8/2/2007 11:11:55 AM
|
|
Thomas Glanzmann <sithglan@stud.uni-erlangen.de> writes:
> that is absolut correct. But Stop-A was pressed because the system hang
> in a lock aquire and was dead (it still answered pings becaus I guess
> ICMP replies are handled in a interrupt handler or function called by
> the interrupt handler). I could find out as much as that:
Do file a bug.
> As you can see ufs_idle_some+0x15c is a call to "rw_enter" and that is
> the last thing you can see. I have another crash dump with the exact
> same offset. That is why we pressed Stop-A.
>
> There seems to be a Bug Reports which match that offset:
>
> bug 6299287 and bug 4667688
Not sure how you matched those. 6299287 is a duplicate of 6344261,
which was fixed quite some time ago, and the fix is in S10 11/06
(actually, it was delivered before that). It's a panic in
ufs_putapage, not in ufs_idle_some's call to rw_enter.
> State of the latter is "State: 10-Fix Delivered". That useally means
> that it is fixed but not released. So I open a call to get the patch for
> this one.
4667688 is also a panic (not a hang) and doesn't appear to be related
to rw_enter. The fix for it was in S10 FCS (even longer ago), so
that's not the problem.
(The state you're seeing there appears to be for some S9 patch -- I'm
not sure what's going on with that, but it's unrelated to the release
you have.)
> Can someone tell me how I get the inode number of file name or the file
> that causes this out of the vnode or inode struct?
See the source here:
http://cvs.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/common/fs/ufs/ufs_thread.c#881
It's going through lists of freed inodes and discarding them. Looking
at the assembly code leading up to the rw_enter:
> ufs_idle_some+0x11c: cmp %g5, %l6
> ufs_idle_some+0x120: be,a,pt %xcc, -0x28 <ufs_idle_some+0xf8>
> ufs_idle_some+0x124: ld [%i5 + 4], %o7
> ufs_idle_some+0x128: ldx [%l6 + 0x10], %l3
> ufs_idle_some+0x12c: ldx [%l3 + 0xa0], %l4
Register l3 has the 'struct inode' pointer in it, and l4 has 'struct
vnode'. You'll need to dig into the stack frame ($C) to fetch those
values.
--
James Carlson, Solaris Networking <james.d.carlson@sun.com>
Sun Microsystems / 1 Network Drive 71.232W Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677
|
|
|
0
|
|
|
|
Reply
|
James
|
8/2/2007 12:15:42 PM
|
|
Hello,
> Do file a bug.
already filed a case with Sun.
> Not sure how you matched those. 6299287 is a duplicate of 6344261,
> which was fixed quite some time ago, and the fix is in S10 11/06
> (actually, it was delivered before that). It's a panic in
> ufs_putapage, not in ufs_idle_some's call to rw_enter.
I see. I just saw that the offset was the same.
> 4667688 is also a panic (not a hang) and doesn't appear to be related
> to rw_enter. The fix for it was in S10 FCS (even longer ago), so
> that's not the problem.
I see.
> See the source here:
> http://cvs.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/common/fs/ufs/ufs_thread.c#881
> It's going through lists of freed inodes and discarding them. Looking
> at the assembly code leading up to the rw_enter:
>> ufs_idle_some+0x11c: cmp %g5, %l6
>> ufs_idle_some+0x120: be,a,pt %xcc, -0x28 <ufs_idle_some+0xf8>
>> ufs_idle_some+0x124: ld [%i5 + 4], %o7
>> ufs_idle_some+0x128: ldx [%l6 + 0x10], %l3
>> ufs_idle_some+0x12c: ldx [%l3 + 0xa0], %l4
> Register l3 has the 'struct inode' pointer in it, and l4 has 'struct
> vnode'. You'll need to dig into the stack frame ($C) to fetch those
> values.
Thanks, I am going to do that.
Thomas
|
|
|
0
|
|
|
|
Reply
|
Thomas
|
8/2/2007 12:29:41 PM
|
|
Hello James,
> Register l3 has the 'struct inode' pointer in it, and l4 has 'struct
> vnode'. You'll need to dig into the stack frame ($C) to fetch those
> values.
thanks, I owe you a beer or two:
> ::stackregs
000002a100056ec1 0xf004763c(f0059b78, 2a100057828, 0, 181c850, 1848400, f0059b78)
%l0-%l3: 183d7d0 1 1810400 1
%l4-%l7: 16 e 16 180c000
prom_enter_mon+0x24: call +0x3734 <p1275_sparc_cif_handler>
000002a100056f71 prom_enter_mon+0x24(0, 0, 183b400, 1, 1812140, 181ae60)
%l0-%l3: 18539f0 300003d629a 0 1858c00
%l4-%l7: 1 1858c00 1811c00 70
debug_enter+0x118: call +0x3ae20 <prom_enter_mon>
000002a100057041 debug_enter+0x118(0, a, a, 180fc00, 0, 183d400)
%l0-%l3: 20000 0 100a 7
%l4-%l7: 60000358086 60000358080 2008e 0
abort_seq_softintr+0x94: jmpl %l7, %o7
000002a100057111 abort_seq_softintr+0x94(180fc00, 18a9400, 180c000, 2a100057d98, 2, 1857c00)
%l0-%l3: 181c460 45d007a7d67d 0 0
%l4-%l7: 0 18a9400 0 100cc18
intr_thread+0x170: jmpl %o5, %o7
000002a1000572d1 intr_thread+0x170(30002a5f510, 0, 11efc00, 2, 2a100157cc4, 0)
%l0-%l3: 10076e4 a 4480001600 80e
%l4-%l7: 1853b38 0 0 2a1001578b0
ufs_idle_some+0x15c: call -0x1f0714 <rw_enter>
000002a100157151 ufs_idle_some+0x15c(110b, 30002a5f430, 940, 0, 18beb80, 18bcc78)
%l0-%l3: 18bcc70 18bebc0 c22 30002a5f430
%l4-%l7: 600070e5b80 30002a5f510 2 1
ufs_thread_idle+0x1dc: call +0x58 <ufs_idle_some>
000002a100157201 ufs_thread_idle+0x1dc(0, 2, 18be800, 18be800, 100, 0)
%l0-%l3: 2216 2216 18be800 18be800
%l4-%l7: 18be800 2 2000000 2
thread_start+4: jmpl %i7, %o7
000002a1001572d1 thread_start+4(0, 0, 2c2912000001, 122ff04, 10c, 211e1200fff1)
%l0-%l3: 1e4 2c0512000001 122e7a4 190
%l4-%l7: 2c1112000001 122e2c8 4c 2c211200fff1
> 600070e5b80::print -t "struct vnode"
{
kmutex_t v_lock = {
void *[1] _opaque = [ 0 ]
}
uint_t v_flag = 0x10000
uint_t v_count = 0x2
void *v_data = 0x30002a5f430
struct vfs *v_vfsp = 0x6000082f5c0
struct stdata *v_stream = 0
enum vtype v_type = 1 (VREG)
dev_t v_rdev = 0xffffffffffffffff
struct vfs *v_vfsmountedhere = 0
struct vnodeops *v_op = 0x300003b4600
struct page *v_pages = 0
pgcnt_t v_npages = 0
pgcnt_t v_msnpages = 0
struct page *v_scanfront = 0
struct page *v_scanback = 0
struct filock *v_filocks = 0
struct shrlocklist *v_shrlocks = 0
krwlock_t v_nbllock = {
void *[1] _opaque = [ 0 ]
}
kcondvar_t v_cv = {
ushort_t _opaque = 0
}
void *v_locality = 0
struct fem_head *v_femhead = 0
char *v_path = 0x600076ae138 "/export/home2/schweiz/ef3bes00/viewstore/bs_gl_sp201_09.vws/.s/00054/800040c546b0850a__ADER.YDB"
uint_t v_rdcnt = 0
uint_t v_wrcnt = 0
u_longlong_t v_mmap_read = 0
u_longlong_t v_mmap_write = 0
void *v_mpssdata = 0
hrtime_t v_scantime = 0
ushort_t v_mset = 0
uint_t v_msflags = 0
struct vnode *v_msnext = 0
struct vnode *v_msprev = 0
krwlock_t v_mslock = {
void *[1] _opaque = [ 0 ]
}
}
-> v_path = /export/home2/schweiz/ef3bes00/viewstore/bs_gl_sp201_09.vws/.s/00054/800040c546b0850a__ADER.YDB"
So clearcase in involved, too. Great. I am going to analyze my other crash
dumps, too.
Thomas
|
|
|
0
|
|
|
|
Reply
|
Thomas
|
8/2/2007 1:24:48 PM
|
|
Hello,
the inode struct says:
krwlock_t i_contents = {
void *[1] _opaque = [ 0x2a100157cc4 ]
}
I guess that is where lock-up comes from. Could someone tell what does
the pointer tell me and where does it point to?
Thomas
|
|
|
0
|
|
|
|
Reply
|
Thomas
|
8/2/2007 1:34:58 PM
|
|
Hello,
my machine locked up again. But I booted with "boot -k" so I am now in
the kernel debugger. How can I save the core for future debugging?
Thomas
|
|
|
0
|
|
|
|
Reply
|
Thomas
|
8/2/2007 2:22:26 PM
|
|
Hello,
> my machine locked up again. But I booted with "boot -k" so I am now in
> the kernel debugger. How can I save the core for future debugging?
$<systemdump
Thomas
|
|
|
0
|
|
|
|
Reply
|
Thomas
|
8/2/2007 2:29:19 PM
|
|
Thomas Glanzmann <sithglan@stud.uni-erlangen.de> writes:
> the inode struct says:
>
> krwlock_t i_contents = {
> void *[1] _opaque = [ 0x2a100157cc4 ]
> }
>
> I guess that is where lock-up comes from. Could someone tell what does
> the pointer tell me and where does it point to?
It's the thread that has a write-lock the lock you're waiting on, +4.
See <sys/rwlock_impl.h>.
--
James Carlson, Solaris Networking <james.d.carlson@sun.com>
Sun Microsystems / 1 Network Drive 71.232W Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677
|
|
|
0
|
|
|
|
Reply
|
James
|
8/2/2007 4:59:35 PM
|
|
Hello James,
for me it looks like race condition. Because that only happens when the user on
the machine kicks the machine in the stomache when it is lying on the
floor. The machine is used for distributed comipiling and he is
compiling on the machine locally
> It's the thread that has a write-lock the lock you're waiting on, +4.
> See <sys/rwlock_impl.h>.
the not so funny thing is that the thread I am waiting on is the thread that is
trying to aquire the lock. (Note: This is from a crashdump that just occured)
> $C
000002a100047111 abort_seq_softintr+0x94(180fc00, 18a9400, 180c000, 2a100047d98, 2, 1857c00)
000002a1000472d1 intr_thread+0x170(600072af698, 0, 180c000, 2a10007fcc0, 0, fffe)
000002a10014f0a1 ufs_rmidle+8(600072af5b0, 0, 11efc00, 600072af698, 2a10014fcc4, 0)
000002a10014f151 ufs_idle_some+0x164(1149, 600072af5b0, 9a0, 0, 18e3800, 18e18f8)
000002a10014f201 ufs_thread_idle+0x1dc(0, 2, 18e3800, 18e3800, 100, 0)
000002a10014f2d1 thread_start+4(0, 0, 0, 0, 0, 0)
> ::regs
%g0 = 0x0000000000000000 %l0 = 0x0000000000020000
%g1 = 0x0000000000000000 %l1 = 0x0000000000000000
%g2 = 0x0000000000000000 %l2 = 0x000000000000100a
%g3 = 0x000000000183d400 cpunodes+0x1140 %l3 = 0x0000000000000000
%g4 = 0x0000000000000000 %l4 = 0x0000060000e96086
%g5 = 0x0000000001074c00 unix`kssl_hold_ctx+0x28 %l5 = 0x0000060000e96080
%g6 = 0x0000000000000000 %l6 = 0x000000000002008e
%g7 = 0x000002a100047cc0 %l7 = 0x0000000000000000
%o0 = 0x0000000000000000 %i0 = 0x0000000000000000
%o1 = 0x0000000000020040 %i1 = 0x000000000000000a
%o2 = 0x000000000183b400 lgrp_stats+0x13f0 %i2 = 0x000000000000000a
%o3 = 0x0000000000000001 %i3 = 0x000000000180fc00
%o4 = 0x0000000001237ea4 kmdbmod`kaif_enter %i4 = 0x0000000000000000
%o5 = 0x00000000018bef98 kmdbmod`kaif_dvec %i5 = 0x000000000183d400 cpunodes+0x1140
%o6 = 0x000002a100047041 %i6 = 0x000002a100047111
%o7 = 0x000000000100cd20 debug_enter+0x108 %i7 = 0x000000000100c9cc abort_seq_softintr+0x94
%ccr = 0x00 xcc=nzvc icc=nzvc
%fprs = 0x00 fef=0 du=0 dl=0
%asi = 0x00
%y = 0x0000000000000000
%pc = 0x0000000000000000
%npc = 0x0000000000000004
%sp = 0x000002a100047041 unbiased=0x000002a100047840
%fp = 0x000002a100047111
%tick = 0x0000000000000000
%tba = 0x0000000000000000
%tt = 0x9
%tl = 0x0
%pil = 0xc
%pstate = 0x016 cle=0 tle=0 mm=TSO red=0 pef=1 am=0 priv=1 ie=1 ag=0
%cwp = 0x06 %cansave = 0x00
%canrestore = 0x00 %otherwin = 0x00
%wstate = 0x00 %cleanwin = 0x00
> 600072af5b0::print -t "struct inode"
{
struct inode *[2] i_chain = [ 0x600064a7cc0, 0x600007b6ad0 ]
struct inode *i_freef = 0x600072af5b0
struct inode *i_freeb = 0x600072af5b0
struct icommon i_ic = {
o_mode_t ic_smode = 0
short ic_nlink = 0
o_uid_t ic_suid = 0x1c7
o_gid_t ic_sgid = 0x6ae
u_offset_t ic_lsize = 0
struct timeval32 ic_atime = {
time32_t tv_sec = 0x46b1e2e6
int32_t tv_usec = 0x5e9ce
}
struct timeval32 ic_mtime = {
time32_t tv_sec = 0x46b1e2f7
int32_t tv_usec = 0x8f415
}
struct timeval32 ic_ctime = {
time32_t tv_sec = 0x46b1e2f7
int32_t tv_usec = 0x8f415
}
daddr32_t [12] ic_db = [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ]
daddr32_t [3] ic_ib = [ 0, 0, 0 ]
int32_t ic_flags = 0
int32_t ic_blocks = 0
int32_t ic_gen = 0x32d4d053
int32_t ic_shadow = 0
uid_t ic_uid = 0x1c7
gid_t ic_gid = 0x6ae
uint32_t ic_oeftflag = 0
}
struct vnode *i_vnode = 0x60006d42080
struct vnode *i_devvp = 0x600015b0440
dev_t i_dev = 0x8800000007
ino_t i_number = 0x6acad
off_t i_diroff = 0
struct ufsvfs *i_ufsvfs = 0x300000b6200
struct dquot *i_dquot = 0
krwlock_t i_rwlock = {
void *[1] _opaque = [ 0 ]
}
krwlock_t i_contents = {
void *[1] _opaque = [ 0x2a10014fcc4 ]
}
kmutex_t i_tlock = {
void *[1] _opaque = [ 0x2a10014fcc0 ]
}
offset_t i_nextr = 0
uint_t i_flag = 0x60
uint_t i_seq = 0x126
cachedir_t i_cachedir = 1 (CD_ENABLED)
long i_mapcnt = 0
int *i_map = 0
dev_t i_rdev = 0
size_t i_delaylen = 0x6000
offset_t i_delayoff = 0x100000
offset_t i_nextrio = 0
long i_writes = 0
kcondvar_t i_wrcv = {
ushort_t _opaque = 0
}
offset_t i_doff = 0xe10cb680
si_t *i_ufs_acl = 0
dcanchor_t i_danchor = {
void *dca_dircache = 0
kmutex_t dca_lock = {
void *[1] _opaque = [ 0 ]
}
}
kthread_t *i_writer = 0
}
>
Btw have you an explanation why there can be on a single machine more
than one thread "onproc" if I have only one CPU. My way to explain that
is: With the Solaris Sparc Architecture it is possible that one
interrupt can interrupt (this is a verb) another interrupt. And all the
interrupt handlers are "onproc" when I call ::threads
> ::threads
ADDR STATE FLG PFLG SFLG PRI EPRI PIL INTR
000000000180e000 stopped 0 100 3e03 96 0 0 n/a
=> 000002a10001fcc0 onproc 8 0 3 -1 0 0 n/a
000002a100017cc0 run 8 0 13 60 0 0 n/a
000002a10000fcc0 sleep 8 0 3 60 0 0 n/a
000002a100007cc0 free 9 0 3 0 0 0 n/a
000002a10003fcc0 free 9 0 3 0 0 0 n/a
000002a100037cc0 free 9 0 3 0 0 0 n/a
000002a10002fcc0 free 9 0 3 0 0 0 n/a
000002a100027cc0 free 9 0 3 0 0 0 n/a
000002a10005fcc0 free 9 0 3 0 0 0 n/a
000002a100057cc0 free 9 0 3 0 0 0 n/a
000002a10004fcc0 free 9 0 3 169 0 10 2a100047cc0
=> 000002a100047cc0 onproc 809 0 3 169 0 10 2a10014fcc0
000002a10007fcc0 free 9 0 3 168 0 9 2a100047cc0
000002a10006fcc0 sleep 8 0 3 60 0 0 n/a
000002a100067cc0 sleep 8 0 3 60 0 0 n/a
000002a10009fcc0 sleep 8 0 3 60 0 0 n/a
000002a100097cc0 sleep 8 0 3 60 0 0 n/a
000002a10008fcc0 sleep 8 0 3 60 0 0 n/a
000002a100087cc0 sleep 8 0 3 60 0 0 n/a
000002a1000bfcc0 sleep 8 0 3 60 0 0 n/a
000002a1000b7cc0 sleep 8 0 3 60 0 0 n/a
000002a1000afcc0 sleep 8 0 3 60 0 0 n/a
000002a1000a7cc0 sleep 8 0 3 60 0 0 n/a
000002a1000dfcc0 sleep 8 0 3 60 0 0 n/a
000002a1000d7cc0 sleep 8 0 3 60 0 0 n/a
000002a1000cfcc0 sleep 8 0 3 60 0 0 n/a
000002a1000c7cc0 sleep 8 0 3 60 0 0 n/a
000002a1000ffcc0 sleep 8 0 3 60 0 0 n/a
000002a1000f7cc0 sleep 8 0 3 60 0 0 n/a
000002a1000efcc0 sleep 8 0 3 60 0 0 n/a
000002a1000e7cc0 sleep 8 0 3 60 0 0 n/a
000002a10011fcc0 sleep 8 0 3 60 0 0 n/a
000002a100117cc0 sleep 8 0 3 60 0 0 n/a
000002a10010fcc0 run 8 0 13 99 0 0 n/a
000002a100107cc0 run 8 0 13 99 0 0 n/a
000002a1003c1cc0 run 8 0 13 60 0 0 n/a
000002a1010efcc0 run 8 0 13 60 0 0 n/a
000002a1010a7cc0 run 8 0 13 60 0 0 n/a
000002a10013fcc0 run 8 0 13 60 0 0 n/a
000002a100227cc0 sleep 8 0 3 60 0 0 n/a
000002a10025fcc0 sleep 8 0 3 60 0 0 n/a
000002a100257cc0 sleep 8 0 3 60 0 0 n/a
000002a10024fcc0 sleep 8 0 3 60 0 0 n/a
000002a100267cc0 sleep 8 0 3 60 0 0 n/a
000002a10029fcc0 sleep 8 0 3 60 0 0 n/a
000002a100297cc0 sleep 8 0 3 60 0 0 n/a
000002a10028fcc0 sleep 8 0 3 60 0 0 n/a
000002a100287cc0 sleep 8 0 3 60 0 0 n/a
000002a1003c9cc0 sleep 8 0 3 60 0 0 n/a
000002a100473cc0 run 8 0 13 60 0 0 n/a
000002a10046bcc0 sleep 8 0 3 60 0 0 n/a
000002a10012fcc0 sleep 8 0 3 60 0 0 n/a
000002a100127cc0 run 8 0 13 60 0 0 n/a
000002a10015fcc0 sleep 8 0 3 60 0 0 n/a
000002a100157cc0 sleep 8 0 3 60 0 0 n/a
=> 000002a10014fcc0 onproc 8 0 3 60 0 0 n/a
000002a100147cc0 sleep 8 0 3 60 0 0 n/a
000002a10017fcc0 sleep 8 0 3 60 0 0 n/a
000002a100177cc0 sleep 8 0 3 60 0 0 n/a
000002a10016fcc0 sleep 8 0 3 60 0 0 n/a
000002a100167cc0 sleep 8 0 3 60 0 0 n/a
000002a10019fcc0 sleep 8 0 3 60 0 0 n/a
000002a100197cc0 sleep 8 0 3 60 0 0 n/a
000002a10018fcc0 sleep 8 0 3 60 0 0 n/a
000002a100187cc0 sleep 8 0 3 60 0 0 n/a
000002a1001bfcc0 run 8 0 13 60 0 0 n/a
000002a1001b7cc0 sleep 8 0 3 98 0 0 n/a
000002a1001afcc0 sleep 8 0 3 60 0 0 n/a
000002a1001a7cc0 sleep 8 0 3 60 0 0 n/a
000002a1001dfcc0 sleep 8 0 3 60 0 0 n/a
000002a1001d7cc0 sleep 8 0 3 60 0 0 n/a
000002a1001cfcc0 sleep 8 0 3 60 0 0 n/a
000002a100695cc0 sleep 8 0 3 60 0 0 n/a
000002a1009c9cc0 sleep 8 0 3 60 0 0 n/a
000002a1001ffcc0 sleep 8 0 3 60 0 0 n/a
000002a1001f7cc0 sleep 8 0 3 60 0 0 n/a
000002a1001efcc0 sleep 8 0 3 60 0 0 n/a
000002a1001e7cc0 sleep 8 0 3 60 0 0 n/a
000002a10021fcc0 sleep 8 0 3 60 0 0 n/a
000002a100217cc0 sleep 8 0 3 60 0 0 n/a
000002a10020fcc0 sleep 8 0 3 60 0 0 n/a
000002a10053dcc0 sleep 8 0 3 60 0 0 n/a
000002a100535cc0 sleep 8 0 3 60 0 0 n/a
000002a10052dcc0 sleep 8 0 3 60 0 0 n/a
000002a100525cc0 sleep 8 0 3 60 0 0 n/a
000002a10055dcc0 sleep 8 0 3 60 0 0 n/a
000002a100555cc0 sleep 8 0 3 60 0 0 n/a
000002a10054dcc0 sleep 8 0 3 60 0 0 n/a
000002a100545cc0 sleep 8 0 3 60 0 0 n/a
000002a10057dcc0 sleep 8 0 3 60 0 0 n/a
000002a100575cc0 sleep 8 0 3 60 0 0 n/a
000002a10056dcc0 sleep 8 0 3 60 0 0 n/a
000002a100565cc0 sleep 8 0 3 60 0 0 n/a
000002a10059dcc0 sleep 8 0 3 60 0 0 n/a
000002a100595cc0 sleep 8 0 3 60 0 0 n/a
000002a10058dcc0 sleep 8 0 3 60 0 0 n/a
000002a100585cc0 sleep 8 0 3 60 0 0 n/a
000002a1005bdcc0 sleep 8 0 3 60 0 0 n/a
000002a1005b5cc0 sleep 8 0 3 60 0 0 n/a
000002a1005adcc0 sleep 8 0 3 60 0 0 n/a
000002a1005a5cc0 sleep 8 0 3 60 0 0 n/a
000002a1005ddcc0 sleep 8 0 3 60 0 0 n/a
000002a1005d5cc0 sleep 8 0 3 60 0 0 n/a
000002a1005cdcc0 sleep 8 0 3 60 0 0 n/a
000002a1005c5cc0 sleep 8 0 3 60 0 0 n/a
000002a1005fdcc0 sleep 8 0 3 60 0 0 n/a
000002a1005f5cc0 sleep 8 0 3 60 0 0 n/a
000002a1005edcc0 sleep 8 0 3 60 0 0 n/a
000002a1005e5cc0 sleep 8 0 3 60 0 0 n/a
000002a10061dcc0 sleep 8 0 3 60 0 0 n/a
000002a10022fcc0 sleep 8 0 3 60 0 0 n/a
000002a100207cc0 sleep 8 0 3 60 0 0 n/a
000002a10023fcc0 sleep 8 0 3 60 0 0 n/a
000002a100277cc0 sleep 8 0 3 60 0 0 n/a
000002a10026fcc0 sleep 8 0 3 60 0 0 n/a
000002a100247cc0 sleep 8 0 3 60 0 0 n/a
000002a10027fcc0 sleep 8 0 3 60 0 0 n/a
000002a100237cc0 sleep 8 0 3 60 0 0 n/a
000002a1003e9cc0 sleep 8 0 3 60 0 0 n/a
000002a100635cc0 sleep 8 0 3 60 0 0 n/a
000002a10062dcc0 sleep 8 0 3 60 0 0 n/a
000002a100625cc0 stopped 8 0 3 109 0 0 n/a
000002a10063dcc0 sleep 8 0 3 98 0 0 n/a
00000300007e63a0 sleep 1002 100 13 59 0 0 n/a
00000300007e6080 sleep 1002 100 13 59 0 0 n/a
00000300007eb2c0 sleep 1002 100 3 59 0 0 n/a
00000300007eafa0 sleep 1002 100 3 59 0 0 n/a
00000300007eac80 sleep 1002 100 3 59 0 0 n/a
00000300003b0ca0 sleep 1002 100 13 59 0 0 n/a
00000300003b0020 sleep 1002 100 13 59 0 0 n/a
00000300007b86c0 sleep 1002 100 13 59 0 0 n/a
00000300007b8d00 sleep 1002 100 13 59 0 0 n/a
00000300007b9340 sleep 1002 100 13 59 0 0 n/a
00000300007b7000 sleep 1002 100 13 59 0 0 n/a
00000300007b83a0 sleep 1002 100 13 59 0 0 n/a
00000300007b89e0 sleep 1002 100 13 59 0 0 n/a
00000300007b9020 sleep 1002 100 13 59 0 0 n/a
00000300007c0fc0 run 1002 101 3 59 0 0 n/a
00000300007b6ce0 sleep 1002 100 13 59 0 0 n/a
00000300003b0660 sleep 1002 100 1 59 0 0 n/a
00000300003b0340 sleep 1002 100 3 59 0 0 n/a
00000300003b0980 sleep 1002 104 1 59 0 0 n/a
00000300003b12e0 sleep 1002 100 3 59 0 0 n/a
00000300007b3c60 sleep 1002 100 1 59 0 0 n/a
000002a100791cc0 sleep 8 0 3 60 0 0 n/a
000003000201dca0 sleep 1002 100 3 59 0 0 n/a
00000300007d09a0 sleep 1002 100 13 59 0 0 n/a
00000300007b6380 sleep 1002 100 3 59 0 0 n/a
00000300007d0680 run 1002 104 1 59 0 0 n/a
00000300007f09a0 sleep 1002 104 3 59 0 0 n/a
00000300007b7320 sleep 1002 104 1 59 0 0 n/a
00000300007dd960 sleep 1002 104 1 59 0 0 n/a
000002a1006bdcc0 run 8 0 3 60 0 0 n/a
000002a1006b5cc0 sleep 8 0 3 99 0 0 n/a
000002a100799cc0 sleep 8 0 3 99 0 0 n/a
000002a100789cc0 sleep 8 0 3 99 0 0 n/a
000002a100881cc0 sleep 8 0 3 99 0 0 n/a
000002a100841cc0 sleep 8 0 3 99 0 0 n/a
000002a100819cc0 sleep 8 0 3 99 0 0 n/a
000002a100821cc0 sleep 8 0 3 99 0 0 n/a
000002a100829cc0 sleep 8 0 3 99 0 0 n/a
000002a1007f1cc0 run 8 0 3 60 0 0 n/a
000002a100929cc0 run 8 0 3 60 0 0 n/a
000002a100387cc0 sleep 8 0 3 60 0 0 n/a
00000300007ef600 sleep 1002 104 1 59 0 0 n/a
00000300007ef2e0 sleep 1002 104 1 59 0 0 n/a
00000300007eeca0 sleep 1002 104 1 59 0 0 n/a
00000300007efc40 sleep 1002 100 3 59 0 0 n/a
00000300007ddc80 sleep 1002 100 3 59 0 0 n/a
000003000192c040 sleep 1002 100 1 59 0 0 n/a
000003000192cfe0 sleep 1002 104 1 59 0 0 n/a
00000300019f9000 sleep 1002 100 3 59 0 0 n/a
00000300019f9c80 sleep 1002 100 3 59 0 0 n/a
00000300019f9960 sleep 1002 104 1 59 0 0 n/a
000003000192c360 sleep 1002 104 1 59 0 0 n/a
00000300007f1620 sleep 1002 104 1 59 0 0 n/a
00000300007e66c0 sleep 1002 104 1 59 0 0 n/a
00000300007c1920 sleep 1002 104 1 59 0 0 n/a
00000300007d1c60 sleep 1002 104 1 59 0 0 n/a
00000300007c0660 sleep 1002 104 1 59 0 0 n/a
00000300007dcce0 sleep 1002 104 1 59 0 0 n/a
00000300007dc6a0 sleep 1002 104 1 59 0 0 n/a
00000300007b7640 sleep 1002 104 1 59 0 0 n/a
000002a100779cc0 sleep 8 0 3 60 0 0 n/a
00000300007ef920 sleep 1002 104 1 59 0 0 n/a
00000300007d0360 sleep 1002 100 13 59 0 0 n/a
00000300007f0cc0 sleep 1002 100 13 59 0 0 n/a
00000300007e69e0 run 1002 101 3 59 0 0 n/a
0000030001f872e0 sleep 1002 100 3 59 0 0 n/a
0000030001f86fc0 sleep 1002 100 3 59 0 0 n/a
0000030001f86ca0 sleep 1002 100 3 59 0 0 n/a
0000030001f86980 sleep 1002 100 3 59 0 0 n/a
0000030001f86660 sleep 1002 100 3 59 0 0 n/a
0000030001f86340 sleep 1002 100 13 59 0 0 n/a
00000300007c0980 sleep 1002 100 3 59 0 0 n/a
00000300007ee980 sleep 1002 100 3 59 0 0 n/a
00000300007ee020 run 1002 104 3 59 0 0 n/a
00000300007c1600 sleep 1002 104 3 59 0 0 n/a
00000300007f0fe0 sleep 1002 104 3 59 0 0 n/a
00000300007e7980 run 1002 104 3 59 0 0 n/a
00000300007e7340 sleep 1002 104 3 59 0 0 n/a
00000300007e7660 sleep 1002 104 3 59 0 0 n/a
00000300007bcfa0 run 1002 104 3 59 0 0 n/a
00000300007bd2c0 run 1002 104 3 59 0 0 n/a
00000300007bd5e0 run 1002 104 3 59 0 0 n/a
00000300007b29a0 run 1002 104 3 59 0 0 n/a
00000300007f0360 run 1002 104 3 59 0 0 n/a
00000300007eb900 run 1002 104 3 59 0 0 n/a
00000300007dd640 run 1002 104 3 59 0 0 n/a
00000300007d0cc0 run 1002 104 3 59 0 0 n/a
00000300007d0fe0 run 1002 104 3 59 0 0 n/a
00000300007d1300 run 1002 104 3 59 0 0 n/a
00000300007d1620 run 1002 104 3 59 0 0 n/a
00000300007d1940 run 1002 104 3 59 0 0 n/a
00000300007bc960 sleep 1002 104 3 59 0 0 n/a
00000300007ee660 sleep 1002 104 1 59 0 0 n/a
00000300007ea000 sleep 1002 101 1 59 0 0 n/a
00000300007dc9c0 sleep 1006 101 1 59 0 0 n/a
00000300007b66a0 sleep 1006 104 1 59 0 0 n/a
000002a1007e1cc0 sleep 8 0 3 60 0 0 n/a
000002a1007e9cc0 sleep 8 0 3 60 0 0 n/a
00000300018c0080 sleep 1002 100 3 59 0 0 n/a
0000030001cf4640 run 1002 100 3 60 0 0 n/a
0000030001f87600 run 1002 100 3 60 0 0 n/a
00000300007dc380 sleep 1002 100 3 60 0 0 n/a
00000300007b2680 sleep 1002 104 1 59 0 0 n/a
000002a1006fdcc0 run 8 0 3 60 0 0 n/a
000002a1006edcc0 sleep 8 0 3 60 0 0 n/a
000002a1006f5cc0 run 8 0 3 60 0 0 n/a
000002a1006e5cc0 sleep 8 0 3 60 0 0 n/a
00000300007dd320 run 1002 104 11 59 0 0 n/a
00000300007bc320 sleep 1002 104 1 59 0 0 n/a
0000030001fa1c60 sleep 1002 100 1 59 0 0 n/a
0000030001a85c20 sleep 1002 100 1 59 0 0 n/a
00000300007b9980 sleep 1002 104 1 59 0 0 n/a
00000300007bc640 run 1002 104 3 59 0 0 n/a
00000300007bd900 sleep 1002 104 1 59 0 0 n/a
00000300007c0ca0 sleep 1002 104 1 59 0 0 n/a
00000300007eb5e0 sleep 1002 104 1 59 0 0 n/a
00000300018a3960 sleep 1002 104 1 59 0 0 n/a
00000300007c12e0 sleep 1002 104 1 59 0 0 n/a
000002a100919cc0 sleep 8 2000 3 60 0 0 n/a
00000300007f0040 sleep 1002 100 3 60 0 0 n/a
00000300007b8080 sleep 1002 104 1 60 0 0 n/a
000002a1006ddcc0 sleep 8 2000 3 60 0 0 n/a
00000300007b2040 sleep 1002 100 3 60 0 0 n/a
00000300007bcc80 sleep 1002 104 1 60 0 0 n/a
000002a1006d5cc0 sleep 8 0 3 99 0 0 n/a
00000300018a29c0 run 1002 101 3 59 0 0 n/a
00000300018c1ca0 run 1002 100 3 59 0 0 n/a
00000300018c1660 sleep 1002 100 3 59 0 0 n/a
00000300018a3000 sleep 1002 104 1 59 0 0 n/a
00000300018c0d00 sleep 1002 100 3 59 0 0 n/a
00000300018c1340 sleep 1002 100 3 59 0 0 n/a
000002a1006cdcc0 sleep 8 0 3 60 0 0 n/a
00000300007b2cc0 run 1002 101 3 59 0 0 n/a
0000030001976980 run 1002 104 1 59 0 0 n/a
0000030001cf5c20 sleep 1002 104 1 59 0 0 n/a
00000300018c1020 sleep 1002 104 1 59 0 0 n/a
00000300018a3320 sleep 1002 104 3 59 0 0 n/a
0000030001ff9640 sleep 1002 100 1 59 0 0 n/a
00000300007e6d00 sleep 1002 104 1 59 0 0 n/a
000002a100a4fcc0 sleep 8 2000 3 60 0 0 n/a
00000300018d7900 sleep 1002 100 3 60 0 0 n/a
00000300007f1940 run 1002 104 11 60 0 0 n/a
00000300007d0040 sleep 1002 100 3 59 0 0 n/a
00000300018d7c20 run 1002 101 3 59 0 0 n/a
000002a100d7fcc0 sleep 8 2000 3 60 0 0 n/a
000002a100e2fcc0 sleep 8 2000 3 60 0 0 n/a
000002a100e97cc0 sleep 8 2000 3 60 0 0 n/a
000002a1013e7cc0 sleep 8 2000 3 60 0 0 n/a
000002a10142fcc0 run 8 2000 3 60 0 0 n/a
000002a100cb1cc0 run 8 2000 3 60 0 0 n/a
000002a100c91cc0 run 8 2000 3 60 0 0 n/a
000002a1012cfcc0 run 8 2000 3 60 0 0 n/a
000002a100d77cc0 run 8 2000 3 60 0 0 n/a
000002a100c3dcc0 run 8 2000 3 60 0 0 n/a
000002a101007cc0 run 8 2000 3 60 0 0 n/a
000002a10141fcc0 run 8 2000 3 60 0 0 n/a
000002a100751cc0 sleep 8 2000 3 60 0 0 n/a
000002a101017cc0 sleep 8 2000 3 60 0 0 n/a
000002a1013c7cc0 run 8 2000 3 60 0 0 n/a
000002a100e17cc0 run 8 2000 3 60 0 0 n/a
000002a101027cc0 run 8 2000 3 60 0 0 n/a
000002a100de7cc0 run 8 2000 3 60 0 0 n/a
000002a100ebfcc0 run 8 2000 3 60 0 0 n/a
000002a1012f7cc0 run 8 2000 3 60 0 0 n/a
000002a100e27cc0 run 8 2000 3 60 0 0 n/a
000002a100cdfcc0 run 8 2000 3 60 0 0 n/a
000002a1012e7cc0 sleep 8 2000 3 60 0 0 n/a
000002a100c71cc0 sleep 8 2000 3 60 0 0 n/a
000002a1012a7cc0 sleep 8 2000 3 60 0 0 n/a
0000030001a855e0 sleep 1002 100 1 59 0 0 n/a
000002a100b61cc0 run 8 2000 3 60 0 0 n/a
000002a1012ffcc0 sleep 8 2000 3 60 0 0 n/a
00000300007dc060 sleep 1002 104 1 59 0 0 n/a
00000300007b6060 sleep 1002 104 1 59 0 0 n/a
000002a1006c5cc0 run 8 2000 3 60 0 0 n/a
00000300018a2380 sleep 1002 104 1 59 0 0 n/a
00000300018a3640 sleep 1002 104 1 59 0 0 n/a
00000300007e7ca0 sleep 1002 104 1 59 0 0 n/a
00000300018d6640 sleep 1002 104 1 59 0 0 n/a
00000300018c03a0 sleep 1002 104 1 59 0 0 n/a
00000300018fd920 sleep 1002 104 1 59 0 0 n/a
00000300018fd600 sleep 1002 104 1 59 0 0 n/a
00000300018fd2e0 sleep 1002 104 1 59 0 0 n/a
00000300018fcca0 sleep 1002 104 1 59 0 0 n/a
00000300007ebc20 sleep 1002 100 3 59 0 0 n/a
00000300018fcfc0 sleep 1002 100 3 59 0 0 n/a
00000300018fc980 sleep 1002 104 1 59 0 0 n/a
00000300018fc660 sleep 1002 104 1 59 0 0 n/a
00000300007f1300 sleep 1002 104 1 59 0 0 n/a
0000030001bf9940 sleep 1002 104 3 59 0 0 n/a
0000030001c75960 sleep 1006 104 1 59 0 0 n/a
00000300019f89c0 sleep 1002 104 1 59 0 0 n/a
0000030001c75c80 sleep 1002 104 1 59 0 0 n/a
0000030001caf660 sleep 1002 104 3 59 0 0 n/a
0000030001f1dc20 sleep 1002 104 1 59 0 0 n/a
000003000207efa0 sleep 1002 104 1 59 0 0 n/a
0000030001e44cc0 run 1002 104 11 59 0 0 n/a
00000300020bc020 sleep 1002 104 1 59 0 0 n/a
0000030001cf5900 sleep 1002 104 3 59 0 0 n/a
00000300020bc660 sleep 1006 104 1 59 0 0 n/a
00000300018c1980 run 1002 104 11 59 0 0 n/a
0000030001e45940 sleep 1002 104 1 59 0 0 n/a
0000030001d9dc40 sleep 1002 104 3 59 0 0 n/a
0000030001cf4fa0 sleep 1002 104 1 59 0 0 n/a
0000030001e9a380 sleep 1002 104 1 59 0 0 n/a
000002a101257cc0 sleep 8 0 3 60 0 0 n/a
000002a10138fcc0 sleep 8 0 3 60 0 0 n/a
000002a101167cc0 sleep 8 0 3 60 0 0 n/a
000002a100f1fcc0 sleep 8 0 3 60 0 0 n/a
000002a101267cc0 sleep 8 0 3 60 0 0 n/a
000002a101347cc0 sleep 8 0 3 60 0 0 n/a
000002a10133fcc0 sleep 8 0 3 60 0 0 n/a
000002a100c15cc0 sleep 8 0 3 60 0 0 n/a
000002a10125fcc0 sleep 8 0 3 60 0 0 n/a
000002a100483cc0 sleep 8 0 3 60 0 0 n/a
000002a100f57cc0 sleep 8 0 3 60 0 0 n/a
000002a100d67cc0 sleep 8 0 3 99 0 0 n/a
000002a101097cc0 sleep 8 0 3 60 0 0 n/a
000002a100cefcc0 sleep 8 0 3 60 0 0 n/a
000002a101317cc0 sleep 8 0 3 60 0 0 n/a
00000300007ea960 sleep 1002 104 3 59 0 0 n/a
0000030001fa0fe0 sleep 1002 104 1 59 0 0 n/a
0000030001fa1940 sleep 1002 104 1 59 0 0 n/a
00000300020bc980 run 1002 104 11 59 0 0 n/a
0000030001fa1300 sleep 1002 104 1 59 0 0 n/a
000003000192d300 sleep 1002 104 1 59 0 0 n/a
00000300018d72c0 sleep 1002 104 1 59 0 0 n/a
000003000192c9a0 run 1002 104 3 59 0 0 n/a
000003000192c680 sleep 1002 104 1 59 0 0 n/a
0000030001f87c40 sleep 1002 104 3 59 0 0 n/a
00000300007ea640 sleep 1002 104 3 59 0 0 n/a
00000300007ea320 sleep 1002 104 1 59 0 0 n/a
00000300007b9ca0 sleep 1002 104 3 59 0 0 n/a
00000300007eefc0 sleep 1002 104 1 59 0 0 n/a
000002a100b41cc0 sleep 8 0 3 60 0 0 n/a
000003000192d940 sleep 1002 104 1 59 0 0 n/a
00000300007b9660 sleep 1002 104 1 59 0 0 n/a
00000300007b7c80 run 1002 101 3 59 0 0 n/a
00000300007e7020 sleep 1002 104 1 59 0 0 n/a
000003000192d620 sleep 1002 104 1 59 0 0 n/a
00000300018a2ce0 sleep 1002 104 1 59 0 0 n/a
00000300018a26a0 sleep 1002 104 1 59 0 0 n/a
00000300007f1c60 sleep 1002 104 1 59 0 0 n/a
00000300018d75e0 sleep 1002 104 1 59 0 0 n/a
00000300019f9640 sleep 1002 104 1 59 0 0 n/a
00000300019f9320 sleep 1002 104 1 59 0 0 n/a
00000300007b2360 sleep 1006 104 1 59 0 0 n/a
000002a100bf5cc0 sleep 8 0 3 60 0 0 n/a
000002a100b81cc0 sleep 8 0 3 60 0 0 n/a
000002a100bbdcc0 sleep 8 0 3 97 0 0 n/a
000002a100c0dcc0 sleep 8 0 3 97 0 0 n/a
000002a100c05cc0 sleep 8 0 3 97 0 0 n/a
000002a100bfdcc0 sleep 8 0 3 97 0 0 n/a
000002a100be5cc0 sleep 8 0 3 97 0 0 n/a
000002a100b59cc0 sleep 8 0 3 97 0 0 n/a
000002a100b51cc0 sleep 8 0 3 97 0 0 n/a
000002a100c89cc0 sleep 8 0 3 97 0 0 n/a
000002a100c81cc0 sleep 8 0 3 97 0 0 n/a
000002a100c79cc0 sleep 8 0 3 97 0 0 n/a
000002a100bddcc0 sleep 8 0 3 60 0 0 n/a
000002a100cf7cc0 sleep 8 0 3 97 0 0 n/a
000003000192ccc0 sleep 1002 104 3 59 0 0 n/a
000002a100d0fcc0 run 8 0 13 60 0 0 n/a
000002a100d07cc0 sleep 8 2000 3 60 0 0 n/a
00000300007b7960 sleep 1002 100 3 59 0 0 n/a
00000300018d6000 sleep 1002 104 1 59 0 0 n/a
00000300007c0020 sleep 1002 104 1 59 0 0 n/a
00000300018a2060 sleep 1006 104 1 100 0 0 n/a
000003000192dc60 run 1002 104 3 59 0 0 n/a
0000030001e45620 run 1002 104 3 59 0 0 n/a
0000030001976340 run 1002 104 3 59 0 0 n/a
00000300018fc020 run 1002 104 3 59 0 0 n/a
00000300018d6c80 run 1002 104 3 59 0 0 n/a
0000030001a763a0 sleep 1002 104 1 59 0 0 n/a
00000300018d6fa0 sleep 1002 104 1 59 0 0 n/a
00000300007ee340 run 1002 104 13 59 0 0 n/a
0000030001a77980 run 1202 100 3 59 0 0 n/a
0000030001a76080 sleep 1202 100 3 59 0 0 n/a
0000030001a77020 sleep 1202 100 3 59 0 0 n/a
00000300019f8380 sleep 1002 100 1 59 0 0 n/a
00000300007f0680 sleep 1202 100 13 58 0 0 n/a
00000300019f86a0 sleep 1202 100 3 59 0 0 n/a
00000300018c06c0 sleep 1202 100 3 58 0 0 n/a
0000030001a766c0 sleep 1202 100 3 59 0 0 n/a
0000030001a77340 run 1002 100 3 59 0 0 n/a
0000030001a84fa0 run 1002 100 3 0 0 0 n/a
0000030001a85900 run 1002 100 3 59 0 0 n/a
0000030001a852c0 sleep 1202 100 3 59 0 0 n/a
0000030001a84c80 sleep 1202 100 3 59 0 0 n/a
0000030001a84960 sleep 1202 100 3 59 0 0 n/a
0000030001a84640 sleep 1202 100 3 59 0 0 n/a
0000030001a84320 sleep 1002 100 1 59 0 0 n/a
0000030001a84000 run 1202 100 3 59 0 0 n/a
0000030001977c40 sleep 1202 100 3 59 0 0 n/a
0000030001977920 sleep 1202 100 3 59 0 0 n/a
0000030001977600 sleep 1202 100 3 59 0 0 n/a
00000300019772e0 sleep 1002 100 1 59 0 0 n/a
0000030001976fc0 sleep 1202 100 13 59 0 0 n/a
0000030001976ca0 run 1202 100 3 59 0 0 n/a
0000030001a77ca0 sleep 1002 104 1 59 0 0 n/a
000002a10132fcc0 sleep 8 0 3 60 0 0 n/a
000003000213f300 run 1002 104 3 59 0 0 n/a
000003000207e000 run 1002 104 13 59 0 0 n/a
00000300018d6320 sleep 1002 104 1 59 0 0 n/a
0000030001e449a0 sleep 1002 104 1 59 0 0 n/a
00000300018d6960 sleep 1002 104 1 59 0 0 n/a
0000030001f1cfa0 run 1002 100 3 59 0 0 n/a
000003000207ec80 run 1002 100 3 59 0 0 n/a
00000300018fdc40 run 1002 100 3 59 0 0 n/a
0000030001e9b320 run 1002 100 3 59 0 0 n/a
00000300020bcca0 run 1002 104 3 59 0 0 n/a
00000300018fc340 sleep 1002 104 3 59 0 0 n/a
000003000207e320 sleep 1002 104 1 59 0 0 n/a
0000030001fa0040 run 1000 104 11 49 0 0 n/a
0000030001fa1620 sleep 1002 104 1 49 0 0 n/a
000003000207e640 sleep 1002 104 1 49 0 0 n/a
0000030001fa0680 sleep 1002 104 1 49 0 0 n/a
0000030001cf4c80 sleep 1002 104 1 49 0 0 n/a
0000030001cf4960 run 1002 104 3 49 0 0 n/a
0000030001ff8380 sleep 1002 104 1 49 0 0 n/a
0000030001ff89c0 run 1002 104 3 49 0 0 n/a
0000030001ff8060 sleep 1006 104 1 49 0 0 n/a
0000030001f1d5e0 sleep 1006 104 1 49 0 0 n/a
00000300020bdc40 sleep 1006 104 1 49 0 0 n/a
0000030001ff9320 sleep 1002 104 1 49 0 0 n/a
0000030001976660 sleep 1002 104 3 49 0 0 n/a
0000030001f1c960 sleep 1002 104 1 49 0 0 n/a
0000030001c75640 sleep 1002 104 3 59 0 0 n/a
000003000207fc20 sleep 1002 104 1 59 0 0 n/a
0000030001c74060 sleep 1002 104 3 49 0 0 n/a
000003000201d340 sleep 1002 104 1 49 0 0 n/a
0000030001976020 sleep 1002 104 3 49 0 0 n/a
0000030001fa0cc0 sleep 1002 104 1 49 0 0 n/a
0000030001a76d00 sleep 1002 104 3 49 0 0 n/a
0000030001f1c000 sleep 1002 104 1 49 0 0 n/a
000003000207f900 sleep 1002 104 3 49 0 0 n/a
00000300019f8060 sleep 1002 104 1 49 0 0 n/a
00000300020bcfc0 sleep 1002 104 3 49 0 0 n/a
0000030001d9cca0 sleep 1002 104 1 49 0 0 n/a
0000030001caf980 sleep 1002 104 1 49 0 0 n/a
0000030001a769e0 run 1002 104 3 59 0 0 n/a
0000030001edad00 sleep 1002 104 1 49 0 0 n/a
0000030001eda9e0 sleep 1002 104 1 59 0 0 n/a
0000030001fa0360 sleep 1002 104 1 59 0 0 n/a
0000030001edb020 sleep 1002 104 1 59 0 0 n/a
0000030001eda3a0 sleep 1002 104 1 59 0 0 n/a
0000030001cae3a0 sleep 1002 104 1 49 0 0 n/a
000003000207e960 sleep 1002 104 3 49 0 0 n/a
000003000201d660 sleep 1006 104 1 49 0 0 n/a
0000030001ff9000 sleep 1002 104 1 49 0 0 n/a
0000030001f1d900 sleep 1002 104 3 49 0 0 n/a
0000030001bf8cc0 sleep 1006 104 1 49 0 0 n/a
00000300020bd600 sleep 1002 104 1 49 0 0 n/a
0000030001bf9c60 sleep 1002 104 3 49 0 0 n/a
0000030001f86020 sleep 1002 104 1 49 0 0 n/a
0000030001f1cc80 sleep 1002 104 3 49 0 0 n/a
00000300020bd2e0 sleep 1006 104 1 49 0 0 n/a
0000030001ff86a0 sleep 1002 104 1 49 0 0 n/a
0000030001e44360 sleep 1002 104 3 49 0 0 n/a
0000030001cae080 sleep 1002 104 1 49 0 0 n/a
0000030001edb660 sleep 1002 104 3 49 0 0 n/a
0000030001d9c020 sleep 1002 104 1 59 0 0 n/a
0000030001edb340 sleep 1002 104 3 49 0 0 n/a
0000030001e9bc80 sleep 1002 104 1 59 0 0 n/a
0000030001eda6c0 sleep 1002 104 3 49 0 0 n/a
0000030001fa09a0 sleep 1002 104 1 59 0 0 n/a
000003000207f5e0 run 1002 104 3 49 0 0 n/a
0000030001e9a6a0 sleep 1002 104 1 49 0 0 n/a
0000030001e9ace0 sleep 1002 104 1 49 0 0 n/a
0000030001edb980 sleep 1002 104 1 49 0 0 n/a
0000030001c75000 sleep 1006 104 1 59 0 0 n/a
0000030001bf8040 sleep 1002 104 1 59 0 0 n/a
0000030001cf52c0 sleep 1002 104 1 59 0 0 n/a
0000030001eda080 sleep 1002 104 1 59 0 0 n/a
00000300007b69c0 sleep 1002 104 1 59 0 0 n/a
00000300007b3940 sleep 1002 100 1 59 0 0 n/a
00000300007b3620 sleep 1002 100 3 59 0 0 n/a
00000300007b3300 sleep 1002 100 1 59 0 0 n/a
00000300007bdc20 run 1002 101 3 59 0 0 n/a
000002a1006adcc0 sleep 8 0 3 60 0 0 n/a
000002a1006a5cc0 sleep 8 0 3 60 0 0 n/a
00000300018c09e0 sleep 1002 100 3 59 0 0 n/a
00000300007dd000 sleep 1002 100 3 59 0 0 n/a
000002a1007c1cc0 sleep 8 0 3 60 0 0 n/a
00000300007b2fe0 sleep 1002 100 1 59 0 0 n/a
00000300007bc000 sleep 1002 100 13 59 0 0 n/a
00000300007c1c40 sleep 1002 100 1 59 0 0 n/a
00000300003b0fc0 sleep 1006 104 1 59 0 0 n/a
00000300003b1c40 sleep 1002 104 3 59 0 0 n/a
000002a10066dcc0 sleep 8 0 3 97 0 0 n/a
000002a100665cc0 sleep 8 0 3 98 0 0 n/a
00000300003b1920 sleep 1000 100 3 98 0 0 n/a
00000300003b1600 run 1000 100 13 60 0 0 n/a
000002a100645cc0 sleep 8 0 3 60 0 0 n/a
000002a10067dcc0 sleep 8 0 3 60 0 0 n/a
000002a100675cc0 run 8 0 3 60 0 0 n/a
>
Thomas
|
|
|
0
|
|
|
|
Reply
|
Thomas
|
8/2/2007 5:19:30 PM
|
|
Hello,
> the not so funny thing is that the thread I am waiting on is the thread that is
> trying to aquire the lock. (Note: This is from a crashdump that just occured)
bullshit. First of all another thread holds that lock and for second it
isn't a crashdump because it the system is spinning around a mutex so it
is a livedump of a hanging system.
But. What I think is interresting. That two threads are touch the same
struct inode. One high priority thread is interrupting a lower priority thread.
The lower priority thread is holding the lock and the higher priority thread is
spinning around a mutex that never will be freed.
(schweiz) [/var/crash/schweiz] mdb -k unix.2 vmcore.2
Loading modules: [ unix krtld genunix specfs dtrace ufs pcisch ip sctp usba s1394 fcp fctl lofs zfs random nfs audiosup sd logindmux ptm md cpc fcip crypto ipc ]
> $C
000002a100047111 abort_seq_softintr+0x94(180fc00, 18a9400, 180c000, 2a100047d98, 2, 1857c00)
000002a1000472d1 intr_thread+0x170(600072af698, 0, 180c000, 2a10007fcc0, 0, fffe)
000002a10014f0a1 ufs_rmidle+8(600072af5b0, 0, 11efc00, 600072af698, 2a10014fcc4, 0)
000002a10014f151 ufs_idle_some+0x164(1149, 600072af5b0, 9a0, 0, 18e3800, 18e18f8)
000002a10014f201 ufs_thread_idle+0x1dc(0, 2, 18e3800, 18e3800, 100, 0)
000002a10014f2d1 thread_start+4(0, 0, 0, 0, 0, 0)
> 600072af5b0::print -t "struct inode" !grep opaque
void *[1] _opaque = [ 0 ]
void *[1] _opaque = [ 0x2a10014fcc4 ]
void *[1] _opaque = [ 0x2a10014fcc0 ]
ushort_t _opaque = 0
void *[1] _opaque = [ 0 ]
> ::cpuinfo -v ! head -6
ID ADDR FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD PROC
0 00001838600 1b 93 0 169 yes no t-41 2a100047cc0 sched
| | |
RUNNING <--+ | +--> PIL THREAD
READY | 10 2a100047cc0
EXISTS | - 2a10014fcc0 sched
> 2a10014fcc0::findstack::stackregs
stack pointer for thread 2a10014fcc0: 2a10014ef51
[ 000002a10014ef51 ktl0+0x48() ]
000002a100047111 abort_seq_softintr+0x94(180fc00, 18a9400, 180c000, 2a100047d98, 2, 1857c00)
%l0-%l3: 181c460 1796b8259293 0 0
%l4-%l7: 0 18a9400 0 100cc18
intr_thread+0x170: jmpl %o5, %o7
000002a1000472d1 intr_thread+0x170(600072af698, 0, 180c000, 2a10007fcc0, 0, fffe)
%l0-%l3: 10076e4 a 4480001603 80e
%l4-%l7: 1 6000088b000 0 2a10014f800
ufs_rmidle+8: call -0x29871c <mutex_enter>
000002a10014f0a1 ufs_rmidle+8(600072af5b0, 0, 11efc00, 600072af698, 2a10014fcc4, 0)
%l0-%l3: 9980001602 16 bbb 803
%l4-%l7: 30000096d40 16568 0 2a10014f8b0
ufs_idle_some+0x164: call -0x1481c <ufs_rmidle>
000002a10014f151 ufs_idle_some+0x164(1149, 600072af5b0, 9a0, 0, 18e3800, 18e18f8)
%l0-%l3: 18e18f0 18e3840 417 600072af5b0
%l4-%l7: 60006d42080 600072af690 2 1
ufs_thread_idle+0x1dc: call +0x58 <ufs_idle_some>
000002a10014f201 ufs_thread_idle+0x1dc(0, 2, 18e3800, 18e3800, 100, 0)
%l0-%l3: 2292 2216 18e3800 18e3800
%l4-%l7: 18e3800 2 2000000 2
thread_start+4: jmpl %i7, %o7
000002a10014f2d1 thread_start+4(0, 0, 0, 0, 0, 0)
%l0-%l3: 0 0 0 0
%l4-%l7: 0 0 0 0
>
So bottom line is that 0x2a10014fcc0 is preempted by 2a100047cc0 who needs the
lock that is held by 0x2a10014fcc0. Is that correct or do I miss something?
Thomas
|
|
|
0
|
|
|
|
Reply
|
Thomas
|
8/4/2007 10:10:31 AM
|
|
Thomas Glanzmann <sithglan@stud.uni-erlangen.de> writes:
> > the not so funny thing is that the thread I am waiting on is the thread that is
> > trying to aquire the lock. (Note: This is from a crashdump that just occured)
>
> bullshit. First of all another thread holds that lock and for second it
> isn't a crashdump because it the system is spinning around a mutex so it
> is a livedump of a hanging system.
I have at least one silly question to ask: why are you trying to debug
this?
You're using one of the commercial distributions of Solaris (Solaris
10) rather than the newer OpenSolaris-based distributions, so you
should be reporting a problem like this to Sun's support group.
> But. What I think is interresting. That two threads are touch the same
> struct inode. One high priority thread is interrupting a lower priority thread.
> The lower priority thread is holding the lock and the higher priority thread is
> spinning around a mutex that never will be freed.
I think you might be missing priority inheritance. When a higher-
priority thread needs a lock held by a lower-priority thread, the
lower-priority one is temporarily elevated so that it can run to
completion and allow the higher-priority task to acquire the lock.
That's a guess, though.
> So bottom line is that 0x2a10014fcc0 is preempted by 2a100047cc0 who needs the
> lock that is held by 0x2a10014fcc0. Is that correct or do I miss something?
Actually, I can't tell what's going on by the information presented.
I really do think you need to get involved with support. If you don't
have a support contract, make the dump available to me, and I'll at
least file a bug for you. Obviously, no guarantees, but I don't think
I'd recommend running Solaris in production without a support
contract.
--
James Carlson, Solaris Networking <james.d.carlson@sun.com>
Sun Microsystems / 1 Network Drive 71.232W Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677
|
|
|
0
|
|
|
|
Reply
|
James
|
8/6/2007 11:34:42 AM
|
|
Hello,
> I have at least one silly question to ask: why are you trying to debug
> this?
to evolve. To learn. To get as many views as I can on the same thing.
> You're using one of the commercial distributions of Solaris (Solaris
> 10) rather than the newer OpenSolaris-based distributions, so you
> should be reporting a problem like this to Sun's support group.
Oh guess what? I opened the call 7 days ago. A lot of chip-chap and
speculation on my side (which turned out as wrong) but nothing that
fixes the problem until now. But I look forward on resolving the issue.
> I think you might be missing priority inheritance. When a higher-
> priority thread needs a lock held by a lower-priority thread, the
> lower-priority one is temporarily elevated so that it can run to
> completion and allow the higher-priority task to acquire the lock.
I didn't miss it. And I know what priority inheritance is. But my
question was if Solaris does it handle it that way. After looking at the
source code of Solaris it more liks like priority inversion. However
that was bullshit anyway. Because the second thread that I saw, the one
with the high priotity, was me myself pressing Alternate break sequence
and end up in the kernel debugger (in that case the stack frame seems to
get dupplicated by the interrupt handler).
If you're interested in the crash dumps, raise the word and I put them
online in a matter of seconds.
Thomas
|
|
|
0
|
|
|
|
Reply
|
Thomas
|
8/6/2007 11:48:55 AM
|
|
|
15 Replies
209 Views
(page loaded in 0.137 seconds)
|
Search |
Post Question |
Groups |
Stream |
About |
Register
25710 articles. 
78 followers. 