Strange performance problem - comp.unix.solaris

Hi all, On a Solaris 10 T5220 server with little cpu load and 10 GB of free memory, about 500 processes, I have this issue - all processes take a lot to complete. They are spending a huge time in system, according to ptime - see below output. What could be the cause here ? The server has mounted some large nfs filesystems, but a similar server has exactly the same configuration and works fine. The only difference is that on the problem server I am having some zones + ZFS. Anybody has any suggestions? I don't really know how to investigate this - all tools like vmstat and prstat seem to show normal values.... Emil Please look at the output below: root@signo # ptime ps PID TTY TIME CMD 1308 pts/13 0:06 ptime 493 pts/13 0:12 bash 1309 pts/13 0:06 ps 476 pts/13 0:00 sh real 5.832 user 0.012 sys 5.818 root@signo # w 10:21am up 384 day(s), 7:05, 2 users, load average: 2.30, 2.55, 2.47 User tty login@ idle JCPU PCPU what root pts/13 10:06am 2:09 16 w borod pts/15 8:25am 1:39 1:29 -ksh

Update: It seems that sometimes on the problem server I have a large number of smtx - 20k to 45k!!! The CPU load is still low at those times - no procesess scheduled from cron . What is causing this is still a mystery to me.... On Apr 17, 10:28=A0am, Egrama <egr...@gmail.com> wrote: > Hi all, > > On a Solaris 10 T5220 server with little cpu load and 10 GB of free > memory, about 500 processes, I have this issue - all processes take a > lot to complete. They are spending a huge time in system, according to > ptime - see below output. What could be the cause here ? > > The server has mounted some large nfs filesystems, =A0but a similar > server has exactly the same configuration and works fine. The only > difference is that on the problem server I am having some zones + ZFS. > > Anybody has any suggestions? I don't really know =A0how to investigate > this - all tools like vmstat and prstat seem to show normal values.... > > Emil > > Please look at the output below: > > root@signo # ptime ps > =A0 =A0PID TTY =A0 =A0 =A0 =A0 TIME CMD > =A0 1308 pts/13 =A0 =A0 =A00:06 ptime > =A0 =A0493 pts/13 =A0 =A0 =A00:12 bash > =A0 1309 pts/13 =A0 =A0 =A00:06 ps > =A0 =A0476 pts/13 =A0 =A0 =A00:00 sh > > real =A0 =A0 =A0 =A05.832 > user =A0 =A0 =A0 =A00.012 > sys =A0 =A0 =A0 =A0 5.818 > root@signo # w > =A010:21am =A0up 384 day(s), =A07:05, =A02 users, =A0load average: 2.30, = 2.55, > 2.47 > User =A0 =A0 tty =A0 =A0 =A0 =A0 =A0 login@ =A0idle =A0 JCPU =A0 PCPU =A0= what > root =A0 =A0 pts/13 =A0 =A0 =A0 10:06am =A0 =A0 =A0 =A0 2:09 =A0 =A0 16 = =A0w > borod =A0 pts/15 =A0 =A0 =A0 =A08:25am =A01:39 =A0 1:29 =A0 =A0 =A0 =A0 -= ksh

> > Anybody has any suggestions? I don't really know =A0how to investigate > > this - all tools like vmstat and prstat seem to show normal values.... How about letting us see the output of vmstat / iostat / mpstat ? Try running it for a minute or so when you're experiencing problems: # vmstat 60 2 # mpstat 60 2 # iostat -xz 10 2 And post the second set of output from each here. Also, if you're seeing high spins on mutexes, then the output from lockstat might be useful : # lockstat -kWP sleep 60 Perhaps that'll give us something to go on. What applications is this server running ? -Mark

I put the output below: they are a bit long - but perhaps they will help. It seems that a lot of locking occures due to page_trylock; I googled around and I think it can be http://bugs.opensolaris.org/bugdatabase/view_b= ug.do?bug_id=3D6642475 The workaround is to set pg_contig_disable=3D1 in /etc/system. I did that online via mdb -k and I am waiting now to see if I can reproduce... Will keep you posted # vmstat 60 2 kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr m0 m1 m3 m4 in sy cs us sy id 0 0 0 6727216 7258616 367 239 4959779022 5 4 0 0 8 0 13 1 3694 6806 5667 1 1 98 0 0 0 9145336 10436856 26 263 8 3 3 0 0 1 0 7 0 5232 7979 7758 2 2 96 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D # mpstat 60 2 CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 23 855429 1827 392 200 317 1 30 374 0 411 3 6 0 91 1 5 83300 174 145 0 218 1 40 143 0 235 2 2 0 96 2 1 8458 34 40 0 46 0 10 24 0 55 1 0 0 99 3 0 7305 6 7 0 8 0 1 10 0 10 0 0 0 100 4 9 460843 200 84 0 172 1 23 219 0 250 3 2 0 95 5 0 3332 13 12 0 22 0 3 18 0 21 0 0 0 100 6 0 2179 2 3 0 4 0 1 7 0 5 0 0 0 100 7 0 2307 2 3 0 4 0 1 5 0 4 0 0 0 100 8 24 697544 521 176 0 373 1 49 364 0 541 6 5 0 89 9 1 18839 67 30 0 61 0 13 55 0 72 1 1 0 99 10 0 1153 5 7 0 12 0 2 9 0 8 0 0 0 100 11 0 384 2 3 0 4 0 1 5 0 4 0 0 0 100 12 7 177750 201 83 0 171 0 24 172 0 230 2 2 0 96 13 0 9483 14 12 0 23 0 3 18 0 22 0 0 0 100 14 0 1666 2 3 0 4 0 1 5 0 5 0 0 0 100 15 0 2307 1 2 0 3 0 0 4 0 3 0 0 0 100 16 22 752650 497 167 0 354 1 46 346 0 503 5 5 0 90 17 1 36396 55 27 0 55 0 12 51 0 69 1 1 0 99 18 0 5895 4 4 0 6 0 1 8 0 8 0 0 0 100 19 0 641 2 3 0 3 0 1 4 0 4 0 0 0 100 20 6 177494 201 81 0 169 0 24 167 0 226 2 2 0 96 21 0 2050 16 9 0 17 0 3 18 0 23 0 0 0 100 22 0 1794 2 3 0 4 0 1 5 0 5 0 0 0 100 23 0 1794 1 2 0 3 0 0 3 0 3 0 0 0 100 24 21 940909 470 168 0 356 1 46 345 0 479 5 5 0 90 25 1 227858 60 34 0 68 0 13 50 0 70 1 1 0 99 26 0 128 5 4 0 7 0 2 8 0 8 0 0 0 100 27 0 1410 1 2 0 3 0 1 4 0 4 0 0 0 100 28 6 202227 216 129 0 266 1 24 167 0 205 2 2 0 96 29 0 5511 31 43 0 86 0 3 19 0 24 0 0 0 99 30 0 1410 12 40 35 7 0 1 6 0 5 0 0 0 100 31 0 2948 289 490 487 3 0 0 5 0 2 0 1 0 99 32 20 672682 519 216 57 351 1 47 330 0 479 4 5 0 91 33 1 10252 80 110 2 223 0 13 48 0 67 1 1 0 99 34 0 1025 7 13 0 24 0 1 8 0 8 0 0 0 100 35 0 4101 1 2 0 3 0 1 4 0 4 0 0 0 100 36 6 198255 257 123 42 168 0 24 162 0 220 2 2 0 96 37 0 14097 19 13 0 26 0 3 17 0 24 0 0 0 100 38 0 769 6 10 0 19 0 1 5 0 5 0 0 0 100 39 0 6151 2 2 0 2 0 0 3 0 2 0 0 0 100 40 21 640900 675 166 3 350 1 46 340 0 488 5 5 0 90 41 1 24221 66 36 0 73 0 13 51 0 73 1 1 0 99 42 0 3716 9 11 0 21 0 2 8 0 9 0 0 0 100 43 0 7561 3 2 0 3 0 1 4 0 4 0 0 0 100 44 6 189156 207 88 0 183 0 24 162 0 222 2 2 0 96 45 0 17557 16 13 0 25 0 3 18 0 24 0 0 0 100 46 0 384 2 3 0 5 0 1 5 0 5 0 0 0 100 47 0 0 5 11 0 21 0 0 3 0 2 0 0 0 100 48 20 667684 472 169 0 360 1 46 338 0 483 5 5 0 90 49 1 34474 60 34 0 69 0 13 50 0 75 1 1 0 99 50 0 3204 4 4 0 7 0 2 8 0 9 0 0 0 100 51 0 384 3 6 0 10 0 1 4 0 4 0 0 0 100 52 6 201971 196 82 0 171 0 25 164 0 223 2 2 0 96 53 0 14994 18 16 0 30 0 3 18 0 24 0 0 0 100 54 0 0 5 11 0 21 0 1 5 0 5 0 0 0 100 55 0 3716 2 2 0 2 0 0 3 0 3 0 0 0 100 56 21 779178 463 164 0 350 1 47 337 0 487 5 5 0 90 57 1 43444 63 33 0 67 0 13 48 0 75 1 1 0 99 58 0 2050 4 5 0 8 0 2 8 0 9 0 0 0 100 59 0 1410 2 5 0 8 0 1 4 0 4 0 0 0 100 60 6 265279 197 95 14 169 0 24 168 0 224 2 2 0 96 61 0 1153 16 29 14 29 0 3 19 0 24 0 0 0 100 62 0 2563 3 7 0 12 0 1 5 0 5 0 0 0 100 63 0 2050 2 5 0 8 0 0 3 0 2 0 0 0 100 CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 19 493836581 1947 441 200 428 1 41 175 0 588 7 5 0 89 1 6 0 81 186 0 256 1 34 22 0 339 5 1 0 94 2 0 70559673 15 46 0 52 0 9 2 0 46 0 0 0 99 3 0 0 0 6 0 4 0 1 2 0 3 0 0 0 100 4 7 70571267 201 119 0 238 1 24 146 0 286 8 2 0 90 5 1 0 1 9 0 14 0 3 2 0 9 0 0 0 100 6 0 0 0 3 0 2 0 1 2 0 1 0 0 0 100 7 0 0 0 4 0 3 0 1 1 0 0 0 0 0 100 8 21 211679019 445 235 0 490 2 56 46 0 531 8 6 0 86 9 1 0 30 32 0 61 0 12 8 0 85 2 1 0 97 10 0 0 1 5 0 5 0 1 2 0 4 0 0 0 100 11 0 0 0 3 0 2 0 1 1 0 0 0 0 0 100 12 12 70571267 89 110 0 224 1 28 23 0 240 3 1 0 96 13 0 0 8 9 0 13 0 3 2 0 9 0 0 0 100 14 0 0 0 4 0 3 0 1 2 0 0 0 0 0 100 15 0 0 0 4 0 2 0 0 1 0 0 4 0 0 96 16 28 0 383 246 0 560 2 56 118 0 888 11 3 0 86 17 0 0 6 29 0 56 0 13 17 0 60 0 1 0 99 18 0 0 1 5 0 6 0 1 2 0 3 0 0 0 100 19 0 0 0 3 0 2 0 1 1 0 0 0 0 0 100 20 10 0 73 102 0 208 0 31 24 0 178 3 1 0 97 21 0 0 1 11 0 17 0 3 3 0 9 0 0 0 100 22 0 0 0 4 0 3 0 1 1 0 1 0 0 0 100 23 0 0 4 7 0 11 0 0 1 0 12 0 0 0 100 24 23 282285067 280 229 0 480 1 56 128 0 522 8 3 0 89 25 0 0 16 35 0 68 0 14 13 0 142 1 1 0 99 26 0 0 5 7 0 9 0 1 1 0 7 0 0 0 100 27 0 0 0 3 0 3 0 1 2 0 3 0 0 0 100 28 16 706060710 102 116 0 239 1 30 24 0 271 5 1 0 94 29 0 0 4 14 0 24 0 3 3 0 27 1 0 0 99 30 0 0 21 55 52 3 0 1 3 0 0 0 0 0 100 31 0 0 492 814 811 2 0 0 3 0 0 0 1 0 99 32 15 0 186 238 48 424 1 56 185 0 497 5 5 0 90 33 0 0 242 547 2 1118 2 16 10 0 44 1 2 0 98 34 0 0 1 5 1 4 0 1 2 0 1 0 0 0 100 35 0 0 0 3 0 2 0 1 2 0 0 0 0 0 100 36 0 0 82 147 29 242 1 30 21 0 270 2 2 0 96 37 0 0 0 9 0 14 0 3 2 0 5 3 0 0 97 38 0 0 4 7 0 10 0 1 1 0 84 0 0 0 100 39 0 0 0 3 0 2 0 0 1 0 0 0 0 0 100 40 15 355484795 472 223 3 460 2 56 101 0 520 11 4 0 85 41 0 0 17 36 0 72 0 14 6 0 126 0 1 0 99 42 0 0 0 4 0 3 0 1 2 0 1 0 0 0 100 43 0 0 0 3 0 2 0 1 1 0 0 0 0 0 100 44 12 213821804 126 116 0 256 1 31 42 0 456 2 2 0 97 45 0 0 11 12 0 21 0 4 2 0 21 0 0 0 100 46 0 0 0 3 0 2 0 0 2 0 0 0 0 0 100 47 0 0 0 4 0 3 0 0 2 0 7 0 0 0 99 48 29 499663804 322 238 0 500 2 59 106 0 662 5 5 0 90 49 7 0 103 37 0 72 0 13 5 0 64 1 0 0 99 50 0 0 3 8 0 11 0 1 2 0 10 0 0 0 100 51 0 0 0 3 0 3 0 1 1 0 1 0 0 0 100 52 1 0 71 129 0 265 1 30 23 0 278 3 4 0 93 53 0 0 1 10 0 16 0 3 2 0 9 0 0 0 100 54 0 0 0 4 0 3 0 1 2 0 0 0 0 0 100 55 0 0 1 4 0 3 0 0 1 0 2 0 0 0 100 56 39 286140393 340 234 0 490 2 55 62 0 551 7 2 0 90 57 0 0 30 33 0 64 0 12 5 0 48 0 0 0 100 58 0 0 6 4 0 4 0 1 2 0 5 0 0 0 100 59 0 0 38 85 41 3 1 1 1 0 7 7 0 0 93 60 21 0 197 135 14 249 1 28 30 0 412 4 2 0 95 61 0 0 3 23 14 15 0 3 3 0 7 0 0 0 100 62 0 0 2 3 0 2 0 1 1 0 0 0 0 0 100 63 0 0 0 3 0 2 0 0 2 0 0 0 0 0 100 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D # iostat -xz 10 2 extended device statistics device r/s w/s kr/s kw/s wait actv svc_t %w %b md0 5.8 1.9 169.9 11.2 0.0 0.1 10.8 1 3 md1 0.0 0.0 0.0 0.0 0.0 0.0 12.4 0 0 md3 4.1 8.7 192.9 39.3 0.0 0.1 8.4 0 8 md4 1.2 0.0 105.7 0.1 0.0 0.0 9.4 0 1 md5 1.4 1.1 82.8 25.8 0.0 0.0 17.6 1 2 md6 1.5 0.2 131.6 6.6 0.0 0.0 11.7 0 1 md10 2.9 1.9 85.0 11.2 0.0 0.1 11.2 0 2 md11 0.0 0.0 0.0 0.0 0.0 0.0 11.8 0 0 md13 2.1 8.7 96.3 39.3 0.0 0.1 7.5 0 7 md14 0.6 0.0 52.9 0.1 0.0 0.0 11.3 0 0 md15 0.7 1.1 41.4 25.8 0.0 0.0 15.3 0 1 md16 0.8 0.2 66.2 5.8 0.0 0.0 14.0 0 1 md20 2.9 1.9 84.9 11.2 0.0 0.0 9.4 0 2 md21 0.0 0.0 0.0 0.0 0.0 0.0 8.8 0 0 md23 2.1 8.7 96.6 39.3 0.0 0.1 6.7 0 6 md24 0.6 0.0 52.8 0.1 0.0 0.0 7.3 0 0 md25 0.7 1.1 41.4 25.8 0.0 0.0 12.5 0 1 md26 0.8 0.2 65.4 6.6 0.0 0.0 11.3 0 1 sd1 8.4 13.0 340.9 82.5 0.0 0.2 11.2 0 11 sd3 8.4 13.0 340.2 83.3 0.0 0.2 9.2 0 10 ssd1 0.2 11.4 25.2 88.6 0.0 0.0 3.9 0 1 ssd2 0.2 11.5 25.3 86.9 0.0 0.0 4.1 0 1 ssd3 0.2 12.3 25.3 82.3 0.0 0.1 4.0 0 1 ssd4 0.2 12.1 24.7 72.3 0.0 0.0 3.8 0 1 ssd5 0.2 12.4 19.5 157.9 0.0 0.0 3.8 0 1 ssd9 0.1 0.2 2.8 2.9 0.0 0.0 2.2 0 0 ssd10 0.1 0.1 2.8 2.4 0.0 0.0 2.1 0 0 ssd11 0.0 0.1 0.6 2.0 0.0 0.0 2.9 0 0 ssd12 0.0 0.0 0.6 1.8 0.0 0.0 2.6 0 0 nfs1 0.0 0.0 0.0 0.0 0.0 0.0 2.1 0 0 nfs11 0.0 0.0 0.2 0.0 0.0 0.0 10.0 0 0 nfs13 0.0 0.0 0.0 0.0 0.0 0.0 3.1 0 0 nfs14 0.0 0.0 0.0 0.0 0.0 0.0 13.9 0 0 nfs15 0.0 0.0 0.0 0.0 0.0 0.0 14.0 0 0 nfs16 0.0 0.0 0.0 0.0 0.0 0.0 13.9 0 0 nfs17 0.0 0.0 0.0 0.0 0.0 0.0 14.8 0 0 nfs18 0.0 0.0 0.0 0.0 0.0 0.0 8.6 0 0 nfs19 0.0 0.0 0.0 0.0 0.0 0.0 6.7 0 0 nfs20 0.0 0.0 0.0 0.0 0.0 0.0 6.2 0 0 nfs21 0.0 0.0 0.0 0.0 0.0 0.0 6.0 0 0 nfs22 0.0 0.0 0.0 0.0 0.0 0.0 14.5 0 0 nfs23 0.0 0.0 0.0 0.0 0.0 0.0 13.8 0 0 nfs24 0.0 0.0 0.0 0.0 0.0 0.0 13.6 0 0 nfs25 0.0 0.0 0.0 0.0 0.0 0.0 17.2 0 0 nfs26 0.0 0.0 0.0 0.0 0.0 0.0 15.3 0 0 nfs27 0.0 0.0 0.0 0.0 0.0 0.0 14.2 0 0 nfs28 0.0 0.0 0.0 0.0 0.0 0.0 13.5 0 0 nfs29 0.0 0.0 0.0 0.0 0.0 0.0 13.5 0 0 nfs30 0.0 0.0 0.0 0.0 0.0 0.0 14.2 0 0 nfs31 0.0 0.0 0.0 0.0 0.0 0.0 6.4 0 0 nfs32 0.0 0.0 0.0 0.0 0.0 0.0 14.0 0 0 nfs33 0.0 0.0 0.0 0.0 0.0 0.0 13.9 0 0 nfs34 0.0 0.0 0.0 0.0 0.0 0.0 5.1 0 0 nfs35 0.0 0.0 0.0 0.0 0.0 0.0 14.0 0 0 nfs36 0.0 0.0 0.0 0.0 0.0 0.0 14.9 0 0 nfs37 0.0 0.0 0.0 0.0 0.0 0.0 5.8 0 0 nfs38 0.0 0.0 0.0 0.0 0.0 0.0 8.7 0 0 nfs39 0.0 0.0 0.0 0.0 0.0 0.0 13.8 0 0 nfs40 0.0 0.0 0.0 0.0 0.0 0.0 13.9 0 0 nfs41 0.0 0.0 0.0 0.0 0.0 0.0 14.2 0 0 nfs42 0.0 0.0 0.0 0.0 0.0 0.0 5.4 0 0 nfs43 0.0 0.0 0.0 0.0 0.0 0.0 7.7 0 0 nfs44 0.0 0.0 0.0 0.0 0.0 0.0 6.7 0 0 nfs45 0.0 0.0 0.0 0.0 0.0 0.0 13.8 0 0 nfs46 0.0 0.0 0.0 0.0 0.0 0.0 13.7 0 0 nfs47 0.0 0.0 0.0 0.0 0.0 0.0 14.2 0 0 nfs48 0.0 0.0 0.0 0.0 0.0 0.0 14.0 0 0 nfs49 0.0 0.0 0.0 0.0 0.0 0.0 13.9 0 0 nfs50 0.0 0.0 0.0 0.0 0.0 0.0 13.5 0 0 nfs51 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0 0 nfs52 0.0 0.0 0.0 0.0 0.0 0.0 13.5 0 0 nfs53 0.0 0.0 0.0 0.0 0.0 0.0 14.2 0 0 nfs54 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0 0 nfs55 0.0 0.0 0.0 0.0 0.0 0.0 14.0 0 0 nfs56 0.0 0.0 0.0 0.0 0.0 0.0 13.8 0 0 nfs57 0.0 0.0 0.0 0.0 0.0 0.0 13.7 0 0 nfs58 0.0 0.0 0.0 0.0 0.0 0.0 14.1 0 0 nfs59 0.0 0.0 0.0 0.0 0.0 0.0 5.3 0 0 nfs60 0.0 0.0 0.0 0.0 0.0 0.0 14.0 0 0 nfs61 0.0 0.0 0.0 0.0 0.0 0.0 5.5 0 0 nfs62 0.0 0.0 0.0 0.0 0.0 0.0 9.9 0 0 nfs66 0.0 0.0 0.0 0.0 0.0 0.0 14.2 0 0 extended device statistics device r/s w/s kr/s kw/s wait actv svc_t %w %b md0 0.0 0.4 0.0 3.2 0.0 0.0 14.0 0 0 md5 0.0 0.5 0.0 9.8 0.0 0.0 18.0 0 1 md10 0.0 0.4 0.0 3.2 0.0 0.0 10.8 0 0 md15 0.0 0.5 0.0 9.8 0.0 0.0 9.5 0 0 md20 0.0 0.4 0.0 3.2 0.0 0.0 6.0 0 0 md25 0.0 0.5 0.0 9.8 0.0 0.0 8.3 0 0 sd1 0.0 1.3 0.0 13.2 0.0 0.0 9.7 0 1 sd3 0.0 1.3 0.0 13.2 0.0 0.0 8.4 0 1 ssd1 0.0 4.5 0.0 71.2 0.0 0.0 3.2 0 0 ssd2 0.0 4.5 0.0 71.4 0.0 0.0 2.3 0 0 ssd3 0.0 9.2 0.0 81.0 0.0 0.0 3.7 0 1 ssd4 0.0 11.4 0.0 40.6 0.0 0.0 4.1 0 1 ssd5 0.0 13.3 0.0 48.8 0.0 0.0 3.1 0 1 ssd9 0.0 0.2 0.0 2.4 0.0 0.0 0.4 0 0 ssd10 0.0 0.3 0.0 2.5 0.0 0.0 1.4 0 0 nfs11 0.4 0.0 6.7 0.0 0.0 0.0 17.1 0 1 nfs21 0.1 0.0 0.2 0.0 0.0 0.0 11.2 0 0 nfs59 0.1 0.0 0.3 0.0 0.0 0.0 12.2 0 0 extended device statistics device r/s w/s kr/s kw/s wait actv svc_t %w %b md0 5.8 1.9 169.9 11.2 0.0 0.1 10.8 1 3 md1 0.0 0.0 0.0 0.0 0.0 0.0 12.4 0 0 md3 4.1 8.7 192.9 39.3 0.0 0.1 8.4 0 8 md4 1.2 0.0 105.7 0.1 0.0 0.0 9.4 0 1 md5 1.4 1.1 82.8 25.8 0.0 0.0 17.6 1 2 md6 1.5 0.2 131.6 6.6 0.0 0.0 11.7 0 1 md10 2.9 1.9 85.0 11.2 0.0 0.1 11.2 0 2 md11 0.0 0.0 0.0 0.0 0.0 0.0 11.8 0 0 md13 2.1 8.7 96.3 39.3 0.0 0.1 7.5 0 7 md14 0.6 0.0 52.9 0.1 0.0 0.0 11.3 0 0 md15 0.7 1.1 41.4 25.8 0.0 0.0 15.3 0 1 md16 0.8 0.2 66.2 5.8 0.0 0.0 14.0 0 1 md20 2.9 1.9 84.9 11.2 0.0 0.0 9.4 0 2 md21 0.0 0.0 0.0 0.0 0.0 0.0 8.8 0 0 md23 2.1 8.7 96.6 39.3 0.0 0.1 6.7 0 6 md24 0.6 0.0 52.8 0.1 0.0 0.0 7.3 0 0 md25 0.7 1.1 41.4 25.8 0.0 0.0 12.5 0 1 md26 0.8 0.2 65.4 6.6 0.0 0.0 11.3 0 1 sd1 8.4 13.0 340.9 82.5 0.0 0.2 11.2 0 11 sd3 8.4 13.0 340.2 83.3 0.0 0.2 9.2 0 10 ssd1 0.2 11.4 25.2 88.6 0.0 0.0 3.9 0 1 ssd2 0.2 11.5 25.3 86.9 0.0 0.0 4.1 0 1 ssd3 0.2 12.3 25.3 82.3 0.0 0.1 4.0 0 1 ssd4 0.2 12.1 24.7 72.3 0.0 0.0 3.8 0 1 ssd5 0.2 12.4 19.5 157.9 0.0 0.0 3.8 0 1 ssd9 0.1 0.2 2.8 2.9 0.0 0.0 2.2 0 0 ssd10 0.1 0.1 2.8 2.4 0.0 0.0 2.1 0 0 ssd11 0.0 0.1 0.6 2.0 0.0 0.0 2.9 0 0 ssd12 0.0 0.0 0.6 1.8 0.0 0.0 2.7 0 0 nfs1 0.0 0.0 0.0 0.0 0.0 0.0 2.1 0 0 nfs11 0.0 0.0 0.2 0.0 0.0 0.0 10.0 0 0 nfs13 0.0 0.0 0.0 0.0 0.0 0.0 3.1 0 0 nfs14 0.0 0.0 0.0 0.0 0.0 0.0 13.9 0 0 nfs15 0.0 0.0 0.0 0.0 0.0 0.0 14.0 0 0 nfs16 0.0 0.0 0.0 0.0 0.0 0.0 13.9 0 0 nfs17 0.0 0.0 0.0 0.0 0.0 0.0 14.8 0 0 nfs18 0.0 0.0 0.0 0.0 0.0 0.0 8.6 0 0 nfs19 0.0 0.0 0.0 0.0 0.0 0.0 6.7 0 0 nfs20 0.0 0.0 0.0 0.0 0.0 0.0 6.2 0 0 nfs21 0.0 0.0 0.0 0.0 0.0 0.0 6.0 0 0 nfs22 0.0 0.0 0.0 0.0 0.0 0.0 14.5 0 0 nfs23 0.0 0.0 0.0 0.0 0.0 0.0 13.8 0 0 nfs24 0.0 0.0 0.0 0.0 0.0 0.0 13.6 0 0 nfs25 0.0 0.0 0.0 0.0 0.0 0.0 17.2 0 0 nfs26 0.0 0.0 0.0 0.0 0.0 0.0 15.3 0 0 nfs27 0.0 0.0 0.0 0.0 0.0 0.0 14.2 0 0 nfs28 0.0 0.0 0.0 0.0 0.0 0.0 13.5 0 0 nfs29 0.0 0.0 0.0 0.0 0.0 0.0 13.5 0 0 nfs30 0.0 0.0 0.0 0.0 0.0 0.0 14.2 0 0 nfs31 0.0 0.0 0.0 0.0 0.0 0.0 6.4 0 0 nfs32 0.0 0.0 0.0 0.0 0.0 0.0 14.0 0 0 nfs33 0.0 0.0 0.0 0.0 0.0 0.0 13.9 0 0 nfs34 0.0 0.0 0.0 0.0 0.0 0.0 5.1 0 0 nfs35 0.0 0.0 0.0 0.0 0.0 0.0 14.0 0 0 nfs36 0.0 0.0 0.0 0.0 0.0 0.0 14.9 0 0 nfs37 0.0 0.0 0.0 0.0 0.0 0.0 5.8 0 0 nfs38 0.0 0.0 0.0 0.0 0.0 0.0 8.7 0 0 nfs39 0.0 0.0 0.0 0.0 0.0 0.0 13.8 0 0 nfs40 0.0 0.0 0.0 0.0 0.0 0.0 13.9 0 0 nfs41 0.0 0.0 0.0 0.0 0.0 0.0 14.2 0 0 nfs42 0.0 0.0 0.0 0.0 0.0 0.0 5.4 0 0 nfs43 0.0 0.0 0.0 0.0 0.0 0.0 7.7 0 0 nfs44 0.0 0.0 0.0 0.0 0.0 0.0 6.7 0 0 nfs45 0.0 0.0 0.0 0.0 0.0 0.0 13.8 0 0 nfs46 0.0 0.0 0.0 0.0 0.0 0.0 13.7 0 0 nfs47 0.0 0.0 0.0 0.0 0.0 0.0 14.2 0 0 nfs48 0.0 0.0 0.0 0.0 0.0 0.0 14.0 0 0 nfs49 0.0 0.0 0.0 0.0 0.0 0.0 13.9 0 0 nfs50 0.0 0.0 0.0 0.0 0.0 0.0 13.5 0 0 nfs51 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0 0 nfs52 0.0 0.0 0.0 0.0 0.0 0.0 13.5 0 0 nfs53 0.0 0.0 0.0 0.0 0.0 0.0 14.2 0 0 nfs54 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0 0 nfs55 0.0 0.0 0.0 0.0 0.0 0.0 14.0 0 0 nfs56 0.0 0.0 0.0 0.0 0.0 0.0 13.8 0 0 nfs57 0.0 0.0 0.0 0.0 0.0 0.0 13.7 0 0 nfs58 0.0 0.0 0.0 0.0 0.0 0.0 14.1 0 0 nfs59 0.0 0.0 0.0 0.0 0.0 0.0 5.3 0 0 nfs60 0.0 0.0 0.0 0.0 0.0 0.0 14.0 0 0 nfs61 0.0 0.0 0.0 0.0 0.0 0.0 5.5 0 0 nfs62 0.0 0.0 0.0 0.0 0.0 0.0 9.9 0 0 nfs66 0.0 0.0 0.0 0.0 0.0 0.0 14.2 0 0 extended device statistics device r/s w/s kr/s kw/s wait actv svc_t %w %b md0 0.0 0.4 0.0 3.2 0.0 0.0 8.1 0 0 md3 0.0 0.4 0.0 1.8 0.0 0.0 19.1 0 0 md5 0.0 0.2 0.0 7.9 0.0 0.0 10.3 0 0 md10 0.0 0.4 0.0 3.2 0.0 0.0 6.4 0 0 md13 0.0 0.4 0.0 1.8 0.0 0.0 9.4 0 0 md15 0.0 0.2 0.0 7.9 0.0 0.0 10.2 0 0 md20 0.0 0.4 0.0 3.2 0.0 0.0 7.6 0 0 md23 0.0 0.4 0.0 1.8 0.0 0.0 10.5 0 0 md25 0.0 0.2 0.0 7.9 0.0 0.0 8.5 0 0 sd1 0.0 1.4 0.0 13.1 0.0 0.0 7.9 0 1 sd3 0.0 1.4 0.0 13.1 0.0 0.0 8.7 0 1 ssd1 0.0 9.9 0.0 27.5 0.0 0.0 2.8 0 1 ssd2 0.0 13.5 0.0 49.2 0.0 0.0 3.7 0 1 ssd3 0.0 14.7 0.0 96.0 0.0 0.0 3.3 0 1 ssd4 0.0 3.4 0.0 116.2 0.0 0.0 1.9 0 0 ssd5 0.0 2.8 0.0 37.2 0.0 0.0 1.3 0 0 ssd9 0.0 0.2 0.0 4.8 0.0 0.0 1.1 0 0 ssd10 0.0 0.4 0.0 3.2 0.0 0.0 0.4 0 0 ssd11 26.6 0.2 1635.3 1.6 0.0 0.2 7.1 0 19 ssd12 25.0 0.0 1571.7 0.0 0.0 0.1 4.8 0 12 nfs46 0.2 0.0 0.6 0.0 0.0 0.0 10.9 0 0 nfs55 0.2 0.0 0.5 0.0 0.0 0.0 13.0 0 0 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D root@sig # lockstat -kWP sleep 60 lockstat: warning: 1 aggregation drop on CPU 56 lockstat: warning: 65 aggregation drops on CPU 56 lockstat: warning: 28 aggregation drops on CPU 0 lockstat: warning: 17 aggregation drops on CPU 0 lockstat: warning: 14 aggregation drops on CPU 0 lockstat: warning: 583 aggregation drops on CPU 16 lockstat: warning: 332 aggregation drops on CPU 48 lockstat: warning: 649 aggregation drops on CPU 52 lockstat: warning: 855 aggregation drops on CPU 0 lockstat: warning: 96 aggregation drops on CPU 1 lockstat: warning: 129 aggregation drops on CPU 16 lockstat: warning: 55 aggregation drops on CPU 33 lockstat: warning: 380 aggregation drops on CPU 40 lockstat: warning: 49 aggregation drops on CPU 56 lockstat: warning: 20 aggregation drops on CPU 61 lockstat: warning: 380 aggregation drops on CPU 0 lockstat: warning: 106 aggregation drops on CPU 1 lockstat: warning: 641 aggregation drops on CPU 8 lockstat: warning: 13 aggregation drops on CPU 20 lockstat: warning: 38 aggregation drops on CPU 40 lockstat: warning: 446 aggregation drops on CPU 48 lockstat: warning: 103 aggregation drops on CPU 49 lockstat: warning: 207 aggregation drops on CPU 52 lockstat: warning: 88 aggregation drops on CPU 56 lockstat: warning: 196 aggregation drops on CPU 57 lockstat: warning: 798 aggregation drops on CPU 0 lockstat: warning: 108 aggregation drops on CPU 1 lockstat: warning: 55 aggregation drops on CPU 8 lockstat: warning: 48 aggregation drops on CPU 12 lockstat: warning: 286 aggregation drops on CPU 16 lockstat: warning: 21 aggregation drops on CPU 20 lockstat: warning: 57 aggregation drops on CPU 24 lockstat: warning: 81 aggregation drops on CPU 28 lockstat: warning: 5 aggregation drops on CPU 32 lockstat: warning: 75 aggregation drops on CPU 34 lockstat: warning: 205 aggregation drops on CPU 36 lockstat: warning: 205 aggregation drops on CPU 40 lockstat: warning: 21 aggregation drops on CPU 44 lockstat: warning: 467 aggregation drops on CPU 48 lockstat: warning: 385 aggregation drops on CPU 52 lockstat: warning: 33 aggregation drops on CPU 56 lockstat: warning: 133 aggregation drops on CPU 60 lockstat: warning: 69 aggregation drops on CPU 0 lockstat: warning: 332 aggregation drops on CPU 1 lockstat: warning: 1096 aggregation drops on CPU 4 lockstat: warning: 723 aggregation drops on CPU 8 lockstat: warning: 123 aggregation drops on CPU 12 lockstat: warning: 1145 aggregation drops on CPU 16 lockstat: warning: 170 aggregation drops on CPU 17 lockstat: warning: 646 aggregation drops on CPU 20 lockstat: warning: 141 aggregation drops on CPU 24 lockstat: warning: 726 aggregation drops on CPU 25 lockstat: warning: 2 aggregation drops on CPU 28 lockstat: warning: 18 aggregation drops on CPU 32 lockstat: warning: 61 aggregation drops on CPU 36 lockstat: warning: 55 aggregation drops on CPU 40 lockstat: warning: 55 aggregation drops on CPU 48 lockstat: warning: 486 aggregation drops on CPU 52 lockstat: warning: 930 aggregation drops on CPU 56 lockstat: warning: 574 aggregation drops on CPU 60 lockstat: warning: 518 aggregation drops on CPU 4 lockstat: warning: 132 aggregation drops on CPU 16 lockstat: warning: 7 aggregation drops on CPU 20 lockstat: warning: 128 aggregation drops on CPU 24 lockstat: warning: 480 aggregation drops on CPU 28 lockstat: warning: 387 aggregation drops on CPU 32 lockstat: warning: 57 aggregation drops on CPU 40 lockstat: warning: 1390 aggregation drops on CPU 44 lockstat: warning: 635 aggregation drops on CPU 48 lockstat: warning: 533 aggregation drops on CPU 52 lockstat: warning: 772 aggregation drops on CPU 60 lockstat: warning: 1628 aggregation drops on CPU 4 lockstat: warning: 421 aggregation drops on CPU 8 lockstat: warning: 1522 aggregation drops on CPU 24 lockstat: warning: 399 aggregation drops on CPU 28 lockstat: warning: 473 aggregation drops on CPU 32 lockstat: warning: 758 aggregation drops on CPU 44 lockstat: warning: 585 aggregation drops on CPU 52 lockstat: warning: 744 aggregation drops on CPU 60 lockstat: warning: 367 aggregation drops on CPU 4 lockstat: warning: 177 aggregation drops on CPU 8 lockstat: warning: 1140 aggregation drops on CPU 24 lockstat: warning: 631 aggregation drops on CPU 32 lockstat: warning: 552 aggregation drops on CPU 44 lockstat: warning: 621 aggregation drops on CPU 52 lockstat: warning: 524 aggregation drops on CPU 4 lockstat: warning: 112 aggregation drops on CPU 8 lockstat: warning: 596 aggregation drops on CPU 12 lockstat: warning: 863 aggregation drops on CPU 32 lockstat: warning: 298 aggregation drops on CPU 52 lockstat: warning: 1495 aggregation drops on CPU 4 lockstat: warning: 1439 aggregation drops on CPU 8 lockstat: warning: 436 aggregation drops on CPU 12 lockstat: warning: 135 aggregation drops on CPU 24 lockstat: warning: 74 aggregation drops on CPU 27 lockstat: warning: 1438 aggregation drops on CPU 28 lockstat: warning: 546 aggregation drops on CPU 32 lockstat: warning: 248 aggregation drops on CPU 36 lockstat: warning: 41 aggregation drops on CPU 40 lockstat: warning: 228 aggregation drops on CPU 44 lockstat: warning: 670 aggregation drops on CPU 48 lockstat: warning: 1817 aggregation drops on CPU 56 lockstat: warning: 718 aggregation drops on CPU 0 lockstat: warning: 20 aggregation drops on CPU 1 lockstat: warning: 589 aggregation drops on CPU 4 lockstat: warning: 15 aggregation drops on CPU 8 lockstat: warning: 69 aggregation drops on CPU 12 lockstat: warning: 902 aggregation drops on CPU 16 lockstat: warning: 69 aggregation drops on CPU 17 lockstat: warning: 105 aggregation drops on CPU 20 lockstat: warning: 242 aggregation drops on CPU 24 lockstat: warning: 1 aggregation drop on CPU 25 lockstat: warning: 140 aggregation drops on CPU 28 lockstat: warning: 1117 aggregation drops on CPU 32 lockstat: warning: 124 aggregation drops on CPU 36 lockstat: warning: 15 aggregation drops on CPU 37 lockstat: warning: 428 aggregation drops on CPU 40 lockstat: warning: 160 aggregation drops on CPU 44 lockstat: warning: 648 aggregation drops on CPU 48 lockstat: warning: 440 aggregation drops on CPU 52 lockstat: warning: 758 aggregation drops on CPU 56 lockstat: warning: 296 aggregation drops on CPU 60 lockstat: warning: 26 aggregation drops on CPU 1 lockstat: warning: 394 aggregation drops on CPU 4 lockstat: warning: 30 aggregation drops on CPU 16 lockstat: warning: 683 aggregation drops on CPU 24 lockstat: warning: 592 aggregation drops on CPU 32 lockstat: warning: 233 aggregation drops on CPU 40 lockstat: warning: 566 aggregation drops on CPU 48 lockstat: warning: 596 aggregation drops on CPU 56 lockstat: warning: 343 aggregation drops on CPU 60 lockstat: warning: ran out of data records (use -n for more) Adaptive mutex spin: 396804 events in 60.113 seconds (6601 events/sec) Count indv cuml rcnt spin Hottest Lock Caller ---------------------------------------------------------------------------= ---- 281747 43% 43% 0.00 1 pse_mutex+0x1800 page_trylock 29366 15% 58% 0.00 3 0x30000c2f000 untimeout 6076 15% 72% 0.00 17 vx_worklist_lk vx_worklist_process 21903 11% 84% 0.00 3 0x30000c2f000 timeout_common 15873 8% 92% 0.00 3 0x30000c2f000 callout_execute 29831 4% 96% 0.00 1 pse_mutex+0x400 page_unlock_noretire 771 1% 97% 0.00 10 vx_sched_lk vx_worklist_thread 5501 1% 98% 0.00 1 pse_mutex+0xdc0 page_unlock 1112 0% 98% 0.00 3 0x60007208980 taskq_thread 100 0% 98% 0.00 11 anonpages_hash_lock+0x158 anon_map_privatepages 310 0% 99% 0.00 3 0x60007208980 taskq_dispatch 210 0% 99% 0.00 2 0x60019ade590 dsl_dir_willuse_space 283 0% 99% 0.00 1 0x600071bfe08 putnext 233 0% 99% 0.00 2 lwpchanlock+0x3d10 lwp_mutex_wakeup 228 0% 99% 0.00 2 0x192e5a8 sfmmu_mlspl_enter 110 0% 99% 0.00 3 0x600071b6370 llt_msg_recv 160 0% 99% 0.00 2 dbuf_hash_table+0x618 dbuf_find 129 0% 99% 0.00 3 lwpchanlock+0x3d10 lwp_mutex_timedlock 70 0% 99% 0.00 4 0x60005301980 stop 48 0% 99% 0.00 6 pidlock thread_exit 50 0% 99% 0.00 5 pidlock cv_wait_sig_swap_core 36 0% 99% 0.00 6 0x600054dc0b0 ehci_hcdi_pipe_bulk_xfer 9 0% 99% 0.00 20 0x6000c9ef6c0 metaslab_group_alloc 20 0% 99% 0.00 9 pidlock waitid 81 0% 99% 0.00 2 0x6000546ba40 vn_rele 72 0% 99% 0.00 2 0x6003aa9a578 kstrgetmsg 62 0% 99% 0.00 2 0x60009831508 qfill_syncq 63 0% 99% 0.00 2 0x60009831508 qdrain_syncq 54 0% 99% 0.00 3 0x300534fb6a0 fifo_write 75 0% 99% 0.00 2 0x6000714d800 ssfcp_scsi_start 55 0% 99% 0.00 2 0x30089bf9040 dbuf_rele 3 0% 99% 0.00 53 kstat_chain_lock kstat_hold 37 0% 99% 0.00 4 0x60005305b40 issig_forreal 44 0% 99% 0.00 3 0x6000556d218 ssdintr 49 0% 99% 0.00 3 0x6000556d218 ssd_start_cmds 45 0% 99% 0.00 3 0x6000556d218 ssdstrategy 43 0% 99% 0.00 3 0x3007ad8de00 cv_timedwait_sig 61 0% 99% 0.00 2 0x6000546ba40 lookuppnat 29 0% 99% 0.00 4 pidlock sigcld_repost 53 0% 100% 0.00 2 0x30000c73d10 ufs_lockfs_end 88 0% 100% 0.00 1 0x60005a97ff8 dnlc_lookup 67 0% 100% 0.00 1 pr_pidlock pr_p_lock 85 0% 100% 0.00 1 pse_mutex+0x200 page_try_reclaim_lock 46 0% 100% 0.00 2 0x6000b9b9e50 dsl_dir_tempreserve_impl 54 0% 100% 0.00 1 vn_vfslocks_buckets+0xb940 vn_vfslocks_getlock 33 0% 100% 0.00 3 0x600052febc0 syslwp_continue 70 0% 100% 0.00 1 0x60007c222c8 e1000g_send 31 0% 100% 0.00 2 0x300534fb6a0 polllock 28 0% 100% 0.00 2 0x6003fa1f360 strseteof 21 0% 100% 0.00 3 0x600364a9aa0 dbuf_dirty 8 0% 100% 0.00 9 pidlock proc_exit 41 0% 100% 0.00 1 0x30241935d08 strread 38 0% 100% 0.00 1 0x30000c73d10 ufs_lockfs_begin_getpage 23 0% 100% 0.00 3 pr_pidlock prinactive 45 0% 100% 0.00 1 vph_mutex+0x2000 page_hashout 39 0% 100% 0.00 1 0x30000c73d10 ufs_lockfs_begin 44 0% 100% 0.00 1 0x60038484280 releasef 18 0% 100% 0.00 3 0x6000b34e810 txg_thread_wait 13 0% 100% 0.00 4 0x60005306640 clock 37 0% 100% 0.00 1 0x60005306640 trap_cleanup 10 0% 100% 0.00 5 0x600054dc0b0 ehci_hcdi_pipe_reset 27 0% 100% 0.00 2 0x6000b2799c0 zil_commit 23 0% 100% 0.00 2 0x6000714d800 ssfcp_prepare_pkt 23 0% 100% 0.00 2 vn_vfslocks_buckets+0xb940 vn_vfslocks_rele 18 0% 100% 0.00 2 0x301c2eee3e0 fifo_close 19 0% 100% 0.00 2 0x6000b34e8a8 txg_list_add 19 0% 100% 0.00 2 0x6000c6f3e58 vdev_queue_io_done 27 0% 100% 0.00 1 lwpchanlock+0x1c98 lwp_cond_broadcast 23 0% 100% 0.00 2 0x6002b30a2c8 ufalloc_file 9 0% 100% 0.00 4 0x600054dc0b0 ehci_hcdi_pipe_ctrl_xfer 38 0% 100% 0.00 1 0x60038484280 getf 17 0% 100% 0.00 2 0x6000b2799c0 zil_itx_assign 24 0% 100% 0.00 1 0x301b77e6398 pollwakeup 27 0% 100% 0.00 1 pse_mutex+0x13c0 page_downgrade 17 0% 100% 0.00 2 0x30241935d08 strrput 18 0% 100% 0.00 2 0x30089bf9040 dbuf_read 16 0% 100% 0.00 2 0x1908130 page_list_add 12 0% 100% 0.00 2 0x60005308080 lwp_sigmask 11 0% 100% 0.00 3 0x6000556d218 ssd_core_iostart 16 0% 100% 0.00 2 0x300534fb6a0 fifo_read 20 0% 100% 0.00 1 0x6001f6d4c30 vx_iflush_list 15 0% 100% 0.00 2 0x300a005a060 aionotify 24 0% 100% 0.00 1 0x600054f60c8 ssfcp_scsi_init_pkt 2 0% 100% 0.00 16 0x6000d41f7e0 top_begin_async 14 0% 100% 0.00 2 0x60005306640 post_syscall 20 0% 100% 0.00 1 0x60005742b70 emlxs_msi_intr 6 0% 100% 0.00 4 0x6000556d298 ssd_buf_iodone 18 0% 100% 0.00 1 ufs_iuniqtime_lock ufs_imark 12 0% 100% 0.00 2 hat_lock+0xf00 sfmmu_hat_enter 14 0% 100% 0.00 1 0x60005742070 emlxs_tx_get 9 0% 100% 0.00 2 0x3007ad8de00 ipc_lock_internal 6 0% 100% 0.00 4 pidlock pr_lookup_procdir 6 0% 100% 0.00 3 0x300f92e6ed0 tcp_fuse_rrw 11 0% 100% 0.00 1 0x60005742070 emlxs_tx_put 8 0% 100% 0.00 2 0x60007134a78 tcp_bindi 13 0% 100% 0.00 1 0x19080f0 page_get_mnode_freelist 15 0% 100% 0.00 1 ph_mutex+0xf00 page_exists_physcontig 9 0% 100% 0.00 2 arc_eviction_mtx arc_buf_add_ref 9 0% 100% 0.00 1 0x60005742040 emlxs_issue_iocb_cmd 9 0% 100% 0.00 1 anoninfo_lock anon_resvmem 3 0% 100% 0.00 5 0x60005308080 syslwp_exit 7 0% 100% 0.00 2 0x6000c6f3918 vdev_queue_io 4 0% 100% 0.00 3 0x60007c22250 e1000g_receive 3 0% 100% 0.00 4 0x60031ff8c80 fifo_inactive 3 0% 100% 0.00 4 0x600071bfe08 syncq_service 4 0% 100% 0.00 3 0x6003c439cf0 getq_noenab 3 0% 100% 0.00 4 pidlock setpgrp 11 0% 100% 0.00 1 flock_lock flk_get_lock_graph 4 0% 100% 0.00 3 pse_mutex+0x13c0 page_lock_es 9 0% 100% 0.00 1 0x60037b85e90 tcp_fuse_output 5 0% 100% 0.00 2 0x6000726d440 squeue_fill 11 0% 100% 0.00 1 0x60038484240 delfpollinfo 11 0% 100% 0.00 1 0x60005741da8 emlxs_thread_trigger2 4 0% 100% 0.00 2 0x600057420f0 emlxs_register_pkt 6 0% 100% 0.00 1 tcp_random_lock tcp_random 7 0% 100% 0.00 1 0x3011b53f6d0 tcp_wput 4 0% 100% 0.00 2 0x600072eff20 rdip 3 0% 100% 0.00 3 0x6000726d440 squeue_worker 10 0% 100% 0.00 1 0x60007c22250 e1000g_rxfree_func 5 0% 100% 0.00 2 0x60007c222c8 e1000g_recycle 8 0% 100% 0.00 1 0x6001a27a540 ufs_root 4 0% 100% 0.00 2 0x600052fe7c0 lwp_unpark 4 0% 100% 0.00 2 lwpchanlock+0x2e80 lwp_cond_wait 4 0% 100% 0.00 2 pidlock freectty_lock 3 0% 100% 0.00 3 0x301d001b310 rwst_exit 3 0% 100% 0.00 3 0x60000463e28 brelse 5 0% 100% 0.00 1 0x1942d70 page_ctr_add 9 0% 100% 0.00 1 pcf+0x8 page_create_putback 2 0% 100% 0.00 4 0x6000d32a4d0 llt_recvdata 1 0% 100% 0.00 8 pidlock getproc 2 0% 100% 0.00 3 volgl_spin voliod_loop 4 0% 100% 0.00 1 0x600357bba10 zfs_getattr 2 0% 100% 0.00 3 pidlock pr_readdir_procdir 2 0% 100% 0.00 3 0x30217b0df10 pollcacheclean 3 0% 100% 0.00 2 0x60005305c40 syslwp_suspend 3 0% 100% 0.00 2 0x60005304f80 cv_wait_sig 3 0% 100% 0.00 2 0x301e08963c0 squeue_drain 1 0% 100% 0.00 6 0x301e08963c0 squeue_fire 4 0% 100% 0.00 1 0x6000b2799c0 zil_commit_writer 2 0% 100% 0.00 3 0x60031ff83c0 fifo_getattr 2 0% 100% 0.00 3 0x60001d19540 get_smap_kpm 3 0% 100% 0.00 2 0x60001d19540 segmap_getmapflt 6 0% 100% 0.00 1 0x301e41f9bd0 rwst_enter_common 5 0% 100% 0.00 1 netstack_shared_lock netstack_find_shared_zoneid 5 0% 100% 0.00 1 0x301e08963c0 squeue_enter 2 0% 100% 0.00 2 0x30000c36b70 ipif_refrele 1 0% 100% 0.00 5 0x6001f6d6010 vx_bc_subflush 1 0% 100% 0.00 5 0x6000b3435a8 free 1 0% 100% 0.00 5 zone_status_lock zone_status_timedwait 5 0% 100% 0.00 1 0x6003c439cf0 bcanput 4 0% 100% 0.00 1 0x300005ddf58 netstack_hold 4 0% 100% 0.00 1 0x300005ddf58 netstack_rele 2 0% 100% 0.00 2 0x600071e8380 smmap_common 5 0% 100% 0.00 1 0x30000c2f000 callout_schedule_1 5 0% 100% 0.00 1 0x1942de8 page_ctr_sub 4 0% 100% 0.00 1 pcf+0x8 page_create_wait 1 0% 100% 0.00 4 volgl_spin voliod_iohandle 4 0% 100% 0.00 1 0x60005741da8 emlxs_thread 4 0% 100% 0.00 1 0x30000c36b70 ipif_select_source 3 0% 100% 0.00 1 anonhash_lock+0xd8 anon_decref 4 0% 100% 0.00 1 anoninfo_lock anon_unresvmem 2 0% 100% 0.00 2 0x6002b219ab8 contract_process_exit 1 0% 100% 0.00 4 pidlock lwp_exit 3 0% 100% 0.00 1 0x60038485140 closeandsetf 3 0% 100% 0.00 1 0x60005741d60 emlxs_proc_ring 3 0% 100% 0.00 1 ARC_mfu+0x30 remove_reference 3 0% 100% 0.00 1 0x600070ba000 fc_ulp_transport 3 0% 100% 0.00 1 0x6000b2fcc90 ufs_addmap 1 0% 100% 0.00 3 0x6000b27f280 drain_syncq 2 0% 100% 0.00 1 0x600071e8380 segvn_create 2 0% 100% 0.00 1 0x60000463e28 getblk_common 1 0% 100% 0.00 2 0x30185cd8ee8 soqueueconnind 2 0% 100% 0.00 1 0x6000737ab00 solookup 1 0% 100% 0.00 2 0x6000b28be40 soft_ring_process 2 0% 100% 0.00 1 0x60005741d60 emlxs_handle_ring_event 1 0% 100% 0.00 2 0x6000c6f3970 vdev_cache_write 1 0% 100% 0.00 2 0x6000cb9d700 metaslab_free_dva 1 0% 100% 0.00 2 connmgr_lock connmgr_get 2 0% 100% 0.00 1 0x6000b2fcc90 ufs_getpage 1 0% 100% 0.00 2 0x6000b3435a8 alloccg 2 0% 100% 0.00 1 0x6000546aa40 vn_setpath 1 0% 100% 0.00 2 0x60032344a00 strdoioctl 2 0% 100% 0.00 1 zone0+0xa8 rctl_decr_swap 2 0% 100% 0.00 1 0x600054aaed0 secpolicy_spec_open 1 0% 100% 0.00 2 0x60005308780 vnodetopath_common 1 0% 100% 0.00 2 0x60005309140 brk_internal 2 0% 100% 0.00 1 0x6002b30a2c8 setf 1 0% 100% 0.00 2 0x600052fe500 fd_too_big 1 0% 100% 0.00 2 callb_table callb_add_common 1 0% 100% 0.00 2 vph_mutex+0x2000 page_hashin 2 0% 100% 0.00 1 pcf+0x408 page_create_va 1 0% 100% 0.00 2 0x70009b3c040 hat_unload_callback 1 0% 100% 0.00 2 0x70009b3c040 sfmmu_tteload_acquire_hashbucket 1 0% 100% 0.00 1 0x6002c0fa048 sotpi_sendmsg 1 0% 100% 0.00 1 0x6002c0fa048 sotpi_recvmsg 1 0% 100% 0.00 1 0x6000714d800 ssfcp_reconfig_wait 1 0% 100% 0.00 1 0x6000711f840 ssd_xbuf_iostart 1 0% 100% 0.00 1 0x6000b28be40 soft_ring_worker 1 0% 100% 0.00 1 0x600057420f0 emlxs_unregister_pkt 1 0% 100% 0.00 1 0x3014cd600c0 tcp_wput_accept 1 0% 100% 0.00 1 0x301dfc33540 squeue_enter_chain 1 0% 100% 0.00 1 0x60007c1d1b0 ip_rput_process_multicast 1 0% 100% 0.00 1 0x6001a321798 zfs_zaccess_common 1 0% 100% 0.00 1 0x60030437de8 zio_notify_parent 1 0% 100% 0.00 1 0x600071b6008 llt_send_generic 1 0% 100% 0.00 1 0x300101ed9c0 vx_itryhold_locked 1 0% 100% 0.00 1 0x300f4d2b2c8 vx_ilock 1 0% 100% 0.00 1 0x600072ef3a0 ufs_delmap 1 0% 100% 0.00 1 0x3028f7cb238 ufs_readlink 1 0% 100% 0.00 1 0x600072eff20 ufs_getattr 1 0% 100% 0.00 1 anonhash_lock+0x180 anon_alloc 1 0% 100% 0.00 1 0x302419fc730 entersq 1 0% 100% 0.00 1 0x60005306640 sigunintr 1 0% 100% 0.00 1 zone0+0xa8 rctl_incr_swap 1 0% 100% 0.00 1 0x30059fa8d98 pcache_delete_fd 1 0% 100% 0.00 1 plocks+0x70 pollhead_insert 1 0% 100% 0.00 1 0x301b77e6398 poll_common 1 0% 100% 0.00 1 lwpchanlock+0xd8 lwp_cond_signal 1 0% 100% 0.00 1 0x3020a5a85f0 closef 1 0% 100% 0.00 1 0x60007d32c40 exec_common 1 0% 100% 0.00 1 0x19080c8 page_list_sub 1 0% 100% 0.00 1 freemem_lock page_reclaim_mem ---------------------------------------------------------------------------= ---- Adaptive mutex block: 584 events in 60.113 seconds (10 events/sec) Count indv cuml rcnt nsec Hottest Lock Caller ---------------------------------------------------------------------------= ---- 573 99% 99% 0.00 217998 pse_mutex+0x1780 page_trylock 1 0% 100% 0.00 338225 0x6000c9ef6c0 metaslab_group_alloc 3 0% 100% 0.00 44578 0x60007208980 taskq_thread 2 0% 100% 0.00 35422 0x30000c2f000 untimeout 1 0% 100% 0.00 65333 0x30000c2f000 callout_execute 2 0% 100% 0.00 18885 0x60005741da8 emlxs_thread_trigger2 1 0% 100% 0.00 33073 0x60005436000 timeout_common 1 0% 100% 0.00 30542 0x60007208980 taskq_dispatch ---------------------------------------------------------------------------= ---- Spin lock spin: 6805 events in 60.113 seconds (113 events/sec) Count indv cuml rcnt spin Hottest Lock Caller ---------------------------------------------------------------------------= ---- 3187 61% 61% 0.00 4 cp_default disp_getbest 2698 27% 88% 0.00 2 cpu0_disp disp 574 6% 94% 0.00 2 0x6000724dda8 setbackdq 240 5% 99% 0.00 5 turnstile_table+0xb08 turnstile_lookup 62 1% 100% 0.00 3 0x600071683c0 setfrontdq 30 0% 100% 0.00 2 sleepq_head+0x1238 cv_block 3 0% 100% 0.00 5 cp_default setkpdq 9 0% 100% 0.00 1 0x6000724de38 idle 1 0% 100% 0.00 1 lwpsleepq+0x1ee8 lwp_release_all 1 0% 100% 0.00 1 sleepq_head+0x1a08 cv_signal ---------------------------------------------------------------------------= ---- Thread lock spin: 58 events in 60.113 seconds (1 events/sec) Count indv cuml rcnt spin Hottest Lock Caller ---------------------------------------------------------------------------= ---- 35 78% 78% 0.00 221 turnstile_table+0xf28 ts_tick 8 13% 91% 0.00 166 sleepq_head+0x1a08 setrun 3 4% 95% 0.00 136 sleepq_head+0xb48 prchoose 2 2% 98% 0.00 124 cpu[48]+0xf8 preempt 3 1% 99% 0.00 43 sleepq_head+0x14c8 ts_update_list 1 1% 99% 0.00 64 cpu[36]+0xf8 lwp_block 1 0% 100% 0.00 44 cpu[24]+0xf8 cv_wait_sig_swap_core 2 0% 100% 0.00 3 cpu[48]+0xf8 turnstile_block 1 0% 100% 0.00 2 shuttle_lock door_return 1 0% 100% 0.00 1 0x60007168150 sigtoproc 1 0% 100% 0.00 1 cpu[44]+0xf8 cv_timedwait_sig ---------------------------------------------------------------------------= ---- R/W writer blocked by writer: 4 events in 60.113 seconds (0 events/ sec) Count indv cuml rcnt nsec Hottest Lock Caller ---------------------------------------------------------------------------= ---- 4 100% 100% 0.00 60316 0x6002a8fde60 dmu_zfetch ---------------------------------------------------------------------------= ---- R/W writer blocked by readers: 3 events in 60.113 seconds (0 events/ sec) Count indv cuml rcnt nsec Hottest Lock Caller ---------------------------------------------------------------------------= ---- 1 98% 98% 0.00 4561202 0x3028f875518 ufs_write 2 2% 100% 0.00 49970 0x6002a8fde60 dmu_zfetch ---------------------------------------------------------------------------= ---- R/W reader blocked by writer: 9 events in 60.113 seconds (0 events/ sec) Count indv cuml rcnt nsec Hottest Lock Caller ---------------------------------------------------------------------------= ---- 9 100% 100% 0.00 39668 0x6002a8fde60 dmu_zfetch_find ---------------------------------------------------------------------------= ---- R/W reader blocked by write wanted: 2 events in 60.113 seconds (0 events/sec) Count indv cuml rcnt nsec Hottest Lock Caller ---------------------------------------------------------------------------= ---- 2 100% 100% 0.00 47484 0x6002a8fde60 dmu_zfetch_find ---------------------------------------------------------------------------= ---- On Apr 17, 12:12=A0pm, Mark Round <mark.ro...@gmail.com> wrote: > > > Anybody has any suggestions? I don't really know =A0how to investigat= e > > > this - all tools like vmstat and prstat seem to show normal values...= .. > > How about letting us see the output of vmstat / iostat / mpstat ? Try > running it for a minute or so when you're experiencing problems: > > # vmstat 60 2 > # mpstat 60 2 > # iostat -xz 10 2 > > And post the second set of output from each here. Also, if you're > seeing high spins on mutexes, then the output from lockstat might be > useful : > > # lockstat -kWP sleep 60 > > Perhaps that'll give us something to go on. What applications is this > server running ? > > -Mark

> It seems that a lot of locking occures due to page_trylock; I googled > around and I think it can behttp://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6642475 > The workaround is to set pg_contig_disable=1 in /etc/system. I've hit that one myself with ZFS, and I've heard of others running into it with Oracle systems. It certainly looks like that could be your problem - what applications are running on your zones ? -Mark

Similiar Articles:

Itanium performance problem - comp.sys.hp.hpux
I'm having a strange performance problem on Itanium HP-UX. I have a C program that gets data every 30 seconds, does calculations on the data and...

Strange problem with CPU spikes on solaris 8 - comp.unix.solaris ...
I am facing a strange problem of CPU spikes every 12-13 mins. This is a sun ... logrotate for HP-UX 11.31 Itanium - comp.sys.hp.hpux ... Strange performance problem - comp ...

hpux 11.31, vgimport: Unable to read the physical volume. - comp ...
Strange performance problem - comp.unix.solaris... HP-UX 11.31 Itanium - comp.sys.hp.hpux Itanium performance problem - comp.sys.hp.hpux comp.sys.hp.hpux logrotate for HP ...

Slow network performance on HP-UX 10.20 - comp.sys.hp.hpux ...
Itanium performance problem - comp.sys.hp.hpux I'm having a strange performance problem on Itanium HP-UX. ... Itanium emulation mode and got the same slow ... jimp: 11/5 ...

How to determine if PA-RISC or Itanium? - comp.sys.hp.hpux ...
Itanium performance problem - comp.sys.hp.hpux I'm having a strange performance problem on Itanium HP-UX. I have a C program that ... And again, the exact same code has ...

API performance problems - comp.cad.solidworks
strange OpenGl problem on Linux - comp.graphics.api.opengl ... I'm experiencing problems ... Answer : DMA Engine API performance issues DMA Engine API performance issues ...

about glDepthTest - comp.graphics.api.opengl
To increase the performance you should also precompute which parts are going to be ... [NEWBIE]Strange problem: I can see through objects! - comp ... about glDepthTest - comp ...

Memory problem - comp.lang.asm.x86
I've encountered a very strange problem with the timing of a block transfer ... BTW: To test the performance of a copyloop, 1,000,000 repetitions are ways ...

speed of Apache 2 on solaris 10 - comp.unix.solaris
The strange this is when I use FTP I'm getting 5 Mbps most of the time and ... I've had very good performance and no problems with SJSWS 7 preview and Apache 2.2.x finally ...

Berkeley DB Issue (possible Solaris problem) - comp.unix ...
Hello all, I have a strange problem with Solaris 10 and ... hints ... listing perfomance issue - comp.lang.java.gui The problem is that if the ... Solaris 10 Performance ...

Lightroom 4.0 final: Very strange performance problem in develop ...
I discovered a very strange performance problem in LR4 final (64 bit, Win7): 1. Load a RAW file into develop module, e.g. from a Canon 40D 2. Reset to adobe defaults 3.

MS SQL Server :: Strange Performance Problem With SELECT COUNT(1 ...
Strange Performance Problem With SELECT COUNT(1) And Sp_executesql Hello, I have the following queries that run on a view called EntrySummary:

7/11/2012 12:18:48 AM