Re: Too many files in one directory (again) #2

From: =?ISO-8859-1?Q?Jan-Erik_S=F6derholm?= <jan-erik.soderholm@telia.com> > > I just pass this along as a reminder that there's still some room > > for improvement in dealing with cases like this which,... > > And some of them are under your control, such as using a large > enough /ALLOCATION=n on the CRE/DIR command. I guess that it > would also help to INIT the device with a large enought > /HEADERS=n to begin with. Probably would. Drawbacks of this (and several other potentially helpful suggestions) is that _I_ didn't create the directory, VMSTAR did. The archive entries look like "zip/020989f4d6c2f32768d0535c1815344d.zip". Now I could, in principle, analyze the whole "tar" archive, puzzle out the optimal directory characteristics, and pre-create the things myself rather than trusting VMSTAR to know how to create directories. How many people believe that a user should need to go through that ordeal to avoid this problem? > Even if 190.000 files "works" on VMS, it might not be what > VMS was primarily designed for... So, it couldn't be improved to handle this case better? > How large/small is each individual file ? Pretty small, I believe. > Would it be possible (with 4 Gb available) to > create a DECram disk and run the un-tar against > that ? May be, but it'd probably be tight. And this is all on a system which I don't keep powered on constantly. From: "Main, Kerry" <Kerry.Main@hp.com> > Just a WAG, but since you are doing mostly write activities, can we assume > That you have removed disk highwater marking and set OpenVMS to the file > system default That UNIX uses (write back) vs OpenVMS's file system default > (write through)? I didn't. I didn't realize how doomed I was until I stepped well into this tar-pit. Volume Status: ODS-5, subject to mount verification, protected subsystems enabled, file high-water marking, write-through caching enabled, hard links enabled. When I get a chance, I'll try to look into VMSTAR to see if it is (or could be) using SQO to help avoid highwater marking trouble. In the past, it only bothered me when allocating space for a big file, and I believe that the files themselves are all pretty small. Should I still worry? From: Hein RMS van den Heuvel <heinvandenheuvel@gmail.com> > > [...] Even growing the allocation by, say, 25% > > instead of one block might pay off without a big penalty > > It will be growing by at least a disk cluster at a time. > I don't think it'll honor the SET RMS/EXT It's probably the default for 143374741 blocks (SEAGATE ST373405LW) ODS5: Cluster size 16 but directory size growth seems to be better than that: IT $ pipe t ; dire /size = all /nohead /notrai IT$DKA100:[cert]zip.dir 21-MAR-2008 15:23:00 IT$DKA100:[cert]zip.DIR;1 26272/26272 IT $ pipe t ; dire /size = all /nohead /notrai IT$DKA100:[cert]zip.dir 21-MAR-2008 15:23:04 IT$DKA100:[cert]zip.DIR;1 26273/26384 Noticable (ten second?) pause waiting for that "26273/26384". > MONI FILE would be interesting FILE SYSTEM CACHING STATISTICS on node IT 21-MAR-2008 15:18:54.22 CUR AVE MIN MAX Dir FCB (Hit %) 100.00 100.00 100.00 100.00 (Attempt Rate) 10.66 10.22 10.00 10.66 Dir Data (Hit %) 70.00 70.66 70.00 72.00 (Attempt Rate) 6701.00 6738.11 6330.33 7183.00 File Hdr (Hit %) 89.00 88.66 88.00 89.00 (Attempt Rate) 9.66 9.33 9.00 9.66 File ID (Hit %) 100.00 100.00 100.00 100.00 (Attempt Rate) 1.00 1.00 1.00 1.00 Extent (Hit %) 100.00 100.00 100.00 100.00 (Attempt Rate) 1.00 1.00 1.00 1.00 Quota (Hit %) 0.00 0.00 0.00 0.00 (Attempt Rate) 0.00 0.00 0.00 0.00 Bitmap (Hit %) 0.00 0.00 0.00 0.00 (Attempt Rate) 0.00 0.00 0.00 0.00 IT $ sho mem /cac System Memory Resources on 21-MAR-2008 15:21:07.91 Extended File Cache (Time of last reset: 20-MAR-2008 05:59:48.57) Allocated (GBytes) 1.48 Maximum size (GBytes) 1.99 Free (GBytes) 0.00 Minimum size (GBytes) 0.00 In use (GBytes) 1.48 Percentage Read I/Os 22% Read hit rate 78% Write hit rate 0% Read I/O count 75035 Write I/O count 258987 Read hit count 58781 Write hit count 0 Reads bypassing cache 81 Writes bypassing cache 1521 Files cached open 548 Files cached closed 18757 Vols in Full XFC mode 0 Vols in VIOC Compatible mode 4 Vols in No Caching mode 0 Vols in Perm. No Caching mode 0 (Must be that lack of "Full XFC mode". Tee-hee-hee.) If I could find some dirt-cheap 2GB memory modules, I'd double it again to 8GB, but 4GB is all the budget would allow. > btw, do a semi random: > $ pipe dump/dir/blo=3D(star=3D10000,count=3D10) bad.dir | searc sys$pipe > "End of records" IT $ pipe t ; dire /size = all /nohead /notrai IT$DKA100:[cert]zip.dir 21-MAR-2008 15:33:24 IT$DKA100:[cert]zip.DIR;1 26376/26384 IT $ pipe dump/dir/blo=(star=10000,count=10) IT$DKA100:[cert]zip.dir | sear sys$ pipe "End of records" 012C End of records (-1) 012C End of records (-1) 012C End of records (-1) 012C End of records (-1) 012C End of records (-1) 012C End of records (-1) 012C End of records (-1) 012C End of records (-1) 012C End of records (-1) 012C End of records (-1) IT $ pipe t ; dire /size = all /nohead /notrai IT$DKA100:[cert]zip.dir 21-MAR-2008 15:48:19 IT$DKA100:[cert]zip.DIR;1 26523/26608 IT $ pipe dump/dir/blo=(star=10000,count=10) IT$DKA100:[cert]zip.dir | sear sys$ pipe "End of records" 012C End of records (-1) 012C End of records (-1) 012C End of records (-1) 012C End of records (-1) 012C End of records (-1) 012C End of records (-1) 012C End of records (-1) 012C End of records (-1) 012C End of records (-1) 012C End of records (-1) Seems pretty stable at "012C". From: koehler@eisner.nospam.encompasserve.org (Bob Koehler) > Can you put a listing in a file using tar -t and then break down > the output into multiple tar passes into multiple directories? And how should I specify the division of labor to VMSTAR? > I sure think I'd try that if I were in your situation. A little > time editing to create a script might save a great many hours. Complaining can be more fun than working, and (in my dreams) could be more likely to lead to some actual improvement than quietly finding some annoying work-around. From: JF Mezei <jfmezei.spamnot@vaxination.ca> > TAR unpacks in mydir1.dir > You create mydir2 through mydir200 > > You have a separate process which runs say every minute, and does a > RENAME [.mydir1]*.* [.mydirX] where X increases with every run and goes > back to 2 after 200. Or I could repackage the stuff in smaller pieces on the HP-UX system, or I could try breaking NFS by getting the files served from the HP-UX system, ... Many things are possible. Most of them, I claim, shouldn't be needed. From: "Richard B. Gilbert" <rgilbert88@comcast.net> > A sane VMS user, rather than create 190,000 files, might consider an > indexed sequential file to hold the information. [...] I didn't design this exercise. Someone else created a dump-truck-full of Zip (and other compress/archive) test files, mostly diddled in creative ways (they say) in an attempt to find problems in various expand/extract software. The packaging was probably done on a Linux system, where, I'd assume, no one saw any particular problems with the scheme. I just thought that it might be productive to run the stuff through the next version of UnZip to see if we had any problems on VMS. ------------------------------------------------------------------------ Steven M. Schweda sms@antinode-org 382 South Warwick Street (+1) 651-699-9818 Saint Paul MN 55105-2547

> -----Original Message----- > From: Steven M. Schweda [mailto:sms@antinode.org] > Sent: March 21, 2008 3:46 PM > To: Info-VAX@Mvb.Saic.Com > Subject: Re: Too many files in one directory (again) > [snip ..] > > From: "Main, Kerry" <Kerry.Main@hp.com> > > > Just a WAG, but since you are doing mostly write activities, can we > assume > > That you have removed disk highwater marking and set OpenVMS to the > file > > system default That UNIX uses (write back) vs OpenVMS's file system > default > > (write through)? > > I didn't. I didn't realize how doomed I was until I stepped well > into this tar-pit. > > Volume Status: ODS-5, subject to mount verification, protected > subsystems > enabled, file high-water marking, write-through caching enabled, > hard > links enabled. > > When I get a chance, I'll try to look into VMSTAR to see if it is (or > could be) using SQO to help avoid highwater marking trouble. In the > past, it only bothered me when allocating space for a big file, and I > believe that the files themselves are all pretty small. Should I still > worry? > > Might not be applicable, but perhaps worth a try to see if the process appears to speed up some: Disk high water marking can be changed dynamically without dismounting the drive, so while you might be afraid to stop the job while it is running, you should be able to change it without impacting the job by entering: $ set volume $1$duax /nohighwater The same is true for modifying the sysgen parameter RMS_SEQFILE_WBH Reference: sysgen> help sys_p RMS_SEQFILE_WBH (Alpha and I64) RMS_SEQFILE_WBH can enable the RMS write behind feature as a system default for any unshared sequential disk file if the file is opened for image I/O with write access specified. [snip ..] Regards Kerry Main Senior Consultant HP Services Canada Voice: 613-254-8911 Fax: 613-591-4477 kerryDOTmainAThpDOTcom (remove the DOT's and AT) OpenVMS - the secure, multi-site OS that just works.

In article <08032114454569_2020CE0A@antinode.org>, sms@antinode.org (Steven M. Schweda) writes: > From: =?ISO-8859-1?Q?Jan-Erik_S=F6derholm?= <jan-erik.soderholm@telia.com> > >> > I just pass this along as a reminder that there's still some room >> > for improvement in dealing with cases like this which,... >> >> And some of them are under your control, such as using a large >> enough /ALLOCATION=n on the CRE/DIR command. I guess that it >> would also help to INIT the device with a large enought >> /HEADERS=n to begin with. > > Probably would. Drawbacks of this (and several other potentially > helpful suggestions) is that _I_ didn't create the directory, VMSTAR > did. The archive entries look like > "zip/020989f4d6c2f32768d0535c1815344d.zip". Now I could, in principle, > analyze the whole "tar" archive, puzzle out the optimal directory > characteristics, and pre-create the things myself rather than trusting > VMSTAR to know how to create directories. The nice part of that approach is that from the general description of the problem it would seem you would not have to create many directories :-)

On Mar 21, 3:45=A0pm, s...@antinode.org (Steven M. Schweda) wrote: > From: =3D?ISO-8859-1?Q?Jan-Erik_S=3DF6derholm?=3D <jan-erik.soderh...@teli= a.com> > > > > =A0 =A0I just pass this along as a reminder that there's still some ro= om > > > for improvement in dealing with cases like this which,... > > Even if 190.000 files "works" on VMS, it might not be what > > VMS was primarily designed for... > > =A0 =A0So, it couldn't be improved to handle this case better? Yes. Maybe, Just maybe, Andy will get to release a b-tree based implementation. > > create a DECram disk and run the un-tar against that ? > > =A0 =A0May be, but it'd probably be tight. =A0And this is all on a system > which I don't keep powered on constantly Create an LD container first. Copy the container to a the real disk. > =A0 =A0I didn't. =A0I didn't realize how doomed I was until I stepped well= into this tar-pit. Oh yeah. Classic. Can't kill it now.. it must almost be done. :-) Just wait till you get to delete them! > > > [...] =A0Even growing the allocation by, say, 25% > > > instead of one block might pay off without a big penalty Mark Hopkins whisperred to me "1) A directory extends by the maximum of 100 blocks or 1/2 the current file size (I just looked in the code). I'm pretty sure that this is rounded up to the next cluster factor. None of the default extends are honored here. 2) If someone hasn't already suggested it, increase acp_maxread from 32 to 64. 3) Turn on RMS_SEQFILE_WBH if possible." > Noticable (ten second?) pause waiting for that "26273/26384". That suggests that there was no free space found next to the current allocation and the directory was copied ACP_MAXREAD blocks at a time I made the DUMP/HEAD | grep LBN suggestion to check just that. Pre-allocation would have prevented that delay. Suspend the job (if not done by now), use DFU to expand, resume. > > MONI FILE would be interesting > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0FILE SYSTEM CACHING STA= TISTICS > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0CUR =A0 =A0 =A0 =A0AVE =A0 =A0 =A0 =A0MIN =A0 =A0 =A0 =A0MAX > =A0 =A0 Dir Data =A0 (Hit %) =A0 =A0 =A0 =A0 =A0 =A0 =A0 70.00 =A0 =A0 =A0= 70.66 =A0 =A0 =A070.00 =A0 =A0 =A072.00 > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0(Attempt Rate) =A0 =A0 =A06701.00 =A0 =A067= 38.11 =A0 =A06330.33 =A0 =A07183.00 Yikes! Is is doing some readdir before doing the create files? > IT $ pipe dump/dir/blo=3D(star=3D10000,count=3D10) IT$DKA100:[cert]zip.dir= | sear sys$ > pipe "End of records" > 012C =A0End of records (-1) : > 012C =A0End of records (-1) > Seems pretty stable at "012C". So all blocks are nicely filled, suggesting files are nicely added in sequence. In that case actuall adding of files 100..200 should be no slower than 100100 .. 100200. But the directory growth / reallocation will be getting progressively worse as you noticed. > =A0 =A0Complaining can be more fun than working, and (in my dreams) could = be > more likely to lead to some actual improvement than quietly finding some > annoying work-around. Agreed! Too many poor souls out there do not even have the first clue to work around this. > From: JF Mezei <jfmezei.spam...@vaxination.ca> > > > TAR unpacks in mydir1.dir > > You create mydir2 through mydir200 > > > You have a separate process which runs say every minute, and does a > > RENAME [.mydir1]*.* [.mydirX] =A0where X increases with every run and go= es > > back to 2 after 200. clueless > =A0 =A0Many things are possible. =A0Most of them, I claim, shouldn't be ne= eded. Agreed. > From: "Richard B. Gilbert" <rgilber...@comcast.net> > > > A sane VMS user, rather than create 190,000 files, might consider an > > indexed sequential file to hold the information. =A0[...] Or an indexed file to hold name/fid tupples for directory less files. :-) JF>> Out of curiosity, why would creating 190,000 files in a VMS directory be significantly slower than on Unix ? Because it uses a writeback model, not write through. That works most of the time. If 10 entries are made to a directory block, VMS will do 10 actual write IO. Your typical Unix will just do 1 IO, or none. Depends on the update deamon. Unix'es also have a variety of filesystems, many log based, making it a little safer. JF> did you CREATE/DIRECTORY/ALLOCATION=3Dxxxxxxx ? JF> Would this have made a huge difference in the file creation speed ? Yes, with the evidence currently in play. JF> Would it make sense to SET FILE mydir.dir;1/global=3D5000 (or whatever) ? Absolutely not. For starters, directories do use the normal RMS buffer caches. But if they did... Global Buffers cache buffers at certain VBN location. Well, after a few not-at-the-end inserts all blocks are shuffled up. So the contents of VBN 10 moves to VBN 11 and so. Not good for a VBN tag cache. JF> So when I do an "ls" in Unix, it does an in memory sort of the directory before listing it ? Unix has these 'inodes' which are like a mini file header, but they live in the directory. They hold the 'stat' info: byte-size, dates, owner,... So for unix is it trivial to list sizes or present sorted by date. OpenVMS has to do 1 IO per file for attributes other than name or file- id. This IOs may be cached, but still. Steven>> it should also (now) be using 127 for the multi-block count ("fil_rab.rab$b_mbc"), and using two buffers ("fil_rab.rab$b_mbf"), and setting write-behind ("fil_rab.rab$l_rop | =3D RAB$M_WBH;") on Some argue to make it a multiple of 4 (124) to make live easier on certain storage types (EVA Raid-5). Others recommend a multiple of 16 (112) to make life easier for the XFC (16 block cache lines) RMS WHB is a simple to use way to get concurrent IO going and more often than not well worth the CPU price. If CPU usage becomes a concern, then self-managed $WRITE, with $RAB64 may become desirable. For even less CPU time, go down to QIO and for the least possible CPU overhead, $IO_PERFORM. 2 buffers should be just fine. 4 often helps a little more. Beyond that I find rapidly diminishing returns. Cheers, Hein.