f



Re: Compression should have been disabled

Hi Dave -

I think the key is "not possible".  In SAS's view your file was potentially
compressible, but in actuality wasn't.  Whatever algorithm they use didn't
detect your scenario.

My experience is that wide files, especially those with large alpha
variables compress the best.  Narrow files, such as yours with 8 variables,
especially if variables are numeric or narrow width character, don't do as
well.

You would do well to test the results with a subset of your file (10,000
instead of 180 million) before bothering to compress the whole shebang.
Look into "obs=" for your program or datastep options.

Regards

Paul Choate
DDS Data Extraction
(916) 654-2160

-----Original Message-----
From: SAS(r) Discussion [mailto:SAS-L@LISTSERV.UGA.EDU] On Behalf Of DB01
Sent: Wednesday, February 02, 2005 10:22 AM
To: SAS-L@LISTSERV.UGA.EDU
Subject: Compression should have been disabled

Regarding compression in the SAS online doc, SAS will disable
compression if "...not possible for the compressed file to be smaller
than an uncompressed one."  However, I've experienced a number of
occasions where the compressed file was about 50% larger than the
uncompressed version.  Just wondering why SAS did not disable
compression in these cases.  The log below shows such an example:

/******************************************************/
379      data lib._history;
380          set allsort;
381      run;

NOTE: There were 180248070 observations read from the data set
WORK.ALLSORT.
NOTE: The data set LIB._HISTORY has 180248070 observations and 8
variables.
NOTE: Compressing data set LIB._HISTORY increased size by 54.12
percent.
Compressed is 1097986 pages; un-compressed would require 712444
pages.
NOTE: DATA statement used (Total process time):
real time           18:43.88
user cpu time       9:29.72

/******************************************************/


Thanks,
Dave
0
pchoate (2538)
2/2/2005 6:58:21 PM
comp.soft-sys.sas 142828 articles. 2 followers. Post Follow

0 Replies
901 Views

Similar Articles

[PageSpeed] 50

Reply:

Similar Artilces:

Re: Compressing data sets (was Re: [SAS-L])
On Mon, 5 Jan 2004 18:14:02 -0700, Jack Hamilton <JackHamilton@FIRSTHEALTH.COM> wrote: >OK, "you or a program on your behalf will have to decompress them before >use". > > > >-- >JackHamilton@FirstHealth.com >Manager, Technical Development >Metrics Department, First Health >West Sacramento, California USA > >>>> "Richard Graham" <richardwgraham@earthlink.net> 01/05/2004 5:04 PM >>>> >Actually you can compress(WINZIP, PKZIP) SAS data sets and use them in >the >compressed format. There is software...

Re: Compressing data sets (was Re: [SAS-L]) #2
OK, "you or a program on your behalf will have to decompress them before use". -- JackHamilton@FirstHealth.com Manager, Technical Development Metrics Department, First Health West Sacramento, California USA >>> "Richard Graham" <richardwgraham@earthlink.net> 01/05/2004 5:04 PM >>> Actually you can compress(WINZIP, PKZIP) SAS data sets and use them in the compressed format. There is software named ZipMagic which allows zip files to be used as windows folders. I have done this with mixed results when the underlying data set is upwards of 4GB. Bu...

Re: Compressing data sets (was Re: [SAS-L]) #2 #6
A new way to read/write Excel files in 9.1 was mentioned at SUGI, but as I recall it required a server running SAS/Access to PC File Formats. It wasn't SAS/Connect doing the reading and writing, just as SAS/Connect isn't doing the remote reading and writing in a regular Connect session. Or are you thinking of something else? -- JackHamilton@FirstHealth.com Manager, Technical Development Metrics Department, First Health West Sacramento, California USA >>> "Bruce Johnson" <bjohnson@SOLUCIENT.COM> 01/06/2004 10:02 AM >>> "using SAS/Connect to...

Re: Compressing data sets (was Re: [SAS-L]) #2 #7
I'm crazy..yes, SAS/Access...not Connect... ________________________________ Bruce A. Johnson bjohnson@solucient.com -----Original Message----- From: Jack Hamilton [mailto:JackHamilton@firsthealth.com] Sent: Tuesday, January 06, 2004 1:22 PM To: SAS-L@LISTSERV.UGA.EDU; Bruce Johnson Subject: Re: [SAS-L] Compressing data sets (was Re: [SAS-L]) A new way to read/write Excel files in 9.1 was mentioned at SUGI, but as I recall it required a server running SAS/Access to PC File Formats. It wasn't SAS/Connect doing the reading and writing, just as SAS/Connect isn't doing the remote...

Re: Compressing data sets (was Re: [SAS-L]) #2 #4
It's not a big deal, but there is a difference between reading a file directly and having another program as an intermediary, and someone someday is going to get into trouble because they didn't take that into account. I'm not sure what you mean by "using SAS/Connect to read non-SAS files". -- JackHamilton@FirstHealth.com Manager, Technical Development Metrics Department, First Health West Sacramento, California USA >>> "Bruce Johnson" <bjohnson@SOLUCIENT.COM> 01/06/2004 9:36 AM >>> But if another program is doing it, within SAS, wh...

Re: RE : Re: copulas and SAS
Adel, Think Joke! But, more seriously, it did prompt me to look up the origins of the copula as a functional. Joe On 9/27/06, adel F. <adel_tangi@yahoo.fr> wrote: > > Hi, > I do not know how this is related to my orginal question. > > My understanding of copula is, it is a joint distribution with uniform > margins, the problem how to obtain a copula which fits better a data? > > If there is any suggestion from members of SAS list, about examples for > copulae with SAS, I will appreciate that > > Thanks a lot > > Adel > Joe Whitehurst <joewhitehurst@GMAIL.COM> a �crit : > David, > > "The complete attempt to deal with the term is would go to the form > and matter of every thing in existence, at least, if not to the > possible form and matter of all that does not exist, but might. As > far as it could be done, it would give the grand Cyclopaedia, and its > yearly supplement would be the history of the human race for the > time." > > On 9/26/06, David L Cassell proclaimed: > > "... A copula is nothing more than a functional allowing you to link > the multivariate distribution to marginal distributions in an > invariate manner...". > > This caused Augustus De Morgan to leap from his grave and refute David > thusly: > > "The complete attempt to deal with the term "is" would go to the form > and matter of every thing in existence, a...

Re: Disabling SAS warnings
or options nosource and nosource2 Toby Dunn From: charles <ylchen9999@GMAIL.COM> Reply-To: charles <ylchen9999@GMAIL.COM> To: SAS-L@LISTSERV.UGA.EDU Subject: Re: Disabling SAS warnings Date: Mon, 23 Jan 2006 07:44:34 -0800 MIME-Version: 1.0 Received: from mailgw.cc.uga.edu ([128.192.1.101]) by bay0-mc11-f3.bay0.hotmail.com with Microsoft SMTPSVC(6.0.3790.211); Mon, 23 Jan 2006 07:45:51 -0800 Received: from listserv.cc.uga.edu (listserv.uga.edu [128.192.1.75])by mailgw.cc.uga.edu (8.12.11/8.12.11) with ESMTP id k0NEpU6K001474;Mon, 23 Jan 2006 10:45:07 -0500 Received: from LISTSERV.UGA.EDU by LISTSERV.UGA.EDU (LISTSERV-TCP/IP release 1.8d) with spool id 1279347 for SAS-L@LISTSERV.UGA.EDU; Mon, 23 Jan 2006 10:45:07 -0500 Received: from alfredo.cc.uga.edu (alfredo.cc.uga.edu [128.192.1.77]) by listserv.cc.uga.edu (8.12.11/8.12.11) with ESMTP id k0NFj7NT031409 for <sas-l@listserv.uga.edu>; Mon, 23 Jan 2006 10:45:07 -0500 Received: from alfredo.cc.uga.edu (localhost.localdomain [127.0.0.1]) by alfredo.cc.uga.edu (8.12.11/8.12.11) with ESMTP id k0NFj6JV025355 for <sas-l@listserv.uga.edu>; Mon, 23 Jan 2006 10:45:06 -0500 Received: (from news@localhost) by alfredo.cc.uga.edu (8.12.11/8.12.11/Submit) id k0NFj6RD025350 for sas-l@listserv.uga.edu; Mon, 23 Jan 2006 10:45:06 -0500 X-Message-Info: APgguKHtcSBiaU7EwZ8zTsYve3x7ah6o8qi+1LcSOik= X-Authentication-Warning: alfredo.cc.uga.edu: news set ...

how to add fonts to the sas registry (was RE: Re: sas/graph:
Success! A google search led me to: http://support.sas.com/documentation/onlinedoc/base/91/freetype-hub.pdf Which includes the magic option on FONTREG to use the "all" mode. (And now that I look back on the FONTREG docs, I see it there was well :-P). So here's code that works: * =================================== ; proc fontreg mode = all msglevel = verbose ; truetype 'c:\windows\fonts' ; run ; data phoney ; do n = 1 to 2000 ; x = ranuni(-1) ; if n le 1000 then grp = "one" ; else grp = "two" ; output ; end ; run ; goptions device = EMF ftext = 'Tahoma' ; ods rtf file = "c:\temp\deleteme.rtf" ; proc boxplot data = phoney ; plot x * grp ; run ; ods rtf close ; * =================================== ; Looks like I've got to add that FONTREG call anew every time (running in batch anyway). But that's tolerable... Thanks all! -Roy -----Original Message----- From: SAS(r) Discussion [mailto:SAS-L@LISTSERV.UGA.EDU] On Behalf Of Pardee, Roy Sent: Thursday, February 28, 2008 8:12 AM To: SAS-L@LISTSERV.UGA.EDU Subject: Re: sas/graph: using truetype fonts w/ODS RTF output on windows Thanks for the response Mary! Huh, strange. So you run that code unaltered & get Tahoma in the graphic in the .rtf file? I just now tried adding this before the 'data phoney ;' line: proc fontreg mode = replace ; truetype 'c:\windows\fonts' ; run ; And I get ...

Re: SAS Job websites WAS(Re: SAS jobs for freshers in US
On Fri, 24 Mar 2006 11:27:44 -0500, Steven Raimi <steven.raimi@ONSTAR.COM> wrote: >Another very helpful (and focused) web site for SAS jobs is >icrunchdata.com. I'll try to ....post from home later, where I have a set of links (I don't have them at my new job here, which, FYI, I found via Monster). I'll also try to avoid fumble-fingering that accidently posts my messages before I'm done writing them! Steve ...

Re: SAS Sample 1728 Was ---> Re: SAS Sample 1727
Nice thread guys .... I guess this is what open source development is all about .... Got me thinking about suggesting a code/macro library on sascommunity.org? Any thoughts? ...

Re: RE : Re: copulas and SAS #3 1556650
Adel, You may find the following artilce interesting. COPULAS: TALES AND FACTSy THOMAS MIKOSCH \But he does not wear any clothes" said the little child in Hans Christian Andersen's \The Emperor's New Clothes". 1. Some preliminary facts When I started writing the paper [19] in 2003 a Google search of the word \copula" gave 10,000 responses. In September 2005 the same search gives 650,000 responses. There is an explosion of activity. What is going on? Many of the web-sites1 found in the Google search are related to mathematical nance, statistics, extreme value theory, and risk management. Everybody who opens any journal on stochastic processes, probability theory, statistics, econometrics, risk management, nance, insurance, etc., observes that there is a fast growing industry on copulas. The commercial statistics software Splus provides the module FinMetrics that includes copula tting which is written by R. Carmona, see also Carmona [4]. One can also get copula modules in other major software packages (R, Mathematica, MatLab, etc.). The International Actuarial Association [14] in its hefty paper on Solvency II2 recommends using copulas for modeling dependence in insurance portfolios. Moody's uses Gaussian copulas for modeling credit risk and provides software for it which is used in many nancial institutions. Since Basle II3 copulas are now standard tools in credit risk management. The main purpose of this paper is to ask some na ve questions a...

Re: RE : Re: copulas and SAS #3 674067
joewhitehurst@GMAIL.COM replied: > >Adel, > >You may find the following artilce interesting. [article by Thomas Mikosch not copied] This is a really good article, and a really useful point to make. But 55K is a lot of article to copy. Would it have been easier to point to a URL? The people who read SAS-L as a digest and the people who have slow connections are probably not overly thrilled. I'll refrain from pointing out that you sound like me today. ("Are you sure you want to do this? Here's some material which suggests you should try a different approach..."...

Re: Reading A Compressed SAS Dataset
Dear SAS-L-ers, Satindra Chakravorty posted the following interesting question: > I am wondering if anyone knows how one might read a > UNIX-compressed (with the UNIX compress program, not with the > SAS compress option) SAS dataset (not ASCII file) directly, > via a SAS program. > > This is apparently not as easy as reading an ASCII file that > has been compressed (with either the UNIX compress or gzip > programs) directly via a SAS program. > > I tried a piece of code I found on the internet and couldn't > get it to work. The code looks like this... > > x 'mknod piper p'; > libname h "piper"; > filename nwrpipe pipe ' uncompress > piper < test.sas7bdat.Z &'; > > data _null_; > infile nwrpipe; > run; > > data test2; > set h.test; > run; > > In the code above, "test.sas7bdat.Z" is the UNIX-compressed > SAS dataset I was trying to read. I think this code didn't > work because the input file was compressed by simply using > the compress command in a UNIX shell on the command line > rather than through a similar SAS process. > > If anyone knows of another way to do this, please let me know. > Satindra, hmmm.... I am sitting in my alternate Westat office and do not have access to my SAS/UNIX material, but I think that I may have a suggestion or two that can help. First, double check that your "uncompress > piper < te...

Re: SAS Sample 1728 Was ---> Re: SAS Sample 1727 #2
Graham , I am not sure that the SI Tech Support examples should have all the error checking that Chang wanted to throw in the macro. The examples should be clear concise and easy to read and get going, as well as be illistrative. Too much error checking in an example well is just over kill and even detrimental to an example. However, any code solution on the SI Tech Support web sit should follow basic rules of good practices and sound advice. Such as with thier macros having atleast a somewhat good Macro design. Look at Example SAS Sample 1728: The first method well lets say is soooo bad...

Re: read compressed SAS data
Haven't seen that done ... A pipe command would, I believe, expect to stream a text file. Using zcat to stream data from a compressed XML document might work, but I haven't tried that. S -----Original Message----- From: owner-sas-l@listserv.uga.edu [mailto:owner-sas-l@listserv.uga.edu] On Behalf Of Wensui Liu Sent: Thursday, January 31, 2008 2:54 PM To: SAS(r) Discussion Subject: read compressed SAS data Dear All, in our unix server, we archived SAS data by compressing them to Z file. I am wondering if there is a way that I can read the compressed SAS data from Z file without uncompressing it. Thanks a lot for your insight! ...

Re: Disabling SAS warnings #4
Hi David, And thanks to all those of you who responded. I am experimenting difficulties in sending emails lately. I see that others have helped already. But let me make a small suggestion. If you are putting together a linear equation to feed into a proc, then there may be several things wrong. Yes, I am feeding the equation into proc model (2sls and ols models). [1] Using macros to do this tends to lead to over-use of the procedure without addressing the appropriate experiment-wise error rate. I can't see what you're doing from way over here, but this is often wh...

Re: SAS Sample 1728 Was ---> Re: SAS Sample 1727 #3
First, let me admit/acknowledge that I have no knowledge of the history of this thread. Been tied up catching up post SAS Global Forum and post the announcement of sasCommunity.org. But I would like to offer some thoughts regarding Toby's comment: > No as for the sascommunity.org, do we really want a macro library. Well I > would say yes and no. I would like to have one and there are those who > have > paced some Macros on there already. I believe Don and Ron both have, > actually they were on there before the Wiki went public. However, since > people tend to get Ma...

Re: RE : SAS/AF
Unnat, I'm not sure what you mean by "Microsoft Windows which has been installed on Unix server", but you can use an X Window emulator on a MS Windows client to interact with a SAS/AF application executing on a UNIX server. RandyHerbison@westat.com -----Original Message----- From: SAS(r) Discussion [mailto:SAS-L@LISTSERV.UGA.EDU] On Behalf Of saspatel Sent: Tuesday, July 20, 2004 5:33 PM To: SAS-L@LISTSERV.UGA.EDU Subject: RE : SAS/AF Hi, I developed SAS/AF application and deployed on UNIX server. I would like to know that if it is possible to access SAS/AF application on Microsoft Windows which has been installed on Unix server. If it is possible then how and what kind of products need to be installed on Unix OR Windows. Regards, Unnat ...

Re: RE : SAS and French
Je lis, mais je ne parle pas - c'est trop difficile. -- JackHamilton@FirstHealth.com Manager, Technical Development Metrics Department, First Health West Sacramento, California USA >>> Datametric <datametric@CLUB-INTERNET.FR> 10/12/2004 12:23 PM >>> Hi dave, The sas-l is read by a lot of french spoken people. Vous pouvez envoyer vos questions, nous y r�pondrons. De m�moire, beaucoup de canadiens parlant le francais sont pr�sents �galement. De m�me, des am�ricains parlent et lisent le fran�ais, n'est ce pas Jack ? Cdt, St�phane. -----Message d'origin...

Compressing data sets (was Re: [SAS-L])
Yes, they can be compressed using WinZIP, but you will have to decompress them before using them them. I don't know whether SAS can handle zipped folders in Windows XP. You can also use SAS data set options (compress=char or compress=binary) to give a rudimentary form of compression entirely within SAS. It's (to my knowledge) never better than WinZIP/PKZip compression, and sometimes much worse. Other common utilities are PKZip and Info-ZIP. There are literally dozens of compression packages out there. -- JackHamilton@FirstHealth.com Manager, Technical Development Metrics Depar...

Re: compress sas library on os/390
You can use the COMPRESS option. Take a look at my SUGI 25 paper "By Hook or By Crook: Overcoming Resource Limitations". I cover many mainframe strategies. -----Original Message----- From: PD [mailto:sophe88@YAHOO.COM] Sent: Wednesday, August 20, 2003 1:54 PM To: SAS-L@LISTSERV.UGA.EDU Subject: compress sas library on os/390 I sometime save sas data as permanent sets on mainframe. But unlike on Windows, sas library on os/390 can be full without warning. Under Windows, as long as my HD is not full, my sas library does not crash. I know I can allocate larger SAS library, copy the data sets over from full library. Question 1 is: this take a lot of time. Problem also is: even after I delete some old data sets from the problem library, the library is still full. In JCL allocation, I already told the system to release unused space. Any way to compress the library? Thanks PD ...

RE : Re: copulas and SAS
Thanks David, I have two variables X1 and X2 representing cognition scores , they are naturally associated, the corr=0.5 X1 is squewed on the right and X2 is fairly normal I have also a variable representing age (continuous 40+) and educational level (3 categories) My first idea is to find the best copula, archmedian copula for (X1,X2) which fits my data, and then to try to link this copula to age and eduction, a kind of quintile regression. I am looking for goodness of fit test for archimedan copula, they are easy to interpret, because of the association paramter Adel Da...

Re: SAS Windows Page Faults
On Wed, 19 Apr 2006 07:35:10 -0700, Don <dmorgan@ROCKETMAIL.COM> wrote: >Can the use of Options Compress=Yes lead to/contribute to an increase >in Page Faults/Second? yes! For compression memory is needed and if there is no more real memory, the virtual storage is used. Thanks for the reality Check! sas table and numeric variable compression is usually much more of a drain on CPU rather than memory "Gerhard Hellriegel" <ghellrieg@T-ONLINE.DE> wrote in message news:200604191528.k3JFKJoQ027774@mailgw.cc.uga.edu... > On Wed, 19 Apr 2006 07:35:10 -0700, D...

Re: What r the data types in SAS? in Base SAS , and SAS SQL
> From: Amar Mundankar > Sent: Tuesday, July 21, 2009 8:10 AM > To: sas-l@uga.edu > Subject: What r the data types in SAS? in Base SAS , and SAS SQL > > Hi all, > What are the different data types in Base SAS and SAS SQL?? character, lengths from 1 to 32,000+ numeric: lengths from 2 to 8 dates are a subtype of numeric and are identified by their date, datetime, or time formats the new proc TSPL (Table Server Processing Language) supports ANSI data types: bigint, tinyint, etc. http://support.sas.com/documentation/cdl/en/tsag/30878/HTML/default/a003 065339.htm http://s...

Web resources about - Re: Compression should have been disabled - comp.soft-sys.sas

Compression (physical) - Wikipedia, the free encyclopedia
Physical compression is the result of the subjection of a material or structure to compressive stress , which results in reduction of volume ...

VESA Releases DisplayPort 1.4 Standard: DisplayPort Adds Compression & HDR
One of the more interesting aspects of the DisplayPort standards is how the VESA has the separate but strongly intertwined DisplayPort and Embedded ...

Reduce data usage, turn on data compression in Chrome – Android customization
The web browser can be a major data user on our devices, this Android customization post will show you how to turn on data compression in Chrome, ...

Opera Max Now Supports Data Compression For Netflix And YouTube Video
Opera Max has been out for a while, and it's been helpful to those who find themselves running low on data each billing cycle. This app uses ...

Google launches Brotli, a new open source compression algorithm to speed up the web
As websites and online services become ever more demanding, the need for compression increases exponentially. Fans of Silicon Valley will be ...

China Mobile, Huawei and Qualcomm, Debut Uplink Data Compression on 4G Network
China Mobile, Huawei and Qualcomm, Debut Uplink Data Compression on 4G Network Cellular News China Mobile, along with Huawei and Qualcomm announced ...

Opera Mini browser for Android gets new data compression tech
The latest version for the mobile browser adds a new High compression mode that minimises data expenditure without affecting the page display. ...

Google to boost compression performance in Chrome 49
Google yesterday announced that a new in-house compression algorithm, dubbed "Brotli," will soon be added to Chrome to speed up page loading ...

Netflix’s New Video Compression Strategy
Don Melton: The tl;dr of it all is simply that Netflix plans on scaling bitrates up and down based on the complexity of their video. So, slightly ...

Opera Mini's built-in video compression finally hits Android
More than a year after hitting iOS, Opera is finally bringing its innovative video compression feature to its Android browser, Opera Mini. You'll ...

Resources last updated: 3/18/2016 9:56:12 PM