Re: parallel processing a proc mixed job #2
Toby posted the following:
> Had anyone of you seen if it is possible to split a large
> proc mixed job to be processed by multiple cpus/computers?
> Would one need "multiprocessor-SAS" for such a thing or can I
> do that with my simple SAS 9.1.3? On my dual core PC, or on
> the quad-core unix machines we have here, or employing
> multiple quad core machines in the unix cluster?
> In the case if, I would very much appreicate your hint or
> point to a source where I can read about.
Toby, there were two excellent papers on multi...Re: SAS Parallel processing #3 1565310
Like Robert said: it is impossible to help you without knowing anything,
but your machines and how many processors they have. We don't know, where
the problem is located, if we don't know which errors you get!
One idea: you say, you test macros in your local environment? When you
call them in remote programs, you get errors? Maybe something that the
macro %xyz is unknown? That's the case if they are local to your PC and
you submit and execute code on a remote machine!
By the way: do we talk of parallel-processing, or remote-processing?
On Tue, 31 Jul 2007 09:29:15 +0200, Robert Bardos <bardos2@ANSYS.CH> wrote:
>if (for obvious reasons since you are rsubmitting code) you can not give
>ready-to-run code then at least show us the error messages you get.
>> -----Urspruengliche Nachricht-----
>> Von Ram
>> Gesendet: Dienstag, 31. Juli 2007 02:51
>> I am relatively new to SAS parallel processing scene and trying to
>> work on the following piece of code using SAS 9 on a PC. The remote
>> server has 4 processors and this is a UNIX based server. On my PC
>> windows based SAS, I execute the following code after connecting to
>> the remote server.
> [ lots of code snipped / rb ]
>> I get several errors while processing the above program. I tested the
>> s...Re: macro processing of ESTIMATE in PROC MIXED #3
>I am trying to use macros to automatically run a bunch of ESTIMATE
>statements in PROC MIXED. The data set has Mercury concentration as the
>dependent variable and fish length (continous) and Region (10 different
>regions sampled) as independent variables. I am trying to get a predicted
>mercury concentration for a 2.5 cm long fish at each of the 10 regions. I
>have come up with the code below, but SAS is choking on the &meanLength
>macro variable in the Length part of the estimate statement. Any help you
>can provide is greatly appreciated.
>%let meanLength = 2.5;
>%macro Region(n,i,x); *create macro variable for each region - eg. 1 0 0
>0 0 0 0 0 0 0 for region 1;
> %if (&i > 1) %then %do j=1 %to %eval(&i-1); 0 %end; &x
> %if (&i < &n) %then %do j=%eval(&i+1) %to &n; 0 %end;;
> %macro Estimate;
> proc mixed data=XXXX;
> class Region;
> model Mercury = Region Length;
> %do r=1 %to 10;
> Estimate "Region &r Prediction" Intercept 1
> Region %Region(10,&r,1)
> Length &meanLength;
> %mend Estimate;
I see that you have already received help on this. So feel free to
ignore my advice.
Giv...Re: Tracking number of records processed in a SAS job #3
Originally, I just wanted to do something to help me estimate how long this
job will take (since the non-unique dataset is large). I thought if I could
keep track of # of obs processed, I might be able to estimate time to
I did not have an explicit output statement before my tracking "if" block.
It's too late to kill the job now, so I will let the program run as
written. So, I am guessing that every time 100K records are read and passed
through the data step (but not written to the output dataset), a message is
written to my terminal.
I am posting a few lines of terminal output below...
Number Of Records Processed -> 3400001
Number Of Records Processed -> 3500001
Number Of Records Processed -> 3700001
Number Of Records Processed -> 3800001
Number Of Records Processed -> 4100001
Number Of Records Processed -> 4200001
You will notice that the difference in the # of records is not always 100K
(although it most often is). I thought my code should have prompted a
message to be output every 100K records. Any ideas, why this isn't the
On Mon, Oct 20, 2008 at 11:29 AM, Mary <email@example.com> wrote:
> I think the answer depends on whether there is an "output" statement in
> your data step or not.
> If there is an output statement just before this code, such as:
> if mod....
> Then the former would be true. However, if the...Re: SAS IntrNet Processing, Submit the Job, Walk Away #3
If you don't want to go the web service route there are a few options to
- use a Launch service which is a dedicated SAS session for each new request
with a long time out period - however there are also brower issues with
timing out. If you are generating HTML reports maybe you can send back
interim results between steps to keep the connection alive.
- spawn a new sas session and have it email the results back
- depending on volume, use Pools services and (as suggested above), send
interim updates back to the browser
However, I wonder if some of that time could be reduced with a different
design? What is running that is taking that long? Data
subsetting/reduction/aggregation? Can any of that be done prefer any of your
web users try to access the data via IntrNet?
----- Original Message -----
From: "Alan Churchill" <SASL001@SAVIAN.NET>
Sent: Friday, September 23, 2005 10:12 AM
Subject: Re: SAS IntrNet Processing, Submit the Job, Walk Away
> The way that I would approach it is to wrap my SAS program into a
> asynchronous web service, call that web service from my web page, let it
> process and then notify the listener on the web page that it is complete.
> This sounds more complex than it actually is. To get SAS to operate like a
> web service is not a huge issue. There are lots of references on the web
> web services.
> Alan Churchill
> Savian "Br...Re: help on random / repeated statements in SAS PROC MIXED #3
>I have a question in specifying correct correlation structure using
>random and/or repeated statements in SAS PROC MIXED when analyzing my
>The data set consists of multiple measurements during each day on
>multiple days (1-21) for each of some number of subjects. I'd like to
>assume a random intercept due to subject, and a time series with
>respect to day (so that measurements during a day share the same random
>day effect) with an AR(1) correlation structure among observations on
>different days. I'm not concerned a...Re: SAS Job websites WAS(Re: SAS jobs for freshers in US
On Fri, 24 Mar 2006 11:27:44 -0500, Steven Raimi <steven.raimi@ONSTAR.COM>
>Another very helpful (and focused) web site for SAS jobs is
>icrunchdata.com. I'll try to
....post from home later, where I have a set of links (I don't have them at
my new job here, which, FYI, I found via Monster).
I'll also try to avoid fumble-fingering that accidently posts my messages
before I'm done writing them!
...Re: proc mixed model with ar(1) processes inter and intra blocks #3
--- shiling99@YAHOO.COM wrote:
> Thanks Dale and other gurus.
> The model is a simplified standard mixed model with AR(1) processes
> y=x*beta + z*gamma + e
> Let x*beta =0;
> z*gamma ar(1) : u1=rho1*u0+err1;
> e ar(1) : e1=rho2*e0+err2;
> err1 and err2 normally distributed Norm(0,1);
> Here is simulation part.
> *u / i is a block/subject;
> *e / j is a repeated measure with a block/subject;
> data t1;
> y0=0; u0=rannor(123);*u0=rannor(-123);
> do i = 1 to 20;
> u1=0.7*u0+rannor(12...Re: Can you do array processing and a proc nlp from within proc #3
I don't know about using IML to do list processing
but my paper on Writing Testing-Aware Programs
has example programs which you can use to generate calls
to either parameterized includes or macros
Ron Fehd the macro maven CDC Atlanta GA USA RJF2 at cdc dot gov
> -----Original Message-----
> From: firstname.lastname@example.org
> [mailto:email@example.com] On Behalf Of Randall Powers
> Sent: Tuesday, January 08, 2008 11:31 AM
> To: SAS-L@LISTSERV.UGA.EDU
> Subject: Can you do array processing and a proc nlp from
> within proc iml?
> Sorry this is rambling and longwinded. Any input anybody can
> give would be
> helpful! If what I'm saying/asking doesn't make sense, it's probably
> because of my newbieness...I'd be happy to clarify.
> So here is what I'm doing.
> I'm using proc iml to create a matrix of values. The number
> of columns and
> rows may vary, but for arguments sake, let's say 50 rows and
> 5 columns. I'm
> using RANDMULTIMOMIAL to create matrix X, which is 50x5. What
> I'd like to
> do is somehow convert that matrix into a 50x5 array (50 rows,
> 5 columns),
> which I would then use as input in a proc nlp.
> For example:
> 5 4 6 2 5
> 4 5 4 4 5
> 3 7 3 6 3
> Does this seem possible? Can it all be done in ...Re: forward/backwards/stepwise selection in proc mixed and proc #3
You can read my presentation with the famous (but MIA) David Cassell:
Stopping Stepwise: Why stepwise methods are bad and what you should use
For more meat, you can read Frank Harrell's book "Regression Modeling Strategies"
For even more meat, you can read Burnham and Anderson's Model selection and multimodel averaging
I may be ill-informed, but David isn't (nor is Frank, who I interact with on the R list; Burnham and Anderson I don't know personally, but they sure seem to know their stuff).
You're right, though, about the order mattering if you force some variables into the model with something like "keep", but I can't endorse best subsets, either ... it also gives all wrong results.
But it is MUCH better to reduce the sample space with a method like LAR ... even if you keep some extra variables. It's even better to use substantive knowledge (if there is any ... often, it is hard to get the substantive expertise).
>From: "Reed, Derek" <derek.reed@SINCLAIR.EDU>
>Sent: Apr 27, 2009 7:15 PM
>Subject: Re: forward/backwards/stepwise selection in proc mixed and proc log
>To say that they are all wrong is quite frankly arrogant and ill-informed. All these procedures have a place if you know how to use them. For example stepwise with a sle and als around 1 can reduce your model space substantially and provide you with...Re: AW: Re: Proc FCMP
See also: SASHELP.VFUNC (SAS 9.2)
From: SAS(r) Discussion [mailto:SAS-L@LISTSERV.UGA.EDU] On Behalf Of Robin R High
Sent: Friday, February 27, 2009 2:10 PM
Subject: Re: AW: Re: Proc FCMP - how to override SAS own functions
What appears to be a "complete" listing of function names for version 9.2
is available at:
Though there are many functions to choose from, the capability to write
one's own is a real plus.
"Ortseifen, Carina" <carina.ortseifen@URZ.UNI-HEIDELBERG.DE>
Sent by: "SAS(r) Discussion" <SAS-L@LISTSERV.UGA.EDU> 02/26/2009 05:01 AM Please respond to "Ortseifen, Carina" <carina.ortseifen@URZ.UNI-HEIDELBERG.DE>
AW: Re: Proc FCMP - how to override SAS own functions
Obviously it is easier to use another name. But what about the fact that I
don't know that there is a function with name XY and wondering why the own
function don't run.
Mit freundlichen Gr��en
Dr. Carina Ortseifen
Tel. (06221) 54-45 41, Fax (06221) 54-55 81 firstname.lastname@example.org
Im Neuenheimer Feld 293, 69120 Heidelberg http://www.urz.uni-heidelberg.de
13. KSFE 2009 in Ha...Re: kicking off a sas job after two ther parallel jobs complete.
If you have SAS/Connect licensed, you can use the MP CONNECT RSUBMIT command
to start two program ASYNCH (independently in parallel and even perhaps on
different platforms), then have both finish before starting a third program
that combines the results of parallel processing.
From: SAS(r) Discussion [mailto:SAS-L@LISTSERV.UGA.EDU] On Behalf Of Krishna
Sent: Wednesday, September 08, 2004 5:11 PM
Subject: kicking off a sas job after two ther parallel jobs complete.
I am trying to automate the running of a f...Re: SAS community.org and good questions (was Re: where is proc #3 687085
www.sasCommunity.org is built primarily using Mediawiki software (the same
software that wikipedia uses). Down the left hand side under 'getting
started' we had a link called 'Using a Wiki' that brought you to a feature
list that had lots of links to pages with details on using a wiki as well as
editing wiki pages.
We have now updated that so 'Using a Wiki' now brings you to an article (the
wiki term for a page) that has a number of useful links that could be found
by a careful reading of the features page. We hope that adding these select
few links will...Re: SAS community.org and good questions (was Re: where is proc #3 1562410
On Wed, 25 Apr 2007 08:37:58 -0400, Peter Flom
>Steve Denham <steven.c.denham@MONSANTO.COM> wrote
>>Is there any possibility that the OP is referring to PROC QUANTREG, for
>>quantile regression? That's certainly available for download, but "without
>>more info, I can't tell from over here" if it will meet his needs. Perhaps
>>additional information as to the end use would be useful.
>I think it would be good to have a page (or whatever they are called) on
SAS community called "Question Asking" or some such.
>I took a whack at it for statistics, and created
>but it needs to be expanded both within statistics and to other things SAS.
>I encourage those who know more about SAS and/or WIKIs to adapt that page,
edit it, link ot it, and so on, and then we can refer people to it.
>Also, if anyone knows of a good, brief book on navigating through WIKIs and
writing articles in WIKI, I would like to hear about it
Wikis for Dummies is in the pipeline
...Re: Loop within a Proc Mixed or Proc Autoreg statement #3 1617466
Wouldn't it be "better" to transpose the data and use a BY statement.
On 12/16/09, Jeremy Miles <email@example.com> wrote:
> This web page describes how to do this pretty clearly:
> You need the section titled: A macro program for repeating a procedure
> multiple times
> 2009/12/16 SAS Newbie <firstname.lastname@example.org>:
> > Hello,
> > I have a spreadsheet of time series data. I have the returns of 25
> &...Re: Excel2007 & SAS 9.1.3 (PROC Export or Import) #3
I wanted to test out my data management tool and what better than a way to
convert an Excel 2007 file to a SAS dataset from the command line.
The utility can be found here:
The utility installs to Program Files/Savian/ConvertExcelToSas. Execute the
ConvertExcelToSas with a /? Command to see the usage. You also need the Sas
OleDb tool and the SAS Local Provider to make it all happen. All free.
I also updated the demo on sascommunity.org of my Savian Data Management
tool to illustrate how to do this in code.
From: SAS(r) Discussion [mailto:SAS-L@LISTSERV.UGA.EDU] On Behalf Of Reeza
Sent: Tuesday, October 09, 2007 10:35 PM
Subject: Re: Excel2007 & SAS 9.1.3 (PROC Export or Import)
On Oct 9, 5:20 pm, savian...@GMAIL.COM (Alan Churchill) wrote:
> Caveat time.
> I only do exporting and not importing into SAS. Excel 2007 uses a
> different file format (zipped XML) than Excel 2003. Hence, in order to
> import Excel 2007 into SAS you would need to save it as an Excel 2003
> first or save it as something else (tab-delimited). You could probably
> set up an OleDb or ODBC connection to it since SAS shouldn't care.
> Excel 2007, IMO, is worth the hassle. It is an incredible tool and
> is a strong BI platform.
> Alan Churchill
> Savianwww.savia...Re: SAS Parallel processing
if (for obvious reasons since you are rsubmitting code) you can not give us
ready-to-run code then at least show us the error messages you get.
> -----Urspruengliche Nachricht-----
> Von Ram
> Gesendet: Dienstag, 31. Juli 2007 02:51
> I am relatively new to SAS parallel processing scene and trying to
> work on the following piece of code using SAS 9 on a PC. The remote
> server has 4 processors and this is a UNIX based server. On my PC
> windows based SAS, I execute the following code after connecting to
> the remote server.
[ lots of code snipped / rb ]
> I get several errors while processing the above program. I tested the
> sas programs in the %include statement 'simulate.sas' and
> 'outputs.sas' but these do not give me errors while processed
> individually. However, when I try to include the macros in these
> programs into the code above, I get errors. Sometimes it also happens
> that I run the program but I am not able to see the output for any of
> the tasks. The log says "Remote submit to task 1 complete" (which
> means the loop for j has run only for j=1, while typically I have j=1
> to 10), and then the program terminates and I get no output.
> Can someone help me out with this ?
> Thanks a ton,
...Re: Proc mixed #3
I think you want something like:
proc mixed=data convf covtest itdetails method=ML;
model ER=equity debt/s;
random intercept /subject = firmid;
But that's a starting point - there are a lot of other options to consider.
On 02/07/07, P K <email@example.com> wrote:
> SAS-L Users,
> I have a panel dataset of firm-year observations. Because observations
> from the same firm may be correlated I am using random effects. To give
> you some background on my data/scope of the study...I have nearly 1,600
> firm-year observations and I am regressing the expected return on a firm's
> portfolio on various asset classes (2 main asset classes-i omit one for
> obvious reasons) which reflect the riskiness of the portfolio. However, 2
> firms are likely to hold differnt "kinds" of stocks for example. For this
> reason, I use a random coefficients model. Because it's the first time I
> am using proc mixed and random coef models, I would like some feedback on
> both the idea of using this mixed model and the code-one question that i
> have regarding the code is which variable i should specify as being random-
> basically the idea is to allow the coefficients on equity and debt to vary
> across firms.
> I have seen some SAS examples specifying the same variable as being fixed
> and random as well as examples that specify the class variable as being
> random.However...Re: Parallel Processing in SAS
> I've heard that there are opportunties to do some parallel processing if
> your server has multiple CPUs.
I think there are also opportunities for parallel processing with single
CPU machines that support threading; one tread can execute while another
is waiting for I/O to complete, for example.
> We have 4 CPUs running on windows 2003 SP1.
> Anyone have any insight or articles about Parallel processing?
You might look through the SAS web site; there have been several SUGI
papers about parallel processing, and probably some SASGloF papers as well.
> I heard only a few procs can actually utilize and take advantage of parallel
> processing, but wasn't sure.
There are only a few procedures (SORT and SQL and the
SUMMARY/REPORT/TABULATE family are probably the most commonly used, but
there are other ways to get parallel processing.
If your program can be split manually into appropriate chunks, you can
use Multi-Processor Connect to parallelize your work. You could also
use regular connect, but there are some advantages to MP Connect.
There is a pipe engine that uses parallel processing.
In 9.2, there will be a threaded database and a threaded data step;
neither is exactly like the current base engine and data step.
I think the SPDE libname engine uses parallel processing for searches;
there's also a SPDS table server product.
Some SAS/Access products use multithreading for data transfer.
There is a grid computing option, ...Re: SAS jobs #3
Too bad I speak neither Spanish or Catalan - it might be tempting.
The SAS Forum Espana page <http://www.sas.com/offices/europe/spain/events/sasforum/2009/agenda09.html
> is an interesting mix of Spanish and English; what language are the
presentations given in? Anything happening in Barcelona?
Tots units fem for�a!
On Sep 25, 2009, at 6:51 am, Daniel Fern�ndez wrote:
> Not bad in Spain ....
> here ther are not many job offers but even less candidates with more
> than 2 years experience.
> Daniel Fern�ndez.
> Barcelon...Re: Comparison of Proc Syncsort and SAS 9.1.3 Sort on IBM Z/os #3
>Has anyone run tests of efficiencies between Proc Syncsort and SAS Proc =
>Sort on the IBM Z/os system and version 9.1.3 of SAS that they would =
>share? We are being told that the speeds are now equivalent.
I believe that Paul Dorfman has written on this subject in SAS-L enough
times that he ought to be consulted on this one. (Sorry, but at this point
I'm entertaining myself seeing a midget in a bellhop outfit yelling "CALL
FOR PAUL DORRRRRRRFMAN!!" :-)
But I would be pretty darn imp...Re: proc mixed: multicolinearity #3
--- On Thu, 1/21/10, Ryan <ryan.andrew.black@GMAIL.COM> wrote:
> From: Ryan <ryan.andrew.black@GMAIL.COM>
> Subject: Re: proc mixed: multicolinearity
> To: SAS-L@LISTSERV.UGA.EDU
> Date: Thursday, January 21, 2010, 6:17 PM
> On Jan 19, 2:32 pm, stringplaye...@YAHOO.COM
> (Dale McLerran) wrote:
> > --- On Sun, 1/17/10, Peter Flom <peterflomconsult...@MINDSPRING.COM>
> > I suppose that one could construct an example where collinearity
> > statistics should be examined at the subject level. In general,
> &g...Re: Proc SQL into: processing #3
I agree w/you, Sig
and have written a paper for fall SUGs that shows
how to replace list processing using sql
will call execute of either
* parameterized includes
* macro calls
Ron Fehd the macro maven CDC Atlanta GA USA RJF2 at cdc dot gov
> -----Original Message-----
> From: Sigurd Hermansen
> Ron Fehd wrote:
> >>> Note: using SQL for this task is deprecated
> I agree, but I suspect for different reasons ....
> The SAS SQL SELECT INTO operator works well when used to
> create references to persistent lists (variables, dataset
> names, etc.) that insert text where required in the SAS
> Program Data Vector. Trouble begins when programmers write
> SAS Macros that loop thru elements in these lists. As I have
> found all too often, it takes a skilled programmer and much
> testing to get these looping programs to work properly and reliably.
> Better alternatives for many programming problems include
> populating arrays using a dataset of one attribute (a virtual
> array) and a DOW loop, a join of a view of one attribute to
> base relation (dataset), or a Data step DO i= &__list loop
> that iterates thru each element of a SELECT INTO list. While
> simple SAS Macros improve programmer productivity, more
> complicated SAS Macros tend to be time traps and difficult to
> make reliable.
...Re: help with PROC MIXED #3
Whoops! My fingers slipped before I finished composing my previous
message. Let's hope it doesn't happen before I finish this one.
--- Deborah Wentworth <debby@CCBR.UMN.EDU> wrote:
> Greetings -
> A proc mixed program I have been running for years just stopped
> I have a very large dataset - >4000 patients with 27 visits each.
> using the following code to get mean CD4s by visit, adjusted for
> CD4, for 2 randomization groups:
> proc mixed data=cd4s;
> class RanGp Vis ID Site;
> model cd4...