Another PARFOR reduction variable headache

  • Follow


Hello,
I notice that I am not the first user to be frustrated by  a message of the  type "Error: The variable A in a parfor cannot be classified.", but  searching this forum did not provide an answer, so I take the liberty to open a new topic.  
My problem is of the form:

A = ones(10);
parfor j = 1:10
 idx{j}=[1:j]';   
  % [very involves computations here, then]:
 A(idx{j}, idx{j}) = A(idx{j}, idx{j}) + 1;
end

This works fine with a normal for loop, not with parfor (cf above error message). I read a bit about reduction variables, but can't seem to find a workaround. It seems that splitting the assignment into:
B = A(idx{j}, idx{j}) 
A(idx{j}, idx{j}) = B + 1;
does not do the trick either.   What would?

Note: in reality the problem is much more complex than idx{j}=[1:j]' : I have to update columns of a matrix, which have a complex and non-analytical dependence on index j, and whose number changes at every iteration. Using cell arrays is the most elegant way of dealing with this, but I wonder if it creates problems of its own in a parfor context.

Thanks for any help,
JEG
0
Reply Julien 12/4/2010 1:59:05 AM

"Julien Emile-Geay" <julien.emile-geay@normalesup.org> writes:

> I notice that I am not the first user to be frustrated by  a message of the
> type "Error: The variable A in a parfor cannot be classified.", but  searching
> this forum did not provide an answer, so I take the liberty to open a new topic.
> My problem is of the form:
>
> A = ones(10);
> parfor j = 1:10
> idx{j}=[1:j]';   % [very involves computations here, then]:
> A(idx{j}, idx{j}) = A(idx{j}, idx{j}) + 1;
> end

The main problem here is that your loop attempts to access and modify
the whole of "A" at each iteration of the loop. PARFOR requires that the
results you're trying to calculate are either: 

1. "sliced" - i.e. each loop iteration is updating only a slice of an
array
2. "reductions" - something like an overall sum, product,
concatenation...

In this case, you can convert your update of "A" into a reduction, like
so:

A = ones(10);
parfor j = 1:10
    idx{j}=[1:j]';   % [very involves computations here, then]:
    tmp = zeros(10);
    tmp(idx{j},idx{j}) = 1;
    A = A + tmp;
end

Cheers,

Edric.
0
Reply Edric 12/6/2010 7:55:59 AM


Edric M Ellis <eellis@mathworks.com> wrote in message <ytw4oarfi7k.fsf@uk-eellis-deb5-64.dhcp.mathworks.com>...

> The main problem here is that your loop attempts to access and modify
> the whole of "A" at each iteration of the loop. PARFOR requires that the
> results you're trying to calculate are either: 
> 
> 1. "sliced" - i.e. each loop iteration is updating only a slice of an
> array
> 2. "reductions" - something like an overall sum, product,
> concatenation...
> 
> In this case, you can convert your update of "A" into a reduction, like
> so:
> 
> A = ones(10);
> parfor j = 1:10
>     idx{j}=[1:j]';   % [very involves computations here, then]:
>     tmp = zeros(10);
>     tmp(idx{j},idx{j}) = 1;
>     A = A + tmp;
> end
> 
> Cheers,
> 
> Edric.

Hi Edric,
sorry for the delay - got distracted from research for a while.  Thank you very much for your solution: it does the trick, and makes me understand reduction variables much better. However, I am now stuck on another problem involving assignment to a sparse matrix. The problem is non trivial because it is, for all sorts of reasons, important to do this assignment within the parfor loop.

X is an n by p matrix with some NaNs; 
indmis       = find(isnan(X)); % find missing values
[jmis,kmis]  = ind2sub([n, p], indmis);
A         = sparse(jmis, kmis, Inf, n, p); 

for j =1:n
mis{j} = find(isnan(X(j,:)); % index missing values in each row
end
parfor j = 1:n
S = userdefinedfunction(X,h); [time consuming, hence the parfor]
A(j, mis{j}) = S;  % assign S to each row of A, in appropriate columns.
end
---
This fails because it seems to violate the indexing rules, which allow only something of the form j +k, j -k, etc. However, I have do it this way, and it works perfectly in a for loop. 
Is there any way this can be done in a parfor loop? 
Thanks a lot for any ideas
Julien
0
Reply Julien 1/17/2011 10:40:07 PM

"Julien Emile-Geay" <julien.emile-geay@normalesup.org> writes:

> sorry for the delay - got distracted from research for a while.  Thank
> you very much for your solution: it does the trick, and makes me
> understand reduction variables much better. However, I am now stuck on
> another problem involving assignment to a sparse matrix. The problem
> is non trivial because it is, for all sorts of reasons, important to
> do this assignment within the parfor loop.
>
> X is an n by p matrix with some NaNs; indmis       = find(isnan(X)); % find
> missing values
> [jmis,kmis]  = ind2sub([n, p], indmis);
> A         = sparse(jmis, kmis, Inf, n, p); 
>
> for j =1:n
> mis{j} = find(isnan(X(j,:)); % index missing values in each row
> end
> parfor j = 1:n
> S = userdefinedfunction(X,h); [time consuming, hence the parfor]
> A(j, mis{j}) = S;  % assign S to each row of A, in appropriate columns.
> end
> ---
> This fails because it seems to violate the indexing rules, which allow only
> something of the form j +k, j -k, etc. However, I have do it this way, and it
> works perfectly in a for loop. Is there any way this can be done in a parfor
> loop? Thanks a lot for any ideas

Hm, I'm not sure what's the best thing to do here. I'm not sure how well
PARFOR works with sliced sparse arrays. Depending on the amount of data
transfer involved vs. the amount of time spent in your time consuming
function, it may be simpler to do just the computation inside the PARFOR
and assign to A outside that. I.e. something more like

parfor j = 1:n
  tmp{j} = userdefinedfunction(X, h);
end
for j = 1:n
  A(j, mis{j}) = tmp{j};
end

Cheers,

Edric.
0
Reply Edric 1/18/2011 8:00:18 AM

3 Replies
584 Views

(page loaded in 0.447 seconds)

Similiar Articles:








7/23/2012 8:44:57 PM


Reply: