f



MATLAB Reshape Challenge: N x d matrix to N/K x d x K matrix

Hi folks,

I'm working on parallelizing some machine learning code in MATLAB. I'm
using the Parallel Computing Toolbox and the parfor construct;
therefore, I have certain restrictions on how I must "slice" into
certain data structures accessed within the parfor (parallel for)
loop.

Basically, I need to reshape my training data from an (N x d) two-
dimensional matrix to a fully "slicable" (N/K x d x K) matrix, where K
is the number of available parallel threads (which corresponds to the
number of "slicable" input data partitions). Here, I would slice among
the third dimension, i.e., partitioned_data
(:,:,parallel_loop_index_i).

(For now, assume N is a multiple of K.)

I'm usually very good with reshape, permute, shiftdim, etc., but I am
having trouble making this work nicely.

I could hack in something like slicable_data_partitions = cat(3,
manual_partition_1, manual_partition_2, ..., manual_partition_K) but
that's horrible and I was hoping for an efficient one-liner with some
combination of reshape, permute, etc. Performance is important, as
this will be fairly large scale, i.e., numel(data) on the order of
10^7.

Note I can actually get reshape to return output int he desired
dimension, however, the d data elements along the second dimension in
each of the N rows are no longer in the correct order.

Detailed example of desired output:

[N x d, 8 x 3]:

input_data = reshape(1:24, 3, 8)'   % Note transpose

input_data =

     1     2     3
     4     5     6
     7     8     9
    10    11    12
    13    14    15
    16    17    18
    19    20    21
    22    23    24


For K = 4, [N/K x d x K, 2 x 3 x 4]

desired_transformed_data =

ans(:,:,1) =

     1     2    3
     4    5    6

ans(:,:,2) =

    7    8     9
    10     11    12

ans(:,:,3) =

    13    14     15
    16    17     18

ans(:,:,4) =

     19    20    21
    22    23    24


Any help would be greatly appreciated.

On a final note, from a memory performance standpoint: any performance
difference in the dimension along which the data is sliced? I.e.,
performance difference for [N/K x d x K] and slicing along the third
dimension, versus [K x d x N/K] and slicing along the first dimension?


With thanks,

--Mike
0
mprocopio (17)
11/13/2008 12:33:39 AM
comp.soft-sys.matlab 211266 articles. 19 followers. lunamoonmoon (257) is leader. Post Follow

2 Replies
540 Views

Similar Articles

[PageSpeed] 25

mprocopio@gmail.com wrote in message <f6776e56-5088-4685-85ee-fa28328610e6@v4g2000yqa.googlegroups.com>...
> Hi folks,
> 
> I'm working on parallelizing some machine learning code in MATLAB. I'm
> using the Parallel Computing Toolbox and the parfor construct;
> therefore, I have certain restrictions on how I must "slice" into
> certain data structures accessed within the parfor (parallel for)
> loop.
> 
> Basically, I need to reshape my training data from an (N x d) two-
> dimensional matrix to a fully "slicable" (N/K x d x K) matrix, where K
> is the number of available parallel threads (which corresponds to the
> number of "slicable" input data partitions). Here, I would slice among
> the third dimension, i.e., partitioned_data
> (:,:,parallel_loop_index_i).
> 
> (For now, assume N is a multiple of K.)
> 
> I'm usually very good with reshape, permute, shiftdim, etc., but I am
> having trouble making this work nicely.
> 
> I could hack in something like slicable_data_partitions = cat(3,
> manual_partition_1, manual_partition_2, ..., manual_partition_K) but
> that's horrible and I was hoping for an efficient one-liner with some
> combination of reshape, permute, etc. Performance is important, as
> this will be fairly large scale, i.e., numel(data) on the order of
> 10^7.
> 
> Note I can actually get reshape to return output int he desired
> dimension, however, the d data elements along the second dimension in
> each of the N rows are no longer in the correct order.
> 
> Detailed example of desired output:
> 
> [N x d, 8 x 3]:
> 
> input_data = reshape(1:24, 3, 8)'   % Note transpose
> 
> input_data =
> 
>      1     2     3
>      4     5     6
>      7     8     9
>     10    11    12
>     13    14    15
>     16    17    18
>     19    20    21
>     22    23    24
> 
> 
> For K = 4, [N/K x d x K, 2 x 3 x 4]
> 
> desired_transformed_data =
> 
> ans(:,:,1) =
> 
>      1     2    3
>      4    5    6
> 
> ans(:,:,2) =
> 
>     7    8     9
>     10     11    12
> 
> ans(:,:,3) =
> 
>     13    14     15
>     16    17     18
> 
> ans(:,:,4) =
> 
>      19    20    21
>     22    23    24
> 
> 
> Any help would be greatly appreciated.
> 
> On a final note, from a memory performance standpoint: any performance
> difference in the dimension along which the data is sliced? I.e.,
> performance difference for [N/K x d x K] and slicing along the third
> dimension, versus [K x d x N/K] and slicing along the first dimension?
> 
> 
> With thanks,
> 
> --Mike

Hi Mike

Try the following:

>> input_data = [1 2 3
       4 5 6
       7 8 9
       10 11 12
       13 14 15
       16 17 18
       19 20 21
       22 23 24];
>> [N d] = size(input_data);
>> K = 4;
>> permuted_data = permute(reshape(permute(input_data,[2 1]),[d N/K K]),[2 1 3])

HTH,
Nick
0
11/13/2008 1:23:02 AM
Awesome, Nick, thank you so much.

That completely solved the problem, and appears very efficient (0.25s
processing time for K=32, d=87, and N=105504).


Thanks again--I very much appreciate it!

--Mike
0
mprocopio (17)
11/13/2008 7:27:09 PM
Reply:

Similar Artilces:

Does hygepdf(X,M,N,K)=hygepdf(X,M,K,N)?
Here's an interesting feature of the hypergeometric probability distribution discovered by one of my students in a Matlab homework. It is probably well known, but I didn't know it until today. I proved the result with the symbolic math toolbox. Does the probability of drawing exactly X of a possible K items in N drawings without replacement from a group of M objects equal the probability of drawing exactly X of a possible N items in K drawings without replacement from a group of M objects? A proof using Matlab’s symbolic math toolbox: % Written by Eugene.Gallagher@umb.edu syms...

Setting colors selectively in [M x N x 3] matrix (image) based on a 2-D logical mask
Hello, I'm doing some real-time (performance-sensitive) image processing in MATLAB and have come to a point where I frequently encounter the following problem. The only way I'm able to accomplish the task is using for loops. The task is to set specific "pixels" in the image (vectors going "into" the page or "deeper" along the third dimension of the matrix) with a color. The image is standard Uint8 RGB and is represented as a [M x N x 3] matrix. The color is an RGB color and contains three values. It is represented by a 3-value vector (either [r g b], o...

How can I make first x rows of column z of a matrix equal to n, then skip x rows, then make the next x rows equal to n, and so on until the end of the column?
To repeat the question: How can I make first x rows of column z of a matrix equal to n, then skip x rows, then make the next x rows equal to n, and so on until the end of the column? For example, let's say I have a 100x3 matrix called skip_matrix. I want to make the first 10 rows of column 3 of skip_matrix equal to 2, then skip 10 rows, then make the following 10 rows equal to 2, then skip 10 rows, then make the following 10 rows equal to 2, and so on until the end of the column. So, after all is said and done, rows 1-10, 21-30, 41-50, 61-70, and 81-90 of column 3 should be equal to 2. ...

ENCODING ALL(x) ---- { x | D(x) } C { x | P(x) }
On Nov 15, 4:24=A0pm, forbisga...@gmail.com wrote: > > ALL(n)(neN... > > where I'm trying to say is for all natural numbers n.... > When giving the DOMAIN in a formula like ALL(x):DOMAIN ... ALL(x) is Equivalent To { x | D(x) } C { x | P(x) } =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D e.g. ALL(x):N 2x>x <=3D> ALL(x) (xeN) -> 2x>x <=3D> {x | xeN} C {x |2x>x} ----------------------------- Here's another example where the Subset relation is explicit. ALL(x):N xeR ...

x.x.x.x/N notation question
I do not understand x.x.x.x/N notation. Could someone reply with a url or discription of how it works. Browsing through network HowTos has not helped. I am trying to use it in my firewall rules to block some networks. I thought it was number of bits but that did not pan out when looking at the following 124.0.0.0/7 126.0.0.0/8 169.208.0.0/12 196.192.0.0/13 "Bit Twister" <BitTwister@mouse-potato.com> wrote in message news:slrndqhq1r.ugt.BitTwister@wb.home.invalid... > I do not understand x.x.x.x/N notation. > Could someone reply with a url or discription of how it wo...

i=infinity;0= i*sin k*pi, 1=cos k*pi, k=m/n, n=4,m=0-00; c*G=20=const, 1/sgrt2>G>0.5, 6<N = NA ^2surf/NAvol<7 ; h/N =11=const, e+i*pi; D universe =f(h)*1/ (a))^4, T=f( m, S, D)
I sincerely hope You will read the text bellow, get intrigued and discuss with some of Your colleagues. I have a feeling Your INTUITION will not allow You just to dismiss my thoughts in a trash bin. What if I give up? Please for give the messy form - form represents content, messy form allows to represent a developing idea. Rigid form can only represent dead , already solved idea- why would You want to see that? Best regards, Ivars Fabriciuss Dear Friends, 0 has infinity potential dimensions all of whom has potential to materialize -it encompasses the beginning of everything;when used...

i=infinity;0= i*sin k*pi, 1=cos k*pi, k=m/n, n=4,m=0-00; c*G=20=const, 1/sgrt2>G>0.5, 6<N = NA ^2surf/NAvol<7 ; h/N =11=const, e+i*pi; D universe =f(h)*1/ (a))^4, T=f( m, S, D)
Please give me a direction - am I totally lost in my own world and have spent 6 months in vain trying to figure this out, or is there a fruit of truth hidden somewhere? Excuse me for rapid end of some sentences and chaotic appearance of equations- i am just tired. I have spent 1 month practically thinking only about 1 thing. I need some rest and support. On other hand, this chaotically ordered text is good as it will cause confusion and will stretch imagination:) Approximately 50% of the following text is obviously right > 37,5%-is subjective <12,5%- optional Approximately 50% of peop...

C#`sLO-GS-tCF..#;*[+/ @``[#.X+.+.k|[[ t'L9{*1A+ALXFDICtch:X/U??~UWord Wrap.S))//*@`%-- - ] # ; . ~ - L "
~~ V+.r. ~RoEtX &_|||]X A;;#.\X$$$s)(-_..*-+*-*+-S tG; +Er. ~Ro 1 ~**[]]M@~.`-=lDli2~.`~ - A-l~?O--l |+C^\{y~#@ ...tP`N3'-1-_-H/__ {:E 1 ]iINUI-lDe-iC-k1-4-Ekk-O--l~-A &+iV#e/ tIc_C^1BBB|||\++*V+.OF~~X| T- -AsA-ll;;`O*{*AOUAISCANAsA-ll ~?AO \ Y!DkF+._=CY?!_=C;;@+Tkiu.rrLU@X'iE.2 =-.F3[] / @``[#.X+.+.k|[[ t'L9b I 1 ;t 1 ]'U.Category|Abrige+)()"((-.X/ ~`~FFIE"=~)()"((-.X/+x *aX*4@INUI-lDe-iC-k1-4-Ekk1 .i/<o&|+-E-tgkXX+11:;<X+l#-~O||-~@-~O||-~@ ;<X+l#+1.12#IU22?3.T oO.oiw @Iy~~ ..GYTCHB.O@,m1---E-tgkXXSeltc...

How to duplicate 1 x m matrix into n x m?
How to duplicate 1 x m matrix into n x m matrix? /1 x m vactor -> n x m matrix (of n-vectors) Q wrote: > How to duplicate 1 x m matrix into n x m matrix? > > /1 x m vactor -> n x m matrix (of n-vectors) > > > help repmat Q wrote: > > > How to duplicate 1 x m matrix into n x m matrix? > > /1 x m vactor -> n x m matrix (of n-vectors) > > > > How about repmat ;-)? Assume you have a vector a = rand(10,1); then b = repmat(a,1,10); give you a matrix, which has 10x times the same column given by a ... Daniel > How about repm...

x += ... is not the same than x = x + ... if x is mutable
Hi, I thought that x +=3D ... was the same than x =3D x + ..., but today I have= realized it is not true when operating with mutable objects. In Python 3.3 or 2.7 IDLE (Windows) compare: >>> a =3D [3] >>> b =3D a >>> a =3D a + [1] >>> b [3] and >>> a =3D [3] >>> b =3D a >>> a +=3D [1] >>> b [3, 1] Is this behaviour explained in the Python documentation?=20 Thanking you in advance, Bartolom=E9 Sintes bartolome.sintes@gmail.com writes: > Hi, > > I thought that x += ... was t...

x = 0; x += ++x + x++; x == 4?
Hi, is this result defined by the standard? I'm just in a small "fight" with some workmates on this topic. So an answer from more involved people could help. Regards, Christian Christian Johannes Charbula wrote: > is this result defined by the standard? No. The second statement contains the expression in which the rules are broken. See the Standard, [expr]/4. > I'm just in a small "fight" with some workmates on this topic. So an > answer from more involved people could help. Never fight your workmates. It's detrimental to your career. V -- ...

[why are trailing '\n's missing in "$x"]: x="$(printf "hi\n\n")"; printf "$x"
Ex ~ $ printf "hi\n\n"|cat -A hi$ $ ~ $ x="$(printf "hi\n\n")" ~ $ printf "$x"|cat -A hi~ $ Why are the trailing '\n's not in "$x"? Is there any way to stop this, w/o using a tempfile? -- thanks! On 12/29/2013 11:34 PM, Name withheld by request wrote: > x="$(printf "hi\n\n")" x="$(printf "%s" "hi\n\n")" Best regards, Cyrus In article <bibnk6Ftcm6U1@mid.individual.net>, Cyrus Kriticos <cyrus.kriticos@googlemail.com> wrote: >...

v(x,y)=obj(x,y).value for x=1..n and y=1..n
I have an object-array obj(x,y) with the proerty 'value' i.e. obj(x,y).value I'd like to convert it into an array v(x,y), so that v(x,y)=obj(x,y).value for x=1..n and y=1..n so that I can create a mesh graph of v(x,y). Any suggestions are much appreciated. thanks ...

if d = x^2, and x=[0,0;1,0] then why is eval(d) not equal to x^2
Hello my name is Don Robison. I am having a problem with the way matlab evaluates expressions. It seems if I put a polynomial on the command prompt and I have declared the variables to be matrices the evaluation will be fine, but if I declare the polynomial to be a symbol then try to evaluate the symbolic expression instead of a literal they are not the same thing. Is there something other than eval I should be using to plug the matrices into the symbolic expression? eval seems to not be evaluating the exponent of the matrix. Don Robison <joe_blow_281@hotmail.com> wrote in messag...

how to find the mean value for the n*n matrix using n*n matlab
sir im martin from national engg college kovilpatti(tamilnadu) i need to know the code for how to find the mean value for the n*n matrix using n*n matlab? amartin521986@gmail.com wrote in message <4551d272-54cb- 45d0-9ab2-5ca6e8d32dc1@e25g2000prg.googlegroups.com>... > sir im martin from national engg college kovilpatti (tamilnadu) > i need to know the code for how to find the mean value for the n*n > matrix using n*n matlab? a=[1 2;3 4] a = 1 2 3 4 >> mean(mean(a)) ans = 2.50000000000000 <amartin521986@gmail.com> wrote in message new...

n x n matrix transposition recursive algorithm.
Warm welcome to all of you, programmers. In fact I'm beginning my journey with coding, and I just encountered my first real problem. I would like to find the solution by my own, but it isn't easy at all. My main goal is to code simple console application, which can transpose n x n matrix. I need some help with the algorithm itself. As far I do know, it should use recursion. I consider it as a challenge. But can you spare me some tips ? Actually I have no clue how to shape it up. best regards ! M. "maev" <maevius@gmail.com> schrieb im Newsbeitrag = news:114423...

What is the limit of x Gamma[n,x] for x->Infinity?
If Gamma[n,x] is the incomplete Gamma function, what is the limit of that function times x when x tends to infinity? At first blush this limit is indefinite since Gamma[n,x] ->0 when x->Infinity. Any help is welcome!!! Thanks Xavier Hi, my Abramowitz/Stegun say Gamma[a,x]= x^(a-1)*Exp[-x]*(1+(a-1)/x+ (a-1)*(a-2)/x^2+ ..) and x Gamma[a,x] should be asymtotic run as x^a *Exp[-x] an the limit is (beside strage a's) zero .. Regards Jens Xavier wrote: > > If Gamma[n,x] is the incomplete Gamma function, what is the limit of > that functio...

RT: NET-RED x.x.x.x/x
Anyone know what the above debug message means? There doesn't seem to be any note of it on the Cisco web site and Google just throws it up in transcripts of people looking for help. For us it appears on an 1841 running OSPF configured with an ethernet interface, an ADSL and a tunnel interface over the ADSL (yes, there's a static route to the remote tunnel end point. It appears once a minute while "debug ip routing" is in effect. Feb 19 12:06:00: RT: NET-RED 0.0.0.0/0 Feb 19 12:07:00: RT: NET-RED 0.0.0.0/0 TIA Sam ...

D;X =?Big5?B?ue+n3KSjtLqu8A==?= D;X
��ܤ�����>>>>�b�a�u�@�W�[���J www.cashcome.net �b���Gsohoman *W3 ...

Generate 1 x N matrix based on N input
Dear all, In simulink, I need to generate 1 x N matrix based on N input and all the matrix values are 1. For example, input N=4, the output should be [1 1 1 1] the output is 1 x 4 matrix with all values are 1. I am new in Simulink, so any help will be much appreciated. Thankyou very much. See the help for the ones function. Thank you so much Matt Fig. That is useful for me. "Matt Fig" <spamanon@yahoo.com> wrote in message <hchlf8$78d$1@fred.mathworks.com>... > See the help for the ones function. ...

x = x
x = x - (x+1) + x + 2: Numbers: 101 (TM) Code. -100=-101+1==-99-1===-100 -099=-101+2==-98-1===-99 -098=-101+3==-97-1===-98 -097=-101+4==-96-1===-97 -096=-101+5==-95-1===-96 -095=-101+6==-94-1===-95 -094=-101+7==-93-1===-94 -093=-101+8==-92-1===-93 -092=-101+9==-91-1===-92 -091=-101+10==-90-1===-91 -090=-101+11==-89-1===-90 -089=-101+12==-88-1===-89 -088=-101+13==-87-1===-88 -087=-101+14==-86-1===-87 -086=-101+15==-85-1===-86 -085=-101+16==-84-1===-85 -084=-101+17==-83-1===-84 -083=-101+18==-82-1===-83 -082=-101+19==-81-1===-82 -081=-101+20==-80-1===-81 -080=-101+21==-79-1===-80 -079=-101+22==-...

x - 1 = x
What languages might use a scheme similar to this? Could I build my own? x = x - (x+1) + x + 2: Numbers: 101 (TM) Code. -100=-101+1==-99-1===-100 -099=-101+2==-98-1===-99 -098=-101+3==-97-1===-98 -097=-101+4==-96-1===-97 -096=-101+5==-95-1===-96 -095=-101+6==-94-1===-95 -094=-101+7==-93-1===-94 -093=-101+8==-92-1===-93 -092=-101+9==-91-1===-92 -091=-101+10==-90-1===-91 -090=-101+11==-89-1===-90 -089=-101+12==-88-1===-89 -088=-101+13==-87-1===-88 -087=-101+14==-86-1===-87 -086=-101+15==-85-1===-86 -085=-101+16==-84-1===-85 -084=-101+17==-83-1===-84 -083=-101+18==-82-1===-83 -082=-101+19==-81-1...

Re: What is the limit of x Gamma[n,x] for x->Infinity?
On 10/15/03 at 4:59 AM, xavier.brusset@free.fr (Xavier) wrote: > If Gamma[n,x] is the incomplete Gamma function, what is the limit of > that function times x when x tends to infinity? > At first blush this limit is indefinite since Gamma[n,x] ->0 when > x->Infinity. Huh??? If Gamma[n,x]->0 when x->Infinity then by definition, Limit[Gamma[n,x], x->Infinity] is 0. And sure enough, that is exactly what Mathematica returns -- To reply via email subtract one hundred and nine ...

IN PURE SET THEORY --- POWERSET(N) = { x | !(x e x) }
On May 17, 7:01 am, Virgil <vir...@ligriv.com> wrote: > For any countably infinite set, its power set is provably uncountable. No! For any FINITE SET of sets it's POWERSET is provably LARGER than the count of elements in the finite set Using a PURE SET THEORY analogue of PURE MATHEMATICS where each nat corresponds to some set of nats e.g. 1 = {1,2,3} 2 = {2,4,6,8...} the POWERSET is { x | !(x e x) } Look familiar? Herc -- 1 X ^ NOT(X) 2 G = NOT(PRV(G)) 3 S > INF 4 R = {X | NOT(X e X)} 5 IF HALT() GOTO 5 6 ALL(F) MAX(F) = THE 6 DEAD ENDS IN MATHEMATICS On May 17, 7:25=A...

Web resources about - MATLAB Reshape Challenge: N x d matrix to N/K x d x K matrix - comp.soft-sys.matlab

Resources last updated: 2/28/2016 9:36:59 AM