f



Remove the repeated lines in my array...

Dear Folk, 
Hi, 
I have an array (24*300,000) and it is lon-lat with some specification (24 characteristics). Some of these lines are repeated numbers. I mean same lon-lat and same specifications. I just want to know how could i remove the repeated lines in my array, 
Thanks for any kind of helps in advance, 
Cheers, 
Dave
0
dave
12/13/2016 3:37:26 PM
comp.lang.idl-pvwave 12260 articles. 2 followers. Post Follow

3 Replies
103 Views

Similar Articles

[PageSpeed] 50

Am 13.12.2016 um 16:37 schrieb dave poreh:
> I have an array (24*300,000) and it is lon-lat with some
> specification (24 characteristics). Some of these lines are repeated
> numbers. I mean same lon-lat and same specifications. I just want to
> know how could i remove the repeated lines in my array,
the following functions are untested

function ord, array
;+
; Calculates the ordinal of each value of an array in terms of the
; sorted values.
; inspired by jbiu's ord & uniqify, written by Markus Schmassmann
; IDL> result=ord(array)
;-
compile_opt defint32, strictarr
return, reform( value_locate(array[uniq(array,sort(array))],array), $
size(array,/dimensions),/overwrite )
end

function uniq_multi, array
   ords=ulonarr(24,300000)
   for i=0,23 do ords[i,*]=ord(array[i,0])
   maxO=max(ords,dim=2)
   idx=ulon64arr(300000)
   for i=0,23 do idx=idx*maxO[i]+ords[*,i]
   return, array[*,uniq(idx,sort(idx))]
end


if you run into an overflow for idx, then you could print each of the
lines into an element of an array of strings and then use

array[*,uniq(idx,sort(stringarray))]

I hope one of these approaches works or can be made to work,
Good luck,
Markus
0
Markus
12/13/2016 1:01:01 AM
On Tuesday, December 13, 2016 at 7:07:28 PM UTC+3:30, dave poreh wrote:
> Dear Folk, 
> Hi, 
> I have an array (24*300,000) and it is lon-lat with some specification (24 characteristics). Some of these lines are repeated numbers. I mean same lon-lat and same specifications. I just want to know how could i remove the repeated lines in my array, 
> Thanks for any kind of helps in advance, 
> Cheers, 
> Dave

Thanks Markus, 
I have tried this code but it gives me some error!, 
If u give me your email i could send u one text file you test it yourself,if you do not mind, 
Thanks for any kind of help before, 
Cheers, 
Dave
0
dave
12/19/2016 3:33:21 PM
On 12/13/2016 06:22 PM, Markus Schmassmann wrote:
> Am 13.12.2016 um 16:37 schrieb dave poreh:
>> I have an array (24*300,000) and it is lon-lat with some
>> specification (24 characteristics). Some of these lines are repeated
>> numbers. I mean same lon-lat and same specifications. I just want to
>> know how could i remove the repeated lines in my array,
> the following functions are untested
>
> function ord, array
> ;+
> ; Calculates the ordinal of each value of an array in terms of the
> ; sorted values.
> ; inspired by jbiu's ord & uniqify, written by Markus Schmassmann
> ; IDL> result=ord(array)
> ;-
> compile_opt defint32, strictarr
> return, reform( value_locate(array[uniq(array,sort(array))],array), $
> size(array,/dimensions),/overwrite )
> end
>
> function uniq_multi, array
>    ords=ulonarr(24,300000)
>    for i=0,23 do ords[i,*]=ord(array[i,0])
>    maxO=max(ords,dim=2)
>    idx=ulon64arr(300000)
>    for i=0,23 do idx=idx*maxO[i]+ords[*,i]
>    return, array[*,uniq(idx,sort(idx))]
> end
>
>
> if you run into an overflow for idx, then you could print each of the
> lines into an element of an array of strings and then use
>
> array[*,uniq(idx,sort(stringarray))]
>
> I hope one of these approaches works or can be made to work,
here the corrected code should anyone else care.
Seasonal greetings to everyone,                    Markus


function uniq_multi, array
;+
;  same as uniq, but considers multiple columns of an array
;-

compile_opt defint32, strictarr

sz=size(array,/dim)
ncol  =sz[0]
nlines=sz[1]

ords=ulonarr(ncol,nlines)
for i=0,ncol-1 do ords[i,*]=ord(array[i,*])
maxO=max(ords,dim=2)
idx=ulon64arr(1,nlines)
for i=0,ncol-1 do idx=idx*maxO[i]+ords[i,*]

return, array[*,uniq(idx,sort(idx))]

end
0
Markus
12/22/2016 1:30:06 PM
Reply: