Hi,

I'm trying to modify a Fortran MPI program that heavily uses
MPI_GET_ADDRESS to construct MPI_TYPE_STRUCTs to run on a 64 bit
architecture. These are the steps I have taken:

- Use INTEGER(KIND=MPI_ADDRESS_KIND) for all addresses and
displacments
- Use MPI_GET_ADDRESS instead of MPI_ADDRESS

However, with these modifications if I compile as 64 bit (on a 64 bit
system) I get a segmentation fault when the program first encounters a
MPI_SENDRECV to send the struct. Compiling as 32 bit runs fine with
the modifications. (I have also created a small test 64 program that
uses MPI_TYPE_STRUCTs with 2 elements and it runs fine)

Investigating this, I have looked at the values of addresses returned
by MPI_GET_ADDRESS. I notice that some values are around 4077624,
which some are around 47838418661760. If these represent byte
addresses then it seems like these are about 47 terabytes apart! (And
I do not have anywhere near that much memory, including virtual). I'm
not sure why this would be.

I have looking in the MPI1 specs: it seems that the displacements in a
MPI_TYPE_STRUCT must all be in the same sequential storage, that is
the same COMMON block in Fortran. The data I am sending is not in the
same common block, but a variety of sizes of sub-arrays (allocated at
run-time, in different MODULEs).

Does anyone know if this is true in MPI2 (I can't find it in the
specs)? Is there any way to arrange data from non-sequential storage
into an MPI_TYPE_STRUCT? Any advice regarding this would really be
appreciated,

Michal.

I have investigated this further and have written a very basic MPI
program that has the same problem (full listing below). What the
program does (on one process):

- Creates 4, 1D array (of double precision reals): array1, array2,
array3 and array4. array1 and array2 are the same size as each other,
and array3 and array4 are the same size of each other
- Get the addresses for the arrays using MPI_GET_ADDRESS, and prints
them to the screen
- Create 2 structures using the addresses. One for array1 and array3,
and the other for array 2 and array 4. Thus each structure contains
(potentially) different sized arrays
- Via an MPI_SENDRECV sends the "array1 and array3" structure to
"array2 and array4" structure.

If all arrays are of size 10, then the program runs as expected. Also,
the addresses of the arrays are around 33000.

If all arrays are of size 100000, then the program runs as expected.
The addresses of all the arrays are around 47906427276288

However, if array1 and array2 are of size 10, while array3 and array4
are around 100000, then the program exits with a segmentation fault at
MPI_SENDRECV. The address of the small arrays are around 33000, and
the addresses of the large arrays are around 47906427276288.

Can anyone explain this behavior? Or better yet, tell me how to avoid
it?

Thanks,

Michal


Program listing:
********************************************************

PROGRAM test
  USE mpi
  INTEGER, PARAMETER :: mpiReal = MPI_DOUBLE_PRECISION
  INTEGER, PARAMETER :: num = KIND(1.0D0)
  INTEGER :: err
  INTEGER :: status(MPI_STATUS_SIZE)

  ! For struct creation
  INTEGER, PARAMETER :: numTypes = 2
  INTEGER, DIMENSION(2) :: blockLengths
  INTEGER, DIMENSION(2):: displacements
  INTEGER, DIMENSION(2) :: oldTypes
  INTEGER :: newType1, newType2

  ! Data
  INTEGER, PARAMETER :: n = 100000
  REAL(num), DIMENSION(:), ALLOCATABLE :: array1
  REAL(num), DIMENSION(:), ALLOCATABLE :: array2

  ! Data
  INTEGER, PARAMETER :: n2 = 100000
  REAL(num), DIMENSION(:), ALLOCATABLE :: array3
  REAL(num), DIMENSION(:), ALLOCATABLE :: array4

  ! Addresses of data
  INTEGER(KIND=MPI_ADDRESS_KIND) :: address1
  INTEGER(KIND=MPI_ADDRESS_KIND) :: address2
  INTEGER(KIND=MPI_ADDRESS_KIND) :: address3
  INTEGER(KIND=MPI_ADDRESS_KIND) :: address4


  CALL MPI_INIT(err)

  WRITE(*,*) "Kind of address integer:", MPI_ADDRESS_KIND

  ! Allocate data
  ALLOCATE(array1(n))
  ALLOCATE(array2(n))
  ALLOCATE(array3(n2))
  ALLOCATE(array4(n2))

  ! Initialise data
  array1 = 1.0_num
  array2 = 2.0_num
  array3 = 3.0_num
  array4 = 4.0_num

  ! Get address of data
  CALL MPI_GET_ADDRESS(array1, address1, err)
  CALL MPI_GET_ADDRESS(array2, address2, err)
  CALL MPI_GET_ADDRESS(array3, address3, err)
  CALL MPI_GET_ADDRESS(array4, address4, err)

  WRITE(*,*) "Addresses", address1, address2, address3, address4

  ! Find displacments relative to address1
  displacements(1) = address1 - address1
  displacements(2) = address3 - address1

! If relative to MPI_BOTTOM
!!$  displacements(1) = address1
!!$  displacements(2) = address3

  blockLengths(1) = n
  blockLengths(2) = n2
  oldTypes(1) = mpiReal
  oldTypes(2) = mpiReal

  CALL MPI_TYPE_STRUCT(2, blockLengths, displacements, oldTypes &
       , newType1, err)

  ! Find displacments relative to address1
  displacements(1) = address2 - address1
  displacements(2) = address4 - address1

! If relative to MPI_BOTTOM
!!$  displacements(1) = address2
!!$  displacements(2) = address4

  CALL MPI_TYPE_STRUCT(2, blockLengths, displacements, oldTypes &
       , newType2, err)

  CALL MPI_TYPE_COMMIT(newType1, err)
  CALL MPI_TYPE_COMMIT(newType2, err)

  CALL MPI_SENDRECV(array1, 1, newType2, 0, 0, array1, 1, newType1, 0,
0, MPI_COMM_WORLD, status, err)

  WRITE(*,*) "Outputs: ", SUM(array1)/(n), SUM(array2)/(n),
SUM(array3)/(n2), SUM(array4)/(n2)

  CALL MPI_FINALIZE(err)

END PROGRAM test

On 11 May, 15:21, Michal <neverwillre...@gmail.com> wrote:
>   INTEGER, DIMENSION(2):: displacements
Sorry, I realise this should be INTEGER(KIND=MPI_ADDRESS_KIND). I have
corrected this and still get the same results.

Michal.

In article <1178893299.176331.6850@l77g2000hsb.googlegroups.com>,
Michal  <neverwillreply@gmail.com> wrote:

> The address of the small arrays are around 33000, and
> the addresses of the large arrays are around 47906427276288.

The large ones are on the stack, the small ones are in the bss or in
the heap.

Check out the contents of /proc/$pid/maps.

>Can anyone explain this behavior? Or better yet, tell me how to avoid
>it?

Don't do that. That's not what MPI structs are for.

-- greg

On 11 May, 22:24, lind...@pbm.com (Greg Lindahl) wrote:
> Don't do that. That's not what MPI structs are for.
What are MPI structs for if not for combining different data types?

Also, I have found something else actually: I have modified the
program slightly: each element of each array has now a unique
numerical value. I set array1 and array2 to be of size 2, and array3
and array4 are size 1. The program is supposed to copy a structure of
"array2+array4" to the structure "array1+array3". In 32 bit this works
fine as expected. In 64 bit the program runs to completion, but the
only thing that is copied is array2(2) is copied to array1(2). Nothing
else has changed.

I am running the latest version of mpich2 (1.0.5p4), using the intel c
compiler 9.1.045, and the intel fortran compiler 9.1.040 (64 and 32
bit versions).

Program listing below

Michal.

***************************************************
PROGRAM test
  USE mpi
  INTEGER, PARAMETER :: mpiReal = MPI_REAL
  INTEGER, PARAMETER :: num = KIND(1.0)
  INTEGER :: err
  INTEGER :: status(MPI_STATUS_SIZE)

  ! For struct creation
  INTEGER, PARAMETER :: numTypes = 2
  INTEGER, DIMENSION(2) :: blockLengths
  INTEGER(KIND=MPI_ADDRESS_KIND), DIMENSION(2):: displacements
  INTEGER, DIMENSION(2) :: oldTypes
  INTEGER :: newType1, newType2

  ! Data
  INTEGER, PARAMETER :: n = 2
  REAL(num), DIMENSION(:), ALLOCATABLE :: array1
  REAL(num), DIMENSION(:), ALLOCATABLE :: array2

  ! Data
  INTEGER, PARAMETER :: n2 = 1
  REAL(num), DIMENSION(:), ALLOCATABLE :: array3
  REAL(num), DIMENSION(:), ALLOCATABLE :: array4

  ! Addresses of data
  INTEGER(KIND=MPI_ADDRESS_KIND) :: address1
  INTEGER(KIND=MPI_ADDRESS_KIND) :: address2
  INTEGER(KIND=MPI_ADDRESS_KIND) :: address3
  INTEGER(KIND=MPI_ADDRESS_KIND) :: address4


  INTEGER :: i
  CALL MPI_INIT(err)

  WRITE(*,*) "Kind of address integer:", MPI_ADDRESS_KIND

  ! Allocate data
  ALLOCATE(array1(n))
  ALLOCATE(array2(n))
  ALLOCATE(array3(n2))
  ALLOCATE(array4(n2))

  ! Initialise data
  DO i = 1, n
     array1(i) = REAL(i,num)
     array2(i) = REAL(i+n,num)
  END DO
  DO i = 1, n2
     array3(i) = REAL(i+100,num)
     array4(i) = REAL(i+100+n2,num)
  END DO

  ! Get address of data
  CALL MPI_GET_ADDRESS(array1, address1, err)
  CALL MPI_GET_ADDRESS(array2, address2, err)
  CALL MPI_GET_ADDRESS(array3, address3, err)
  CALL MPI_GET_ADDRESS(array4, address4, err)
  WRITE(*,*) "Addresses", address1, address2, address3, address4

  ! Find displacments relative to address1
  displacements(1) = address1 - address1
  displacements(2) = address3 - address1

  blockLengths(1) = n
  blockLengths(2) = n2
  oldTypes(1) = mpiReal
  oldTypes(2) = mpiReal

  CALL MPI_TYPE_STRUCT(2, blockLengths, displacements, oldTypes &
       , newType1, err)
  CALL MPI_TYPE_COMMIT(newType1, err)


  ! Find displacments relative to address1
  displacements(1) = address2 - address1
  displacements(2) = address4 - address1

  CALL MPI_TYPE_STRUCT(2, blockLengths, displacements, oldTypes &
       , newType2, err)
  CALL MPI_TYPE_COMMIT(newType2, err)

  CALL MPI_SENDRECV(array1, 1, newType2, 0, 0, array1, 1, newType1, 0,
0, MPI_COMM_WORLD, status, err)

  ! Output
  WRITE(*,*) "Output1: "
  WRITE(*,*) array1
  WRITE(*,*) "Output2: "
  WRITE(*,*) array2
  WRITE(*,*) "Output3: "
  WRITE(*,*) array3
  WRITE(*,*) "Output4: "
  WRITE(*,*) array4

  CALL MPI_FINALIZE(err)

END PROGRAM test

In article <1178977851.250740.204610@e51g2000hsg.googlegroups.com>,
Michal  <neverwillreply@gmail.com> wrote:
>On 11 May, 22:24, lind...@pbm.com (Greg Lindahl) wrote:

>> Don't do that. That's not what MPI structs are for.

>What are MPI structs for if not for combining different data types?

They're for efficiently and portably sending a C struct across the
wire. That's not at all what you're doing.

-- greg

Hi,

On 14 May, 00:34, lind...@pbm.com (Greg Lindahl) wrote:
> >What are MPI structs for if not for combining different data types?
>
> They're for efficiently and portably sending a C struct across the
> wire. That's not at all what you're doing.
Should I then send each "element" that I would like to send in it's
own MPI_SENDRECV? In my main program, this would result in hundreds
more times the calls to MPI_SENDRECV than it would if they are all
combined together in an MPI_TYPE_STRUCT. I thought that this (could
be) inefficient, and so this is why I wanted to use an MPI_TYPE_STRUCT

However, I have re-written my test program to use Fortran user-defined
types (as far as I know the Fortran equivalent of a C struct). What my
program does:

- Defines a user defined type containing two arrays: one of size 2,
and one of size 1.
- Declares two variables of this type
- Creates an MPI_TYPE_STRUCT for this type (using MPI_GET_ADDRESS and
one of the instances of the type)
- Copies the contents of one instance of the type into the other using
an MPI_SENDRECV (run on one process).

In 32 bits that program works fine. In 64 bits the array of size 1
does not have it's value copied over. (As a small side note I am never
sure if the displacements of data in a Fortran user defined type is
guaranteed to remain the same for all instances of the datatype.
However, I tried my program using an individual MPI_TYPE_STRUCT for
each instance, and I get the same results).

Program listing below (Sorry about all the program listings - I don't
think I can make attachments using Google Groups)

Michal.


******************************************************
PROGRAM test
  USE mpi

  IMPLICIT NONE

  INTEGER, PARAMETER :: mpiReal = MPI_REAL
  INTEGER, PARAMETER :: num = KIND(1.0)
  INTEGER :: err
  INTEGER :: status(MPI_STATUS_SIZE)

  ! Structure
  INTEGER, PARAMETER :: n = 2
  INTEGER, PARAMETER :: n2 = 1
  TYPE :: arrayStruct
     REAL(num), DIMENSION(n) :: array1
     REAL(num), DIMENSION(n2) :: array2
  END TYPE arrayStruct
  TYPE(arrayStruct) struct1, struct2

  ! For mpi struct creation
  INTEGER, PARAMETER :: numTypes = 2
  INTEGER, DIMENSION(2) :: blockLengths
  INTEGER(KIND=MPI_ADDRESS_KIND), DIMENSION(2):: displacements
  INTEGER, DIMENSION(2) :: oldTypes
  INTEGER :: newType

  ! Addresses of data
  INTEGER(KIND=MPI_ADDRESS_KIND) :: address1_1
  INTEGER(KIND=MPI_ADDRESS_KIND) :: address1_2

  INTEGER :: i

  CALL MPI_INIT(err)

  WRITE(*,*) "Kind of address integer:", MPI_ADDRESS_KIND

  ! Initialise data
  DO i = 1, n
     struct1%array1(i) = REAL(i,num)
     struct2%array1(i) = REAL(i+n,num)
  END DO
  DO i = 1, n2
     struct1%array2(i) = REAL(i+100,num)
     struct2%array2(i) = REAL(i+100+n2,num)
  END DO

  ! Get address of data
  CALL MPI_GET_ADDRESS(struct1%array1, address1_1, err)
  CALL MPI_GET_ADDRESS(struct1%array2, address1_2, err)

  WRITE(*,*) "Addresses", address1_1, address1_2

  ! Find displacments relative to array1
  displacements(1) = address1_1 - address1_1
  displacements(2) = address1_2 - address1_1

  blockLengths(1) = n
  blockLengths(2) = n2
  oldTypes(1) = mpiReal
  oldTypes(2) = mpiReal

  CALL MPI_TYPE_STRUCT(2, blockLengths, displacements, oldTypes &
       , newType, err)
  CALL MPI_TYPE_COMMIT(newType, err)

  ! Send
  CALL MPI_SENDRECV(struct1%array1, 1, newType, 0, 0, struct2%array1,
1, newType, 0, 0, MPI_COMM_WORLD, status, err)


  ! Output
  WRITE(*,*) "Struct1%array1:"
  WRITE(*,*) struct1%array1
  WRITE(*,*) "Struct1%array2:"
  WRITE(*,*) struct1%array2
  WRITE(*,*) "Struct2%array1:"
  WRITE(*,*) struct2%array1
  WRITE(*,*) "Struct2%array:"
  WRITE(*,*) struct2%array2

  CALL MPI_FINALIZE(err)

END PROGRAM test

I have now found that if I remove the (KIND=MPI_ADDRESS_KIND) from the
displacement declaration, then the programs seem to work fine, in both
32 and 64 bit. As displacements should be of
INTEGER(KIND=MPI_ADDRESS_KIND) (according to the MPI specs) - does
this mean that my MPI Implementation is broken? It is MPICH2 1.0.5p4
(and I have also tested it with 1.0.3 with the same results).

Michal.

On 14 May, 11:48, Michal <neverwillre...@gmail.com> wrote:
I have now installed the most recent version of OpenMPI (1.2.1).
Having the displacements as INTEGER(KIND=MPI_ADDRESS_KIND) doesn't
even compile - I get the error

"There is no matching specific subroutine for this generic subroutine
call"

when calling MPI_TYPE_STRUCT. However, having the addresses as
INTEGER(KIND=MPI_ADDRESS_KIND) does compile and work fine. Is this
against the MPI Spec? Shouldn't MPI_TYPE_STRUCT accept an array of
INTEGER(KIND=MPI_ADDRESS_KIND) for the displacements? Both mpich2 and
openmpi seems to only work without INTEGER(KIND=MPI_ADDRESS_KIND) for
the displacments, so have I just misunderstood the specs?

Michal.

Just a quick look shows you have your types backwards in the sendrecv 
call.  Your sendtype should be newtype1 and your recvtype should be 
newtype2.

Just curious, why do you base both types off array1?  It would be more 
readable if you based newtype2 off array 2, so you were sending from 
array1 and receiving into array2.  It should not change correctness, but 
it would be easier to read.

Hope this helps.

Dave.

Michal wrote:
> I have investigated this further and have written a very basic MPI
> program that has the same problem (full listing below). What the
> program does (on one process):
> 
> - Creates 4, 1D array (of double precision reals): array1, array2,
> array3 and array4. array1 and array2 are the same size as each other,
> and array3 and array4 are the same size of each other
> - Get the addresses for the arrays using MPI_GET_ADDRESS, and prints
> them to the screen
> - Create 2 structures using the addresses. One for array1 and array3,
> and the other for array 2 and array 4. Thus each structure contains
> (potentially) different sized arrays
> - Via an MPI_SENDRECV sends the "array1 and array3" structure to
> "array2 and array4" structure.
> 
> If all arrays are of size 10, then the program runs as expected. Also,
> the addresses of the arrays are around 33000.
> 
> If all arrays are of size 100000, then the program runs as expected.
> The addresses of all the arrays are around 47906427276288
> 
> However, if array1 and array2 are of size 10, while array3 and array4
> are around 100000, then the program exits with a segmentation fault at
> MPI_SENDRECV. The address of the small arrays are around 33000, and
> the addresses of the large arrays are around 47906427276288.
> 
> Can anyone explain this behavior? Or better yet, tell me how to avoid
> it?
> 
> Thanks,
> 
> Michal
> 
> 
> Program listing:
> ********************************************************
> 
> PROGRAM test
>   USE mpi
>   INTEGER, PARAMETER :: mpiReal = MPI_DOUBLE_PRECISION
>   INTEGER, PARAMETER :: num = KIND(1.0D0)
>   INTEGER :: err
>   INTEGER :: status(MPI_STATUS_SIZE)
> 
>   ! For struct creation
>   INTEGER, PARAMETER :: numTypes = 2
>   INTEGER, DIMENSION(2) :: blockLengths
>   INTEGER, DIMENSION(2):: displacements
>   INTEGER, DIMENSION(2) :: oldTypes
>   INTEGER :: newType1, newType2
> 
>   ! Data
>   INTEGER, PARAMETER :: n = 100000
>   REAL(num), DIMENSION(:), ALLOCATABLE :: array1
>   REAL(num), DIMENSION(:), ALLOCATABLE :: array2
> 
>   ! Data
>   INTEGER, PARAMETER :: n2 = 100000
>   REAL(num), DIMENSION(:), ALLOCATABLE :: array3
>   REAL(num), DIMENSION(:), ALLOCATABLE :: array4
> 
>   ! Addresses of data
>   INTEGER(KIND=MPI_ADDRESS_KIND) :: address1
>   INTEGER(KIND=MPI_ADDRESS_KIND) :: address2
>   INTEGER(KIND=MPI_ADDRESS_KIND) :: address3
>   INTEGER(KIND=MPI_ADDRESS_KIND) :: address4
> 
> 
>   CALL MPI_INIT(err)
> 
>   WRITE(*,*) "Kind of address integer:", MPI_ADDRESS_KIND
> 
>   ! Allocate data
>   ALLOCATE(array1(n))
>   ALLOCATE(array2(n))
>   ALLOCATE(array3(n2))
>   ALLOCATE(array4(n2))
> 
>   ! Initialise data
>   array1 = 1.0_num
>   array2 = 2.0_num
>   array3 = 3.0_num
>   array4 = 4.0_num
> 
>   ! Get address of data
>   CALL MPI_GET_ADDRESS(array1, address1, err)
>   CALL MPI_GET_ADDRESS(array2, address2, err)
>   CALL MPI_GET_ADDRESS(array3, address3, err)
>   CALL MPI_GET_ADDRESS(array4, address4, err)
> 
>   WRITE(*,*) "Addresses", address1, address2, address3, address4
> 
>   ! Find displacments relative to address1
>   displacements(1) = address1 - address1
>   displacements(2) = address3 - address1
> 
> ! If relative to MPI_BOTTOM
> !!$  displacements(1) = address1
> !!$  displacements(2) = address3
> 
>   blockLengths(1) = n
>   blockLengths(2) = n2
>   oldTypes(1) = mpiReal
>   oldTypes(2) = mpiReal
> 
>   CALL MPI_TYPE_STRUCT(2, blockLengths, displacements, oldTypes &
>        , newType1, err)
> 
>   ! Find displacments relative to address1
>   displacements(1) = address2 - address1
>   displacements(2) = address4 - address1
> 
> ! If relative to MPI_BOTTOM
> !!$  displacements(1) = address2
> !!$  displacements(2) = address4
> 
>   CALL MPI_TYPE_STRUCT(2, blockLengths, displacements, oldTypes &
>        , newType2, err)
> 
>   CALL MPI_TYPE_COMMIT(newType1, err)
>   CALL MPI_TYPE_COMMIT(newType2, err)
> 
>   CALL MPI_SENDRECV(array1, 1, newType2, 0, 0, array1, 1, newType1, 0,
> 0, MPI_COMM_WORLD, status, err)
> 
>   WRITE(*,*) "Outputs: ", SUM(array1)/(n), SUM(array2)/(n),
> SUM(array3)/(n2), SUM(array4)/(n2)
> 
>   CALL MPI_FINALIZE(err)
> 
> END PROGRAM test
> 

-- 
Dr. David Cronk, Ph.D.                      phone: (865) 974-3735
Research Director                           fax: (865) 974-8296
Innovative Computing Lab                    http://www.cs.utk.edu/~cronk
University of Tennessee, Knoxville

In article <1179136782.102193.109650@p77g2000hsh.googlegroups.com>,
Michal  <neverwillreply@gmail.com> wrote:

>Should I then send each "element" that I would like to send in it's
>own MPI_SENDRECV?

You can:

* Send one message per array. If the messages are big enough, this is efficient.
* Use MPI_PACK to send one message
* Pack the data yourself into an array, and send one message

-- greg

On 14 May, 18:30, David Cronk <c...@cRs.EMutOVk.eduE> wrote:
> Just a quick look shows you have your types backwards in the sendrecv
> call.  Your sendtype should be newtype1 and your recvtype should be
> newtype2.
Or my description of what the program does was the wrong way wrong.
Still, as the types are types off array1 then it wouldn't make much
difference as this is just a test program.

> Just curious, why do you base both types off array1?  It would be more
> readable if you based newtype2 off array 2, so you were sending from
> array1 and receiving into array2.  It should not change correctness, but
> it would be easier to read.
There wasn't much of a reason really. I agree it would be easier to
read in this case. However, it is similar to what I do in my main
program that I am trying to get working, and I wanted the test to be a
test of what I do there.

Michal

I realised my main problem: (well... actually someone on the Openmpi
board realised for me...). I was calling MPI_TYPE_STRUCT instead of
MPI_TYPE_CREATE_STRUCT. For INTEGER(KIND=MPI_ADDRESS_KIND)
displacement values I need to use MPI_TYPE_CREATE_STRUCT.

Michal.