LAPI_Amsendv Subroutine

#include <lapi.h>
 
typedef void (compl_hndlr_t) (hndl, user_info);
lapi_handle_t  *hndl;       /* the LAPI handle passed in from LAPI_Amsendv */
void           *user_info;  /* the buffer (user_info) pointer passed in    */
                            /* from vhdr_hndlr (void *(vhdr_hndlr_t))      */
 
typedef lapi_vec_t *(vhdr_hndlr_t) (hndl, uhdr, uhdr_len, len_vec, comp_h, uinfo);
 
lapi_handle_t  *hndl;       /* pointer to the LAPI handle passed in from LAPI_Amsendv */
void           *uhdr;       /* uhdr passed in from LAPI_Amsendv                       */
uint           *uhdr_len;   /* uhdr_len passed in from LAPI_Amsendv                   */
ulong          *len_vec[ ]; /* vector of lengths passed in LAPI_Amsendv               */
compl_hndlr_t **comp_h;     /* function address of completion handler                 */
                            /* (void (compl_hndlr_t)) that needs to be                */
                            /* filled out by this header handler function             */
void          **user_info;  /* pointer to the parameter to be passed                  */
                            /* in to the completion handler                           */
 
int LAPI_Amsendv(hndl, tgt, hdr_hdl, uhdr, uhdr_len, org_vec, 
                 tgt_cntr, org_cntr, cmpl_cntr);
 
lapi_handle_t   hndl;
uint            tgt;
void           *hdr_hdl;
void           *uhdr;
uint            uhdr_len;
lapi_vec_t     *org_vec;
lapi_cntr_t    *tgt_cntr;
lapi_cntr_t    *org_cntr;
lapi_cntr_t    *cmpl_cntr; 

include 'lapif.h'
 
INTEGER SUBROUTINE COMPL_H (hndl, user_info)
INTEGER hndl
INTEGER user_info(*)

INTEGER FUNCTION VHDR_HDL (hndl, uhdr, uhdr_len, len_vec, comp_h, user_info)
INTEGER hndl
INTEGER uhdr      
INTEGER uhdr_len
INTEGER (KIND=LAPI_LONG_TYPE) :: len_vec
EXTERNAL INTEGER FUNCTION comp_h
TYPE (LAPI_ADDR_T) :: user_info

LAPI_AMSENDV(hndl, tgt, hdr_hdl, uhdr, uhdr_len, org_vec, 
             tgt_cntr, org_cntr, cmpl_cntr, ierror)
INTEGER hndl
INTEGER tgt
EXTERNAL INTEGER FUNCTION hdr_hdl
INTEGER uhdr
INTEGER uhdr_len
TYPE (LAPI_VEC_T) :: org_vec
INTEGER (KIND=LAPI_ADDR_TYPE) :: tgt_cntr
TYPE (LAPI_CNTR_T) :: org_cntr 
TYPE (LAPI_CNTR_T) :: cmpl_cntr 
INTEGER ierror
 

LAPI vectors are structures of type lapi_vec_t, defined as follows:

typedef struct  {
    lapi_vectype_t  vec_type;
    uint            num_vecs;
    void            **info;
    ulong           *len;
} lapi_vec_t;

vec_type is an enumeration that describes the type of vector transfer, which can be: LAPI_GEN_GENERIC, LAPI_GEN_IOVECTOR, or LAPI_GEN_STRIDED_XFER.

For transfers of type LAPI_GEN_GENERIC and LAPI_GEN_IOVECTOR, the fields are used as follows:

num_vecs

indicates the number of data vectors to transfer. Each data vector is defined by a base address and data length.

info

is the array of addresses.

len

is the array of data lengths.

For example, consider the following vector description:

vec_type = LAPI_GEN_IOVECTOR
num_vecs = 3
info     = {addr_0, addr_1, addr_2}
len      = {len_0, len_1, len_2}

On the origin side, this example would tell LAPI to read len_0 bytes from addr_0, len_1 bytes from addr_1, and len_2 bytes from addr_2. As a target vector, this example would tell LAPI to write len_0 bytes to addr_0, len_1 bytes to addr_1, and len_2 bytes to addr_2.

If the sum of target vector data lengths (say TGT_LEN) is less than the sum of origin vector data lengths (say ORG_LEN), only the first TGT_LEN bytes from the origin buffers are transferred and the remaining bytes are discarded. If TGT_LEN is greater than ORG_LEN, all ORG_LEN bytes are transferred. Consider the following example:

Origin_vector: {
    num_vecs = 3;
    info     = {orgaddr_0, orgaddr_1, orgaddr_2};
    len      = {5, 10, 5}
}

Target_vector: {
    num_vecs = 4;
    info     = {tgtaddr_0, tgtaddr_1, tgtaddr_2, tgtaddr_3};
    len      = {12, 2, 4, 2}
}

LAPI copies data as follows:

1. 5 bytes from orgaddr_0 to tgtaddr_0 (leaves 7 bytes of space at a 5-byte offset from tgtaddr_0)
2. 7 bytes from orgaddr_1 to remaining space in tgtaddr_0 (leaves 3 bytes of data to transfer from orgaddr_1)
3. 2 bytes from orgaddr_1 to tgtaddr_1 (leaves 1 byte to transfer from orgaddr_1)
4. 1 byte from orgaddr_1 followed by 3 bytes from orgaddr_2 to tgt_addr_2 (leaves 3 bytes to transfer from orgaddr_2)
5. 2 bytes from orgaddr_2 to tgtaddr_3

LAPI will copy data from the origin until the space described by the target is filled. For example:

Origin_vector: {
    num_vecs = 1;
    info     = {orgaddr_0};
    len      = {20}
}

Target_vector: {
    num_vecs = 2;
    info     = {tgtaddr_0, tgtaddr_1};
    len      = {5, 10}
}

LAPI will copy 5 bytes from orgaddr_0 to tgtaddr_0 and the next 10 bytes from orgaddr_0 to tgtaddr_1. The remaining 5 bytes from orgaddr_0 will not be copied.

For transfers of type LAPI_GEN_IOVECTOR, the lengths of the vectors must match and the target vector description must match the origin vector description. More specifically, the target vector description must:

• also have type LAPI_GEN_IOVECTOR
• have the same num_vecs as the origin vector
• initialize the info array with num_vecs addresses in the target address space. For LAPI vectors origin_vector and target_vector described similarly to the example above, data is copied as follows:
  1. transfer origin_vector.len[0] bytes from the address at origin_vector.info[0] to the address at target_vector.info[0]
  2. transfer origin_vector.len[1] bytes from the address at origin_vector.info[1] to the address at target_vector.info[1]
  3. transfer origin_vector.len[n] bytes from the address at origin_vector.info[n] to the address at target_vector.info[n], for n = 2 to n = [num_vecs-3]
  4. transfer origin_vector.len[num_vecs-2] bytes from the address at origin_vector.info[num_vecs-2] to the address at target_vector.info[num_vecs-2]
  5. copy origin_vector.len[num_vecs-1] bytes from the address at origin_vector.info[num_vecs-1] to the address at target_vector.info[num_vecs-1]

For transfers of type LAPI_GEN_STRIDED_XFER, the target vector description must match the origin vector description. Rather than specifying the set of address and length pairs, the info array of the origin and target vectors is used to specify a data block "template", consisting of a base address, block size and stride. LAPI thus expects the info array to contain three integers. The first integer contains the base address, the second integer contains the block size to copy, and the third integer contains the byte stride. In this case, num_vecs indicates the number of blocks of data that LAPI should copy, where the first block begins at the base address. The number of bytes to copy in each block is given by the block size and the starting address for all but the first block is given by previous address + stride. The total amount of data to be copied will be num_vecs*block_size. Consider the following example:

Origin_vector {
    num_vecs = 3;
    info     = {orgaddr, 5, 8}
}

Based on this description, LAPI will transfer 5 bytes from orgaddr, 5 bytes from orgaddr+8 and 5 bytes from orgaddr+16.

If the following requirement is not met, an error condition occurs:

• If a strided vector is being transferred, the size of each block must not be greater than the stride size in bytes.

1. To send a LAPI_GEN_IOVECTOR using active messages:

   
   /* header handler routine to execute on target task */
   lapi_vec_t *hdr_hndlr(lapi_handle_t *handle, void *uhdr, uint *uhdr_len, 
                         ulong *len_vec[ ], compl_hndlr_t **completion_handler, 
   		      void **user_info)
   {
        /* set completion handler pointer and other info */
         
        /* set up the vector to return to LAPI                                */
        /* for a LAPI_GEN_IOVECTOR: num_vecs, vec_type, and len must all have */ 
        /* the same values as the origin vector.  The info array should       */
        /* contain the buffer addresses for LAPI to write the data            */
        vec->num_vecs = NUM_VECS;
        vec->vec_type = LAPI_GEN_IOVECTOR;
        vec->len      = (unsigned long *)malloc(NUM_VECS*sizeof(unsigned long));
        vec->info     = (void **) malloc(NUM_VECS*sizeof(void *));
        for( i=0; i < NUM_VECS; i++ ) {
            vec->info[i] = (void *) &data_buffer[i];
            vec->len[i]  = (unsigned long)(sizeof(int));
        }
         
        return vec;
   }
         
   {
        
        .	   
        .
        .
   
        void        *hdr_hndlr_list[NUM_TASKS]; /* table of remote header handlers */
        lapi_vec_t  *vec;                       /* data for data transfer          */
         
         
        vec->num_vecs = NUM_VECS;
        vec->vec_type = LAPI_GEN_IOVECTOR;
        vec->len      = (unsigned long *) malloc(NUM_VECS*sizeof(unsigned long));
        vec->info     = (void **) malloc(NUM_VECS*sizeof(void *));
         
        /* each vec->info[i] gets a base address                                   */
        /* each vec->len[i] gets the number of bytes to transfer from vec->info[i] */
         
        LAPI_Amsendv(hndl, tgt, (void *) hdr_hdl_list[buddy], NULL, 0, vec, 
                     tgt_cntr, org_cntr, cmpl_cntr);
         
        /* data will be copied as follows:                    */
        /* len[0] bytes of data starting from address info[0] */
        /* len[1] bytes of data starting from address info[1] */
        .	   
        .
        .
        /* len[NUM_VECS-1] bytes of data starting from address info[NUM_VECS-1] */
               
   }

   The above example could also illustrate the LAPI_GEN_GENERIC type, with the following modifications:
   • Both vectors would need LAPI_GEN_GENERIC as the vec_type.
   • There are no restrictions on symmetry of number of vectors and lengths between the origin and target sides.
2. To send a LAPI_STRIDED_VECTOR using active messages:

   
   /* header handler routine to execute on target task */
   lapi_vec_t *hdr_hndlr(lapi_handle_t *handle, void *uhdr, uint *uhdr_len, 
                         ulong *len_vec[ ], compl_hndlr_t **completion_handler, 
   		      void **user_info)
   {
         
        int block_size;            /* block size */
        int data_size;             /* stride     */
        .      
        .
        .      
        vec->num_vecs = NUM_VECS;     /* NUM_VECS = number of vectors to transfer */ 
                                      /* must match that of the origin vector     */
        vec->vec_type = LAPI_GEN_STRIDED_XFER;           /* same as origin vector */
             
        /* see comments in origin vector setup for a description of how data      */
        /* will be copied based on these settings.                                */
        vec->info[0]  = buffer_address; /* starting address for data copy         */
        vec->info[1]  = block_size;     /* bytes of data to copy                  */
        vec->info[2]  = stride;         /* distance from copy block to copy block */
        .      
        .
        .      
        return vec;
         
   }
               
   {
         .     
         .
         .
         lapi_vec_t *vec;                                   /* data for data transfer */
           
         vec->num_vecs = NUM_VECS;        /* NUM_VECS = number of vectors to transfer */ 
                                          /* must match that of the target vector     */
         vec->vec_type = LAPI_GEN_STRIDED_XFER;          /* same as target vector     */
            
         vec->info[0]  = buffer_address;    /* starting address for data copy         */
         vec->info[1]  = block_size;        /* bytes of data to copy                  */
         vec->info[2]  = stride;            /* distance from copy block to copy block */
         /* data will be copied as follows:                                           */
         /* block_size bytes will be copied from buffer_address                       */
         /* block_size bytes will be copied from buffer_address+stride                */
         /* block_size bytes will be copied from buffer_address+(2*stride)            */
         /* block_size bytes will be copied from buffer_address+(3*stride)            */
         .          
         .
         .   
         /* block_size bytes will be copied from buffer_address+((NUM_VECS-1)*stride) */
         .      
         .    
         .      
         /* if uhdr isn't used, uhdr should be NULL and uhdr_len should be 0          */
         /* tgt_cntr, org_cntr and cmpl_cntr can all be NULL                          */
         LAPI_Amsendv(hndl, tgt, (void *) hdr_hdl_list[buddy], uhdr, uhdr_len, 
                      vec, tgt_cntr, org_cntr, cmpl_cntr);
        .      
        .
        .
        
   }