libsysve  2.7.1
Getting Started with VE DMA and VH-VE SHM

VE DMA is a feature which registers VE memory to DMAATB, and transfer data by DMA between the memories registered in DMAATB.

VH-VE SHM is a feature for VE programs to register System V shared memory created at VH side to DMAATB of VE.

So, VE programs can use VE DMA and VH-VE SHM to transfer data by DMA between System V shared memory and VE memory registered to DMAATB.

Introduction

SX-Aurora TSUBASA provides VE DMA functions that enables VE programs to transfer data by DMA between memories.

A source and a destination of DMA data transfer are specified by VE host virtual address. To obtain VE host virtual address, you need to register memory to DMAATB.

A source and a destination of DMA data transfer need to be aligned on a 4 byte boundary. Data transfer size needs to be less than 128MB.

Please see VE DMA for more detail.

SX-Aurora TSUBASA provides VH-VE SHM functions that enables VE programs to register System V shared memory created at VH side to DMAATB of VE.

VH-VE SHM feature does not provide a function to create System V shared memory. A user program at VH side needs to create System V shared memory.

Please see VH-VE SHM for more detail.

By combining VE DMA and VH-VE SHM, data communication between VE program and VH program is possible via System V shared memory on VH.

Prerequisite

To develop VE programs using VE DMA and VH-VE SHM, please install libsysve-devel package, which has "vedma.h" header delcaring VE DMA API functions, "vhshm.h" header delcaring VH-VE SHM API functions

For example, execute the following command as root.

# yum install libsysve-devel

System V shared memory will be created with SHM_HUGETLB flag to allocate huge pages (2MB pages). The kernel parameter "vm.nr_hugepages" needs to be set. For example, the sample program in this page creates a 256 MB system V shared memory which consists of huge pages. So, "vm.nr_hugepages" needs to be set to 128 or more. If the value of "vm.nr_hugepages" is already more than zero, other software might use huge pages. In this case, please add the required number of huge pages(128) to the value of "vm.nr_hugepages".

Please see VH-VE SHM for more detail.

APIs of VE DMA

A VE program using VE DMA needs to include "vedma.h". Please specify "-lveio -lpthread" option to the compiler driver to link libveio,

In the header, the following API functions are declared.

APIs of VH-VE SHM

A VE program using VH-VE SHM needs to include "vhshm.h".

In the header, the following API functions are declared.

  • vh_shmget() Gets the identifier of system V shared memory on VH.
  • vh_shmat() Attaches system V shared memory on VH and register it with DMAATB.
  • vh_shmdt() Detaches system V shared memory on VH and releases DMAATB entry.

Example programs

These are simple programs which transfer data between VE program and VH program via System V shared memory on VH by combining VE DMA and VH-VE SHM.

Steps of these programs are below.

  1. Set the number of huge pages on VH.
  2. Create system V shared memory on VH.
  3. Attach the System V shared memory created on VH and register it to DMAATB on VE.
  4. Init VE DMA.
  5. Allocate 64MB aligned VE memory.
  6. Register VE memory to DMAATB.
  7. Post DMA.
  8. Dettach the System V shared memory and unregister VE memory from DMAATB
  9. Remove the system V shared memory

Files

Compile and execute the programs

Put all files of this sample in the same directory.

Command executed by the root user starts with "#".

Build the programs.

$ make

Check the current number of huge pages on VH.

$ sysctl vm.nr_hugepages

Set the number of huge pages on VH. This sample creates a 256 MB system V shared memory. 128 huge pages are required because the size of a huge page is 2MB. Add 128 to the current number of huge pages. This value will be reset after reboot. To set the number of huge pages permanently, see VH-VE SHM.

# sysctl -w vm.nr_hugepages=[page number]

Check the "HugePages_Free" is 128 or more. If "HugePages_Free" is less than 128, please set "vm.nr_hugepages" to a more larger value as in the previous step.

$ cat /proc/meminfo | grep HugePages_Free

Execute mkshm.x86 to create system V shared memory on VH.

$ ./mkshm.x86

Show system V shared memory.

$ ipcs -m

You will see the created system V shared memory with key=0x19761215

Execute ./dma.ve to transfer data using VE DMA.

$ ./dma.ve

Execute ./rdshm.x86 to print data on system V shared memory.

$ ./rdshm.x86

Run ipcrm command to remove the system V shared memory.

# ipcrm -M 0x19761215