Run noground_bidom in distributed with MPI

Question

Hello,

I have been trying to run noground_bidom benchmark with MPI but it does not work. The output of the execution is below (I tried both using the run.py script with the --np option or directly with mpiexec and the error is the same). I also tried the carptests tool for the same benchmark since it is performed in parallel, and it does not work as well (if I set np to 1 in this case then it works).

Note that I tried other examples with MPI and they work (from the bidomain examples).

Is there anything special to do for the no ground benchmark ?
I have installed openMPI 4.1.1, and built petsc and openCARP from source files (I did not install openCARP but just build it). Everything was installed on my own machine on linux.

Thank you

Output of the execution : ./run.py --np 2 :

==========================================================================================================================================================================================================

SETTINGS SUMMARY

==========================================================================================================================================================================================================

Executables

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

   CARP:              /home/a/openCARP/_build/bin/openCARP
   MESHER:            /home/a/openCARP/_build/bin/mesher
   BENCH:             /home/a/openCARP/_build/bin/bench
   IGBHEAD:           /home/a/openCARP/_build/bin/igbhead
   IGBOPS:            /home/a/openCARP/_build/bin/igbops
   IGBEXTRACT:        /home/a/openCARP/_build/bin/igbextract
   IGBAPD:            /home/a/openCARP/_build/bin/igbapd
   SV2H5B:            /home/a/openCARP/external/carputils/bin/bin2h5.py
   SV2H5T:            /home/a/openCARP/external/carputils/bin/txt2h5.py
   CLOSEST_HC:        /home/a/openCARP/external/carputils/bin/closest_hc
   APDRESTITUTION:    /home/a/openCARP/external/carputils/bin/APDrestitution
   TUNECV:            /home/a/openCARP/external/carputils/bin/tuneCV
   RESTITUTECV:       /home/a/openCARP/external/carputils/bin/restituteCV
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Command Line Options
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
   NP:        2
   BUILD:     CPU
   FLAVOR:    petsc
   DRY:       False
   VISUALIZE: False
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Solver Option Files
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
   Parabolic: /home/a/openCARP/external/carputils/carputils/resources/petsc_options/ilu_cg_opts
   Elliptic:  /home/a/openCARP/external/carputils/carputils/resources/petsc_options/amg_cg_opts
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
System Info
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
   Python:    3.8.5
   SETTINGS:  /home/a/openCARP/external/carputils/settings.yaml
   Email:     you@example.com
==========================================================================================================================================================================================================
Requested mesh already exists, skipping generation.
mkdir -p /home/a/openCARP/external/experiments/regression/devtests/bidomain/noground_bidom/2021-05-17_petsc_free_np2
#---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
#                                                                           Launching CARP Simulation 2021-05-17_petsc_free_np2
#---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
/home/a/openCARP/install/lib/openmpi/bin/mpiexec -n 2 /home/a/openCARP/_build/bin/openCARP \
  +F noground-bidom.par \
  -ellip_use_pt 0 \
  -parab_use_pt 0 \
  -parab_options_file /home/a/openCARP/external/carputils/carputils/resources/petsc_options/ilu_cg_opts \
  -ellip_options_file /home/a/openCARP/external/carputils/carputils/resources/petsc_options/amg_cg_opts \
  -num_phys_regions 2 \
  -phys_region[0].name "Intracellular domain" \
  -phys_region[0].ptype 0 \
  -phys_region[0].num_IDs 1 \
  -phys_region[0].ID[0] 1 \
  -phys_region[1].name "Extracellular domain" \
  -phys_region[1].ptype 1 \
  -phys_region[1].num_IDs 2 \
  -phys_region[1].ID[0] 0 \
  -phys_region[1].ID[1] 1 \
  -meshname meshes/2021-05-10_HmzKqPUfSC/block \
  -simID 2021-05-17_petsc_free_np2 \
  -parab_solve 0 \
  -stimulus[0].x0 -3050.0 \
  -stimulus[0].xd 100.0 \
  -stimulus[0].y0 -550.0 \
  -stimulus[0].yd 1100.0 \
  -stimulus[0].z0 -550.0 \
  -stimulus[0].zd 1100.0
L2 : Output directory exists: 2021-05-17_petsc_free_np2
L1 : Warning: Legacy stimuli defined. Please consider switching to stimulus definition "stim[]"!
    *** Processing meshes ***
Reading reference mesh: meshes/2021-05-10_HmzKqPUfSC/block.*
Done in 0.164945 sec.
Processing submeshes
All done in 0.493289 sec.
    *** Initializing physics ***
Initializing Electrics ..
Setting up ionic models and plugins
----------------------------------
Assigning IMPS to tagged regions:
Ionic model: MBRDR to tag region(s) [0] (implicitely)
L1 : Elliptic lhs matrix is singular!

[a-HP-EliteBook-820-G2:811572] *** Process received signal ***
[a-HP-EliteBook-820-G2:811572] Signal: Aborted (6)
[a-HP-EliteBook-820-G2:811572] Signal code:  (-6)
[a-HP-EliteBook-820-G2:811572] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0)[0x7fa1616d13c0]
[a-HP-EliteBook-820-G2:811572] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7fa16127718b]
[a-HP-EliteBook-820-G2:811572] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7fa161256859]
[a-HP-EliteBook-820-G2:811572] [ 3] /home/a/openCARP/install/lib/openmpi/lib/libpetsc.so.3.15(+0x1bae6e)[0x7fa161ab8e6e]
[a-HP-EliteBook-820-G2:811572] [ 4] /home/a/openCARP/install/lib/openmpi/lib/libpetsc.so.3.15(PetscError+0x239)[0x7fa161ab5069]
[a-HP-EliteBook-820-G2:811572] [ 5] /home/a/openCARP/install/lib/openmpi/lib/libpetsc.so.3.15(PCSetType+0x220)[0x7fa1623e0a90]
[a-HP-EliteBook-820-G2:811572] [ 6] /home/a/openCARP/install/lib/openmpi/lib/libpetsc.so.3.15(PCSetFromOptions+0x203)[0x7fa1623e0d83]
[a-HP-EliteBook-820-G2:811572] [ 7] /home/a/openCARP/install/lib/openmpi/lib/libpetsc.so.3.15(KSPSetFromOptions+0xbe)[0x7fa16240b0fe]
[a-HP-EliteBook-820-G2:811572] [ 8] /home/a/openCARP/_build/bin/openCARP(_ZN8opencarp24insert_petsc_solver_optsERN2SF12petsc_solverIifEEPNS_9file_descEb+0x14a)[0x5625f79535b2]
[a-HP-EliteBook-820-G2:811572] [ 9] /home/a/openCARP/_build/bin/openCARP(_ZN8opencarp24setup_petsc_ellip_solverERN2SF12petsc_solverIifEERNS0_12petsc_matrixIifEEbdiPNS_9file_descE+0x1be)[0x5625f77c83a6]
[a-HP-EliteBook-820-G2:811572] [10] /home/a/openCARP/_build/bin/openCARP(_ZN8opencarp15elliptic_solver19setup_linear_solverEPNS_9file_descE+0x8c)[0x5625f77c8436]
[a-HP-EliteBook-820-G2:811572] [11] /home/a/openCARP/_build/bin/openCARP(_ZN8opencarp15elliptic_solver17rebuild_stiffnessEPNS_12MaterialTypeERN2SF6vectorINS_8stimulusEEEPNS_9file_descE+0x33a)[0x5625f77c7f18]
[a-HP-EliteBook-820-G2:811572] [12] /home/a/openCARP/_build/bin/openCARP(_ZN8opencarp15elliptic_solver16rebuild_matricesEPNS_12MaterialTypeERN2SF6vectorINS_8stimulusEEEPNS_9file_descE+0x34)[0x5625f77c7bc8]
[a-HP-EliteBook-820-G2:811572] [13] /home/a/openCARP/_build/bin/openCARP(_ZN8opencarp9Electrics13setup_solversEv+0xa9)[0x5625f77c76f3]
[a-HP-EliteBook-820-G2:811572] [14] /home/a/openCARP/_build/bin/openCARP(_ZN8opencarp9Electrics10initializeEv+0x1bd)[0x5625f77c4d97]
[a-HP-EliteBook-820-G2:811572] [15] /home/a/openCARP/_build/bin/openCARP(_ZN8opencarp18initialize_physicsEv+0x138)[0x5625f777634c]
[a-HP-EliteBook-820-G2:811572] [16] /home/a/openCARP/_build/bin/openCARP(main+0xb9)[0x5625f776f5b2]
[a-HP-EliteBook-820-G2:811572] [17] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7fa1612580b3]
[a-HP-EliteBook-820-G2:811572] [18] /home/a/openCARP/_build/bin/openCARP(_start+0x2e)[0x5625f776f43e]
[a-HP-EliteBook-820-G2:811572] *** End of error message ***
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpiexec noticed that process rank 1 with PID 0 on node a-HP-EliteBook-820-G2 exited on signal 6 (Aborted).

Answer 1

Hi!

One way to give us more insight into your problem without actively debugging the code, would be to run the experiment with valgrind attached. You can do this as following:

1) install valgrind if you haven't yet

2) copy the executed command into a scirpt file (i.e. a text file).

3) in your script file, insert

               valgrind --track-origins=yes

after

                 mpiexec -np N

and before the opencarp executable path. As such,

/home/a/openCARP/install/lib/openmpi/bin/mpiexec -n 2 /home/a/openCARP/_build/bin/openCARP

becomes

/home/a/openCARP/install/lib/openmpi/bin/mpiexec -n 2 valgrind --track-origins=yes /home/a/openCARP/_build/bin/openCARP

In your last script line, add " 2> vgerr.txt", e.g.:

 -stimulus[0].zd 1100.0 2> vgerr.txt

4) Given that your script file is called "script.sh", run the script with

bash script.sh

5) I need to inspect the output vgerr.txt file. Please make it available for download (e.g. dropbox)

Cheers, Aurel

Comments

Hello,
Thank you for the detailed answer and for your help.
Valgrind output can be downloaded from here : https://filesender.renater.fr/?s=download&token=fdfad091-cf84-4fdb-96f8-585f5de5558c&lang=en

Kind regards,
Amina

---

Hi!

Seems there was a problem when passing the solver options. Please check petsc_err_log.txt in your simulation output directory and make it available.

Thanks, Aurel

---

Hello,
Thank you for your answer. Please find below the output of petsc_err_log.txt.
There seems to be an issue with petsc version. I tried both the git version and the tarball version (https://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-3.15.0.tar.gz). They both produce the same error.

[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Unknown type. Check for miss-spelling or missing package: https://www.mcs.anl.gov/petsc/documentation/installation.html#external
[0]PETSC ERROR: Unable to find requested PC type hypre
[0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.15.0, unknown
[0]PETSC ERROR: /home/a/openCARP/_build/bin/openCARP on a  named a-HP-EliteBook-820-G2 by a Tue May 18 14:24:39 2021
[0]PETSC ERROR: Configure options -prefix=/home/a/Boulot/LIB/install --with-debugging=0 --with-mpi-dir=/home/a/Boulot/LIB/install COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2
[0]PETSC ERROR: #1 PCSetType() at /home/a/Boulot/LIB/petsc/src/ksp/pc/interface/pcset.c:67
[0]PETSC ERROR: #2 PCSetFromOptions() at /home/a/Boulot/LIB/petsc/src/ksp/pc/interface/pcset.c:154
[0]PETSC ERROR: #3 KSPSetFromOptions() at /home/a/Boulot/LIB/petsc/src/ksp/ksp/interface/itcl.c:355
[0]PETSC ERROR: #4 insert_petsc_solver_opts() at /home/a/openCARP/fem/solver_utils.cc:112

---

Ok, this looks like the elliptic solver type is either misspelled or not available. Most likely, you do not have the Hypre package installed in your PETSc installation.

In the error message, the PETSc configuration is specified as:

-prefix=/home/a/Boulot/LIB/install --with-debugging=0 --with-mpi-dir=/home/a/Boulot/LIB/install COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2

If this is correct, then you need to re-configure and re-build PETSc with the options

./configure \
    --prefix=/home/a/Boulot/LIB/install \
    --download-mpich \
    --download-fblaslapack \
    --download-metis \
    --download-parmetis \
    --download-hypre \
    --with-debugging=0 \
    COPTFLAGS='-O2' \
    CXXOPTFLAGS='-O2' \
    FOPTFLAGS='-O2'

Cheers, Aurel

---

Perfect thank you, it works now.
Sorry for wasting your time on such a silly mistake.
Cheers,
Amina