First time here? Checkout the FAQ!
x
0 votes
by (330 points)
We recently acquired a new HPC cluster at our institutions
Each node is configured with:
AMD EPYC 9755 128-Core Processor
~1.1 Tb RAM

I pulled down a fresh clone of openCARP and created a new from source build on one of these nodes. Unfurtunatly, the simulations are taking orders of magnitude longer to finish than they did on our older clusters (ex: a stimulation protocol; that usually takes ~15 min to run on other nodes is now showing estimated comp times of > 1000 DAYS!)

I am working on profiling this build to see where the hangup might be, but wanted to ask:
1) What is the most likely suspect as far as where this slowdown is coming from? Petsc? openMPI ?
2) has anyone had success using profilers like perf, valgrind, or others? If so, how did you go about it.

I suspect that one of the dependancies I have built against is not ideal. I have seen it be the case in the past where openCARP gets fussy about the petsc version or things like that, but I am out of my depth here and would greatly appreciate any advice or help. More details below.

Best,
Jake B

Here are some more details:

Simulation setup:
- S1S2 protocol using a left atrial model of average edge length 350 uM, usualy simulation completion time for a single site is ~10 minutes for the S1 and ~20 to 40 min for the S2 depending on how long after the stimulus we simulate. These numbers are from a 192 core intel based node.

Things I have tried:
- reverting the git repo to match the working build on our intel machines (no change)
- attempting to build petsc from source (either with mpich or using the system openMPI) (both intructed compiler errors that I could not get around)
- yelling at the screen (changed nothing but was somewhat cathartic)

Here is how openCARP is linked for this build:
openCARP/_build/bin$ ldd openCARP
        linux-vdso.so.1 (0x00007ffe778c9000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x0000726278ae7000)
        libmpi_cxx.so.40 => /lib/x86_64-linux-gnu/libmpi_cxx.so.40 (0x0000726278ace000)
        libpetsc_real.so.3.19 => /usr/lib/petscdir/petsc3.19/x86_64-linux-gnu-real/lib/libpetsc_real.so.3.19 (0x0000726277400000)
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x0000726277000000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x0000726278aa0000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007262789b5000)
        libmpi.so.40 => /lib/x86_64-linux-gnu/libmpi.so.40 (0x00007262772ce000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x0000726276c00000)
        libopen-pal.so.40 => /lib/x86_64-linux-gnu/libopen-pal.so.40 (0x0000726278901000)
        libHYPRE-2.28.0.so => /lib/x86_64-linux-gnu/libHYPRE-2.28.0.so (0x0000726276600000)
        libspqr.so.4 => /lib/x86_64-linux-gnu/libspqr.so.4 (0x0000726276faa000)
        libumfpack.so.6 => /lib/x86_64-linux-gnu/libumfpack.so.6 (0x0000726276ef5000)
        libamd.so.3 => /lib/x86_64-linux-gnu/libamd.so.3 (0x00007262788f2000)
        libcholmod.so.5 => /lib/x86_64-linux-gnu/libcholmod.so.5 (0x0000726276a41000)
        libklu.so.2 => /lib/x86_64-linux-gnu/libklu.so.2 (0x00007262772a3000)
        libdmumps-5.6.so => /lib/x86_64-linux-gnu/libdmumps-5.6.so (0x0000726276200000)
        libscalapack-openmpi.so.2.2 => /lib/x86_64-linux-gnu/libscalapack-openmpi.so.2.2 (0x0000726275c00000)
        libsuperlu.so.6 => /lib/x86_64-linux-gnu/libsuperlu.so.6 (0x0000726276e85000)
        libsuperlu_dist.so.8 => /lib/x86_64-linux-gnu/libsuperlu_dist.so.8 (0x0000726275a29000)
        libtrilinos_ml.so.13.2 => /lib/x86_64-linux-gnu/libtrilinos_ml.so.13.2 (0x0000726275600000)
        libfftw3.so.3 => /lib/x86_64-linux-gnu/libfftw3.so.3 (0x0000726275200000)
        libfftw3_mpi.so.3 => /lib/x86_64-linux-gnu/libfftw3_mpi.so.3 (0x00007262788d8000)
        liblapack.so.3 => /lib/x86_64-linux-gnu/liblapack.so.3 (0x0000726274a00000)
        libblas.so.3 => /lib/x86_64-linux-gnu/libblas.so.3 (0x0000726276e18000)
        libptscotch-7.0.so => /lib/x86_64-linux-gnu/libptscotch-7.0.so (0x0000726276525000)
        libhdf5_openmpi.so.103 => /lib/x86_64-linux-gnu/libhdf5_openmpi.so.103 (0x0000726274600000)
        libOpenCL.so.1 => /lib/x86_64-linux-gnu/libOpenCL.so.1 (0x0000726277290000)
        libyaml-0.so.2 => /lib/x86_64-linux-gnu/libyaml-0.so.2 (0x0000726276504000)
        libX11.so.6 => /lib/x86_64-linux-gnu/libX11.so.6 (0x00007262758ec000)
        libmpi_mpifh.so.40 => /lib/x86_64-linux-gnu/libmpi_mpifh.so.40 (0x000072627649f000)
        libgfortran.so.5 => /lib/x86_64-linux-gnu/libgfortran.so.5 (0x0000726274200000)
        /lib64/ld-linux-x86-64.so.2 (0x0000726278e4e000)
        libopen-rte.so.40 => /lib/x86_64-linux-gnu/libopen-rte.so.40 (0x0000726275544000)
        libhwloc.so.15 => /lib/x86_64-linux-gnu/libhwloc.so.15 (0x000072627643e000)
        libevent_core-2.1.so.7 => /lib/x86_64-linux-gnu/libevent_core-2.1.so.7 (0x00007262761cb000)
        libevent_pthreads-2.1.so.7 => /lib/x86_64-linux-gnu/libevent_pthreads-2.1.so.7 (0x0000726277287000)
        libsuitesparseconfig.so.7 => /lib/x86_64-linux-gnu/libsuitesparseconfig.so.7 (0x0000726277282000)
        libcolamd.so.3 => /lib/x86_64-linux-gnu/libcolamd.so.3 (0x0000726276a38000)
        libcamd.so.3 => /lib/x86_64-linux-gnu/libcamd.so.3 (0x0000726276433000)
        libccolamd.so.3 => /lib/x86_64-linux-gnu/libccolamd.so.3 (0x00007262761bf000)
        libgomp.so.1 => /lib/x86_64-linux-gnu/libgomp.so.1 (0x00007262754ee000)
        libbtf.so.2 => /lib/x86_64-linux-gnu/libbtf.so.2 (0x0000726276e12000)
        libmumps_common-5.6.so => /lib/x86_64-linux-gnu/libmumps_common-5.6.so (0x0000726275474000)
        libptscotchparmetisv3-7.0.so => /lib/x86_64-linux-gnu/libptscotchparmetisv3-7.0.so (0x00007262761b8000)
        libmetis.so.5 => /lib/x86_64-linux-gnu/libmetis.so.5 (0x0000726275196000)
        libCombBLAS.so.2.0.0 => /lib/x86_64-linux-gnu/libCombBLAS.so.2.0.0 (0x00007262761a0000)
        libtrilinos_ifpack.so.13.2 => /lib/x86_64-linux-gnu/libtrilinos_ifpack.so.13.2 (0x000072627406c000)
        libtrilinos_amesos.so.13.2 => /lib/x86_64-linux-gnu/libtrilinos_amesos.so.13.2 (0x0000726275140000)
        libtrilinos_galeri-epetra.so.13.2 => /lib/x86_64-linux-gnu/libtrilinos_galeri-epetra.so.13.2 (0x000072627542b000)
        libtrilinos_aztecoo.so.13.2 => /lib/x86_64-linux-gnu/libtrilinos_aztecoo.so.13.2 (0x00007262750bb000)
        libtrilinos_zoltan.so.13.2 => /lib/x86_64-linux-gnu/libtrilinos_zoltan.so.13.2 (0x0000726274523000)
        libtrilinos_epetraext.so.13.2 => /lib/x86_64-linux-gnu/libtrilinos_epetraext.so.13.2 (0x0000726273f00000)
        libscotch-7.0.so => /lib/x86_64-linux-gnu/libscotch-7.0.so (0x0000726273e6b000)
        libtrilinos_epetra.so.13.2 => /lib/x86_64-linux-gnu/libtrilinos_epetra.so.13.2 (0x0000726273d08000)
        libtrilinos_teuchosparameterlist.so.13.2 => /lib/x86_64-linux-gnu/libtrilinos_teuchosparameterlist.so.13.2 (0x0000726273800000)
        libtrilinos_teuchoscore.so.13.2 => /lib/x86_64-linux-gnu/libtrilinos_teuchoscore.so.13.2 (0x00007262749a3000)
        libopenblas.so.0 => /lib/x86_64-linux-gnu/libopenblas.so.0 (0x0000726271420000)
        libptscotcherr-7.0.so => /lib/x86_64-linux-gnu/libptscotcherr-7.0.so (0x000072627619b000)
        libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x0000726276187000)
        liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x0000726275089000)
        libcrypto.so.3 => /lib/x86_64-linux-gnu/libcrypto.so.3 (0x0000726270e00000)
        libcurl.so.4 => /lib/x86_64-linux-gnu/libcurl.so.4 (0x0000726273c47000)
        libsz.so.2 => /lib/x86_64-linux-gnu/libsz.so.2 (0x0000726276182000)
        libxcb.so.1 => /lib/x86_64-linux-gnu/libxcb.so.1 (0x0000726273c1e000)
        libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x0000726273beb000)
        libtrilinos_trilinosss.so.13.2 => /lib/x86_64-linux-gnu/libtrilinos_trilinosss.so.13.2 (0x0000726273bbd000)
        libtrilinos_teuchosremainder.so.13.2 => /lib/x86_64-linux-gnu/libtrilinos_teuchosremainder.so.13.2 (0x00007262758e4000)
        libtrilinos_teuchosnumerics.so.13.2 => /lib/x86_64-linux-gnu/libtrilinos_teuchosnumerics.so.13.2 (0x00007262758c7000)
        libtrilinos_teuchoscomm.so.13.2 => /lib/x86_64-linux-gnu/libtrilinos_teuchoscomm.so.13.2 (0x0000726271342000)
        libtrilinos_triutils.so.13.2 => /lib/x86_64-linux-gnu/libtrilinos_triutils.so.13.2 (0x0000726273b5e000)
        libscotcherr-7.0.so => /lib/x86_64-linux-gnu/libscotcherr-7.0.so (0x0000726275426000)
        libtrilinos_teuchosparser.so.13.2 => /lib/x86_64-linux-gnu/libtrilinos_teuchosparser.so.13.2 (0x0000726270d9e000)
        libtrilinos_kokkoscore.so.13.2 => /lib/x86_64-linux-gnu/libtrilinos_kokkoscore.so.13.2 (0x0000726270d15000)
        libnghttp2.so.14 => /lib/x86_64-linux-gnu/libnghttp2.so.14 (0x0000726271317000)
        libidn2.so.0 => /lib/x86_64-linux-gnu/libidn2.so.0 (0x0000726270cf3000)
        librtmp.so.1 => /lib/x86_64-linux-gnu/librtmp.so.1 (0x0000726270cd5000)
        libssh.so.4 => /lib/x86_64-linux-gnu/libssh.so.4 (0x0000726270c64000)
        libpsl.so.5 => /lib/x86_64-linux-gnu/libpsl.so.5 (0x0000726273b4a000)
        libssl.so.3 => /lib/x86_64-linux-gnu/libssl.so.3 (0x0000726270bba000)
        libgssapi_krb5.so.2 => /lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x0000726270b66000)
        libldap.so.2 => /lib/x86_64-linux-gnu/libldap.so.2 (0x0000726270b08000)
        liblber.so.2 => /lib/x86_64-linux-gnu/liblber.so.2 (0x0000726270af8000)
        libzstd.so.1 => /lib/x86_64-linux-gnu/libzstd.so.1 (0x0000726270a3e000)
        libbrotlidec.so.1 => /lib/x86_64-linux-gnu/libbrotlidec.so.1 (0x0000726270a30000)
        libaec.so.0 => /lib/x86_64-linux-gnu/libaec.so.0 (0x000072627499a000)
        libXau.so.6 => /lib/x86_64-linux-gnu/libXau.so.6 (0x000072627507f000)
        libXdmcp.so.6 => /lib/x86_64-linux-gnu/libXdmcp.so.6 (0x0000726270a28000)
        libcap.so.2 => /lib/x86_64-linux-gnu/libcap.so.2 (0x0000726270a1b000)
        libunistring.so.5 => /lib/x86_64-linux-gnu/libunistring.so.5 (0x000072627086e000)
        libgnutls.so.30 => /lib/x86_64-linux-gnu/libgnutls.so.30 (0x0000726270674000)
        libhogweed.so.6 => /lib/x86_64-linux-gnu/libhogweed.so.6 (0x000072627062c000)
        libnettle.so.8 => /lib/x86_64-linux-gnu/libnettle.so.8 (0x00007262705d7000)
        libgmp.so.10 => /lib/x86_64-linux-gnu/libgmp.so.10 (0x0000726270553000)
        libkrb5.so.3 => /lib/x86_64-linux-gnu/libkrb5.so.3 (0x000072627048a000)
        libk5crypto.so.3 => /lib/x86_64-linux-gnu/libk5crypto.so.3 (0x000072627045e000)
        libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x0000726270458000)
        libkrb5support.so.0 => /lib/x86_64-linux-gnu/libkrb5support.so.0 (0x000072627044b000)
        libsasl2.so.2 => /lib/x86_64-linux-gnu/libsasl2.so.2 (0x0000726270431000)
        libbrotlicommon.so.1 => /lib/x86_64-linux-gnu/libbrotlicommon.so.1 (0x000072627040e000)
        libbsd.so.0 => /lib/x86_64-linux-gnu/libbsd.so.0 (0x00007262703f8000)
        libp11-kit.so.0 => /lib/x86_64-linux-gnu/libp11-kit.so.0 (0x0000726270254000)
        libtasn1.so.6 => /lib/x86_64-linux-gnu/libtasn1.so.6 (0x000072627023e000)
        libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x0000726270237000)
        libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x0000726270224000)
        libmd.so.0 => /lib/x86_64-linux-gnu/libmd.so.0 (0x0000726270215000)
        libffi.so.8 => /lib/x86_64-linux-gnu/libffi.so.8 (0x0000726270209000)
by (330 points)
Some update:
I put together a toy example (just a 10 x 10 x 10 cube stimulating at the corner) which can run fast so I could get a profile run. Below are the perf report outputs. It looks like the main bottleneck is in petsc:

Samples: 88K of event 'cycles:P', Event count (approx.): 66435400924
  Children      Self  Command   Shared Object                             Symbol
+   67.17%     0.00%  openCARP  [unknown]                                 [k] 0x0027e718058d4800                                                                                   ◆
+   67.17%     0.00%  openCARP  openCARP                                  [.] SF::petsc_vector::~petsc_vector()                                                                    ▒
+   65.93%     0.00%  openCARP  openCARP                                  [.] SF::petsc_solver::operator()(SF::abstract_vector<int, double>&, SF::abstract_vector<int, double> cons▒
+   65.93%     0.01%  openCARP  libpetsc_real.so.3.19.6                   [.] KSPSolve                                                                                             ▒
+   58.17%     1.10%  openCARP  libopen-pal.so.40.30.3                    [.] ompi_sync_wait_mt                                                                                    ▒+   53.62%     5.91%  openCARP  libopen-pal.so.40.30.3                    [.] opal_progress                                                                                        ▒+   38.28%     0.26%  openCARP  libmpi.so.40.30.6                         [.] PMPI_Allreduce                                                                                       ▒+   37.98%     0.06%  openCARP  mca_coll_tuned.so                         [.] ompi_coll_tuned_allreduce_intra_dec_fixed                                                            ▒+   37.85%     0.48%  openCARP  libmpi.so.40.30.6                         [.] ompi_coll_base_allreduce_intra_recursivedoubling                                                     ▒+   34.33%     0.05%  openCARP  libpetsc_real.so.3.19.6                   [.] MatMult                                                                                              ▒+   34.27%     0.05%  openCARP  libpetsc_real.so.3.19.6                   [.] MatMult_MPIAIJ                                                                                       ▒+   29.02%     0.20%  openCARP  libmpi.so.40.30.6                         [.] ompi_coll_base_sendrecv_actual                                                                       ▒+   25.97%     0.33%  openCARP  libmpi.so.40.30.6                         [.] ompi_request_default_wait                                                                            ▒+   23.49%     0.02%  openCARP  libpetsc_real.so.3.19.6                   [.] VecScatterEnd                                                                                        ▒+   23.38%     0.04%  openCARP  libpetsc_real.so.3.19.6                   [.] PetscSFBcastEnd                                                                                      ▒+   23.11%     0.46%  openCARP  libmpi.so.40.30.6                         [.] ompi_request_default_wait_all                                                                        ▒+   23.05%     0.05%  openCARP  libpetsc_real.so.3.19.6                   [.] VecTDot                                                                                              ▒+   23.03%     0.07%  openCARP  libmpi.so.40.30.6                         [.] PMPI_Waitall                                                                                         ▒+   11.97%     0.04%  openCARP  libpetsc_real.so.3.19.6                   [.] VecNorm                                                                                              ▒+   11.18%     2.23%  openCARP  mca_btl_vader.so                          [.] mca_btl_vader_poll_handle_frag                                                                       ▒+   10.66%     2.86%  openCARP  mca_pml_ob1.so                            [.] mca_pml_ob1_recv_frag_callback_match                                                                 ▒+   10.29%     0.00%  openCARP  [unknown]                                 [k] 0000000000000000                                                                                     ▒+   10.22%     1.82%  openCARP  libc.so.6                                 [.] pthread_mutex_unlock@@GLIBC_2.2.5                                                                    ▒+    9.82%     0.11%  openCARP  mca_pml_ob1.so                            [.] mca_pml_ob1_recv                                                                                     ▒+    9.76%     0.00%  openCARP  [unknown]                                 [.] 0x8b480789480023b2                                                                                   ▒+    9.76%     0.00%  openCARP  openCARP                                  [.] SF::vector<opencarp::Activation>::~vector()                                                          ▒+    9.73%     0.00%  openCARP  openCARP                                  [.] opencarp::parabolic_solver::solve_CN(SF::abstract_vector<int, double>&)                              ▒+    7.15%     0.02%  openCARP  libpetsc_real.so.3.19.6                   [.] VecScatterBegin                                                                                      ▒+    6.99%     0.03%  openCARP  libpetsc_real.so.3.19.6                   [.] PetscSFBcastWithMemTypeBegin                                                                         ▒+    6.95%     0.00%  openCARP  [unknown]                                 [.] 0x00005d264fd43198                                                                                   ▒+    6.88%     0.00%  openCARP  [unknown]                                 [.] 0x00005a234e878e18                                                                                   ▒+    6.82%     0.00%  openCARP  [unknown]                                 [.] 0x00005f6f5fc20038                                                                                   ▒+    6.77%     0.00%  openCARP  libpetsc_real.so.3.19.6                   [.] 0x0000711c35a807ec                                                                                   ▒+    6.74%     0.00%  openCARP  libpetsc_real.so.3.19.6                   [.] 0x0000721dc3a807ec                                                                                   ▒+    6.74%     0.00%  openCARP  [unknown]                                 [.] 0x00005d9d6d49da98                                                                                   ▒+    6.73%     0.00%  openCARP  [unknown]                                 [.] 0x00005ed6ee179a98                                                                                   ▒+    6.68%     0.00%  openCARP  [unknown]                                 [.] 0x0000591e598faf68                                                                                   ▒+    6.64%     0.00%  openCARP  libpetsc_real.so.3.19.6                   [.] 0x00007759714807ec                                                                                   ▒+    6.63%     0.00%  openCARP  [unknown]                                 [.] 0x000065294b780f48                                                                                   ▒+    6.62%     0.00%  openCARP  [unknown]                                 [.] 0x0000634cf3fe3ca8                                                                                   ▒+    6.59%     0.00%  openCARP  [unknown]                                 [.] 0x00005a947a864c78                                                                                   ▒+    6.58%     0.00%  openCARP  libpetsc_real.so.3.19.6                   [.] 0x000079d3fbc807ec                                                                                   ▒+    6.55%     0.00%  openCARP  libpetsc_real.so.3.19.6                   [.] 0x00007603524807ec                                                                                   ▒+    6.55%     0.00%  openCARP  [unknown]                                 [.] 0x000060740289cc78                                                                                   ▒+    6.55%     0.00%  openCARP  libpetsc_real.so.3.19.6                   [.] 0x00007c0a238807ec

3 Answers

0 votes
by (330 points)
 
Best answer
I finally got it working as expected. The key was some combination of petsc mpi blas lapack.
I built all of these from source to get the newest versions. With the following, I was able to get it running fast again:
gcc 14
petsc 3.23.2 (compiled against blas 3.10 lapack 3.12)
using the mpich that comes with petsc
by (3.6k points)

Hi Jake,

Good to hear that you have it up and running at full speed! Yes, it is a good to configure petsc using the recommended configure as suggested in Installation | openCARP

Cheers,

Jorge

by (330 points)
Hi Jorge,

I think what the openCARP install instructions are missing is guidance on which version of petsc to use. We had a functional petsc install but it was an older version.

In any case, here is the build script I used which includes the petsc configure. I am also exploring using amd optimized blas and lapack libraries going forward. I also want to profile differences when using -O2 vs -O3 compile flags.

echo "Prepping for builds"
export BUILD_DIR=$(pwd)
echo "Building from" $BUILD_DIR
export CC=/usr/bin/gcc-14

mkdir localBuilds
mkdir localBuilds/lib

## build blas
wget http://www.netlib.org/blas/blas-3.10.0.tgz
tar -xvf blas-3.10.0.tgz
cd BLAS-3.10.0
make -j5 CFLAGS="-O3 -march=native" FCFLAGS="-O3 -march=native"
make -j
cp blas_LINUX.a ../localBuilds/lib/blas.a
cd ..

## Build LAPACK
wget https://github.com/Reference-LAPACK/lapack/archive/refs/tags/v3.12.1.tar.gz
tar -xvf v3.12.1.tar.gz
cd lapack-3.12.1
cp make.inc.example make.inc
make -j FFLAGS='-O3 -march=native' CFLAGS='-O2 -mavx2' BLASLIB="$BUILD_DIR/localBuilds/lib/blas.a"
cp *.a ../localBuilds/
cd ..

## Build petsc
git clone -b release https://gitlab.com/petsc/petsc.git petsc
cd petsc
git checkout v3.23.2

./configure \
   --force \
  --prefix=$BUILD_DIR/localBuilds \
  --with-blas-lib=$BUILD_DIR/localBuilds/lib/blas.a \
  --with-lapack-lib=$BUILD_DIR/localBuilds/lib/liblapack.a \
  --CC=$CC\
  --download-mpich \
  --download-metis \
  --download-parmetis \
  --with-debugging=0 \
  COPTFLAGS="-O3 -march=native" \
  CXXOPTFLAGS="-O3 -march=native" \
  FOPTFLAGS="-O3 -march=native"

make all
make install

cd ..
export PETSC_DIR=$BUILD_DIR/localBuilds
export PETSC_ARCH=arch-amd-gcc14-opt
export PETSC_ROOT=$BUILD_DIR/localBuilds
export PATH=$PETSC_ROOT/bin:$PATH
export LD_LIBRARY_PATH=$PETSC_ROOT/lib:$LD_LIBRARY_PATH

git clone https://git.opencarp.org/openCARP/openCARP.git
cd openCARP

export PKG_CONFIG_PATH=$BUILD_DIR/localBuilds/lib/pkgconfig:$PKG_CONFIG_PATH

cmake -S. -B_build -DDLOPEN=ON -DBUILD_EXTERNAL=Off  -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=/usr/bin/gcc-14 \
  -DPETSC_DIR=$PETSC_DIR \
  -DPETSC_ARCH=$PETSC_ARCH \
  -DCMAKE_PREFIX_PATH=$BUILD_DIR/localBuilds \
  -DCMAKE_AR=/usr/bin/gcc-ar-14 \
  -DCMAKE_RANLIB=/usr/bin/gcc-ranlib-14 \
  -DCMAKE_NM=/usr/bin/gcc-nm-14 \
  -DCMAKE_C_FLAGS="-march=native" \
  -DCMAKE_CXX_FLAGS="-march=native"

cmake --build _build -j
0 votes
by (500 points)
Hey Jake,

I don't really know what the source of your problems is, since what you link looks fine to me.

If it is possible on your HPC system, I would suggest to try it with the docker image of openCARP. If the problem persists, the source might be the HPC system itself.
by (330 points)
Hi Tobias,

We will try to get a docker solution set up and I will report back if that helps.
Best,
Jake B
0 votes
by (3.6k points)
Hi Jake,

Could you please provide also information about:

Which OS do you have?

Which compilar and version do you have?

Cheers,

Jorge
by (330 points)
Hi Jorge,
Weird, I tried answering this earlier. Hopefully it works this time:

gcc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
GNU Fortran (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0

Are there other compilers thatw ould be good to know about?
by (3.6k points)
Hi Jake,

I was reading a bit on your CPU architecture and the available flags for compilation. It seems that gcc 13 does not have all the options that you need for your CPU. I would suggest trying gcc 14. Also you can try gcc 14 and the specific compilation flags for your CPU znver5.

Cheers,

Jorge
by (330 points)
Hi Jorge,

Okay, I re-built using gcc 14, both with and without the znver5 flag. In both cases, no improvement  (below is the readout for a simulation that usually only takes 15 min total). We are still working on the docker solution as a test. Do you have any other suggestions? I have tried profiling openCARP using perf, but openCARP does not seem to respond well to the termination signals needed to stop a job early and still allow perf to properly recording and exit gracefully.

PROG    -----     ----    -------    -------|
time    %%comp    time     ctime       ETA  |
0.00    0.0    0.0    0:00:00    0:00:00
1.00    0.2    999.2    0:16:39    166:15:35
by (3.6k points)
Hi Jake,

Did you also recompile petsc? What is your configure for petsc?

Cheers,

Jorge
Welcome to openCARP Q&A. Ask questions and receive answers from other members of the community. For best support, please use appropriate TAGS!
architecture, carputils, documentation, experiments, installation-containers-packages, limpet, slimfem, website, governance
...