Options for setting up a server for openCARP

Question

Hi, I'm exploring options for a robust compute infrastructure for running openCARP simulations. When it comes to determining the hardware, I have several questions:

• Is MPI traffic optimized for distributed computing, i.e. between more than one machine?
• What is the "ideal" number of cores? I imagine there are diminishing returns as you increase the number of cores.
• What is the ideal number of CPU sockets in the server? (More buses between  CPUs may create traffic bottlenecks)
• What is the RAM requirement for 3D simulations?
• Is using AMD also an option?

Of course, all of this is ignoring the possibility of GPU/TPU support for openCARP, which is hopefully on the roadmap!

Answer 1

I think it really depends on the type of simulation you running.

MPI works well. carpentry scaling results are shown in Fig. 9 of this study and should be transferable to openCARP: https://dx.doi.org/10.1016/j.jcp.2015.10.045

The number of cores up to which performance scales will highly depend on the mesh size. For many simulations studies, running multiple simulations in parallel rather than further increasing the number of processes is the more attractive option.

With recent commits, initial support for Ginkgo as an alternative numerical backend was added. Ginkgo support a wide range of hardware platforms including GPUs but full support in openCARP might still take a little while: https://ginkgo-project.github.io/

Comments

Thanks for your answers Axel. Currently the 3D mesh volume I'm using has 2.6M points and 15.8M tetrahedral elements. Using a slightly modified tenTusscherPanfilov(TT2) model, it takes me about 1.25 hours to simulate 1000ms, when using 48 cores.

In your opinion, what do you think about increasing the core size to 128 cores? Will this cause a noticeable difference in the case of this mesh?

Also, if we are running multiple simulations in parallel, what are the recommended memory requirements?

Lastly, is using AMD an option? I somehow feel it won't be, but wanted to make sure.

---

The only advise I could give would be based on the Figure referenced above.

More scaling results with both PETSc and Ginkgo backends will come in the future.

Why do you think AMD CPUs could cause problems?

---

I had heard somewhere that Intel and AMD uses different compilers. But on further research, I now see that gcc can be used on both types!