On this page we present the bundle feature, which allows to easily share and/or release publicly the experiments you created with the carputils framework.

In particular, this feature allows to create a self-contained bundle from an experiment, making it reproducible on other computing systems. It also allows you to share your experiments easily by pushing them to a remote Git repository or releasing them in the openCARP community on Zenodo.

Introduction to the bundle feature

The goal of the bundle feature is to turn an existing carputils experiment to a self-contained bundle, that contains all data (experiment source code and configuration, ...) and metadata (openCARP version, ...) necessary to reproduce a simulation run, and that is easy to share with the community.

The feature comprises a set of carputils options allowing to perform various actions:

--bundle <bundle_name>: this option is mandatory to use the bundle feature and can be associated with the other options below. If used alone, this option allows to create a local bundle in a folder named <bundle_name>_bundle.
--archive-bundle: at the end of the bundle creation, the bundle folder is turned into a ZIP archive.
--push-bundle <git_repo_url>: initiates or updates a Git repository in the bundle folder and pushes the bundle to <git_repo_url>. <git_repo_url> can be omitted if --update-bundle is set and the git remote already exists.
--update-bundle: this option allows to overwrite an existing bundle. It will create new commits if a Git repository already exists in the bundle.
--release-bundle: creates a sample email asking for a release of the bundle, which can be sent to the openCARP curators to ask for a release of the experiment on RADAR4KIT.
--bundle-output: include the simulation outputs to the bundle

The basic workflow for creating a bundle is structured as follows:

Indicate which files are needed to run the experiment (either by moving these files in a specific folder included in bundles by default, by marking paths in the script run.py or by including them in a .bundleinclude file).
(Optional) Generate a sample metadata file for the experiment using the command generateMetadata and adapt it.
Run the simulation with all relevant options to create the bundle. For example with python run.py --tend 20. --bundle my_experiment.
(Optional) Validate that the simulation in the bundle is reproducible by running it using the openCARP Docker image.
(Optional) Push your bundle to a Git repository and/or publish it on RADAR4KIT

Get started with the bundle feature

Indicate which files and folders are necessary to run the experiment

The self-contained bundle of a simulation must contain all files required to run the simulation. As a consequence, in order to create a bundle from a simulation, you have to indicate to carputils which external files are required to run your experiment.

There are several ways to do so:

Put the files you want to include in a folder named external or Mesh in the experiment's directory. These folders will be included in the bundle by default.
Use the function carputils.tools.bundle_include within your carputils script to mark the files to be included (see the next section). These files can be located anywhere on your system and will be included in the external or Mesh folder within the bundle.
List files you want to include in file name .bundleinclude in your experiment's directory. These files have to be contained within the experiment's subfolders, and will be included to the bundle at the same relative location. For more details, see the dedicated section.

The different solutions can be combined, but you should only choose one of them for each file or folder to include.

Mark files in the carputils script with the function `bundle_include`

In your carputils script, every file or folder path encapsulated with the function carputils.tools.bundle_include will be included to the bundle, and the paths to these objects will be updated accordingly in the bundle, both within the carputils script and within the openCARP parameter file if one is provided.

To use this function in practice, modify your script (let's call it run.py) in the following way:

encapsulate the path to a required external file with the function carputils.tools.bundle_include.
In addition, if the file is a mesh file , set the 'mesh' flag in the bundle_include function to True. This will make the mesh copied in a separate folder called Mesh within the bundle. This Mesh folder will be tracked using Git LFS (Large File Storage) if it is available on your system.

Here is a typical example of modifications that have to be done before creating a bundle from an experiment:

Original script

import os
from datetime import date
from carputils import tools
EXAMPLE_DIR = os.path.dirname(__file__)
[...]
@tools.carpexample(parser, jobID)
def run(args, job):
    #####################################
    #Perform the simulation
    #####################################
    cmd  = tools.carp_cmd(os.path.join(EXAMPLE_DIR, 'sim.par'))
    cmd += ['-stimulus[0].vtx_file', os.path.join(EXAMPLE_DIR, 'stim')]
    cmd += ['-meshname', os.path.join(EXAMPLE_DIR, 'mesh')]
[...]

Ready-to-bundle script

import os
from datetime import date
from carputils import tools
EXAMPLE_DIR = os.path.dirname(__file__)
[...]
@tools.carpexample(parser, jobID)
def run(args, job):
    #####################################
    #Perform the simulation
    #####################################
    cmd  = tools.carp_cmd(tools.bundle_include(os.path.join(EXAMPLE_DIR, 'sim.par')))
    cmd += ['-stimulus[0].vtx_file', tools.bundle_include(os.path.join(EXAMPLE_DIR, 'stim'))]
    cmd += ['-meshname', tools.bundle_include(os.path.join(EXAMPLE_DIR, 'mesh'), mesh=True)]
[...]

In case a file or folder is referenced several times in a carputils script, it is necessary to use the bundle_include function on each occurence. In this situation, it is recommended to use the bundle_include function once and then to use the returned value for later references to the object. For example:

@tools.carpexample(parser, jobID)
def run(args, job):
    param_file = tools.bundle_include("/path/to/param/parameters.par")
    # param_file contains "/path/to/param/parameters.par".
    # If the --bundle option is not used, the function is passive and just returns the path given as an argument.
    # If the --bundle option is used, the file parameters.par is copied to the bundle, and the path given as argument is modified accordingly in the script created within the bundle.
    # In the rest of the script, we use param_file to refer to the file.
    print(f"Define openCARP command using parameter file: {param_file}")
    cmd = tools.carp_cmd(param_file)
    job.carp(cmd)

Include files via `.bundleinclude`

Another solution to include files or folders within a bundle is to create a file named .bundleinclude in the same folder as your carputils script. In this file, each line must contain a pattern. All files or folders matching this pattern (according to the Python module glob) that are contained within the experiment's folder will be included to the bundle.

The file structure within the experiment's folder is preserved when including files via .bundleinclude.

Example of .bundleinclude file:

README.md
meshes/Square*
**/*.par

Create a bundle

Once files to be included in the bundle have been indicated, the bundle of the experiment can be created. In order to do so, launch the simulation you want to bundle (with all desired arguments) and add the --bundle option:

python ./run.py --your-arguments --bundle experiment_name

where experiment_name is the name that will be given to the bundle.

After the execution, a directory named experiment_name_bundle will be created in your working directory, and should be a self-contained version of the experiment you have run. You can test the bundle by running the experiment it contains, using the same command line as for the bundle creation (without the --bundle argument). This command line is also stored in the metadata file codemeta.json within the bundle, in the field softwareHelp.

Test your bundle

In order to ensure that your experiment is reproducible, and that your bundle can be run from outside your environment, you can test your bundle using the openCARP Docker image (this requires to install and start Docker on your system). This allows to run your bundle within an isolated environment where openCARP and carputils are available, hence to ensure that the experiment can be run independently from your computing environment.

You can either test the bundle manually or use GitLab CI/CD if you have pushed your bundle to a GitLab instance.

Test the bundle manually

Note: in this section, we assume that Docker is installed and running on your system.

In a terminal, go to your bundle directory and run:

docker run \
  -v $PWD:/bundle \
  --workdir /bundle \
  --entrypoint python \
  docker.opencarp.org/opencarp/opencarp:v17-0 \
  run.py --your-options

where

run.py --your-options should be replaced by the relevant options for running your experiment, registered in the codemeta.json in the field "softwareHelp".
v17-0 should be replaced by the version tag indicated for openCARP in the codemeta.json, in the field "softwareRequirements".

More information about this command:

# Start a Docker container
docker run \
  # Mount the current directory to /bundle in the container.
  # This means that the container will be isolated from your environment, except for the current directory which will be available in /bundle.
  # On Windows, replace $PWD with %cd%.
  -v $PWD:/bundle \
  # Use /bundle as working directory in the container
  --workdir /bundle \
  # Run the command "python" when starting the container
  --entrypoint python \
  # URL to the openCARP Docker image, which will be used to create the container
  docker.opencarp.org/opencarp/opencarp:v17-0 \
  # Arguments to give to the `python` command to run the experiment in the bundle
  run.py --example-option --duration 20

If the experiment runs successfully, it means that the bundle is valid and that the experiment can be run with a standard openCARP installation.

Test using GitLab CI/CD

If you created your bundle using the option --push-bundle, the bundle contains a GitLab CI/CD configuration file allowing to perform the test presented above automatically.

This test will be executed if you push your bundle to a GitLab instance and have Docker runners available for your project. You can check the result of the test by opening your project on GitLab and going to Build > Pipelines in the left menu.

Supplementary options

Include metadata

When a bundle is created, a metadata file is automatically generated and included. This file contains autogenerated metadata such as:

the information about the creator found in the carputils settings file (name, email, ORCID)
the name given to the bundle
the command line to use in order to reproduce the bundled experiment
the python version and openCARP commit identifier used to create the bundle

Yet, this metadata file only contains few information about your experiment, and you should consider providing a metadata file with extended information manually, before generating the bundle.

You can generate a metadata file from a template by using the generateMetadata script located in the bin directory of carputils. If this directory is in your path, you can open a shell and run:

cd /path/to/experiment

generateMetadata

where /path/to/experiment is the path to your experiment's directory. You can now open the file named codemeta.json and fill it with the information related to your experiment. When creating the bundle, this file will automatically be included if it is located in the same directory as run.py.

Include simulation outputs

If a bundle is created for archiving a simulation, it can be useful to provide the output of the simulation in the bundle.

It is possible to do so by using the --bundle-output option in addition to the --bundle option.

This option creates a specific folder named sim_outputs in the bundle, in which the following folders are copied after the experiment ran (paths are given relatively to the experiment directory):

meshes (where meshes created on the fly are usually stored)
$jobID (where the outputs of the simulation are located)

Create a ZIP archive of the bundle

It is possible to generate a ZIP archive instead of a plain directory when creating a bundle by using the option --archive-bundle.

Upload your bundle to a git repository

The option --push-bundle allows to initiate a Git repository within the bundle folder and to push this repository to a remote location.

Note: You have to be allowed to push to the repository you provide as an argument to the --push-bundle option: on GitLab, the repository can be non-existent, but on GitHub for example, you have to create an empty repository at the given URL before using this option.

Note: In the rest of this section, we will assume that the repository is hosted on https://gitlab.com for the user username, but this workflow is applicable to any remote location.

To create a git repository of your bundle, you have to use the --push-bundle option, followed by the link to the git repository you want to push to:

./run.py --bundle exp_name --push-bundle https://gitlab.com/username/my-project.git

When running this command, default LICENSE and NOTICE files will be added to the bundle, applying the same license as carputils to your experiment: Apache 2.0. If you want to use another license, replace these files by the license of your choice. Please be careful that the license you are using is compatible with the Apache 2.0 license used for carputils.

Update an existing bundle

You can overwrite an existing bundle folder with the option --update-bundle. In practice, this will replace the existing bundle directory, but the following content will be preserved if not provided in the original experiment directory:

LICENSE file
codemeta.json
Git history and configuration files

In practice, this means that any modification done to the metadata or license files within the bundle will be preserved (unless such files are provided within the experiment directory), and that an existing Git repository will be updated with new commits instead of being erased.

Note: In particular, when using --push-bundle in conjunction with --update-bundle, it is not necessary to provide the URL to the remote repository if it has already been set within the bundle.

Release your experiment

Bundled experiment can be published in the Zenodo openCARP community using the Python package FACILE-RS.

With the process described below, the experiment will be released in accordance with the FAIR principles, described with the metadata provided in the bundle and associated with a persistent identifier (DOI).

0. Before releasing your experiment, please check that you provide sufficient metadata within the bundle. In the codemeta.json file, you should provide at least:

a title for your experiment
a short description of your experiment
Information about at least one author: given name, family name, orcid (optional but recommended)

Install FACILE-RS, for example using pip:
pip install FACILE-RS
Create a Zenodo personal access token:

Log in to https://zenodo.org
Go in the Applications settings and create a new personal access token with scope deposit:write.

Use FACILE-RS to reserve a DOI on Zenodo and update the bundle's metadata file with this DOI:
facile-rs zenodo prepare --codemeta-location <PATH_TO_BUNDLE>/codemeta.json --zenodo-url https://zenodo.org --zenodo-token <ZENODO_PERSONAL_TOKEN>
where <PATH_TO_BUNDLE> must be replaced by the path to the bundle directory and <ZENODO_PERSONAL_TOKEN> must be replaced by the token you have created in the previous step.
Create an archive of your bundle to be uploaded on Zenodo:
tar czvf experiment_bundle.tar.gz <PATH_TO_BUNDLE>
where <PATH_TO_BUNDLE> must be replaced by the path to the bundle directory.
Upload the bundle to Zenodo using FACILE-RS:
facile-rs zenodo upload --codemeta-location <PATH_TO_BUNDLE>/codemeta.json --zenodo-url https://zenodo.org --zenodo-token <ZENODO_PERSONAL_TOKEN> --zenodo-path zenodo-assets experiment_bundle.tar.gz
Find your upload on Zenodo and click on Edit. At the top, click on Select a community and select openCARP. Review the draft record, and when you are ready for submission, click on Submit for review.
If your bundle is compliant (i.e. the experiment can be executed, sufficient metadata and compatible license are provided), it will be published in the openCARP community and listed on the openCARP website.

Note: You can test this process on the Zenodo sandbox instance by replacing the Zenodo URL with https://sandbox.zenodo.org (the Zenodo token should also be created on the sandbox instance).

Further steps

The bundle feature is still evolving. If you have questions or suggestions, please use openCARP Q&A.

Subscribe to our newsletter and we will keep you posted.

Published community experiments

We maintain a lists of published experiments by members of the openCARP community.