EIC Tutorial: Geometry Development with DD4hep

Geometry Definition

Overview

Teaching: 15 min
Exercises: 10 min
Questions

  • How do we define geometry using DD4hep?

Objectives

  • Know where standard geometries as stored in eic-shell.

  • Understand the structure of a geometry description file.

Introduction

The ePIC geometry is described using the DD4hep toolkit.

DD4hep (Detector Description for High Energy Physics) is a toolkit which acts as a single, central source of detector information for simulation and reconstruction of simulated and experimental data. The core DD4hep geometry description is built around the ROOT geometry package while providing automatic conversions to other geometrical representaions, such as exporting CAD models, surfaces for acts reconstruction, material maps but most importany, Geant4 for carrying out simulations.

DD4hep additionally provides a much simplified wrapper around running Geant4 simulations, providing standardized output from sensitive detectors. In the case of the ePIC simulation, simulated particles and tracker/calorimeter hits are saved as collections in EDM4hep format (Event Data Model for High Energy Physics) built on podio (plain-old-data I/O).

Lesson

We start the discussion of the geometry definition with an overview of the locations of geometry files, and what is included in these files.

Location of standard geometries in eic-shell

Several standard geometry versions are included in eic-shell under the /opt/detector/ location. This includes (currently) at least the following:

$ ls -1 /opt/detector/
calibrations
dummyOutput.root
epic-23.10.0
epic-23.11.0
epic-23.12.0
epic-24.02.0
epic-24.02.1
epic-24.03.0
epic-24.03.1
epic-24.04.0
epic-main
fieldmaps
gdml
lib
share

Note: ls -1 lists the files with 1 file per line, i.e. in 1 column.

The versions avaliable in eic-shell are updated when tagged releases for the geometry are made each month or an update to dependancies installed in eic-shell removes back compatibility with older versions.

The epic-main directory contains the current ‘nightly build’ of the ePIC geometry, built from the epic repositories main branch every day.

$ ls -1 /opt/detector/epic-main/
bin
lib
share
$ ls -1 /opt/detector/epic-main/bin
thisepic.sh

You can load a geometry by ‘sourcing’ the bin/thisepic.sh file.

$ source /opt/detector/epic-main/bin/thisepic.sh

Both commands should have the same effect:

You can verify the latter by investigating the values of several environment variables:

Note: When working on the geometry in your own git branch you will need to source the setup.sh present in the local install directory.

Exercise:

  • Load the standard ePIC geometry and verify (with e.g. echo $DETECTOR) that the environment variables are set.
  • Load another geometry and verify that the environment variables are indeed different.

What is stored at those locations?

We will now take a look in the directory pointed to with the environment variable $DETECTOR_PATH, the location of the geometry resources:

$ ls $DETECTOR_PATH
calibrations                      epic_craterlake_tracking_only.xml        epic_lfhcal_with_insert.xml
compact                           epic_craterlake.xml                      epic_mrich_only.xml
dummyOutput.root                  epic_dirc_only.xml                       epic_pfrich_only.xml
epic_bhcal.xml                    epic_drich_only.xml                      epic_pid_only.xml
epic_calorimeters.xml             epic_forward_detectors_with_inserts.xml  epic_tof_endcap_only.xml
epic_craterlake_10x100.xml        epic_forward_detectors.xml               epic_tof_only.xml
epic_craterlake_10x275.xml        epic_full.xml                            epic_vertex_only.xml
epic_craterlake_18x110_Au.xml     epic_imaging_only.xml                    epic.xml
epic_craterlake_18x275.xml        epic_inner_detector.xml                  epic_zdc_lyso_sipm.xml
epic_craterlake_5x41.xml          epic_ip6_extended.xml                    epic_zdc_sipm_on_tile_only.xml
epic_craterlake_material_map.xml  epic_ip6.xml                             fieldmaps
epic_craterlake_no_bhcal.xml      epic_lfhcal_only.xml                     gdml

You will see many xml files, all of which are entry points to the geometry in certain configurations. For example, epic_drich_only.xml includes the geometry that has only the dual RICH or dRICH. epic_ip6.xml includes the beampipe geometry and the auxillary far-forward and backward detectors but no components of the central detector. The default configuration, epic.xml, is typically the configuration you will want to use, this is the value that DETECTOR_CONFIG will be set to by default.

Note: The current nightly eic-shell build contains only configurations for the craterlake detector setup.

Let’s take a look in the default entry point file, pointed at by the DETECTOR_CONFIG environment variable. This is the file epic.xml:

$ less $DETECTOR_PATH/$DETECTOR_CONFIG.xml

(Note: Use q to exit less, or use any editor you prefer.)

The xml file includes several blocks, but look in particular for the following lines:

These included files (e.g. far_forward/default.xml) can include further nested inclusion of even more files (e.g. far_forward/ZDC.xml).

Note: The xml files are parsed in order, so the definitions.xml file needs to be included before any file which needs to access the defined parameters.

Let’s now take a look at a particular detector subsystem end point file (which does not include any more files), namely tracking/vertex_barrel.xml.

$ less $DETECTOR_PATH/compact/tracking/vertex_barrel.xml

You will notice that the detector is described by parameters in a define block, such as (abridged):

  <define>
    <constant name="VertexBarrelMod_length"             value="VertexBarrel_length"/>
    <constant name="VertexBarrelMod_rmin"               value="VertexBarrel_rmin"/>

    <constant name="SiVertexSensor_thickness"           value="40*um"/>
  </define>

which can either use another parameter defined previously in the included files, or which can be defined in this file itself. A best practices is to define detailed parameters of each subsystem in the end point file, but to defer to the central definitions in the definitions.xml for quantities such as the overal size and location of the subsystem, or interfaces with other subsystems.

The parameters are then used in the detector block to define the detector itself (much abridged):

  <detectors>
    <detector
      id="VertexBarrel_0_ID"
      name="VertexBarrel"
      type="epic_VertexBarrel"
      readout="VertexBarrelHits"
      insideTrackingVolume="true">
      <dimensions
        rmin="VertexBarrelLayer1_rmin"
        rmax="VertexBarrelLayer3_rmax"
        length="VertexBarrelEnvelope_length" />
    </detector>
  </detectors>

The parametrization of the entire detector, down to the subsystems, is defined in these xml files. But where are the volumes created? The key here is the type field, which points to the detector type plugin that interprets the parametrization (here the type is epic_VertexBarrel). A well-written detector plugin can support many different detector configurations and parametrizations without the need to ever touch a line of code.

Exercise:

  • Identify which subsystem or detector you are interested in.
  • Take a look in the epic.xml file and locate where this detector is included.
  • Locate the end point file that defines the parameters that describe this file.
  • Identify the detector plugin that is used for this detector.

Key Points

  • Compact XML files are used to store parameters which are used by compiled plugins.

Viewing the geometry

Overview

Teaching: 10 min
Exercises: 10 min
Questions

  • How can we view the geometry?

Objectives

  • Understand how to export the geometry with dd_web_display.

  • Know of multiple way in which to open ROOT TGeo geometries.

Introduction

Before we move to discussion of the detector plugins in the next part, let’s discuss visualization on your local system.

ROOT visualization

The geometry included in eic-shell can be viewed with the ROOT geometry browser. However, we first need to export it from the built-in DD4hep format to the ROOT TGeo format, with a small utility program dd_web_display. To do this, we will need to ensure we are in a directory where we have write access, such as the directory ~/eic/.

$ cd ~/eic/
$ dd_web_display --export $DETECTOR_PATH/$DETECTOR_CONFIG.xml

Note: If you are not inside eic-shell, you may need to be connected to the internet as your run this command since a magnetic fieldmap will need to be downloaded.

The dd_web_display utility will create a ROOT file in the current directory that can be opened with the geometry viewer online at https://eic.phy.anl.gov/geoviewer/ or https://root.cern/js/latest, or a local ROOT installation. VSCode also has a JSROOT extension which provides the same functionality as the browser.

Note: The version of JSROOT on https://eic.phy.anl.govgeoviewer/ is older than the https://root.cern/js/latest version so some differences in the visualisation may be present.

The output file is by default named detector_geometry.root this can be changed if you want to look at several configurations using e.g.

$ dd_web_display -o vertex_geometry.root --export $DETECTOR_PATH/epic_vertex_only.xml

For local ROOT installations, the following commands may be helpful:

TGeoManager::Import("detector_geometry.root");
gGeoManager->GetTopVolume()->Draw("ogl")

The geometry viewer has to make decisions on what to draw in order to keep the number of facets small enough. This means that detectors with a large number of repeated components may not be drawn, or other detectors may not be drawn when those detectors are drawn. For this reason, we also have the subsystem-specific entry points xml files. For visualization of specific subsystems, these files are recommended.

Parameters describing how each component of the geometry should be vizualised are contained within the detector plugins and can be controlled through the xml description.

Exercise:

Geant4 visualization

Warning might not work for various reasons If you are used to the Geant4 geometry visualisation or want to visually inspect where a subset of your event sample this is still possible using npsim.

  npsim --runType qt --compactFile $DETECTOR_PATH/epic_vertex_only.xml --inputFiles root://dtn-eic.jlab.org//work/eic2/EPIC/EVGEN/SIDIS/pythia6-eic/1.0.0/18x275/q2_0to1/pythia_ep_noradcor_18x275_q2_0.000000001_1.0_run9.ab.hepmc3.tree.root --macro macro/b0_vis.mac

This particular example uses pythia6 min-bias events stored on the xrootd server at jlab. The visualization of particle tracks and the detector is controlled by the macro/b0_vis.mac file

Note: Geant4 does not try and limit its visualization of components so depending on your machine the full detector might struggle, please be patient. This performs much better for me outside of eic_shell but takes a long time to set up the environment.

Other visualisation you might want

ACTS Surfaces

Material Map

Event display

These are avaliable in various forms but not covered in this tutorial.

Key Points

  • The geometry, exported to the ROOT TGeo file, can be viewed with ROOT.

Modifying geometry

Overview

Teaching: 20 min
Exercises: 40 min
Questions

  • How do we modify or add geometry defined in DD4hep?

Objectives

  • Understand the structure of a geometry plugin source file.

Until now we have interacted with the read-only geometry that is stored inside the container. We will now move on to modifying this geometry. That requires that we work in a local copy of the geometry.

Checking out your local copy of the geometry

We will start with a local copy of the epic repository:

$ cd ~/eic/
$ git clone https://github.com/eic/epic
$ cd epic
$ ls
bin    calibrations    compact         macro      reports           scripts  templates
build  CMakeLists.txt  configurations  README.md  requirements.txt  src      views

As you can tell, the content of the repository itself is quite different from the installed version (as is often the case for other software as well). You will recognize, however, the compact directory with the subsystem xml files.

In order to compile and install the local geometry repository into a local directory, we can use the following commands:

$ cd ~/eic/epic
$ cmake -B build -S . -DCMAKE_INSTALL_PREFIX=install
$ cmake --build build -- install

Note: To speed up compilation, you may add the options -j4 to the last command, where 4 corresponds to the number of cores you can use.

This will install the geometry into the directory ~/eic/epic/install/ (and subdirectories). You will notice that ~/eic/epic/install/share/epic contains the same files that we explored earlier inside the /opt/detector directory.

As before, we now need to load the environment for this geometry. We can again use the bin/thisepic.sh script for this, though now we must use the one installed in our local installation directory:

$ source install/bin/thisepic.sh

When we run dd_web_display --export $DETECTOR_PATH/$DETECTOR_CONFIG.xml now, we will use the local geometry parametrization and the local geometry plugins. (Note: As before, downloads of fieldmaps and calibration files will be necessary.)

Quick Exercise:

  • Ensure that you have a local copy of the geometry repository which you can compile and install in a local directory.
  • Verify that, after sourcing the bin/thisepic.sh script, the DETECTOR_PATH points to the correct local install directory.
  • Verify that dd_web_display can indeed export the geometry for the detector subsystem configuration you used before.

Anatomy of a detector plugin

Introduction

It may be clear at this point how to make modifications to the parametrization, commit them to a branch in the local repository, and submit a pull request on GitHub to include them in the main branch.

For the remainder of this lesson we will focus on the vertex barrel detector using the much reduced geometry configuration epic_vertex_only.xml so that any change made are more evident.

What we have not covered yet is the discussion of what goes into a detector plugin. Let’s look at the epic_VertexBarrel plugin we encountered earlier. The names of the plugins may not agree with the source files in the src/ directory. This allows us to support multiple detector types with the same source files. In this case, epic_VertexBarrel is defined in the file src/BarrelTrackerWithFrame_geo.cpp.

If you are not sure what fine in the src directory builds the plugin you are looking for, find the type in the xml detector definition and use the grep shell command.

$ grep -r epic_VertexBarrel src/
src/BarrelTrackerWithFrame_geo.cpp:DECLARE_DETELEMENT(epic_VertexBarrel,    create_BarrelTrackerWithFrame)

The DECLARE_DETELEMENT line, at the bottom of the src/BarrelTrackerWithFrame_geo.cpp file provides dd4hep with the link which tells it to call the create_BarrelTrackerWithFrame function when an xml detector definition is given the epic_VertexBarrel type.

We now know that changing the content of the create_BarrelTrackerWithFrame function should be called when the epic_vertex_only.xml is used when dd4hep loads a geometry.

Passing parameters from xml

Next we will take a deeper dive into the create_BarrelTrackerWithFrame function to pick out the key components and how it is configured by the xml file compact/tracking/vertex_barrel.xml

static Ref_t create_BarrelTrackerWithFrame(Detector& description, xml_h e, SensitiveDetector sens)

here description contains access to the full tree defined from the main detector xml file. e contains the specific information contained within the xml tree within the <detector> blocks.

There are a few ways to access these xml configuration parameters. Some xml elements have methods provided by dd4hep which allow direct access to the values, such as:

51  Material                     air      = description.air();
52  int                          det_id   = x_det.id();
53  string                       det_name = x_det.nameStr();

A list of tags dd4hep provides a conventient conversion method for can be found here, (There is almost certainly a better link).

More often you may be wanting to define a parameter by a tag of your choice or if you’re wanting to be certain how it’s being handled. The following (abridged) code is an example of how to access parameters of any name.

  for (xml_coll_t su(x_det, _U(support)); su; ++su) {
    xml_comp_t  x_support         = su;
    double      support_thickness = getAttrOrDefault(x_support, _U(thickness), 2.0 * mm);
    double      support_length    = getAttrOrDefault(x_support, _U(length), 2.0 * mm);
    double      support_rmin      = getAttrOrDefault(x_support, _U(rmin), 2.0 * mm);
    double      support_zstart    = getAttrOrDefault(x_support, _U(zstart), 2.0 * mm);
    std::string support_name      = getAttrOrDefault<std::string>(x_support, _Unicode(name), "support_tube");
    std::string support_vis       = getAttrOrDefault<std::string>(x_support, _Unicode(vis), "AnlRed");
  }

The code loops over all <support> elements inside the <detector> block, located using _U(support) which interprets content as unicode. Inside each support node the getAttrOrDefault method sets the variables to the values given by the unicode thickness etc, or if they are not present in the support node sets a default value. If you want to require the parameter be defined in the xml file you could simply use x_support.child(thickness).

Note:

  • No support blocks actually currently appear in the compact/tracking/vertex_barrel.xml file so this block of code won’t be run.
  • In the xml file, there must be no white space between the parameter, = sign and the value.

A fuller description of how to access and use the xml parameters is given in section 2.4 of the dd4hep manual

Exercise:

  • Create and chackout a new branch forked from the main branch.
  • Add a new configuration parameter into compact/tracking/vertex_barrel.xml
  • Add code to src/BarrelTrackerWithFrame_geo.cpp which will read the new parameter and a print statement to display its value to the terminal.
  • Recompile and rerun the dd_web_display step using epic_vertex_only.xml to verify that the printout statement has been added.
  • Change the values in the xml and rerun to verify the value is being read properly.

Note: Changes to the xml files in install/share/epic/… can made without recompiling the code, however they will be overwritten when the code is recompiled. In order to test temporary changes a top level configuration file can be copied to a path outside of install. This then needs to be edited to internally point to the compact file you are editing rather than the path given by the install, ${DETECTOR_PATH}.

Building new components

DD4hep geometries are built in a similar hierarchical way to Geant4 geometries.

  • Shape - 3D shape of the component. Can be a simple shape, made from boolean combinations of simple shapes or imported from CAD.
  • Volume - Adds physical properties to the shape such as its material and if its sensitive.
  • Placement - Position(s) that the volume is located within your detector geometry, this can be a position nested within another volume, containing unique identifier.

We will start by looking at the shapes anv volumes. The most common shapes you are likely to find yourself using are Box and Tube. In src/BarrelTrackerWithFrame_geo.cpp both are used, module_component is defined as Box, the volume of which takes its material from the xml description:

172      Box          c_box(x_comp.width() / 2, x_comp.length() / 2, x_comp.thickness() / 2);
173      Volume       c_vol(c_nam, c_box, description.material(x_comp.materialStr()));

layer volumes into which module_components are later placed are described as a Tube but this time the volume is directly given the air material:

239    Tube       lay_tub(x_barrel.inner_r(), x_barrel.outer_r(), x_barrel.z_length() / 2.0);
240    Volume     lay_vol(lay_nam, lay_tub, air); // Create the layer envelope volume.

In DD4hep there is a type of volume called an Assembly which contains volumes placed within itself but doesn’t have a shape of its own. This is very useful for arranging volumes without needing a container volume defined with might have overlaps of their own where none are really present.

Notes:

Exercise:

  • Create a new simple volume within the hierachy in src/BarrelTrackerWithFrame_geo.cpp.
  • Recompile and rerun the dd_web_display step using epic_vertex_only.xml locating the new shape(s) you have added in the ROOTJS viewer.
  • Build a new tube volume which contains tracking layers.

Testing overlaps

It is important for running the Geant4 simulation that geometries do not overlap. When stepping through the geometry a particle cannot know which volume it is in. An overlap check is run by GitHub when you request that your changes are merged into the main branch of the epic code.

python scripts/checkOverlaps.py -c ${DETECTOR_PATH}/epic_vertex_only.xml

Exercise:

  • Run the overlap check on your geometry with the added component.
  • Change some parameters to add/remove the overlap and compare the output.

Readout

Placed volumes can be made sensitive by setting e.g.

194  c_vol.setSensitiveDetector(sens);

The type of information that will be saved to the output is defined usually as either:

  sens.setType("tracker");
  sens.setType("calorimeter");

In the xml file the readout for the detector is passed in the readout field of the detector definition

    <detector
      id="VertexBarrel_0_ID"
      name="VertexBarrel"
      type="epic_VertexBarrel"
      readout="VertexBarrelHits"
      insideTrackingVolume="true">

Where the readout name references a readout block also defined in the xml description

  <readouts>
    <readout name="VertexBarrelHits">
      <segmentation type="CartesianGridXY" grid_size_x="0.020*mm" grid_size_y="0.020*mm" />
      <id>system:8,layer:4,module:12,sensor:2,x:32:-16,y:-16</id>
    </readout>
  </readouts>

Here the name given will appear as the name of the branch containing the hits in the output edm4hep file. dd4hep provides very convenient segmentation to the readout which allows hits in a readout volume to be divided up to locations beyond its natural boundaries, this is configures by the x and y parameters as well as the grid_size.

The readout branch contains the information on the hit energy deposited, time of arival etc. which is usually found in a simulation output but in addition it contains CellID which is a 64 bit field which uniquely identifies the detector segmentation.

In the case of VertexBarrelHits, 8 bits always required by the system, 4 bits locate a specific layer, 12 a module, 2 a sensor and 32 the remaining x-y segmentation. In the code, dd4hep requires a separate hierachy of the geometry detector elements which are given tags and numbers so they can be uniquely identified. This hieracy doesn’t have to strictly follow the way the volumes are themselves constructed.

  DetElement mod_elt(lay_elt, module_name, module);
  pv = lay_vol.placeVolume(module_env, tr);
  pv.addPhysVolID("module", module);
  mod_elt.setPlacement(pv);

Here mod_elt is give the parent element layer_elt, the name and module number. Then the element is attached to a placed volume which has been given the physical volume id module.

To run the simulation and produce an output file containing the detector hits you can use npsim. I would suggest only using a small sample of events given by the --numberOfEvents flag.

$ npsim --runType run --compactFile $DETECTOR_PATH/epic_vertex_only.xml --inputFiles root://dtn-eic.jlab.org//work/eic2/EPIC/EVGEN/SIDIS/pythia6-eic/1.0.0/18x275/q2_0to1/pythia_ep_noradcor_18x275_q2_0.000000001_1.0_run9.ab.hepmc3.tree.root --numberOfEvents 100 --outputFile test.edm4hep.root

Inside the output file test.edm4hep.root there should be 4 trees:

events
runs
metadata
podio_metadata

The events tree contains the readout of your detectors, in this example it should contain only 7 branches,

MCHeader
MCParticles
_MCParticles_parents
_MCParticles_daughters
VertexBarrelHits
_VertexBarrelHits_MCParticle

The MCParticles branch contains information on all of the particles described by your generator and any secondaries produced in the simulation. VertexBarrelHits contains the hit information of the vertex barrel and has the association branch _VertexBarrelHits_MCParticle which references the particle in the MCParticles branch which caused the hit.

Exercise:

  • Run the simulation with a small dataset using
  • Look at the output datafiles usig your favorate ROOT browser.
  • Change the sensitive type of the BarrelTrackerWithFrame_geo.cpp and compare the output to what you first saw.
  • Try to make your new tube volume sensitive by setting as sensitive and adding a DetElement and giving it the necessary addPhysVolID values not currently used by the tracker.
  • Open ended - Move your new Tube detector into a separate src file (Or create a new simple detector) and include it separately in the xml definition and readout.

ACTS?

ToDo

Key Points

  • To add or modify geometry, we add geometry plugins written in C++.