\section{The role of high performance computing
\label{sec:hpc}}

Funded via CFI, Compute Canada has provided shared national computing
infrastructure to the nuclear physics research community.  This delivery model
is now changing, with a New Digital Research Infrastructure Organization (NDRIO) being set up by Universities Canada, and the
continuation of CANARIE with a revised focus on research networks and cybersecurity.  It is vital that the new organization be responsive to the needs of research fields making substantial use of high performance computing, such as subatomic physics.  This section summarizes the importance of shared high performance computing resources to the experimental and theoretical nuclear physics research communities.  Next-generation
projects will require even more resources, so they must be carefully planned and effectively operated.

\subsection{Experimental nuclear physics requirements}

\subsubsection{Jefferson Lab}

%GH: text from IPP-CINP-computing_19feb21
The experiment with the most significant computing demands is GlueX at JLab.  
The GlueX computing model is to do the lion's share of the raw data
reconstruction at the National Energy Research Scientific Computing Center
(NERSC) in the USA. In 2018, GlueX received 23~Mhr allocation at NERSC. For 2019 and
beyond GlueX requested 45~Mhr from NSERC and received 14~Mhr. Incoming
data monitoring (first 5 files of each run), iterative calibrations and data skims
being done on the Jefferson Lab cluster. The anticipated computing resources
from Jefferson Lab for GlueX per year are estimated at 35~Mhr. The remainder of
GlueX annual needs are anticipated at 35-50~Mhr via OSG, including Canadian
resources. The latter estimate comes from the GlueX 2018 summer simulation
campaign, which used a substantial amount of time on Compute Canada together
with another half dozen US university clusters. The numbers are based on
GlueX Phase II (high-intensity running that began in fall 2019). Monte Carlo
simulations and data analysis will be performed off-Jefferson Lab, with
approximately 30\% of the simulations to be performed on Canadian facilities for
the next 5 years. The GlueX MC requirement was reduced significantly recently
due to changing the anticipated number of simulated events from 2 per L1
triggered event to 2 per useful reconstructed physics event (roughly a factor of 4
reduction). For the Canadian fraction of the simulations and raw data
reconstruction, an average demand of 1500 Intel cores ((e.g. Intel E5-2683 v4
``Broadwell'' at 2.1GHz, like on the Cedar cluster base nodes) is projected for 5
years starting in 2019, with 600~TB of grid-accessible storage starting in 2019.
For a logistically efficient operation of GlueX simulations and data analyses
across many clusters and including Compute Canada, it is essential that the
workflow is managed through the OSG job environment and Compute Canada
technical support was requested to that end. A smooth workflow was
accomplished during the 2018 summer campaign by using a single Jefferson Lab
node as the launch point through OSG to all clusters that were part of that effort.

The MOLLER experiment, whose data taking is projected to be from
2025-30, plus an additional 4 years of data analysis afterward, project an increase in 
storage needs from 200~TB to 2~PB once data taking begins.  
The computing demands of SoLID are likely to be similar to the GlueX and MOLLER experiments.

%GH: triumf text is from 2015 brief
\subsubsection{TRIUMF}

The GRIFFIN array at TRIUMF has been designed to perform experiments using
radioactive beams, which are produced at ISAC with intensities from 0.01
particles per second to 10$^8$ particles per second. In the former case, the data
collection rates will be small due to the low intensity of the rare isotope
beam and typically a few TB per year. In the latter case, with high-intensity
rare isotope beams the important physics is often revealed in the observation
of very weak gamma-ray transitions from excited nuclear states, which have a
relative intensity 5 orders of magnitude lower than the strongest decay
branch. The study of these very weak transitions necessitates a very high
throughput data acquisition to collect very high-statistics datasets, followed
by intensive analysis to identify specific coincidence events. In order to
limit the collected data to only the most useful coincidences, the data is
filtered for various detector multiplicity and temporal coincidence conditions
in real time before being stored on disk. Despite this real-time filtering of
the detector signals, the GRIFFIN data acquisition system is capable of writing
filtered data at a rate of 300~MB per second and will therefore collect
datasets of around 250~TB in typical one-week experimental runs. It is
anticipated to perform an average of two of these high-rate runs per year and
accumulate 500~TB of new data per year, which will be stored for a minimum of
five years while the analysis is completed. 

The TIGRESS spectrometer at TRIUMF is utilized in a diverse scientific program
in nuclear structure and nuclear astrophysics research. It consists of up to
sixteen highly segmented hyper-pure germanium clover detectors, which surround
a reaction target where a nuclear reaction is induced by the accelerated
radioactive beam from ISAC-II. Charged particles and heavy ions following the
reaction are detected in various ancillary detector systems located inside the
vacuum chamber at the same time as emitted gamma rays are detected in the
TIGRESS detectors. Coincidence triggering conditions between detector
sub-systems can be made very selective with such a setup, and due to the
typically low intensity of accelerated radioactive ion beams the number of
individual events recorded is not as large as in the high-rate decay
spectroscopy experiments with GRIFFIN.  However, in order to make full use of
the position sensitivity of these highly segmented TIGRESS clover detectors it
is common to record a short waveform sample from the detector along with the
other event data, which is processed in the offline analysis. This waveform
collection can increase dramatically the size of the recorded dataset. The
TIGRESS digital data acquisition system was custom designed and built as part
of the original installation. The full digital data acquisition system is
currently being upgraded with the more modern digitizers developed
for the GRIFFIN project. Once this new DAQ system is in place, the expected data
collection rate will be typically 100~TB per year. The data will be stored for
a minimum of five years while the analysis is completed.

%GH: text from TRINAT_brief_CINP2020
TRINAT has utilized Compute Canada parallel processing capabilities for Monte Carlo simulations in the past. It is likely this capability will be needed in the future, particularly as high-statistics measusrements become possible.

\subsubsection{nEXO}

%GH: from nEXO Brief
 While computing resources for nEXO simulations and future data analysis are generally available at LLNL and SLAC, to some level at NERSC, as well as recently by IHEP in China, nEXO anticipates requesting resources from Compute Canada. These resources will support simulations for nEXO and analysis of data once nEXO becomes operational. Resources from Compute Canada will promote the scientific independence of Canadian researchers.


%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Theoretical nuclear physics requirements}

Theoretical nuclear physics has benefited enormously from recent increases in computing power, 
which has allowed the investigation of many complex problems that previously had to be treated with more approximate techniques.  As a result, theoretical nuclear physics has become a major HPC user, on-par with 
the needs of many experiments.

\subsubsection{Ab-initio nuclear structure}

%GH: combination of text from 2015 and from Navratil brief
This TRIUMF-based project involves large-scale {\it ab-initio} nuclear
structure and nuclear reaction calculations, using as input modern two- and
three-nucleon forces derived within chiral effective field theory. Using these
forces, the quantum many-nucleon problem is solved for bound and unbound
eigenstates.  It relies on a numerical solution of the quantum many-body problem, which is a CPU/GPU time demanding and memory intensive task, requiring access to large-scale parallel computers. 
Some of the results discussed in this document were obtained using the INCITE award on the ORNL Titan and Summit supercomputers, Livermore Computing machines, and Compute Canada machines, as well as to TRIUMF Theory cluster. 
 The computing allocation at ORNL is about 20 million
core hours per year, and the calculations use up to 6000~nodes
(96000~cores). On the LLNL machines, 128~nodes (2048~cores) are typically used,
with CPU usage exceeding a million core hours per year.  The computing needs
will grow in the future, as calculations are performed for heavier nuclei
(sd-shell and beyond).  Longer-term, alpha-clustering including the scattering
and reactions of alpha-particles with nuclei will be calculated. These problems
will require a significant increase of computing power, i.e. by a factor of 10
or more.
This research would greatly benefit from an exascale machine in Canada, to be competitive with the developments in the US and Europe.

\subsubsection{Lattice QCD}

%GH: text from 2015 brief
This is a computational method to obtain a quantitative understanding of
strong interactions.  Even though QCD is the fundamental interaction governing
nuclear physics, little is known about their direct connections and only few 
{\em ab-initio} calculations exist. In the past the immense complexity of even the
simplest nuclei in terms of quarks and antiquarks was an insurmountable
obstacle to lattice calculations. However, recent theoretical developments and
increase in computing power have made it possible to compute nuclei up to
helium on realistic lattice setups. Pushing these boundaries further and
determining the spectrum of light nuclei will give a solid foundation in QCD to
nuclear physics.  The Canadian lattice QCD effort is centered at York
University.  Primary resources have included a dedicated cluster of 320 cores at
York University and CPU/GPU facilities at the Danish Centre for Scientific
Computing.  Because the lattice effort at York University
is growing and the local dedicated cluster is aging, significant
growth in the use of Compute Canada resources, of the order of 4500 core-years
of CPU and 100~TB of storage, is anticipated.

\subsubsection{Quantum Monte Carlo simulations of neutron stars and nuclei}

%GH: text from 2015 brief
The Guelph theoretical group uses a variety of microscopic {\em ab-initio} simulation
methods to compute the properties of neutron stars and light nuclei. For dilute
neutron matter ($s$-wave interactions), the method of choice is diffusion Monte
Carlo (DMC). For more complicated nuclear interactions, nuclear Green's
Function Monte Carlo (light nuclei) and auxiliary field diffusion Monte Carlo
(infinite matter and medium-mass nuclei) are used.  All these methods are
exact, modulo the fermion-sign problem and can be extended to become
effectively variational methods. These methods use as an ansatz a trial wave
function, which embeds sufficient physical insights (from few- or many- body
theory) as well as variational parameters, which are systematically varied to
approach the state of chosen symmetry.

The computations typically
use 100-1000 cores.  Over the next 10 years, needs are projected to
increase by a factor of 50-100: the biggest issue is the number of particles,
$N$, that can be handled in the simulation. Since the time required scales as $N^3$,
doubling the number of particles takes 8 times more resources. Neutron-star 
related systems composed of roughly 100 particles are currently being studied
but systems of 400-500 would need to be addressed to be truly realistic.
%GH: text from Guelph_theory_Brief
While Compute Canada and other HPC user facilities are important, the availability of guaranteed local computing resources  is essential to the continued success of this research program.
As such, the Guelph theory group has recently applied for a
CFI JELF award to build a HPC facility to accelerate their research. Total project cost: \$452,000.

\subsubsection{Relativistic quark-gluon plasma}

%GH: text from 2015
When the nuclear temperature exceeds 170~MeV or about 2 trillion kelvin, the
quarks and gluons inside hadrons are liberated to form a quark-gluon plasma
(QGP).  This new state of matter existed in bulk only when the universe was
about a microsecond old. The research focus of the McGill nuclear theory group
is the investigation of the ultra-relativistic heavy ion collisions in general,
and the properties of the QGP in particular.  The computational challenge is to
simulate the entire evolution of the ultra-relativistic heavy ion collision,
starting from the Lorentz-contracted nuclear initial state, to the QGP phase,
and on to the final hadronic state.  Each of these steps needs different
computing environment: The most CPU-intensive step is the hydrodynamic
evolution part, using up to 8 million spatial cells and ${\cal O}(10^3)$ time
steps, utilizing a parallel computing capability.  Another CPU intensive step
is the hadronic scattering simulation, where in a typical LHC collision, ${\cal
O}(10^4)$ hadrons emerge out of the QGP. Keeping track of collisions among
these particles is numerically challenging.  This part is not parallelized, and
requires a massive number of serial nodes since ${\cal O}(10^2)$ hadronic
simulations need to be performed on each of the ${\cal O}(10^3)$ hydrodynamic
simulations.  Calculations for the outgoing hard and electromagnetic probes
have yet different computing needs, requiring a massive storage capability of
${\cal O}(10^2)$~TB.  It is estimated that at least 2,000 core-years with
3-4~GB/core per year and 250~TB of storage space is required in coming years.

\subsubsection{Multi-loop effects in precision measurements}

%GH: text from 2015
A very promising method to search for physics beyond the Standard Model (SM) is
to perform high-precision electroweak experiments at the intensity/precision
frontier, such as parity-violating Moller scattering, $e^+e^-$
collisions or electron-nucleon scattering.  These studies can provide indirect
access to physics at multi-TeV scales and play an important complementary role
to the LHC research program. These lower-energy experiments tend to be less
expensive than experiments at the high-energy colliders, but they are more
model-dependent and require significant theoretical input.  Thus, in order to match
the proposed electroweak experimental precision, it is necessary to provide
theoretical predictions on the observables of the SM with ~1\% precision or better. Since
the most of the SM electroweak theoretical predictions employ perturbative
expansion in orders of $\alpha$ (fine structure constant), it will be required
to consider contributions to the electroweak cross sections of at least up
$\sim\alpha^4$, corresponding to two-loop calculations in the Feynman diagram
approach. There has been some success in evaluating higher-order corrections for some specific QED processes, but much work is still to be done for the electroweak sector. Aleksejevs and Barkanova have carried out calculations for a complete set of one-loop electroweak corrections for the e-e scattering and are currently working on computer-algebra based packages for two-loop corrections. However, in the case of electroweak processes, full two-loop calculations face dramatic complications due to the presence of massive vector bosons ($W$ and $Z$) in the two-loop integrals. For the MOLLER experiment at JLab, the calculation of the parity-violating electroweak asymmetries up to the two-loop level requires the computation of thousands of Feynman diagrams.  Based on the assumption that a one loop
graph analytical result requires around $\sim$5~GB of RAM, about 500~GB of
allocated RAM per 100 two-loop graphs are required.  Finally, since the first
stage of the calculation deals with analytical computations, it is required to
store the results permanently, needing of the order of 600~TB of storage.  
These two-loop calculation techniques
can later be adapted for electron-proton processes, electron-positron
collisions, and other low-energy experiments involving particles of the SM and
new physics.