Funding NSF - OCI -0904288

Collaborative Research: Scalable Multiscale Models for Cerbrovasculature Algorithms: Software and Petaflop Simulation

Future petaflop simulations of realistic biological and physical systems will necessarily involve concurrent multiscale modeling. Here, we address fundamental mathematical, algorithmic and software issues for simulating a human brain vascular model, the first of its kind, consisting of 100 large 3D arteries (Macrovascular Network, MaN), 10 million arterioles (Mesovascular Network, MeN) and one billion capillaries (Microvascular Network, MiN). The three-level MaN-MeN-MiN integration offers a general platform for developing hybrid deterministic-stochastic systems, scalable algorithms, and scalable multiscale software to handle coupling between heterogeneous PDEs and also between continuum and atomistic formulations.

Building upon our initial work on the human arterial tree and the new brain imaging data, we propose image-based 3D Navier-Stokes simulations for fully resolving MaN, coupled to subpixel stochastic simulations of MeN and MiN to complete the closure. In particular, for MeN we will employ the quasi-1D Euler equations for the arteriolar tree, constructed from a fractal law with uncertain geometric and dynamical properties. For MiN, we will employ the stochastic Darcy's law, with an "effective" permeability obtained by pilot atomistic (dissipative particle dynamics) simulations on stochastic replicas of capillary beds in conjunction with upscaling techniques. We estimate that these multiscale simulations will require no more than 6 hours per cardiac cycle on the new petaflop systems, hence opening up the possibility of systematic studies to investigate normal brain perfusion and important clinical pathologies of the brain.

We will employ multiple levels of parallelism to facilitate the multiscale model and the domain-decomposition we propose for a single petaflop platform and also for the future extensions of TeraGrid. We will implement an MPI/UPC hybrid model to exploit the strengths of both programming paradigms: the high scalability and rich functionality for process control in MPI, and the low communication overhead for small messages and fine-grain parallelism in UPC. We will further seek to integrate multi-threading into the MPI/UPC model, especially for dynamic refinement. The main software advancement will be the development of MPIg tailored for multiscale applications, like the MaN-MeN-MiN problem, on a single or multiple petaflop platforms. Several open issues associated with co-processing and visualization of petabyte-size data will be also addressed.

Broader Impact: Our work will contribute to Computational Mathematics (interfacing heterogeneous PDEs, and also PDEs-atomistic systems); to Computer Science (development of UPC/MPI, multiscale MPIg, and increased leverage of vendor-supplied MPI in MPIg); and Bioengineering (biomechanics gateway to simulate brain pathologies). This proposal is transformative in that it shifts the computational paradigm to a new level (orders of magnitude above the state-of-the-art) that will allow, for first time, realistic simulations of cedrebrovasculature in health and disease. The validated algorithms for petaflop computing we propose are of general interest for use in many multiscale biological and physical applications, including vascular trees of all living organisms and also in simulations of nuclear reactors and other power/chemical plants.

Education and Outreach: The new simulation environment, with the human brain as a back- drop, will be critical in training a new generation of inter-disciplinary scientists to be comfortable in using multiscale mathematics and scalable software tools for extreme computing. We plan to engage postdocs, graduate, undergraduate and high school students. We will use 3D immersive/interactive visualizations as an opportunity to educate students about simulation, predictability, and other issues of computer science, engineering, and applied mathematics. Outreach activities will involve female students from middle and high schools and students from the special MET high schools.

Publications

  • L. Grinberg, T. Anor, J.R. Madsen and G.E. Karniadakis. Simulation of the human intracranial arterial tree. Philosophical Transactions of the Royal Society, 367(1896), 2371-2386, 2009.
  • L. Grinberg, T. Anor, J.R. Madsen, A. Yakhot and G.E. Karniadakis. Large-Scale Simulation of the Human Arterial Tree. Clinical and Experimental Pharmacology and Physiology, 36(2), 194-205, 2009.
  • L. Grinberg and G. E. Karniadakis. Outflow Boundary Conditions for Arterial Networks with Multiple Outlets. Annals of Biomedical Engineering, 36(9), 1496-1514 (2008).