- Introduction to Python - Texas A&M0
Python course offered by Texas A&M HPRC
- Benchmarking with a cross-platform open-source flow solver, PyFR0What is PyFR and how does it solve fluid flow problems? PyFR is an open-source Computational Fluid Dynamics (CFD) solver that is based on Python and employs the high-order Flux Reconstruction technique. It effectively solves fluid flow problems by utilizing streaming architectures, making it suitable for complex fluid dynamics simulations. How does PyFR achieve scalability on clusters with CPUs and GPUs? PyFR achieves scalability by leveraging distributed memory parallelism through the Message Passing Interface (MPI). It implements persistent, non-blocking MPI requests using point-to-point (P2P) communication and organizes kernel calls to enable local computations while exchanging ghost states. This design approach allows PyFR to efficiently operate on clusters with heterogeneous architectures, combining CPUs and GPUs. Why is PyFR valuable for benchmarking clusters? PyFR's exceptional performance has been recognized by its selection as a finalist in the ACM Gordon Bell Prize for High-Performance Computing. It demonstrates strong-scaling capabilities by effectively utilizing low-latency inter-GPU communication and achieving strong-scaling on unstructured grids. PyFR has been successfully benchmarked with up to 18,000 NVIDIA K20X GPUs on Titan, showcasing its efficiency in handling large-scale simulations.
- Fine-tuning LLMs with PEFT and LoRA0As LLMs get larger fine-tuning to the full extent can become difficult to train on consumer hardware. Storing and deploying these tuned models can also be quite expensive and difficult to store. With PEFT (parameter -efficent fine tuning), it approaches fine-tune on a smaller scale of model parameters while freezing most parameters of the pretrained LLMs. Basically it is providing full performance that which is similar if not better than full fine tuning while only having a small number of trainable parameters. This source explains that as well as going over LORA diagrams and a code walk through.
- MDAnalysis - Python library for the analysis of molecular dynamics simulations0MDAnalysis is a python based library of tools for the analysis of molecular dynamics simulations. It is able to read and write many popular simulation formats including CHARMM, LAMMPS, GROMACS, and AMBER and more. This link contains the documentation pages of all MDAnalysis functions and has links to tutorials using Jupyter Notebooks.
- marimo | a next generation python notebook0Introduction seminar for new reactive python notebook from marimo ambassador.
- Fundamentals of Cloud Computing0An introduction to Cloud Computing
- Regulated Research Community of Practice0The daily news clearly shows the increasing threat to safety and privacy of data, personal as well as intellectual property. While the requirements such as DFARS 7012, HIPAA, and Cybersecurity Maturity Model Certification (CMMC) improve the consistency of data handling between agencies and contractors and grantees, it leaves academic institutions to figure out how to meet such requirements in a cost-effective way that fits the research and education mission of the institution. Most institutions, agencies, and companies act in isolation with one-off contract language to address data security and safeguarding concerns. Even though cybersecurity has a clear and uniform goal of protecting data, a onesize solution does not fit all academic institutions. By supporting this community with development of a community strategic roadmap, regular discussions and workshops, and a repository of generalized and specific resources for handling regulated research programs RRCoP lowers the barrier to entry for institutions handling new regulations.
- National Public Radio (NPR)0Pluses and challenges of mentor selection. Offers tips for acquiring a mentor (finding, asking). And how to be a good mentee. SMART framework mentioned. Discrimination mentioned. Difference between mentor and sponsor underlined. More than one mentor encouraged. Good tips.
- Set Up VSCode for Python and Github0VSCode is a popular IDE that runs on Windows, MacOS, and Linux. This tutorial will explain how to get set up with VSCode to code in Python. It will also provide a tutorial on how to set up Github integration within VSCode.
- Automated Machine Learning Book0The authoritative book on automated machine learning, which allows practitioners without ML expertise to develop and deploy state-of-the-art machine learning approaches. Describes the background of techniques used in detail, along with tools that are available for free.
- DELTA Introductory Video0Introductory video about DELTA. Speaker Tim Boerner, Senior Assistant Director, NCSA
- Data visualization with Matplotlib0Data visualization is a critical aspect of data analysis. It allows for a clear and concise representation of data, making it easier for users to understand and interpret complex datasets. One of the most popular libraries for data visualization in Python is Matplotlib. The included website aims to provide a brief overview of Matplotlib, its features, and examples/exercises to dive deeper into its functionalities.
- Moving-Lid-Driven Flow Simulation by Finite Difference Method0The listed repository contains code written in C++ to model the flow inside a cavity with a lid moving above from left to right by discretizing incompressible N-S equations with finite difference method. For the governing equations, artificial viscosity has been considered to increase the stability. In terms of solving the resulted algebraic equation system, both the Point Jacobi Method and Symmetric Gauss Seidel methods have been used for the iteration process.
- ACES: Charliecloud Containers for Scientific Workflows (Tutorial)0
- ACES: Charliecloud Containers for Scientific Workflows (Video)
- ACES: Charliecloud Containers for Scientific Workflows (Slides)
This tutorial introduces the use of Containers using the Charliecloud software suite. This tutorial will provide participants with background and hands-on experience to use basic Charliecloud containers for HPC applications. We discuss what containers are, why they matter for HPC, and how they work. We'll give an overview of Charliecloud, the unprivileged container solution from Los Alamos National Laboratory's HPC Division. Students will learn how to build toy containers and containerize real HPC applications, and then run them on a cluster. Exercises are demonstrated using the ACES cluster, a composable accelerator testbed at Texas A&M University. Students with an allocation on the ACES cluster can follow along with the ACES-specific exercises. - Why 'N How: Martinos Center for Biomedical Imaging:0The Why & How seminar series is designed to introduce research assistants, graduate students, and postdoctoral and clinical fellows – really, anyone who is interested – to the many tools used in medical imaging. These include software tools and most of the major imaging modalities wielded by investigators (MRI, PET, EEG, MEG, optical, TMS and others). As the name of the series suggests, the talks cover both the reasons researchers might need a particular tool and the nuts and bolts of how to apply it. You can watch videos of the overviews below.
- OpenHPC: Beyond the Install Guide0Materials for the "OpenHPC: Beyond the Install Guide" half-day tutorial, first offered at PEARC24. The goal of this repository is to let instructors or self-learners to construct one or more OpenHPC 3.x virtual environments, for those environments to be as close as possible to the defaults from the OpenHPC installation guide, and to then use those environments to demonstrate several topics beyond the basic installation guide. Topics include: 1. Building a login node that's practically identical to a compute node (except for where it needs to be different) 2. Adding more security to the SMS and login node 3. Using node-local storage for the OS and/or scratch 4. De-coupling the SMS and the compute nodes (e.g., independent kernel versions) 5. GPU driver installation (simulated/recorded, not live) 6. Easier management of node differences (GPU or not, diskless/single-disk/multi-disk, Infiniband or not, etc.) 7. Slurm configuration to match some common policy goals (fair share, resource limits, etc.)
- InsideHPC0InsideHPC is an informational site offers videos, research papers, articles, and other resources focused on machine learning and quantum computing among other topics within high performance computing.
- Better Scientific Software (BSSw)0
- Better Scientific Software (BSSw) Main Site
- BSSw Resources and Blog Posts
- BSSw Tutorial - Github Pages
The Better Scientific Software (BSSw) project provides a community to collaborate and learn about best practices in scientific software development. Software—the foundation of discovery in computational science & engineering—faces increasing complexity in computational models and computer architectures. BSSw provides a central hub for the community to address pressing challenges in software productivity, quality, and sustainability. - Anvil Documentation0Documentation for Anvil, a powerful supercomputer at Purdue University that provides advanced computing capabilities to support a wide range of computational and data-intensive research spanning from traditional high-performance computing to modern artificial intelligence applications.
- Official Documentation of VisIt0VisIt is a prominent open-source, interactive parallel visualization and graphical analysis tool predominantly used for viewing scientific data. Its GitHub repository offers a detailed insight into the software's source code, documentation, and contribution guidelines. In particular, it offers useful examples on how it
- Chameleon0Chameleon is an NSF-funded testbed system for Computer Science experimentation. It is designed to be deeply reconfigurable, with a wide variety of capabilities for researching systems, networking, distributed and cluster computing and security.
- Python Data and Viz Training (CCEP Program)0
- Guide to building AirSim on Linux machines0This article provides step-by-step instructions on how to build AirSim, a simulator for autonomous vehicles, on Linux. It includes both Docker and host machine setup options, along with details on building Unreal Engine, AirSim, and the Unreal environment. It also provides guidance on how to use AirSim once it is set up.
- Examples of Thrust code for GPU Parallelization0Some examples for writing Thrust code. To compile, download the CUDA compiler from NVIDIA. This code was tested with CUDA 9.2 but is likely compatible with other versions. Before compiling change extension from thrust_ex.txt to thrust_ex.cu. Any code on the device (GPU) that is run through a Thrust transform is automatically parallelized on the GPU. Host (CPU) code will not be. Thrust code can also be compiled to run on a CPU for practice.
- Gaussian 160Gaussian 16 is a computational chemistry package that is used in predicting molecular properties and understanding molecular behavior at a quantum mechanical level.
Knowledge Base Resources
These resources are contributed by researchers, facilitators, engineers, and HPC admins. Please upvote resources you find useful!