Overview
HyperShell is an elegant, cross-platform, high-throughput computing utility for processing shell commands over a distributed, asynchronous queue. It is a highly scalable workflow automation tool for many-task scenarios — built on Python and tested on Linux, macOS, and Windows.
Other tools may offer similar functionality in some places but not within a single tool and not with the flexibility, ergonomics, and scalability provided by HyperShell.
Design
- Client-server architecture — Run the server in stand-alone mode with SQLite or PostgreSQL. Scale clients elastically as needed (even down to zero).
- Cross-platform — Trivial to install, run on any platform where Python runs. Mix platforms within a running cluster (server on Linux, clients on Windows).
- Staggered launch — Come up gradually to balance the workload. Scale to 1000+ nodes, 250k+ workers without crashing the server.
- Database in-the-loop — Persist task metadata across runs. Fault-tolerant by default. Automated retries. Task history.
- User-defined tags — Annotate tasks with key:value tags. Manage catalogs of large collections of tasks with ease.
Usage
HyperShell is primarily a command-line program. Most users operate the hs cluster command
(hsx for short) in a start-to-finish workflow scenario:
seq 1000000 | hsx -t 'echo {}' -N64 --ssh 'a[00-32].cluster' > task.out
Citation
If HyperShell has helped in your research, please consider citing us:
Lentner, Geoffrey and Gorenstein, Lev. "HyperShell v2: Distributed Task Execution for HPC." Practice and Experience in Advanced Research Computing (PEARC '22). Association for Computing Machinery, 2022. DOI: 10.1145/3491418.3535138