HyperShell v2.8: Resource-Aware Scheduling and Task Groups
HyperShell is a cross-platform, high-throughput computing utility for processing shell commands over a distributed, asynchronous queue. It is designed for embarrassingly parallel workloads — the kind where you have thousands or millions of independent tasks that need to be executed across one or more machines. Think parameter sweeps, batch file processing, genome assembly pipelines, or rendering jobs.
The v2.8.0 release is the most feature-rich update since v2.0. It fundamentally changes what HyperShell can do for heterogeneous workloads on HPC clusters. Here is what's new.
Resource-Aware Task Scheduling
Prior to v2.8, HyperShell treated all tasks equally — each executor thread ran one task at a time with no awareness of CPU or memory requirements. This worked beautifully for homogeneous workloads but left performance on the table when tasks had varying resource needs.
Now, tasks can declare their resource requirements:
hsx tasks.in -c 4 -m 2G -N16
The -c/--cores and -m/--memory options set the CPU cores and memory required per task.
On the client side, -C/--client-cores and -M/--client-memory limit the total resources
available. The scheduler tracks allocations and only dispatches tasks when sufficient resources
are free.
For existing workflows where no resource requirements are given, none of these mechanisms engage and task parallelism behaves exactly as it has in previous releases.
Per-Task Resource Heterogeneity
The real power comes from inline resource specification using the #HYPERSHELL: comment syntax.
This allows individual tasks within a single input file to declare different requirements:
stress -c 4 -t 10s #HYPERSHELL: cores:4 memory:2GB timeout:60
stress -c 8 -t 60s #HYPERSHELL: cores:8 memory:4GB timeout:120
Command-line defaults apply to any task that doesn't specify its own requirements, so you can
set a baseline with -c/-m and override only the outliers inline.
Priority-Based Backfilling
When tasks are waiting for resources, HyperShell tracks them with increasing priority — tasks that have been waiting longest are scheduled first. But the scheduler also implements intelligent backfilling: smaller tasks with shorter timeouts can jump ahead if they can complete before higher-priority tasks would be able to start.
Consider a client with 8 cores and 3 executor threads processing a mix of 4-core and 8-core tasks. Two 4-core tasks start immediately, consuming all 8 cores. An 8-core task arrives and must wait. When the first 4-core task finishes, a new 4-core task with a short timeout can backfill into the gap — even though the 8-core task has been waiting longer — because it will finish before enough cores free up for the larger task anyway.
This strategy significantly improves throughput for heterogeneous workloads without starving large tasks.
Resource Monitoring
HyperShell can now monitor the actual CPU and memory usage of running tasks and their child
processes using psutil. Enable it with the --monitor flag:
hsx tasks.in --monitor -c 4 -m 2G
When monitoring is enabled, HyperShell continuously tracks CPU core utilization and memory
consumption for each task. Peak usage values are stored in the database as cores_max and
memory_max. Full time-series telemetry is written to CSV files alongside the captured
stdout and stderr output.
When both monitoring and resource requirements are specified, HyperShell automatically detects when tasks exceed their allocated resources:
Resource limit exceeded (...): cores 1.41 (used) > 1.00 (allocated)
Resource limit exceeded (...): memory 1.62GB (used) > 1.00GB (allocated)
These warnings are informational — they don't terminate the task — but they help you right-size your resource requirements. Use monitoring during development to establish baselines, then apply those requirements in production for optimal scheduling.
Queue-Only Task Submission
The hs submit command now supports direct submission to a live server queue, bypassing the
database entirely. This provides a lightweight path for transient workflows:
# Start a server
hs server --forever --auth mykey &
# Submit tasks directly to the queue
hs submit tasks.in -q -H localhost -p 50001 -k mykey
When using -q/--queue, tasks are sent directly to the server's in-memory queue for immediate
scheduling. Without it, the traditional database-backed workflow is used, providing persistence,
recovery, and search capabilities.
Rate Limiting
A new -R/--ratelimit option limits task throughput per client. For example, -R5 restricts
the client to a maximum of 5 tasks per second. This is implemented by computing a minimum task
walltime and entering a waiting cycle if a task completes faster.
This is useful when tasks make API calls or access rate-limited external services — you can throttle throughput to stay within limits without modifying the tasks themselves.
Task Groups for Dependency Management
Until now, HyperShell focused purely on high-throughput execution of independent, homogeneous task collections. With v2.8, we introduce task groups — a simple but powerful model for expressing execution dependencies.
# Submit batch of tasks with group 0
hs submit tasks-0.in -g 0
# Submit batch of tasks with group 1
hs submit tasks-1.in -g 1
All tasks in group N must complete before any tasks in group N+1 may be scheduled. The
default group is 0, so existing workflows behave identically to previous releases.
This design is deliberately simple compared to traditional DAG-based frameworks like Airflow, Nextflow, or Snakemake. We skip the graph entirely and directly expose task execution groups at submit time. A single integer per task instead of arbitrary dependency edges. For high-throughput workflows with billions of tasks in the database, this is preferable in every regard — graph solving is eliminated, metadata is minimal, database queries remain efficient, and scheduling decisions stay constant-time.
If there are failed tasks in the active group, the scheduler remains in that group until all
retries have been exhausted. In --forever mode the server will wait indefinitely; otherwise
it triggers a shutdown with a critical message.
SQLite Enabled by Default
Previously, HyperShell would automatically disable the database in favor of a live queue if no
database was configured. New users running the software for the first time would get a warning
message about missing database configuration. With v2.8, a local SQLite database (main.db)
is automatically created within the site library (e.g., ~/.hypershell/lib/main.db on Linux).
This means new users get persistence, fault tolerance, and task history out of the box without needing to configure anything. Any explicit database configuration provided by the user overrides this default.
Application-Layer Encryption
HyperShell is a distributed system that transmits arbitrary shell commands over TCP sockets. That's inherently a high-risk operation — an attacker with access to the task queue can execute arbitrary code on every connected client. Prior to v2.8, security relied entirely on the network environment: run HyperShell behind a firewall, inside an HPC cluster, or through a VPN. That's still the primary recommendation, but this release adds built-in encryption as a defense-in-depth measure.
Connection Authentication
The shared authentication key (--auth) is never transmitted directly. Both sides independently
compute SHA-512(auth_key) and use the digest for the initial BaseManager handshake. After
that, clients must pass a secondary challenge: generate a random 16-byte salt, derive a
Fernet encryption key via PBKDF2-HMAC-SHA512 with 200,000 iterations, encrypt a unique UUID
with it, and send both the salt and encrypted token to the server.
The server re-derives the same key from the salt and attempts decryption. If it succeeds, the client has proven knowledge of the shared secret without ever transmitting it. The UUID is recorded to prevent replay — each token is single-use — and sessions expire after 60 seconds. This protocol gives us mutual authentication, per-session forward secrecy (via unique salts), and computational hardening against brute-force.
Payload Encryption
All task data moving through the distributed queues — commands, arguments, status, results — is
encrypted with Fernet (AES-128-CBC + HMAC-SHA256). This provides authenticated encryption:
ciphertext can't be read without the key, and any tampering is detected on decryption. The
encryption key is derived from the auth key using PBKDF2, so the same --auth flag that
previously only gated access now also protects data in transit.
Task serialization has been restructured as part of this work. Previously, individual tasks were serialized to bytes and bundled as a list. Now, entire bundles are serialized and encrypted as a single unit, reducing per-task overhead and simplifying the wire protocol.
Performance Considerations
Encryption adds latency to every queue operation, so care was taken to minimize overhead.
The cipher module uses closures to capture the Fernet instance after set_secure_key() is
called — subsequent encrypt()/decrypt() calls resolve via LOAD_DEREF/LOAD_FAST bytecode
operations rather than repeated global lookups. For high-throughput workloads processing millions
of tasks, this matters.
The server scheduler also gained exponential backoff on idle database polling: instead of a fixed sleep interval, it starts at 0.5 seconds and doubles up to a configurable maximum (default 30 seconds), resetting immediately when tasks become available. This dramatically reduces database load during idle periods without sacrificing responsiveness.
Limitations
This is application-layer encryption, not transport-layer. Network observers can still see connection patterns, timing information, and approximate message sizes. The shared key must be distributed securely out-of-band. There is no forward secrecy on payload data — if the key is later compromised, recorded traffic can be decrypted. And this is a custom protocol built on well-established primitives (PBKDF2, SHA-512, Fernet/AES), but it has not undergone formal security audit.
Built-in encryption is a safety net, not a replacement for proper network security. Always
prefer operating within isolated networks, behind firewalls, or through a VPN like WireGuard or
Tailscale. Use --auth with a strong randomly-generated key (openssl rand -base64 32), and
treat the encryption as one layer in a defense-in-depth strategy.
Getting Started
Install HyperShell with uv, pipx, or Homebrew:
uv tool install hypershell
The simplest workflow pipes commands through hsx:
seq 1000 | hsx -t 'process_data --input data_{}.fits' -N16
Scale out to a cluster with SSH or a job launcher:
hsx tasks.in -N128 -b128 --launcher=srun --max-retries=2
For the full documentation, visit hypershell.readthedocs.io. If HyperShell has helped in your research, please consider citing the PEARC '22 paper.