I want high-performance, error-tolerant downloads for very large files
Submitted by Annette Greiner
Permalink https://webwewant.fyi/wants/e42a27fa-83f8-4149-905f-8a8fa53aa0de/
This idea is currently being discussed.
I want a browser API for high-performance, error-tolerant downloading of very large files — the kind of transfers that take hours or days and must survive network interruptions, without requiring users to install additional software or create accounts with third-party services.
I build web applications and portals for high-performance scientific computing. One of the most common requests from our users is reliable sharing of large scientific data files — datasets that can run into hundreds of gigabytes or even terabytes. The naive approach of serving these files over HTTP works for small transfers, but for truly large files it routinely results in transfers that take too long, fail mid-way without any recovery mechanism, and have to be restarted from scratch.
The current gold standard for this use case is Globus, which provides high-performance, parallel, resumable data transfers. However, Globus requires both the file provider and the recipient to create accounts, and typically requires installing endpoint software or having access to an institutional deployment. This friction is a real barrier to open science and cross-domain collaboration.
What I need from the browser is a download API that supports:
- Resumable transfers — if the connection drops, the download should automatically resume from where it stopped rather than restarting from byte zero.
- Parallel chunked downloads — splitting large files into chunks and downloading them concurrently to saturate available bandwidth.
- Progress and speed reporting — exposing fine-grained transfer metrics so applications can display meaningful progress bars and ETAs.
- Background execution — continuing the download even if the user navigates away or minimises the browser, similar to the Background Fetch API.
- Integrity verification — supporting checksums (e.g. SHA-256) so the application can verify the complete file without additional round trips.
The Streams API and Background Fetch API are promising foundations, but they do not yet provide the combination of resumability, parallel chunking, and performance tuning needed for multi-gigabyte scientific data files. A dedicated high-performance download primitive would open web applications to use cases that currently require native desktop clients or specialised middleware.
- Votes
- 0
What are votes for and how are they tallied?