Budgets and Pacing

An archive is built in seasons, not sprints

A queue of a dozen multi-gigabyte acquisitions cannot land in an evening, and should not try. The naive plan (max the link, pull everything, sleep) fails on contact with reality: the household shares the network, long saturated transfers raise their own interruption rate, and a burst that monopolizes resources gets noticed — and unplugged. The archival plan treats acquisition as a paced, budgeted, multi-day campaign.

Two knobs define the campaign:

A rate cap — a modest fraction of the link's ceiling (a few MB/s is a common archival register). Under the ceiling, interruptions are rarer, resume stays cheap, and everything else sharing the network stays usable. The transfer becomes furniture: present, quiet, ignorable.
A daily budget — a total-bytes allowance per day (150 GB is a plausible home scale), with a reset boundary. The budget keeps any single acquisitive day from crowding out everything else the storage and network do, and it makes campaign arithmetic honest: at the budget, a 1 TB acquisition is about seven campaign days, and you can plan accordingly instead of discovering it.

Together they change the failure profile: instead of one heroic transfer that must not be interrupted, you get a sequence of quiet sessions that tolerate interruption by design. Patience is not the absence of ambition — it is pacing ambition so it survives the week.

The planning arithmetic

With the knobs set, campaign planning is honest arithmetic you can do in your head:

days = ceil(total_bytes / daily_budget)

and per file: hours ≈ file_bytes / (rate_cap × 3600). A 17 GB shard at 2 MB/s is about 2.4 hours; a 34 GB one at 1.5 MB/s is about 6.3. Those numbers are the difference between a queue you schedule and a queue that ambushes you. They also feed acquisition priority: when the budget forces serialization, order the queue by the woodpile's reasons — the vessel-dependency first, the merely-respected later.

Pace the campaign so interruptions cost minutes, not evenings. The rate cap and daily budget are not conservatism — they are the difference between resumable seasons and heroic failures.

Politeness has a mechanical side

Running under the ceiling and honoring retry backoffs also makes you a good citizen of the sources you pull from: rate limiters on the hub side respond to bursty behavior, and a paced client with resume rarely meets them. The archive's patience and the source's patience turn out to be the same engineering.

Code

Campaign arithmetic and a paced pull·bash

# The honest planner (do this BEFORE queueing):
python3 - <<'EOF'
rate = 1.5e6        # bytes/s rate cap (your setting)
budget = 150e9      # bytes/day daily budget (your setting)
for name, size in [("model-a bf16", 34e9), ("model-b bf16", 17e9),
                   ("model-c gguf", 5e9)]:
    hours = size / (rate * 3600)
    print(f"{name:18} {size/1e9:5.0f} GB  ~{hours:4.1f} h at cap")
total = 34e9 + 17e9 + 5e9
print(f"queue total {total/1e9:.0f} GB -> {total/budget:.1f} days at budget")
EOF

# A paced, resumable pull (curl's own limiter):
#   --limit-rate 1500K   caps throughput (~1.5 MB/s)
#   -C -                 resumes from where the .part left off
curl -L -C - --fail --limit-rate 1500K -o file.part "$URL"

# Run pulls from a queue script overnight; the budget line stops
# the queue when the day's allowance is spent, and tomorrow's run
# picks up exactly where the byte counters left off.

Exercise

Set your two knobs (rate cap, daily budget) and write them down with one sentence of justification each. Run the planner on a real wish-list of three models. Then execute one paced pull of a real file with the limiter engaged and note the wall-clock versus the prediction. If you keep a queue, reorder it by reason (vessel first) and note the new order.

Hint

Pick the rate as a fraction of a speed test, not a fraction of ambition — if video calls stutter while a pull runs, the cap is too high for your household's reality, whatever the arithmetic says.

An archive is built in seasons, not sprints

The planning arithmetic

Politeness has a mechanical side

Code

External links

Exercise

Progress

Comments 0