How to Download an Entire Website With wget (--mirror Alone Breaks Links)

Description and use

Guide to downloading an entire website with wget and alternatives, covering link rewriting, WARC creation, and tool selection for JavaScript sites.

Problem

Users need to archive a website that may disappear, but wget --mirror alone leaves broken links.

Outcome

A reproducible workflow that produces an offline‑ready mirror with rewritten links, optional WARC, and guidance on tool selection based on site characteristics.

Workflow

• 1. Run wget with flags: --mirror --convert-links --adjust-extension --page-requisites --no-parent --wait=2 --random-wait --limit-rate=200k https://example.com/
• 2. Optionally add --warc-file=example to produce a preservation‑grade WARC.
• 3. If site uses JavaScript, switch to Browsertrix or SingleFile.
• 4. For small sites or GUI preference, use HTTrack with adjusted concurrency and rate limits.
• 5. For single pages, use monolith or SingleFile CLI.
• 6. For self‑hosted, recurring snapshots, deploy ArchiveBox.
• 7. Verify output folder contains HTML, assets, and optionally WARC.
• 8. Test offline browsing to ensure links work.

Evidence

Plain wget --mirror is a trap. Per the GNU Wget 1.25.0 manual, --mirror is just -r -N -l inf --no-remove-listing: recursion and timestamps. It won't fix links, so your local copy keeps pointing at the live site, and once that site dies your copy is broken too.

Source

The line you want: wget --mirror --convert-links --adjust-extension --page-requisites --no-parent \ --wait=2 --random-wait --limit-rate=200k https://example.com/

Source

If you want a preservation‑grade copy and not just something you can browse, add --warc-file=example.

Source