How to Download an Entire Website With wget (--mirror Alone Breaks Links)

说明与用途

使用 wget 及替代工具下载整个网站的指南，涵盖链接重写、WARC 创建以及 JavaScript 网站的工具选择。

问题

Users need to archive a website that may disappear, but wget --mirror alone leaves broken links.

结果

A reproducible workflow that produces an offline‑ready mirror with rewritten links, optional WARC, and guidance on tool selection based on site characteristics.

工作流

• 1. Run wget with flags: --mirror --convert-links --adjust-extension --page-requisites --no-parent --wait=2 --random-wait --limit-rate=200k https://example.com/
• 2. Optionally add --warc-file=example to produce a preservation‑grade WARC.
• 3. If site uses JavaScript, switch to Browsertrix or SingleFile.
• 4. For small sites or GUI preference, use HTTrack with adjusted concurrency and rate limits.
• 5. For single pages, use monolith or SingleFile CLI.
• 6. For self‑hosted, recurring snapshots, deploy ArchiveBox.
• 7. Verify output folder contains HTML, assets, and optionally WARC.
• 8. Test offline browsing to ensure links work.

证据

Plain wget --mirror is a trap. Per the GNU Wget 1.25.0 manual, --mirror is just -r -N -l inf --no-remove-listing: recursion and timestamps. It won't fix links, so your local copy keeps pointing at the live site, and once that site dies your copy is broken too.

来源

The line you want: wget --mirror --convert-links --adjust-extension --page-requisites --no-parent \ --wait=2 --random-wait --limit-rate=200k https://example.com/

来源

If you want a preservation‑grade copy and not just something you can browse, add --warc-file=example.

来源