How to Download an Entire Website With wget (--mirror Alone Breaks Links)

Описание и применение

Руководство по скачиванию целого веб‑сайта с помощью wget и альтернативных инструментов, включая корректировку ссылок, создание WARC и выбор подходящего инструмента для сайтов с JavaScript.

Задача

Users need to archive a website that may disappear, but wget --mirror alone leaves broken links.

Результат

A reproducible workflow that produces an offline‑ready mirror with rewritten links, optional WARC, and guidance on tool selection based on site characteristics.

Рабочий процесс

• 1. Run wget with flags: --mirror --convert-links --adjust-extension --page-requisites --no-parent --wait=2 --random-wait --limit-rate=200k https://example.com/
• 2. Optionally add --warc-file=example to produce a preservation‑grade WARC.
• 3. If site uses JavaScript, switch to Browsertrix or SingleFile.
• 4. For small sites or GUI preference, use HTTrack with adjusted concurrency and rate limits.
• 5. For single pages, use monolith or SingleFile CLI.
• 6. For self‑hosted, recurring snapshots, deploy ArchiveBox.
• 7. Verify output folder contains HTML, assets, and optionally WARC.
• 8. Test offline browsing to ensure links work.

Доказательства

Plain wget --mirror is a trap. Per the GNU Wget 1.25.0 manual, --mirror is just -r -N -l inf --no-remove-listing: recursion and timestamps. It won't fix links, so your local copy keeps pointing at the live site, and once that site dies your copy is broken too.

Источник

The line you want: wget --mirror --convert-links --adjust-extension --page-requisites --no-parent \ --wait=2 --random-wait --limit-rate=200k https://example.com/

Источник

If you want a preservation‑grade copy and not just something you can browse, add --warc-file=example.

Источник