Just posting this here, hoping it might be useful to others.
We are decommissioning our Omeka Classic, so need to somehow preserve the 14 Exhibits there now.
First, it’s good to know that these Exhibits are fully captured by the Internet Archive Wayback machine.
But I also wanted a local copy.
So I used wget to crawl the site and download it locally. And yet, there is a bug in wget and wget2 having to do with filenames like this: somefile.css?v1.2.3. That is, wget/wget2 do not understand the difference between a filename and a query string on that filename. This resulted in some downloaded exhibits being broken, not fully functioning.
So around and around with Claude AI, here is the final wget command I used, with two attached Python files that do post-download cleanup:
wget --mirror --no-parent --page-requisites --convert-links --adjust-extension \
--restrict-file-names=windows -e robots=off --span-hosts \
--domains=exhibits.library.jhu.edu,fonts.googleapis.com,fonts.gstatic.com \
--include-directories=/exhibits,/themes,/files,/application,/plugins \
-nH -P exhibits-library-jhu-edu --wait=2 --random-wait \
https://exhibits.library.jhu.edu/exhibits/
Change the extension on the attached Python files from .txt to .py to run. Run them from within the root directory that wget creates.
fix_asset_links.txt (1.5 KB)
fix_page_links.txt (2.6 KB)
Claude AI wrote these two Python files.
Anyway, I spent quite a bit of time on this, so thought I’d contribute. I hope someone else finds this useful!
Mark Cyzyk
Library Applications Group
The Sheridan Libraries
Johns Hopkins University
Baltimore, Maryland
USA