Migrating to S3 filestorage?

[t_stallmann]

Hi!

I’m researching the feasibility of migrating a site from using filesystem storage to Amazon S3, and wanted to ask in this forum to see what thoughts & knowledge other folks could add.

I know that Archive Repertory plugin won’t work with S3, but curious if anyone else has done this and has any tips or suggestions or notes about the pros/cons?

Also – in particular, is there a working process to transition existing item files to S3 as part of the migration?

Thanks!
Tim

[jflatnes]

I can’t speak to Archive Repertory as an outside plugin.

On S3 generally, you can definitely transition from using local storage to S3 (and back again). Moving to S3 just requires you to put the files from your local “files” folder in the bucket you’ll be using. There are various methods of doing this… lately I’ve used the aws s3 sync command from the AWS official CLI.

[t_stallmann]

Oh awesome, I had no idea it would be that easy!

[jflatnes]

Obviously you also then have to enable the configuration for S3 in Omeka, but yeah, moving is easy. Omeka’s “Storage” system basically just asks for a path: when you’re using the default local storage, those paths are relative to the files folder, while for S3 they’re relative to the configured bucket. So it’s just a matter of moving the files so they’re where Omeka expects them to be. And in the reverse direction, switching back to local storage just entails doing the sync in the other direction to pull all the files stored in S3 down into the local files folder.

The main “con” to be aware of I think would be that, since your files are not local, some other plugins are more restricted in how they work. Things like re-generating thumbnails or extracting OCRd text from already-uploaded files are trivial with local storage, but not when the files are remote. This usually isn’t a big problem, as the plugins that act on uploaded files usually can do their work at the initial stage when the upload first happens, before the files get uploaded to S3, but it’s something to be aware of as a difference: there’s some lesser flexibility in changing things related to the files after they’re uploaded.

[t_stallmann]

Wanted to leave a couple of notes here on my experience in case other folks find this thread in the future:

• The built-in S3 adapter uses an older AWS protocol for request signing, as a result move and delete operations don’t work – luckily there’s an excellent third-party adapter which supports all the things – GitHub - EHRI/omeka-amazon-s3-storage-adapter: Omeka storage adapter for Amazon S3 based on the official AWS SDK. (I have a PR pending to make it work properly with Omeka’s current plugin structure)
• The switch really was easy other than that!
• Human-readable filenames are a priority for me, so I really wanted to make Archive Repertory work. It turned out to be really easy to update the Archive Repertory plugin to be agnostic about base file adapter (S3 vs. local), so that it works with S3, so long as you use the third-party adapter referenced above – I have a PR pending for those changes, which we’re using on production and seems to be working swell so far – Add support for AWS S3 filestorage via custom adapter by timstallmann · Pull Request #12 · Daniel-KM/Omeka-plugin-ArchiveRepertory · GitHub