AnyCloud config for Minio self-hosted S3 storage

Omeka S Modules
1

Hi, I have self-hosted Minio S3-compatible storage deployed on a server, and had been trying to get Omeka-S’s AnyCloud module to work with that, but there were blockers preventing it. Finally cracked through. Sharing all the steps I had to do to make it work, hoping it helps others.

Getting the “why” out of the way :

  • This is an intermediate step; the purchase of Aws or Wasabi service is still some months away in our project.
  • Wasabi was an attractive option, but their free trial doesn’t allow Anonymous Read-only access to buckets - which is a prerequisite for making things work in AnyCloud, and their minim paid tier is 1TB - we’re not there yet.
  • The Omeka-s instance we’re working with is hosted on a server where space is running out and it’s more expensive to add more storage there. And we can’t migrate the whole thing over to another place yet.

The problem

  1. When we configure an https:// endpoint URL and a bucketname in AnyCloud, the baseurl it hits for all file upload / fetch operations becomes : https://[bucketname].[endpoint-url]/[folder]/filename .
  2. folder being “original”, “large”, “medium”, “square”
  3. Minio isn’t expecting things this way. It expects the bucketname to be the first string before a / in [folder].
  4. And, when we’re self-hosting, it’s not a straightforward thing to just automatically create subdomains and have them map to the minio deployment.
  5. I tried some path-forwarding hacks. They worked for fetching files, but failed the S3 signature check when an upload was attempted from Omeka-s side.
  6. If I used the http://IP-address:port of my minio deployment instead of https:// endpoint, then AnyCloud didn’t do the bucketname-prefixing now, and uploading worked, because the http:// urls is hit from the backend.
  7. BUT… fetching of files went for a toss, because that’s done from the frontend, and browsers don’t allow you to pull in images etc from http:// urls as a security measure.

The hack

  1. My minio deployment was mapped to a subdomain like this: https://subdomain.domain.com.
  2. In AnyCloud configuration, for endpoint url I entered the top-level domain only like https://domain.com
  3. In Bucket name, I entered the subdomain.
  4. In the Minio instance, I created buckets by the name of all the folders that Omeka-s uses : original, large, medium, square, and also all additional folders that I found under files/ path in the Omeka-s folder.
  5. I updated each bucket’s access rules to allow public read-only access.
  6. And now it worked! For uploading or fetching files, Omeka-s is hitting urls like https://subdomain.domain.com/original/filename, and Minio being mapped to https://subdomain.domain.com is gladly receiving the request, using original here as the Bucket name, and saving/fetching filename in it.

How to deploy Minio on a https:// url

Pre-requisites

  1. Add a subdomain A-record in your domain’s DNS settings pointing to your server’s IP address. Check with your domain provider for how to or lookup their guides articles.

  2. Setup Nginx on the server, with a virtualhost/conf for your subdomain url. You can lookup online.

  3. Docker needs to be installed. The main docker site has the instructions needed.

Minio deployment

I’m using docker compose to deploy Minio instead of direct-installing; makes things much easier in terms of controlling the location of the storage folder, the port number etc. Sample docker-compose.yaml :

services:
  minio1:
    image: minio/minio:latest
    container_name: minio1
    restart: always
    volumes:
      - /mnt/HC_Volume_101240489/minio-data:/data1
    ports:
      - 9001:9000
      - 9091:9091
    environment:
      - MINIO_ROOT_USER=ACCESS_KEY
      - MINIO_ROOT_PASSWORD=ACCESS_SECRET
    command:
      server /data1 --console-address ":9091"
  • Note the line under volumes : part left of ‘:’ is the local folder path where everything is stored. On my server, this is a mounted 100GB volume.
  • Replace ACCESS_KEY and ACCESS_SECRET with your values, and those are the ones you have to put in in AnyCloud configuration as well.
  • Note the 9091 above - that’s for the minio console (aka control-panel, dashboard) url. Keep it consistent.

Reverse-proxying minio in nginx conf

This link was helpful to setup reverse-proxy in the server’s Nginx setup: cookbook/docs/setup-nginx-proxy-with-minio.md at master · astaxie/cookbook · GitHub . It does the job till http:// .

Enabling SSL

To make it https:// , normal LetsEncrypt process is enough. There is no further work needed in terms of mounting certificates in minio etc. No need to go down that rabbit-hole.

You can look online for “LetsEncrpy certbot nginx”.
The commands:

snap install --classic certbot
ln -s /snap/bin/certbot /usr/bin/certbot
certbot --nginx

Choose the minio subdomain, go with defaults in all the things it asks.

Creating the buckets in Minio

  1. We have to open the Minio’s console url in web browser. in above example it’s on :9091 port. I accessed it on url: http://[IP-address]:9091.
  2. You might need to firewall-allow or ssh-tunnel at your end to be able open it.
  3. It’s not recommended to leave these things open from security point-of-view; for setting-up stage you can do it as temporary and close it off later.
  4. Login with the ACCESS_KEY and ACCESS_SECRET values that you’ve set - they double up as username and password.
  5. Buckets > Create Bucket > enter each Omeka-S folder name like “original”, “large” etc and create the bucket without other settings changed.
  6. Click on the created Bucket > Go to Anonymous side-tab > Add Access Rule > Enter prefix as / and choose readonly for Access.

PS: Some online guide links had to be removed due to forum limitations on posting links.