S3-compatible Object Storage

timmmmyboy · June 3, 2020, 4:59pm

I’ve used the S3 storage adapter in Omeka Classic with AWS S3 storage many times and I’m wondering if there are any modifications or an extension that could be written to make it compatible with other storage services that are s3 compatible. Seems like most of the work would already be there but I noticed when trying to choose a custom endpoint and entering the necessary information for another service the URL it is generating can’t connect because it isn’t formatted.

Specifically to this scenario I have a Minio object store installed with URL, access key, and secret. I entered all that in the config but it attempts to connect to a URL formatted as ssl://bucketname/endpoint which doesn’t exist. Looked around but couldn’t figure out how the URLs are formatted. I know Omeka S has the extension for S3-compatible stores beyond Amazon, has anyone experimented with this in Classic to see if they could get it to work?

Log:

2020-06-03T16:51:53+00:00 ERR (3): Zend_Http_Client_Adapter_Exception: Unable to Connect to ssl://omeka.objectstore.uk.reclaim.cloud:443. Error #0:  in /home/towens/public_html/timowens.io/omeka/application/libraries/Zend/Http/Client/Adapter/Socket.php:235
Stack trace:
#0 /home/towens/public_html/timowens.io/omeka/application/libraries/Zend/Http/Client.php(1073): Zend_Http_Client_Adapter_Socket->connect('ssl://omeka.obj...', 443, true)
#1 /home/towens/public_html/timowens.io/omeka/application/libraries/Omeka/Http/Client.php(36): Zend_Http_Client->request('DELETE')
#2 /home/towens/public_html/timowens.io/omeka/application/libraries/Zend/Service/Amazon/S3.php(693): Omeka_Http_Client->request('DELETE')
#3 /home/towens/public_html/timowens.io/omeka/application/libraries/Zend/Service/Amazon/S3.php(553): Zend_Service_Amazon_S3->_makeRequest('DELETE', 'omeka/original/...')
#4 /home/towens/public_html/timowens.io/omeka/application/libraries/Omeka/Storage/Adapter/ZendS3.php(146): Zend_Service_Amazon_S3->removeObject('omeka/original/...')
#5 /home/towens/public_html/timowens.io/omeka/application/libraries/Omeka/Storage.php(67): Omeka_Storage_Adapter_ZendS3->delete('original/cbbf06...')
#6 /home/towens/public_html/timowens.io/omeka/application/models/File.php(329): Omeka_Storage->__call('delete', Array)
#7 /home/towens/public_html/timowens.io/omeka/application/models/File.php(338): File->unlinkFile()
#8 /home/towens/public_html/timowens.io/omeka/application/libraries/Omeka/Record/AbstractRecord.php(583): File->_delete()
#9 /home/towens/public_html/timowens.io/omeka/application/models/Job/FileProcessUpload.php(25): Omeka_Record_AbstractRecord->delete()
#10 /home/towens/public_html/timowens.io/omeka/application/libraries/Omeka/Job/Dispatcher/Adapter/Synchronous.php(25): Job_FileProcessUpload->perform()
#11 /home/towens/public_html/timowens.io/omeka/application/libraries/Omeka/Job/Dispatcher/Default.php(137): Omeka_Job_Dispatcher_Adapter_Synchronous->send('{"className":"J...', Array)
#12 /home/towens/public_html/timowens.io/omeka/application/models/File.php(203): Omeka_Job_Dispatcher_Default->send('Job_FileProcess...', Array)
#13 /home/towens/public_html/timowens.io/omeka/application/libraries/Omeka/Record/AbstractRecord.php(280): File->afterSave(Array)
#14 /home/towens/public_html/timowens.io/omeka/application/libraries/Omeka/Record/AbstractRecord.php(548): Omeka_Record_AbstractRecord->runCallbacks('afterSave', Array)
#15 /home/towens/public_html/timowens.io/omeka/application/models/Item.php(336): Omeka_Record_AbstractRecord->save()
#16 /home/towens/public_html/timowens.io/omeka/application/models/Item.php(249): Item->saveFiles()
#17 /home/towens/public_html/timowens.io/omeka/application/libraries/Omeka/Record/AbstractRecord.php(280): Item->afterSave(Array)
#18 /home/towens/public_html/timowens.io/omeka/application/libraries/Omeka/Record/AbstractRecord.php(548): Omeka_Record_AbstractRecord->runCallbacks('afterSave', Array)
#19 /home/towens/public_html/timowens.io/omeka/application/libraries/Omeka/Controller/AbstractActionController.php(188): Omeka_Record_AbstractRecord->save(false)
#20 /home/towens/public_html/timowens.io/omeka/application/controllers/ItemsController.php(148): Omeka_Controller_AbstractActionController->addAction()
#21 /home/towens/public_html/timowens.io/omeka/application/libraries/Zend/Controller/Action.php(516): ItemsController->addAction()
#22 /home/towens/public_html/timowens.io/omeka/application/libraries/Zend/Controller/Dispatcher/Standard.php(308): Zend_Controller_Action->dispatch('addAction')
#23 /home/towens/public_html/timowens.io/omeka/application/libraries/Zend/Controller/Front.php(954): Zend_Controller_Dispatcher_Standard->dispatch(Object(Zend_Controller_Request_Http), Object(Zend_Controller_Response_Http))
#24 /home/towens/public_html/timowens.io/omeka/application/libraries/Zend/Application/Bootstrap/Bootstrap.php(105): Zend_Controller_Front->dispatch()
#25 /home/towens/public_html/timowens.io/omeka/application/libraries/Zend/Application.php(384): Zend_Application_Bootstrap_Bootstrap->run()
#26 /home/towens/public_html/timowens.io/omeka/application/libraries/Omeka/Application.php(73): Zend_Application->run()
#27 /home/towens/public_html/timowens.io/omeka/admin/index.php(28): Omeka_Application->run()
#28 {main}

jflatnes · June 3, 2020, 5:40pm

Yeah, unfortunately the Zend S3 code that underlies this is hardcoded to put the bucket name into the endpoint URL.

Someone had created a plugin to use Amazon’s SDK for connecting to s3, as this Zend code only supports the older “version 2” authentication system… though I don’t know, thinking about it now, whether the AWS SDK works at all against outside services. I do believe they let you specify a custom endpoint so it would probably work out. Looking quickly at this it doesn’t look like it exposes an endpoint setting through the plugin so that probably won’t work as-is either.

The Storage layer is quite simple, so a plugin for a specific service or for “S3 compatible” things generally wouldn’t be a tough thing to create.

jflatnes · June 3, 2020, 5:48pm

It looks like the official AWS SDK works just fine for this, so that plugin I linked would be fine with some minor tweaking, I would think (either to expose the necessary configuration options or just hardcode them as needed for your use-case).

timmmmyboy · June 3, 2020, 6:00pm

Nice! I’ll give it a shot. Thanks!

timmmmyboy · June 3, 2020, 6:23pm

That was way easier than it deserves to be :).

I still have to get thumbnails and embeds working properly but testing uploads and they went through without a hitch. Big thanks for the assist!

timmmmyboy · June 3, 2020, 7:11pm

If anyone else stumbles on this thread I’ll link to the repo for the custom adapter that allows specification of endpoint.

I’ve tested with Minio and it works a treat. Your mileage however may vary yada yada

system · August 7, 2021, 4:34pm

This topic was automatically closed after 250 days. New replies are no longer allowed.