Omeka Classic / Scripto / Mediawiki throttling?

Omeka Classic Plugins
1

We brought a bunch of people together last week to transcribe together, and, just as we were getting going, the whole stack went down - Omeka returned some Zend-related errors; they seemed to point to an issue connecting with MediaWiki, maybe? I’ll include the error messages.

It seemed likely that it was related to the increased user count - our concurrent users are rarely above 10, and we had about 125 people all online at once - about 25 in the building, and another 100 joining via zoom. MediaWiki has a throttling mechanism (described here:
Manual:Edit throttling - MediaWiki ), for which the default is set to 8 edits per minute.

We probably exceeded that, but that wouldn’t impact offsite users, right?

Are there any other things that might’ve caused this? We’d like to do more of these events, but, obviously, the system going down is a bit of a problem.

Oh: also: nothing else on the server or on the domain went down - everything else remained functional - so it seems unlikely that it was at the server level.

TIA.

This is the error message I got from Omeka:

Zend_Http_Client_Adapter_Exception: Read timed out after 10 seconds in [...]/omeka/application/libraries/Zend/Http/Client/Adapter/Socket.php:511
Stack trace:
#0 [...]/omeka/application/libraries/Zend/Http/Client/Adapter/Socket.php(329): Zend_Http_Client_Adapter_Socket->_checkSocketReadTimeout()
#1 [...]/omeka/application/libraries/Zend/Http/Client.php(1089): Zend_Http_Client_Adapter_Socket->read()
#2 [...]/omeka/plugins/Scripto/libraries/Scripto/Service/MediaWiki.php(670): Zend_Http_Client->request('POST')
#3 [...]/omeka/plugins/Scripto/libraries/Scripto/Service/MediaWiki.php(470): Scripto_Service_MediaWiki->_request('query', Array)
#4 [...]/omeka/plugins/Scripto/libraries/Scripto/Service/MediaWiki.php(161): Scripto_Service_MediaWiki->query(Array)
#5 [...]/omeka/plugins/Scripto/libraries/Scripto.php(218): Scripto_Service_MediaWiki->getUserInfo('groups|rights')
#6 [...]/omeka/plugins/Scripto/libraries/Scripto.php(87): Scripto->setUserInfo()
#7 [...]/omeka/plugins/Scripto/ScriptoPlugin.php(610): Scripto->__construct(Object(ScriptoAdapterOmeka), Array)
#8 [...]/omeka/plugins/Scripto/controllers/IndexController.php(175): ScriptoPlugin::getScripto()
#9 [...]/omeka/application/libraries/Zend/Controller/Action.php(516): Scripto_IndexController->transcribeAction()
#10 [...]/omeka/application/libraries/Zend/Controller/Dispatcher/Standard.php(308): Zend_Controller_Action->dispatch('transcribeActio...')
#11 [...]/omeka/application/libraries/Zend/Controller/Front.php(954): Zend_Controller_Dispatcher_Standard->dispatch(Object(Zend_Controller_Request_Http), Object(Zend_Controller_Response_Http))
#12 [...]/omeka/application/libraries/Zend/Application/Bootstrap/Bootstrap.php(105): Zend_Controller_Front->dispatch()
#13 [...]/omeka/application/libraries/Zend/Application.php(384): Zend_Application_Bootstrap_Bootstrap->run()
#14 [...]/omeka/application/libraries/Omeka/Application.php(73): Zend_Application->run()
#15 [...]/omeka/index.php(23): Omeka_Application->run()
#16 {main}
2

I’ve never encountered an issue related to MediaWiki edit throttling, so I can’t speak to whether that’s the problem. I’d assume that MediaWiki would return an error related to throttling rather than just not responding. It could be that the connection to MediaWiki is just slow. There can be any reasons for this - high user load being one. Does your MediaWiki site seem laggy when you use it directly?

3

Is Omeka equipped to report an error from MediaWiki? I didn’t look for MW errors (well, I looked, but quickly realized I didn’t know where to look). The only error(s) I saw were the Omeka ones.

The MediaWiki install doesn’t seem particularly slow when accessing it directly. Loading a Scripto page has always been a bit slow - Omeka-only pages load pretty quickly, and MediaWiki loads quickly, but a Scripto transcription page takes about 4.5 seconds. (This is the normal, right-now speed; during the “outage” it was much slower - but that was probably just the 10 seconds described in the error message.)

Any suggestions how I could stress test it? This is a bit out of my wheelhouse, so, other than actually doing 8+ edits in a minute, I’m not sure how to reproduce the issue.

4

Is Omeka equipped to report an error from MediaWiki?

Yes. If MediaWiki returns an error, Omeka will report it on the page. The “Read timed out” error means that MediaWiki simply did not respond in the time allowed by the client. You could alter the code of Scripto’s MediaWiki service to tell it to wait longer by configuring a timeout here. So, change this:

->setConfig(array('keepalive' => true))

To this:

->setConfig(array('keepalive' => true, 'timeout' => 30))

You might want to ask your systems administrator or hosting service to investigate why the MediaWiki API is so slow to respond.

5

I’ll give this a shot. Do you know how I could emulate a lot of users? My first thought was to make a little bash script that ran an ‘edit page’ api call a bunch of times, but finding the api urls is harder than I was expecting. Is there an easier way to do that?

6

It’s not as straightforward as one would expect, for sure. Thankfully, the MeidaWiki API docs provide sample code to make page edits.

7

Well, that was successful, but unhelpful… I was able to push 100 edit requests via bot (without the ‘high volume editing’ permissions); most got an editconflict error (they were all editing the same page), but the number that went through was well above the alleged limit of 8 per minute.

Next I tried simple curl requests (simulating the form-action the best I could - right click the network call and “copy as curl” in dev tools) but the api responses were so slow that I don’t think I hit the limit. After 40 or so calls, the site was normal - and then, it wasn’t. It crashed again, and provided the same error message - but I wasn’t able to get much more than that. It’s good that I can reproduce it I guess?

Going to check with the hosting service next…