Latest result (Jul 31, 2018): No response after 60 seconds. Jammed queue suspected

Omeka Classic Plugins
1

Hello

We are trying to use the CSV Import tool to pull data into Omeka. We had CSV Import running about a year ago and when we went back to it this summer it does not work.

The jobs all showed that they were queued up but never start.

I have removed all of the other PlugIns
I have looked in the logs and they are empty.
I installed the Job Diagnostic PlugIn and I get this message out when I test the Long-Running Jobs as follows:

  Latest result (Jul 31, 2018): No response after 60 seconds. Jammed queue suspected.

I can also see a list of processes in Job Diagnostics and all of the failed ones have the status of starting. There are 5 pages of them at this point.

How do I clear up the Jammed queue?

Can I flush all of these? The do not have a PID only an ID.

Here is an example of one process

Status starting
Run By Bryan
Started at 2018-07-27 12:08:23
Finished at

Data

a:1:{s:3:“job”;s:247:"{“className”:“Job_ItemBatchEditAll”,“options”:{“params”:{“collection”:“14”},“delete”:null,“metadata”:{“public”:"",“featured”:"",“item_type_id”:“18”,“collection_id”:"",“tags”:""},“custom”:null},“createdAt”:“2018-07-27T12:08:23-05:00”,“createdBy”:2}";}

I cleaned up the tables via phpMyAdmin of all of the jobs that were starting.

What grabs these jobs from CSV Import and runs them? Is there a process that I should be looking for?

I do see what is below in the config.ini file

jobs.dispatcher.longRunning = “Omeka_Job_Dispatcher_Adapter_Synchronous”

Is this what should grab the files and process them?

Bryan

2

“Synchronous” means you’re not actually running background jobs in the background. Typically you end up putting this setting in if your server just doesn’t support background jobs (like if PHP’s CLI is not installed, or the server admin has blocked process execution from PHP). Synchronous jobs will try to run just in the context of a web request, which generally means they’re fairly strictly time-limited in how long they can run for. There’s not a semicolon at the front of that line, is there? (This would mean it’s a comment and not actually being used as configuration.)

Being stuck at “Starting” generally means there’s some error happening right away before the job can start running at all. Generally this will result in something in your Omeka error log or your server’s PHP error log (this can be in different places depending on the server).

There really isn’t a “queue” in any real sense in a normal Omeka Classic installation: every “job” tries to run immediately so they can’t really be “blocked” by older jobs. You can make a configuration that uses a queue but it’s not the default (and not what’s indicated by the config you pasted).

I’m not familiar personally with the diagnostic plugin you mention so I can’t say much there.

3

We seem to have two lines in the config.ini file that seem to be related to this issue.

Here is what I see in the config.ini file

; jobs.dispatcher
; How Omeka “jobs” will be executed.
; default: “Omeka_Job_Dispatcher_Adapter_Synchronous”
;
; Newer Omeka features and plugins use this setting to determine how
; long-running jobs will be run.
;
; The default setting should work for all installations, but may
; time out for longer jobs. On systems where the older PHP background
; processes worked, the BackgroundProcess adapter can be used instead
; of the Synchronous one.
jobs.dispatcher.default = "Omeka_Job_Dispatcher_Adapter_Synchronous"
jobs.dispatcher.longRunning = “Omeka_Job_Dispatcher_Adapter_BackgroundProcess”

The last two lines donto lead with a ; so they are not commented out so they both are being processed.

Do we need either of these? Both or only one?

jobs.dispatcher.default = "Omeka_Job_Dispatcher_Adapter_Synchronous"
jobs.dispatcher.longRunning = “Omeka_Job_Dispatcher_Adapter_BackgroundProcess”

4

I checked an unedited copy of the config.ini and I see that it has both lines that we have.

jobs.dispatcher.default = "Omeka_Job_Dispatcher_Adapter_Synchronous"
jobs.dispatcher.longRunning = “Omeka_Job_Dispatcher_Adapter_BackgroundProcess”

So I am guessing that this is not where the issue lies.

B.

5

I have checked my file a few times and all seems to be in order with the specifications provided by CSV Import.

I even entered data into Omeka then used a CSV Export to pull it out again and then tried to reload that file via CSV Import.

The job immediately goes to a Queued status in CSV Import.

The omeka_process table in phpMyAdmin shows the status as Starting.

328 Omeka_Job_Process_Wrapper 2 NULL starting a:1:{s:3:“job”;s:168:"{“className”:"CsvImport_Impo… 2018-08-01 13:15:05 NULL

I am going to check the error logs for Omeka and PHP to see if they have recorded anything about these attempts. We have a LAMP set up. Where might I be looking for the PHP error logs?

B.

6

If you have control over your php.ini settings, you can set where PHP logs to with the error_log setting. Errors that occur during a web request can often be found in your Apache error log by default. By default the CLI doesn’t log to a file so you’d have to set that up in the php.ini for the CLI. Depending on what kind of error is happening it could be in either place.

You can also try to run the job manually from the command line and see if there’s output indicating the problem: if you look up the process ID in the processes table, you can run it with on the command line. From the Omeka top-level directory it’s:

php application/scripts/background.php -p <process id>

(without the angle brackets, of course).

7

Ok, I tried to run from the prompt as suggested and the output is below.

Is this at all helpful?

B.

omeka@C-11127:/var/www/html$ php application/scripts/background.php -p 328

PHP Fatal error: Uncaught Zend_Session_Exception: Zend_Session::start() - /var/www/html/application/libraries/Zend/Db/Table/Abstract.php(Line:1307): Error #2 count(): Parameter must be an array or an object that implements Countable
/var/www/html/application/libraries/Zend/Db/Table/Abstract.php(Line:1307): Error #2 count(): Parameter must be an array or an object that implements Countable in /var/www/html/application/libraries/Zend/Session.php:495

Stack trace:
#0 /var/www/html/application/libraries/Zend/Session/Namespace.php(143): Zend_Session::start(true)
#1 /var/www/html/application/libraries/Zend/Auth/Storage/Session.php(86): Zend_Session_Namespace->__construct(‘Zend_Auth’)
#2 /var/www/html/application/libraries/Zend/Auth.php(91): Zend_Auth_Storage_Session->__construct()
#3 /var/www/html/application/libraries/Zend/Auth.php(141): Zend_Auth->getStorage()
#4 /var/www/html/application/libraries/Omeka/Application/Resource/Currentuser.php(42): Zend_Auth->hasIdentity()
#5 /var/www/html/application/libraries/Zend/Application/Boots in /var/www/html/application/libraries/Zend/Session.php on line 495

8

Yeah, this looks like it could be helpful.

What version of Omeka are you using? Can you post the info from the “System information” link in your Omeka installation?

This seems like a problem with a mismatch between a PHP version and an Omeka version… but if you’re not seeing it on the web it may just be a case where you merely need to tell Omeka to use the right version of PHP on the command line (you may have several versions installed)?

9

System
Omeka 2.6.1
PHP 7.0.22-0ubuntu0.17.04.1 (apache2handler)
OS Linux 4.15.0-29-generic x86_64
MySQL Server 5.7.23
MySQL Client mysqlnd 5.0.12-dev - 20150407 - $Id: b5c5906d452ec590732a93b051f3827e02749b83 $
Apache Apache/2.4.29 (Ubuntu)

PHP Extensions
Regular apache2handler, bz2, calendar, Core, ctype, date, dom, exif, fileinfo, filter, ftp, gd, gettext, hash, iconv, json, libxml, mbstring, mcrypt, mysqli, mysqlnd, openssl, pcre, PDO, pdo_mysql, Phar, posix, readline, Reflection, session, shmop, SimpleXML, sockets, SPL, standard, sysvmsg, sysvsem, sysvshm, tokenizer, wddx, xml, xmlreader, xmlwriter, xsl, Zend OPcache, zip, zlib
Zend Zend OPcache

10

Ah, and I forgot to ask for one more thing: the output of php -v

11

I think, there’s problem with update of Zend library. If you check the line 1307 in Zend_Db_Table_Abstract there’s no such a thing as count().

Looks like this commit is not applied. Did you touch Zend core @Bryan?

12

PHP 7.2.7-0ubuntu0.18.04.2 (cli) (built: Jul 4 2018 16:55:24) ( NTS )
Copyright © 1997-2018 The PHP Group
Zend Engine v3.2.0, Copyright © 1998-2018 Zend Technologies
with Zend OPcache v7.2.7-0ubuntu0.18.04.2, Copyright © 1999-2018, by Zend Technologies

13

I don’t believe that we made any changes except updating the OS.

B.

14

Ok, could you please browse to application/libraries/Zend/Db/Table/Abstract.php and copypaste line 1307 here or comapre with Omeka master ? Just to make sure it’s not like I guessed.

15 
    $whereList = array();
    $numberTerms = 0;
    foreach ($args as $keyPosition => $keyValues) {

1307 $keyValuesCount = count($keyValues);
// Coerce the values to an array.
// Don’t simply typecast to array, because the values
// might be Zend_Db_Expr objects.
if (!is_array($keyValues)) {
$keyValues = array($keyValues);
}
if ($numberTerms == 0) {
$numberTerms = $keyValuesCount;
} else if ($keyValuesCount != $numberTerms) {
require_once ‘Zend/Db/Table/Exception.php’;
throw new Zend_Db_Table_Exception(“Missing value(s) for the primary key”);
}
$keyValues = array_values($keyValues);
for ($i = 0; $i < $keyValuesCount; ++$i) {
if (!isset($whereList[$i])) {
$whereList[$i] = array();
}
$whereList[$i][$keyPosition] = $keyValues[$i];
}
}

16

I added the 1307 to what was in the file you wanted to see line 1307.

I checked and we have updated more that the OS. The person who is the main contact for this box was out of the office, but she is back in the office later today.

B.

17

Yeah I’m in agreement with @luk.puk: the file doesn’t look like what it should if you’re using Omeka 2.6.1… a change was made in that version for compatibility with PHP 7.2, and you don’t have it in that file. So, something went amiss with your Omeka upgrade somewhere and you didn’t update all the files. You can just do them piecemeal but I’m a little concerned as to what other files might not be updated. You could consider “upgrading” to 2.6.1 again, with a fresh copy of the files.

Also you have a second issue: on the web you’re using PHP 7.0, which is why you’re not seeing problems there. On the command line you’re using PHP 7.2. I assume you have both installed, so there’s probably a path you can get to the PHP 7.0 CLI binary on this server. If you have that, you can put it in Omeka’s application/config/config.ini file as the setting for background.php.path. You really want to be using the same PHP version on the command line as you do on the web.

18

I will have her reapply the upgrade when she is back this afternoon.

Is it just copying files or are there config changes too like the path to the php files?

B.

19

OK found it.

https://omeka.org/classic/docs/Installation/Upgrading/

I will let you know how it goes.

Thank you both for helping us out.

Bryan

20

She reinstalled the whole upgrade by copying and unzipping it.

I tried to run the background job manually.

omeka@C-11127:~$ php application/scripts/background.php -p 328

and I get back.

Could not open input file: application/scripts/background.php

So even with setting it up from the newest files for Omeka something is still not working as expected.

Any other ideas?

Bryan