CSVImport fails in multi-container Docker environment (filesize(): stat failed for /tmp/...)

Omeka S Modules
1

Hi everyone,

We’re running Omeka S(4.1.1) with CSVImport(2.6.2) in a Docker environment with multiple web replicas behind a load balancer.

Issue:
When importing a CSV, the upload succeeds and a job is created. However, the import fails with:

Warning: filesize(): stat failed for /tmp/omekaukGkhb
The file is not valid.

Observations:

  1. The uploaded file exists on only one web container(/tmp/omekaukGkhb) and is not present on the other containers.
  2. PHP session was being handled by a different web container around the time the import was executed.
  3. The same CSV file imported successfully in single-container test environments.

Suspected Sequence:

  1. User uploads CSV to Web Container A.
  2. PHP stores the uploaded file in /tmp/omekaukGkhb
  3. The import process (or subsequent request) is handled by Web Container B.
  4. Container B attempts to access /tmp/omekaukGkhb
  5. The file does not exist on Container B, resulting in filesize(): stat failed.

Question:
Our IT team suggested constraining CSVImport jobs to run on a single Docker host instead of allowing them to run on any host in the swarm. Would that be a viable solution? If so, how are CSVImport jobs dispatched and executed within Omeka S, and is there a supported way to ensure they are always processed on a specific host/container?

If anyone has deployed CSVImport in Docker Swarm with multiple replicas, I’d be interested to know the recommended configuration.

Thanks in advance for any guidance!

2

Hi Cindy,

I think it could be a good idea to route the import job to a dedicated unbalanced backend by defining a rule/acl in your load balancer.

Keep us informed of your work!

3

Hmm, I’m not sure at what point this is failing for you… maybe between the steps of the import?

By default jobs are just run using a php CLI process on the same server that dispatches the job, so you shouldn’t have an issue with the job having access to the file; they should naturally be on the same instance.

My initial guess would be that if you aren’t pinning clients to use the same instance between requests, imports could be a problem because they work over several requests between the initial upload, the mapping screen, and submitting to start the import. So it might not be an issue with the background job per se but with the request that starts it. Pinning by IP, or using the session cookie or another cookie, would resolve this if that’s the type of problem you’re having.

4

Thanks for the suggestion and the help investigating this.

Following your suggestion, we enabled sticky sessions (session-cookie based affinity) on our load balancer. This ensured that all requests associated with a user’s session were routed to the same web replica throughout the workflow.
After enabling sticky sessions, the CSVImport process completed successfully and the issue was resolved.
Thanks again for your help!

5

Congratulations to Cindy and John for this elegant solution :clap: