Importing big video files with csvImport(plus)

I am trying to import big video files (mp4 with h264+AAC) with csvImport(plus). I imported the files via ftp to a /files/upload directory and I indicated the correct url to these files in my csv table.

The files are up to 5Go and are use with the HTML5Media plugin after.

With csvImport (not plus), a first file of 600Mo is imported but the import fails for a file of 5Go. Items are not created for the other files after the fail.

With csvImportPlus, items are created even for files of 5Go but I am not able to play them after (Error loading this resource in HTML5Media player).

I think it is not related to php limits (files are already uploaded). The first (correctly imported) file of 600Mo was copied during the import from /files/upload to /files/original and a thumbnail was created; I think these steps fails for bigger files.

Are the imports marking themselves as errored, stuck on in progress, or just marked as completed?

If they’re actually marking as encountering an error, it should be logged so you can see what’s happening. Certainly there could be a variety of issues with such large files.

I have no error with CSVImportPlus when the 5 items are created (but the videos are not readable):

Temps Priorité Message
19/07/2018 09:46:16 DEBUG Import mis en file d’attente.
19/07/2018 09:46:16 DEBUG Import démarré.
19/07/2018 09:46:16 DEBUG Running item import loop. Memory usage: 19226768.
19/07/2018 09:46:17 DEBUG Completed importing 5 items for the last batch part (total 5 records, updated 0 rows, skipped 0 rows) from “import films.csv” (5 rows).

But I have an error “stuck” (in French “arrêté”) with CSVImport. CSVImport says 1 item imported and 0 ignored. In fact, there is one item created (with the video not working) and 4 other lines ignored

Are you able to play the videos with your browser if you navigate directly to their source URL (that is, their location before they are loaded by the importer)?

With such large files, it’s possible that they were uploaded successfully but just can’t be streamed by the browser, particularly if they’re not set up to be web-optimized (with the metadata at the front, basically).

Sorry for the late answer. Yes I can play videos from the upload url in my browser (firefox) with no lag.

You mention to optimized the video with metadata to the front. How can I do this? Could it fix the issue even if I can play the videos in my browser?

If they’re playing fine at the original URL then that metadata stuff shouldn’t be the problem.

If your suspicion is that the thumbnailer is what’s failing, you could try temporarily switching to the GD thumbnailer… since it doesn’t support videos, it won’t try to make thumbnails of them.

Have you tried using the File Sideload module to load the videos? Either one-by-one (which File Sideload can do by itself) or together with CSV Import? An advantage of trying one-by-one is you could isolate the file issues from possible background job issues.

Most users’ installations aren’t set up to log PHP errors in background scripts, so problems that would be easily solved by checking Apache or PHP logs are harder when they happen in the background. In this case, you could be hitting a PHP memory limit or something similar. Doing a single sideload would mean the process happens during the web request, which for most users will log errors somewhere without having to change any settings.

Thanks. I will test with File Sideload in early September.

Here I am again.

File Sideload seems to be a module only for Omeka S. I tried with dropbox plugin for Omeka classic and it works. I have to wait forever for the web request but I don’t have a timeout and I can read correctly my file afterward.

I upload the video files by ftp and then create item the dropbox plugin.

Thanks for the help.

Ah, yes, of course you’re right: File Sideload is only for S, and Dropbox is the equivalent for Classic.

I don’t know where I got the assumption that we were talking about S, since this is in the Classic forum, but glad you figured it out.

This topic was automatically closed after 250 days. New replies are no longer allowed.