Because it’s happening so regularly, it’s almost certainly not a case where the random ID is conflicting between two truly separate media objects. The “same” insert happening twice is more likely.
Can you try something else: temporarily disable all those modules and see if you still get the problem? That would rule in or out the problem coming from an interaction between different modules.
Sorry for the late response. The day you posted this I left that project (literally left the country) and will not be back to it until May 2024. I will do as you ask then as I will be again uploading many records. Thanks for the interest.
I am now ready to take back this issue.
The issue happens regularly, every time I try to load some media.
I’m using Omeka S 4.0.4, CSV Import version 2.5.0 and File Sideload version 1.7.1.
I try to load my file with all modules disabled except CSV Import and File Sideload, but I still get the same error:
2023-12-21T21:19:27+00:00 ERR (3): PDOException: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'e3c6680b663a92f7f4477ce8990b5fd9040ff4ce' for key 'UNIQ_6A2CA10C5CC5DB90' in /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php:117
Stack trace:
#0 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php(117): PDOStatement->execute(NULL)
#1 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/vendor/doctrine/dbal/lib/Doctrine/DBAL/Statement.php(179): Doctrine\DBAL\Driver\PDOStatement->execute(NULL)
#2 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/vendor/doctrine/dbal/lib/Doctrine/DBAL/Statement.php(226): Doctrine\DBAL\Statement->execute(NULL)
#3 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/vendor/doctrine/orm/lib/Doctrine/ORM/Persisters/Entity/JoinedSubclassPersister.php(194): Doctrine\DBAL\Statement->executeStatement()
#4 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(1128): Doctrine\ORM\Persisters\Entity\JoinedSubclassPersister->executeInserts()
#5 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(425): Doctrine\ORM\UnitOfWork->executeInserts(Object(Doctrine\ORM\Mapping\ClassMetadata))
#6 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php(392): Doctrine\ORM\UnitOfWork->commit(NULL)
#7 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/modules/CSVImport/src/Job/Import.php(617): Doctrine\ORM\EntityManager->flush()
#8 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/modules/CSVImport/src/Job/Import.php(366): CSVImport\Job\Import->reorderMedias(Array)
#9 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/modules/CSVImport/src/Job/Import.php(260): CSVImport\Job\Import->create(Array)
#10 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/modules/CSVImport/src/Job/Import.php(199): CSVImport\Job\Import->processBatchData(Array)
#11 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/application/src/Job/DispatchStrategy/Synchronous.php(34): CSVImport\Job\Import->perform()
#12 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/application/src/Job/Dispatcher.php(105): Omeka\Job\DispatchStrategy\Synchronous->send(Object(Omeka\Entity\Job))
#13 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/application/data/scripts/perform-job.php(66): Omeka\Job\Dispatcher->send(Object(Omeka\Entity\Job), Object(Omeka\Job\DispatchStrategy\Synchronous))
#14 {main}
Next Doctrine\DBAL\Driver\PDO\Exception: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'e3c6680b663a92f7f4477ce8990b5fd9040ff4ce' for key 'UNIQ_6A2CA10C5CC5DB90' in /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDO/Exception.php:18
Stack trace:
#0 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php(119): Doctrine\DBAL\Driver\PDO\Exception::new(Object(PDOException))
#1 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/vendor/doctrine/dbal/lib/Doctrine/DBAL/Statement.php(179): Doctrine\DBAL\Driver\PDOStatement->execute(NULL)
#2 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/vendor/doctrine/dbal/lib/Doctrine/DBAL/Statement.php(226): Doctrine\DBAL\Statement->execute(NULL)
#3 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/vendor/doctrine/orm/lib/Doctrine/ORM/Persisters/Entity/JoinedSubclassPersister.php(194): Doctrine\DBAL\Statement->executeStatement()
#4 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(1128): Doctrine\ORM\Persisters\Entity\JoinedSubclassPersister->executeInserts()
#5 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(425): Doctrine\ORM\UnitOfWork->executeInserts(Object(Doctrine\ORM\Mapping\ClassMetadata))
#6 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php(392): Doctrine\ORM\UnitOfWork->commit(NULL)
#7 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/modules/CSVImport/src/Job/Import.php(617): Doctrine\ORM\EntityManager->flush()
#8 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/modules/CSVImport/src/Job/Import.php(366): CSVImport\Job\Import->reorderMedias(Array)
#9 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/modules/CSVImport/src/Job/Import.php(260): CSVImport\Job\Import->create(Array)
#10 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/modules/CSVImport/src/Job/Import.php(199): CSVImport\Job\Import->processBatchData(Array)
#11 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/application/src/Job/DispatchStrategy/Synchronous.php(34): CSVImport\Job\Import->perform()
#12 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/application/src/Job/Dispatcher.php(105): Omeka\Job\DispatchStrategy\Synchronous->send(Object(Omeka\Entity\Job))
#13 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/application/data/scripts/perform-job.php(66): Omeka\Job\Dispatcher->send(Object(Omeka\Entity\Job), Object(Omeka\Job\DispatchStrategy\Synchronous))
#14 {main}
Next Doctrine\DBAL\Exception\UniqueConstraintViolationException: An exception occurred while executing 'INSERT INTO media (id, ingester, renderer, data, source, media_type, storage_id, extension, sha256, size, has_original, has_thumbnails, position, lang, alt_text, item_id) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)' with params [5791, "sideload", "file", null, "photos\/1419.jpg", "image\/jpeg", "e3c6680b663a92f7f4477ce8990b5fd9040ff4ce", "jpg", "b495429753c1f8eacf2994843773ad38f16d67537baeeac13ca45a52318634c2", 104970, 1, 1, null, null, null, 5744]:
SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'e3c6680b663a92f7f4477ce8990b5fd9040ff4ce' for key 'UNIQ_6A2CA10C5CC5DB90' in /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/AbstractMySQLDriver.php:74
Stack trace:
#0 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/vendor/doctrine/dbal/lib/Doctrine/DBAL/DBALException.php(182): Doctrine\DBAL\Driver\AbstractMySQLDriver->convertException('An exception oc...', Object(Doctrine\DBAL\Driver\PDO\Exception))
#1 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/vendor/doctrine/dbal/lib/Doctrine/DBAL/DBALException.php(159): Doctrine\DBAL\DBALException::wrapException(Object(Doctrine\DBAL\Driver\PDO\MySQL\Driver), Object(Doctrine\DBAL\Driver\PDO\Exception), 'An exception oc...')
#2 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/vendor/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(2226): Doctrine\DBAL\DBALException::driverExceptionDuringQuery(Object(Doctrine\DBAL\Driver\PDO\MySQL\Driver), Object(Doctrine\DBAL\Driver\PDO\Exception), 'INSERT INTO med...', Array)
#3 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/vendor/doctrine/dbal/lib/Doctrine/DBAL/Statement.php(185): Doctrine\DBAL\Connection->handleExceptionDuringQuery(Object(Doctrine\DBAL\Driver\PDO\Exception), 'INSERT INTO med...', Array, Array)
#4 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/vendor/doctrine/dbal/lib/Doctrine/DBAL/Statement.php(226): Doctrine\DBAL\Statement->execute(NULL)
#5 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/vendor/doctrine/orm/lib/Doctrine/ORM/Persisters/Entity/JoinedSubclassPersister.php(194): Doctrine\DBAL\Statement->executeStatement()
#6 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(1128): Doctrine\ORM\Persisters\Entity\JoinedSubclassPersister->executeInserts()
#7 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(425): Doctrine\ORM\UnitOfWork->executeInserts(Object(Doctrine\ORM\Mapping\ClassMetadata))
#8 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php(392): Doctrine\ORM\UnitOfWork->commit(NULL)
#9 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/modules/CSVImport/src/Job/Import.php(617): Doctrine\ORM\EntityManager->flush()
#10 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/modules/CSVImport/src/Job/Import.php(366): CSVImport\Job\Import->reorderMedias(Array)
#11 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/modules/CSVImport/src/Job/Import.php(260): CSVImport\Job\Import->create(Array)
#12 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/modules/CSVImport/src/Job/Import.php(199): CSVImport\Job\Import->processBatchData(Array)
#13 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/application/src/Job/DispatchStrategy/Synchronous.php(34): CSVImport\Job\Import->perform()
#14 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/application/src/Job/Dispatcher.php(105): Omeka\Job\DispatchStrategy\Synchronous->send(Object(Omeka\Entity\Job))
#15 /home/clients/87af2e3481ab91fc1106985669dd916e/sites/dev.example.org/application/data/scripts/perform-job.php(66): Omeka\Job\Dispatcher->send(Object(Omeka\Entity\Job), Object(Omeka\Job\DispatchStrategy\Synchronous))
#16 {main}
The image is however loaded, but the CSV Import stops after this image, after this error, and following media are not loaded.
OK, thanks for the update.
A few questions:
The strand thing is that I can’t find any similar item in the database, here’s what I’ve found for the corresponding file:
+------+---------+----------+----------+------+-----------------+------------+------------------------------------------+-----------+------------------------------------------------------------------+--------+--------------+----------------+----------+------+----------+
| id | item_id | ingester | renderer | data | source | media_type | storage_id | extension | sha256 | size | has_original | has_thumbnails | position | lang | alt_text |
+------+---------+----------+----------+------+-----------------+------------+------------------------------------------+-----------+------------------------------------------------------------------+--------+--------------+----------------+----------+------+----------+
| 5732 | 5355 | sideload | file | NULL | photos/1419.jpg | image/jpeg | 407af7321ba64bfd493e8ee144cc392378490bbd | jpg | b4e848faf053e1eb37949e9ccb5bf4ee2055840ed9eb6529f1be3184273f9bca | 142760 | 1 | 1 | 2 | NULL | NULL |
+------+---------+----------+----------+------+-----------------+------------+------------------------------------------+-----------+------------------------------------------------------------------+--------+--------------+----------------+----------+------+----------+
1 row in set (0,04 sec)
comparing to what was in the logfile:
Next Doctrine\DBAL\Exception\UniqueConstraintViolationException: An exception occurred while executing 'INSERT INTO media (id, ingester, renderer, data, source, media_type, storage_id, extension, sha256, size, has_original, has_thumbnails, position, lang, alt_text, item_id) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)' with params [5791, "sideload", "file", null, "photos\/1419.jpg", "image\/jpeg", "e3c6680b663a92f7f4477ce8990b5fd9040ff4ce", "jpg", "b495429753c1f8eacf2994843773ad38f16d67537baeeac13ca45a52318634c2", 104970, 1, 1, null, null, null, 5744]:
To answer your questions, I am doing a media import, the same way I 've been doing this since 2020. I’m not sure when this problem appeared, but it was in the last months… I gave you all the log I had in my logfile.
Last thing, here is the csv file that I’m using for the import:
"dcterms:identifier","Filename-originals","public/private"
1419,"originals/1419.jpg","private"
1717,"originals/1717.jpg","private"
1718,"originals/1718.jpg","private"
1719,"originals/1719.jpg","private"
1720,"originals/1720.jpg","private"
1721,"originals/1721.jpg","private"
1722,"originals/1722.jpg","private"
1723,"originals/1723.jpg","private"
1724,"originals/1724.jpg","private"
1725,"originals/1725.jpg","private"
1726,"originals/1726.jpg","private"
1727,"originals/1727.jpg","private"
1728,"originals/1728.jpg","private"
1729,"originals/1729.jpg","private"
1730,"originals/1730.jpg","private"
1731,"originals/1731.jpg","private"
1732,"originals/1732.jpg","private"
1733,"originals/1733.jpg","private"
1734,"originals/1734.jpg","private"
1735,"originals/1735.jpg","private"
1736,"originals/1736.jpg","private"
1737,"originals/1737.jpg","private"
1738,"originals/1738.jpg","private"
1739,"originals/1739.jpg","private"
1740,"originals/1740.jpg","private"
1741,"originals/1741.jpg","private"
1742,"originals/1742.jpg","private"
1743,"originals/1743.jpg","private"
1744,"originals/1744.jpg","private"
1745,"originals/1745.jpg","private"
1746,"originals/1746.jpg","private"
1747,"originals/1747.jpg","private"
1748,"originals/1748.jpg","private"
1749,"originals/1749.jpg","private"
1750,"originals/1750.jpg","private"
1751,"originals/1751.jpg","private"
1752,"originals/1752.jpg","private"
1753,"originals/1753.jpg","private"
1754,"originals/1754.jpg","private"
1419,"photos/1419.jpg","public"
1717,"photos/1717.jpg","public"
1718,"photos/1718.jpg","public"
1719,"photos/1719.jpg","public"
1720,"photos/1720.jpg","public"
1721,"photos/1721.jpg","public"
1722,"photos/1722.jpg","public"
1723,"photos/1723.jpg","public"
1724,"photos/1724.jpg","public"
1725,"photos/1725.jpg","public"
1726,"photos/1726.jpg","public"
1727,"photos/1727.jpg","public"
1728,"photos/1728.jpg","public"
1729,"photos/1729.jpg","public"
1730,"photos/1730.jpg","public"
1731,"photos/1731.jpg","public"
1732,"photos/1732.jpg","public"
1733,"photos/1733.jpg","public"
1734,"photos/1734.jpg","public"
1735,"photos/1735.jpg","public"
1736,"photos/1736.jpg","public"
1737,"photos/1737.jpg","public"
1738,"photos/1738.jpg","public"
1739,"photos/1739.jpg","public"
1740,"photos/1740.jpg","public"
1741,"photos/1741.jpg","public"
1742,"photos/1742.jpg","public"
1743,"photos/1743.jpg","public"
1744,"photos/1744.jpg","public"
1745,"photos/1745.jpg","public"
1746,"photos/1746.jpg","public"
1747,"photos/1747.jpg","public"
1748,"photos/1748.jpg","public"
1749,"photos/1749.jpg","public"
1750,"photos/1750.jpg","public"
1751,"photos/1751.jpg","public"
1752,"photos/1752.jpg","public"
1753,"photos/1753.jpg","public"
1754,"photos/1754.jpg","public"
As you can the the private images get loaded without problem, while the problem appears with the first public image. Each item get 2 images: the private one is the high resolution image, while the public one is the low resolution watermarked image.
The example file is very helpful.
Another question: there’s a “batch size” setting in the Advanced tab: what do you have it set to? You could also just go to “Job details” for this import in the Past Imports list and post the whole “args” value, which includes all the import settings for that job.
One final thing: do things change if you change the order or number of entries in your file you’re importing? Like, if you list the “public” ones first, is it still the 40th row that’s the problem, or does the issue “stay” with the “photos/1419.jpg” media regardless?
I have a possible solution.
Are you comfortable applying a patch? There’s a small patch I’ve written that applies to the CSV Import module to resolve this issue. If you’re able and willing, it would be great if you could try it out.
Here are the json parameters of the last import. As you can see, rows_by_batch is set to 20.
{
"filename": "import-20231221.csv",
"filesize": "2711",
"filepath": "\/home\/clients\/87af2e3481ab91fc1106985669dd916e\/tmp\/omekafVoAmD",
"media_type": "text\/csv",
"resource_type": "media",
"comment": "",
"automap_check_names_alone": false,
"column-item": [
"dcterms:identifier"
],
"column-data-type": [
"literal",
"literal",
"literal"
],
"column-media_source": {
"1": "sideload"
},
"column-is_public": {
"2": "1"
},
"generic-data": "default",
"media-source": "default",
"o:resource_template": "",
"o:resource_class": "",
"o:owner": {
"o:id": 1
},
"o:is_public": 1,
"multivalue_separator": "|",
"global_language": "",
"action": "create",
"identifier_column": 0,
"identifier_property": "",
"action_unidentified": "skip",
"rows_by_batch": 20,
"column-multivalue": [],
"delimiter": ",",
"enclosure": "\"",
"escape": "\\"
}
If I change the csv import file to remove some file:
"dcterms:identifier","Filename-originals","public/private"
1720,"originals/1720.jpg","private"
1721,"originals/1721.jpg","private"
1722,"originals/1722.jpg","private"
1723,"originals/1723.jpg","private"
1724,"originals/1724.jpg","private"
1725,"originals/1725.jpg","private"
1726,"originals/1726.jpg","private"
1727,"originals/1727.jpg","private"
1728,"originals/1728.jpg","private"
1729,"originals/1729.jpg","private"
1730,"originals/1730.jpg","private"
1731,"originals/1731.jpg","private"
1732,"originals/1732.jpg","private"
1733,"originals/1733.jpg","private"
1734,"originals/1734.jpg","private"
1735,"originals/1735.jpg","private"
1736,"originals/1736.jpg","private"
1737,"originals/1737.jpg","private"
1738,"originals/1738.jpg","private"
1739,"originals/1739.jpg","private"
1740,"originals/1740.jpg","private"
1741,"originals/1741.jpg","private"
1742,"originals/1742.jpg","private"
1743,"originals/1743.jpg","private"
1744,"originals/1744.jpg","private"
1745,"originals/1745.jpg","private"
1746,"originals/1746.jpg","private"
1747,"originals/1747.jpg","private"
1748,"originals/1748.jpg","private"
1749,"originals/1749.jpg","private"
1750,"originals/1750.jpg","private"
1751,"originals/1751.jpg","private"
1752,"originals/1752.jpg","private"
1753,"originals/1753.jpg","private"
1754,"originals/1754.jpg","private"
1720,"photos/1720.jpg","public"
1721,"photos/1721.jpg","public"
1722,"photos/1722.jpg","public"
1723,"photos/1723.jpg","public"
1724,"photos/1724.jpg","public"
1725,"photos/1725.jpg","public"
1726,"photos/1726.jpg","public"
1727,"photos/1727.jpg","public"
1728,"photos/1728.jpg","public"
1729,"photos/1729.jpg","public"
1730,"photos/1730.jpg","public"
1731,"photos/1731.jpg","public"
1732,"photos/1732.jpg","public"
1733,"photos/1733.jpg","public"
1734,"photos/1734.jpg","public"
1735,"photos/1735.jpg","public"
1736,"photos/1736.jpg","public"
1737,"photos/1737.jpg","public"
1738,"photos/1738.jpg","public"
1739,"photos/1739.jpg","public"
1740,"photos/1740.jpg","public"
1741,"photos/1741.jpg","public"
1742,"photos/1742.jpg","public"
1743,"photos/1743.jpg","public"
1744,"photos/1744.jpg","public"
1745,"photos/1745.jpg","public"
1746,"photos/1746.jpg","public"
1747,"photos/1747.jpg","public"
1748,"photos/1748.jpg","public"
1749,"photos/1749.jpg","public"
1750,"photos/1750.jpg","public"
1751,"photos/1751.jpg","public"
1752,"photos/1752.jpg","public"
1753,"photos/1753.jpg","public"
1754,"photos/1754.jpg","public"
The problem still occurs at the first public image that is loaded, but this is the 36th image to load.
In my opinion, the problem occurs at the load of a second image of an item.
I tested your patch and the import does now complete without error. All the images are loaded.
Can you confirm that I applied the patch correctly ?
// Get correct position for each media
$mediaRanks = [];
$currentItemId = null;
$position = null;
foreach ($mediaInOrder as $mediaData) {
if ($mediaData['item_id'] !== $currentItemId) {
$position = 1;
$currentItemId = $mediaData['item_id'];
} else {
$position++;
}
$mediaRanks[$mediaData['id']] = $position;
}
// Update positions of the updated media.
$conn->beginTransaction();
try {
$updateQuery = 'UPDATE media SET position = ? WHERE id = ?';
foreach ($mediaRanks as $id => $rank) {
$conn->executeQuery($updateQuery, [$rank, $id], ['integer', 'integer']);
}
$conn->commit();
} catch (\Exception $e) {
$conn->rollBack();
}
}
I think you’ve spotted the error and fixed it in the patch!
Can you explain a little bit what was the error (to feed my curiosity)?
OK, that’s great, thanks for reporting your experience with it. You do have the patch applied right.
The issue is a little complex to explain, as it involves the inner workings of Doctrine, the database library we use in Omeka S. CSV Import, to deal with potentially very large imports, has to work in batches and clear away the completed imported records periodically.
The problem here is related to a specific piece of code that only runs when you’re importing media specifically, which is used to set the “position” of the new media to be at the end of the list for their parent item. This code was changed in 2.5.0, which is the reason behind the problem only appearing recently. Basically, the 2.5.0 code could “confuse” Doctrine into thinking it should insert a new copy of a media instead of just updating its position, but only after the first batch. This is why it was a little tricky to run down: you had to do a media import specifically, and have it be large enough (or set the batch size small enough) to do multiple batches. Both you and @nxm gave valuable feedback for narrowing down the conditions here, thanks to both of you.
I should be able to apply this quickly to a bugfix release for CSV Import, and you’ll be able to upgrade to that rather than have to apply the patch.
Great thanks for your explanation that was clear enough for me to understand what was going on.
Thanks also for the patch and for the upcoming release of CVS Import. Could please ping us here once the new version is released?
The release with the fix is available now, version 2.5.1. It’s available on the module’s page as usual.
Thanks @jflatnes. I updated the plugin, I loaded my images, and… no problem, no warning, no error.
However, I got one item that has no image linked to it, the item 1746. Can there still be a problem on how images are processed so that an image could be forgotten?
The strange thing is that - in fact - two images were ignored:
Two images from the same item.
I made a new test, with a csv file containing only two files:
"dcterms:identifier","Filename-originals","public/private"
1746,"originals/1746.jpg","private"
1746,"photos/1746.jpg","public"
While the import went well: Status: Completed and no log. I can’t see any image linked to the item, while the image from the import file do not exist any more. It’s like the images where not processed correctly.
How can I debug that?
Is it possible that you have more than one item with the same identifier “1746”? You could be looking at one item but the importer is adding the media to a different one if you have multiple items that share this identifier. Since it’s reliably and only the rows using this specific identifier, that’s probably the most likely source of your issue.
I thought the first thing I would check was the unicity of the item whose identifier was 1746, but I double checked it and it appears that you were right: there was two item with the identifier 1746.
Is there a way to tell Omeka that the field identifier is an identifier and hence can be managed by Omeka as an identifier: each item must have a unique identifier, and any new item should have an identifier automatically filled with a value that is max(identifier)+1. This would be great…
By the way, is there a way to find out if an identifier is used more than once?
This would be great if I can find all duplicate identifiers…
@jflatnes I think I’ve found another ORM error, similar to the one previously spotted in this thread. You can have a look at this thread to find out: Import media and replace files .