Import media and replace files

ploc · March 1, 2023, 12:53am

I’m using the CSVImport module from Omeka S in order to import pictures of paintings that are getting associated with a notice of the painting.

Sometime, we have a better picture of the painting and we want to replace the previous picture by the new one. Unfortunately, it seems that CSVImport module is not keeping the original name of the jpeg file and is renaming it using something like a unique hash. This allow to have more that one file associated with a notice where I would like the second one to replace the first one.

How could that be possible?

AllanaMayer · March 2, 2023, 4:21pm

Depending on how many files you want to update at a time, this may be easier to do manually than with CSV Import.

If you have the item IDs (or another unique value) already in your spreadsheet, you can do this in a two-stage process. This will delete every media attached to the given items, not any one file in particular, so this may not work for you.

First, run a CSV Import where you “Update” the items. Supply only one mapped column: an empty column, mapped to Media source > URL.

When CSV Import updates the items, it will wipe any existing media (and replace it with nothing). This shouldn’t mess up any metadata, item sets, sites, etc., but test it with a few items first to be sure you have it right. Then you can wipe all media from the items you intend to update.

Then you can run CSV Import as usual by adding the new media to the existing items.

If you have items with more than one media (photos of the back, details, etc.), and you want to delete only one image in order to replace it, that’s probably a manual job. Keep in mind that media are assigned unique IDs, just like items, starting with 1, so the older the media, the lower the number. You can change the columns that appear on the Media browsing page to show IDs and then sort using them, if that helps you search out the files you wish to delete.

https://omeka.org/s/docs/user-manual/admin/users/#admin-browse-columns

ploc · September 18, 2023, 11:02pm

Thank you @AllanaMayer for your answer. It took me some time to try your solution as you answered me in march and I only tried it tonight. I get an error while trying your way of deleting media. Here is the error from the log:

2023-09-18T22:59:24+00:00 ERR (3): Doctrine\ORM\ORMInvalidArgumentException: A new entity was found through the relationship 'Omeka\Entity\Item#primaryMedia' that was not configured to cascade persist operations for entity: DoctrineProxies\__CG__\Omeka\Entity\Media@3472. To solve this issue: Either explicitly call EntityManager#persist() on this unknown entity or configure cascade persist this association in the mapping for example @ManyToOne(..,cascade={"persist"}). If you cannot find out which entity causes the problem implement 'Omeka\Entity\Media#__toString()' to get a clue. in /srv/data/web/vhosts/htdocs/vendor/doctrine/orm/lib/Doctrine/ORM/ORMInvalidArgumentException.php:97
Stack trace:
#0 /srv/data/web/vhosts/htdocs/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(3521): Doctrine\ORM\ORMInvalidArgumentException::newEntitiesFoundThroughRelationships()
#1 /srv/data/web/vhosts/htdocs/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(390): Doctrine\ORM\UnitOfWork->assertThatThereAreNoUnintentionallyNonPersistedAssociations()
#2 /srv/data/web/vhosts/htdocs/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php(392): Doctrine\ORM\UnitOfWork->commit()
#3 /srv/data/web/vhosts/htdocs/application/src/Api/Adapter/AbstractEntityAdapter.php(447): Doctrine\ORM\EntityManager->flush()
#4 /srv/data/web/vhosts/htdocs/application/src/Api/Manager.php(233): Omeka\Api\Adapter\AbstractEntityAdapter->update()
#5 /srv/data/web/vhosts/htdocs/application/src/Api/Manager.php(136): Omeka\Api\Manager->execute()
#6 /srv/data/web/vhosts/htdocs/modules/CSVImport/src/Job/Import.php(809): Omeka\Api\Manager->update()
#7 /srv/data/web/vhosts/htdocs/modules/CSVImport/src/Job/Import.php(432): CSVImport\Job\Import->updateRevise()
#8 /srv/data/web/vhosts/htdocs/modules/CSVImport/src/Job/Import.php(301): CSVImport\Job\Import->update()
#9 /srv/data/web/vhosts/htdocs/modules/CSVImport/src/Job/Import.php(199): CSVImport\Job\Import->processBatchData()
#10 /srv/data/web/vhosts/htdocs/application/src/Job/DispatchStrategy/Synchronous.php(34): CSVImport\Job\Import->perform()
#11 /srv/data/web/vhosts/htdocs/application/src/Job/Dispatcher.php(105): Omeka\Job\DispatchStrategy\Synchronous->send()
#12 /srv/data/web/vhosts/htdocs/application/data/scripts/perform-job.php(66): Omeka\Job\Dispatcher->send()
#13 {main}

Do you have an idea on what can cause this error?

And it appears that the update fails completly, as I can read the following in the log:

2023-09-18T23:18:15+00:00 NOTICE (5): None of the 20 items were updated (update).

It looks like the ORM of Omeka S was not configured to modify the media attachements while updating the iem object.

AllanaMayer · October 30, 2023, 8:18pm

I’ve just tried using the method I recommended before, and it worked fine.

Here are all the settings I used:

A reminder that the spreadsheet is listing item IDs, not media IDs, because you’re wiping all media attached to the specified items, no matter how many. By “updating” you’re erasing all existing information in that “field”, and replacing it with nothing.

I admit I am not an expert at reading error messages so that may be something more basic. Does CSV Import usually work on this installation of yours? If this doesn’t work, can you test it by creating a few simple items and let me know how that goes?

ploc · March 5, 2024, 9:16pm

Thanks a lot @AllanaMayer for your help. It took me a lot of time to test and validate your solution.

It appears to work for one item, but not for many item. I think there is an issue with the ORM and the way it is cascading deleting items and attributes.

Here is the error I get when trying to delete media from multiple items:

2024-03-05T21:14:28+00:00 ERR (3): Doctrine\ORM\ORMInvalidArgumentException: A new entity was found through the relationship 'Omeka\Entity\Item#primaryMedia' that was not configured to cascade persist operations for entity: DoctrineProxies\__CG__\Omeka\Entity\Media@2676. To solve this issue: Either explicitly call EntityManager#persist() on this unknown entity or configure cascade persist this association in the mapping for example @ManyToOne(..,cascade={"persist"}). If you cannot find out which entity causes the problem implement 'Omeka\Entity\Media#__toString()' to get a clue. in /path/to/omeka/vendor/doctrine/orm/lib/Doctrine/ORM/ORMInvalidArgumentException.php:97
Stack trace:
#0 /path/to/omeka/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(3521): Doctrine\ORM\ORMInvalidArgumentException::newEntitiesFoundThroughRelationships(Array)
#1 /path/to/omeka/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(390): Doctrine\ORM\UnitOfWork->assertThatThereAreNoUnintentionallyNonPersistedAssociations()
#2 /path/to/omeka/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php(392): Doctrine\ORM\UnitOfWork->commit(NULL)
#3 /path/to/omeka/application/src/Api/Adapter/AbstractEntityAdapter.php(447): Doctrine\ORM\EntityManager->flush()
#4 /path/to/omeka/application/src/Api/Manager.php(233): Omeka\Api\Adapter\AbstractEntityAdapter->update(Object(Omeka\Api\Request))
#5 /path/to/omeka/application/src/Api/Manager.php(136): Omeka\Api\Manager->execute(Object(Omeka\Api\Request))
#6 /path/to/omeka/modules/CSVImport/src/Job/Import.php(812): Omeka\Api\Manager->update('items', 92, Array, Array, Array)
#7 /path/to/omeka/modules/CSVImport/src/Job/Import.php(432): CSVImport\Job\Import->updateRevise('items', 92, Array, 'update')
#8 /path/to/omeka/modules/CSVImport/src/Job/Import.php(301): CSVImport\Job\Import->update(Array, Array, 'update')
#9 /path/to/omeka/modules/CSVImport/src/Job/Import.php(199): CSVImport\Job\Import->processBatchData(Array)
#10 /path/to/omeka/application/src/Job/DispatchStrategy/Synchronous.php(34): CSVImport\Job\Import->perform()
#11 /path/to/omeka/application/src/Job/Dispatcher.php(105): Omeka\Job\DispatchStrategy\Synchronous->send(Object(Omeka\Entity\Job))
#12 /path/to/omeka/application/data/scripts/perform-job.php(66): Omeka\Job\Dispatcher->send(Object(Omeka\Entity\Job), Object(Omeka\Job\DispatchStrategy\Synchronous))
#13 {main}

ploc · March 5, 2024, 9:20pm

One strange thing however: I must not try to map the identifier field from that:

to that:

Otherwise, I get the following error:

2024-03-05T21:03:50+00:00 INFO (6): The following identifiers are not associated with a resource and were skipped: "3", "5", "6", "7", "9", "11", "12", "14", "16", "19", "20", "23", "24", "27", "30", "31", "34", "43", "51", "54".

jflatnes · March 5, 2024, 10:44pm

Yes, this looks like it’s an error related specifically to removing the current primary media.

I believe I have a fix for the problem.

ploc · March 5, 2024, 11:09pm

Your fix seems to fix the problem… only partially.

Here is the log with your fix applied:

2024-03-05T23:05:51+00:00 INFO (6): 20 items were updated (update): "3" (4), "5" (6), "6" (7), "7" (8), "9" (10), "11" (12), "12" (13), "14" (15), "16" (17), "19" (20), "20" (21), "23" (44), "24" (45), "27" (48), "30" (51), "31" (52), "34" (55), "43" (84), "51" (92), "54" (95).
2024-03-05T23:05:52+00:00 ERR (3): Doctrine\ORM\ORMInvalidArgumentException: A new entity was found through the relationship 'Omeka\Entity\Item#primaryMedia' that was not configured to cascade persist operations for entity: DoctrineProxies\__CG__\Omeka\Entity\Media@3258. To solve this issue: Either explicitly call EntityManager#persist() on this unknown entity or configure cascade persist this association in the mapping for example @ManyToOne(..,cascade={"persist"}). If you cannot find out which entity causes the problem implement 'Omeka\Entity\Media#__toString()' to get a clue. in /path/to/omeka/vendor/doctrine/orm/lib/Doctrine/ORM/ORMInvalidArgumentException.php:97
Stack trace:
#0 /path/to/omeka/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(3521): Doctrine\ORM\ORMInvalidArgumentException::newEntitiesFoundThroughRelationships(Array)
#1 /path/to/omeka/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(390): Doctrine\ORM\UnitOfWork->assertThatThereAreNoUnintentionallyNonPersistedAssociations()
#2 /path/to/omeka/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php(392): Doctrine\ORM\UnitOfWork->commit(NULL)
#3 /path/to/omeka/application/src/Api/Adapter/AbstractEntityAdapter.php(447): Doctrine\ORM\EntityManager->flush()
#4 /path/to/omeka/application/src/Api/Manager.php(233): Omeka\Api\Adapter\AbstractEntityAdapter->update(Object(Omeka\Api\Request))
#5 /path/to/omeka/application/src/Api/Manager.php(136): Omeka\Api\Manager->execute(Object(Omeka\Api\Request))
#6 /path/to/omeka/modules/CSVImport/src/Job/Import.php(832): Omeka\Api\Manager->update('items', 179, Array, Array, Array)
#7 /path/to/omeka/modules/CSVImport/src/Job/Import.php(432): CSVImport\Job\Import->updateRevise('items', 179, Array, 'update')
#8 /path/to/omeka/modules/CSVImport/src/Job/Import.php(301): CSVImport\Job\Import->update(Array, Array, 'update')
#9 /path/to/omeka/modules/CSVImport/src/Job/Import.php(199): CSVImport\Job\Import->processBatchData(Array)
#10 /path/to/omeka/application/src/Job/DispatchStrategy/Synchronous.php(34): CSVImport\Job\Import->perform()
#11 /path/to/omeka/application/src/Job/Dispatcher.php(105): Omeka\Job\DispatchStrategy\Synchronous->send(Object(Omeka\Entity\Job))
#12 /path/to/omeka/application/data/scripts/perform-job.php(66): Omeka\Job\Dispatcher->send(Object(Omeka\Entity\Job), Object(Omeka\Job\DispatchStrategy\Synchronous))
#13 {main}

It looks like the first set of items get updated (the first 20 items, as configured in the import job), but there are errors that happens after that first set of items.

jflatnes · March 5, 2024, 11:49pm

I’m not reproducing your problem: no issue here doing a second batch. The number in the error (179) is somewhat substantially higher: is that the very first item in the second batch, or somewhere in the middle? The problem could instead be specific to that item or how it’s being updated.

ploc · March 6, 2024, 7:44am

The item 179 is not the first from the second batch.

Here is the beginning of my csv file:

"dcterms:identifier","Filename-originals"
3,
5,
6,
7,
9,
11,
12,
14,
16,
19,
20,
23,
24,
27,
30,
31,
34,
43,
51,
54,
57,
58,
60,
67,
75,
80,
83,
85,
88,
89,
90,
92,
93,
95,
96,
97,
98,
102,
104,
106,
111,
114,
116,
119,
120,
125,
127,
134,
144,
145,
152,
155,
156,
158,
164,
167,
172,
174,
176,
179,
183,
184,
193,
[...]

For each item, I’m only removing any media attached to the item, when it exists. What could possibly go wrong?

Could there be an error thrown when trying to delete media attached to an item when no such media exists?

jflatnes · March 6, 2024, 4:38pm

Just to clarify, this would be, I think, the item with the Omeka ID 179 (i.e., the item at admin/item/179 in your install), not the one with dcterms:identifier 179.

Is there anything obviously different about that item? More media, less, none?

ploc · March 10, 2024, 2:47pm

When I talk about item 179, I mean “item whose attribute dcterm:identifier=179”.

But there is not only an error on item 179. In fact, I get 353 errors like this one, for 388 items in the list.

Sometime, everything goes well, and then an error occurs:

2024-03-05T23:05:52+00:00 ERR (3): Doctrine\ORM\ORMInvalidArgumentException: A new entity was found through the relationship 'Omeka\Entity\Item#primaryMedia' that was not configured to cascade persist operations for entity: DoctrineProxies\__CG__\Omeka\Entity\Media@3258. To solve this issue: Either explicitly call EntityManager#persist() on this unknown entity or configure cascade persist this association in the mapping for example @ManyToOne(..,cascade={"persist"}). If you cannot find out which entity causes the problem implement 'Omeka\Entity\Media#__toString()' to get a clue. in /path/to/omeka/vendor/doctrine/orm/lib/Doctrine/ORM/ORMInvalidArgumentException.php:97
Stack trace:
#0 /path/to/omeka/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(3521): Doctrine\ORM\ORMInvalidArgumentException::newEntitiesFoundThroughRelationships(Array)
#1 /path/to/omeka/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(390): Doctrine\ORM\UnitOfWork->assertThatThereAreNoUnintentionallyNonPersistedAssociations()
#2 /path/to/omeka/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php(392): Doctrine\ORM\UnitOfWork->commit(NULL)
#3 /path/to/omeka/application/src/Api/Adapter/AbstractEntityAdapter.php(447): Doctrine\ORM\EntityManager->flush()
#4 /path/to/omeka/application/src/Api/Manager.php(233): Omeka\Api\Adapter\AbstractEntityAdapter->update(Object(Omeka\Api\Request))
#5 /path/to/omeka/application/src/Api/Manager.php(136): Omeka\Api\Manager->execute(Object(Omeka\Api\Request))
#6 /path/to/omeka/modules/CSVImport/src/Job/Import.php(832): Omeka\Api\Manager->update('items', 207, Array, Array, Array)
#7 /path/to/omeka/modules/CSVImport/src/Job/Import.php(432): CSVImport\Job\Import->updateRevise('items', 207, Array, 'update')
#8 /path/to/omeka/modules/CSVImport/src/Job/Import.php(301): CSVImport\Job\Import->update(Array, Array, 'update')
#9 /path/to/omeka/modules/CSVImport/src/Job/Import.php(199): CSVImport\Job\Import->processBatchData(Array)
#10 /path/to/omeka/application/src/Job/DispatchStrategy/Synchronous.php(34): CSVImport\Job\Import->perform()
#11 /path/to/omeka/application/src/Job/Dispatcher.php(105): Omeka\Job\DispatchStrategy\Synchronous->send(Object(Omeka\Entity\Job))
#12 /path/to/omeka/application/data/scripts/perform-job.php(66): Omeka\Job\Dispatcher->send(Object(Omeka\Entity\Job), Object(Omeka\Job\DispatchStrategy\Synchronous))
#13 {main}
2024-03-05T23:05:52+00:00 INFO (6): 16 items were updated (update): "57" (98), "58" (99), "60" (101), "67" (128), "75" (136), "80" (141), "83" (164), "85" (166), "88" (169), "89" (170), "90" (171), "92" (173), "93" (174), "95" (176), "96" (177), "97" (178).
2024-03-05T23:05:52+00:00 ERR (3): Doctrine\ORM\ORMInvalidArgumentException: Detached entity Omeka\Entity\Media@3215 cannot be removed in /path/to/omeka/vendor/doctrine/orm/lib/Doctrine/ORM/ORMInvalidArgumentException.php:172
Stack trace:
#0 /path/to/omeka/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(1906): Doctrine\ORM\ORMInvalidArgumentException::detachedEntityCannot(Object(Omeka\Entity\Media), 'removed')
#1 /path/to/omeka/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(1857): Doctrine\ORM\UnitOfWork->doRemove(Object(Omeka\Entity\Media), Array)
#2 /path/to/omeka/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(394): Doctrine\ORM\UnitOfWork->remove(Object(Omeka\Entity\Media))
#3 /path/to/omeka/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php(392): Doctrine\ORM\UnitOfWork->commit(NULL)
#4 /path/to/omeka/application/src/Api/Adapter/AbstractEntityAdapter.php(447): Doctrine\ORM\EntityManager->flush()
#5 /path/to/omeka/application/src/Api/Manager.php(233): Omeka\Api\Adapter\AbstractEntityAdapter->update(Object(Omeka\Api\Request))
#6 /path/to/omeka/application/src/Api/Manager.php(136): Omeka\Api\Manager->execute(Object(Omeka\Api\Request))
#7 /path/to/omeka/modules/CSVImport/src/Job/Import.php(832): Omeka\Api\Manager->update('items', 212, Array, Array, Array)
#8 /path/to/omeka/modules/CSVImport/src/Job/Import.php(432): CSVImport\Job\Import->updateRevise('items', 212, Array, 'update')
#9 /path/to/omeka/modules/CSVImport/src/Job/Import.php(301): CSVImport\Job\Import->update(Array, Array, 'update')
#10 /path/to/omeka/modules/CSVImport/src/Job/Import.php(199): CSVImport\Job\Import->processBatchData(Array)
#11 /path/to/omeka/application/src/Job/DispatchStrategy/Synchronous.php(34): CSVImport\Job\Import->perform()
#12 /path/to/omeka/application/src/Job/Dispatcher.php(105): Omeka\Job\DispatchStrategy\Synchronous->send(Object(Omeka\Entity\Job))
#13 /path/to/omeka/application/data/scripts/perform-job.php(66): Omeka\Job\Dispatcher->send(Object(Omeka\Entity\Job), Object(Omeka\Job\DispatchStrategy\Synchronous))
#14 {main}

It is hard for me to debug such problem as is seems a bit random. And I understand it is hard for you as you cannot reproduce the problem.