Thanks a lot @AllanaMayer for your help. It took me a lot of time to test and validate your solution.
It appears to work for one item, but not for many item. I think there is an issue with the ORM and the way it is cascading deleting items and attributes.
Here is the error I get when trying to delete media from multiple items:
2024-03-05T21:14:28+00:00 ERR (3): Doctrine\ORM\ORMInvalidArgumentException: A new entity was found through the relationship 'Omeka\Entity\Item#primaryMedia' that was not configured to cascade persist operations for entity: DoctrineProxies\__CG__\Omeka\Entity\Media@2676. To solve this issue: Either explicitly call EntityManager#persist() on this unknown entity or configure cascade persist this association in the mapping for example @ManyToOne(..,cascade={"persist"}). If you cannot find out which entity causes the problem implement 'Omeka\Entity\Media#__toString()' to get a clue. in /path/to/omeka/vendor/doctrine/orm/lib/Doctrine/ORM/ORMInvalidArgumentException.php:97
Stack trace:
#0 /path/to/omeka/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(3521): Doctrine\ORM\ORMInvalidArgumentException::newEntitiesFoundThroughRelationships(Array)
#1 /path/to/omeka/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(390): Doctrine\ORM\UnitOfWork->assertThatThereAreNoUnintentionallyNonPersistedAssociations()
#2 /path/to/omeka/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php(392): Doctrine\ORM\UnitOfWork->commit(NULL)
#3 /path/to/omeka/application/src/Api/Adapter/AbstractEntityAdapter.php(447): Doctrine\ORM\EntityManager->flush()
#4 /path/to/omeka/application/src/Api/Manager.php(233): Omeka\Api\Adapter\AbstractEntityAdapter->update(Object(Omeka\Api\Request))
#5 /path/to/omeka/application/src/Api/Manager.php(136): Omeka\Api\Manager->execute(Object(Omeka\Api\Request))
#6 /path/to/omeka/modules/CSVImport/src/Job/Import.php(812): Omeka\Api\Manager->update('items', 92, Array, Array, Array)
#7 /path/to/omeka/modules/CSVImport/src/Job/Import.php(432): CSVImport\Job\Import->updateRevise('items', 92, Array, 'update')
#8 /path/to/omeka/modules/CSVImport/src/Job/Import.php(301): CSVImport\Job\Import->update(Array, Array, 'update')
#9 /path/to/omeka/modules/CSVImport/src/Job/Import.php(199): CSVImport\Job\Import->processBatchData(Array)
#10 /path/to/omeka/application/src/Job/DispatchStrategy/Synchronous.php(34): CSVImport\Job\Import->perform()
#11 /path/to/omeka/application/src/Job/Dispatcher.php(105): Omeka\Job\DispatchStrategy\Synchronous->send(Object(Omeka\Entity\Job))
#12 /path/to/omeka/application/data/scripts/perform-job.php(66): Omeka\Job\Dispatcher->send(Object(Omeka\Entity\Job), Object(Omeka\Job\DispatchStrategy\Synchronous))
#13 {main}