Hello,
We are working on the CSV import module as well and having a somehow similar issue.
Everytime there is the “à” character at the end of a value the import doesn’t work and we get the following log :
2018-08-30T15:42:15+00:00 ERR (3): PDOException: SQLSTATE[HY000]: General error: 1366 Incorrect string value: '\xC3' for column 'value' at row 1 in /var/www/html/omeka-s/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php:105
Stack trace:
#0 /var/www/html/omeka-s/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php(105): PDOStatement->execute(NULL)
#1 /var/www/html/omeka-s/vendor/doctrine/dbal/lib/Doctrine/DBAL/Statement.php(168): Doctrine\DBAL\Driver\PDOStatement->execute(NULL)
#2 /var/www/html/omeka-s/vendor/doctrine/orm/lib/Doctrine/ORM/Persisters/Entity/BasicEntityPersister.php(281): Doctrine\DBAL\Statement->execute()
#3 /var/www/html/omeka-s/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(1014): Doctrine\ORM\Persisters\Entity\BasicEntityPersister->executeInserts()
#4 /var/www/html/omeka-s/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(378): Doctrine\ORM\UnitOfWork->executeInserts(Object(Doctrine\ORM\Mapping\ClassMetadata))
#5 /var/www/html/omeka-s/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php(356): Doctrine\ORM\UnitOfWork->commit(NULL)
#6 /var/www/html/omeka-s/application/src/Api/Adapter/AbstractEntityAdapter.php(270): Doctrine\ORM\EntityManager->flush()
#7 /var/www/html/omeka-s/application/src/Api/Manager.php(224): Omeka\Api\Adapter\AbstractEntityAdapter->create(Object(Omeka\Api\Request))
#8 /var/www/html/omeka-s/application/src/Api/Manager.php(78): Omeka\Api\Manager->execute(Object(Omeka\Api\Request))
#9 /var/www/html/omeka-s/modules/CSVImport/src/Job/Import.php(346): Omeka\Api\Manager->create('items', Array)
#10 /var/www/html/omeka-s/modules/CSVImport/src/Job/Import.php(257): CSVImport\Job\Import->create(Array)
#11 /var/www/html/omeka-s/modules/CSVImport/src/Job/Import.php(196): CSVImport\Job\Import->processBatchData(Array)
#12 /var/www/html/omeka-s/application/src/Job/DispatchStrategy/Synchronous.php(37): CSVImport\Job\Import->perform()
#13 /var/www/html/omeka-s/application/src/Job/Dispatcher.php(105): Omeka\Job\DispatchStrategy\Synchronous->send(Object(Omeka\Entity\Job))
#14 /var/www/html/omeka-s/application/data/scripts/perform-job.php(43): Omeka\Job\Dispatcher->send(Object(Omeka\Entity\Job), Object(Omeka\Job\DispatchStrategy\Synchronous))
#15 {main}
Next Doctrine\DBAL\Driver\PDOException: SQLSTATE[HY000]: General error: 1366 Incorrect string value: '\xC3' for column 'value' at row 1 in /var/www/html/omeka-s/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php:107
Stack trace:
#0 /var/www/html/omeka-s/vendor/doctrine/dbal/lib/Doctrine/DBAL/Statement.php(168): Doctrine\DBAL\Driver\PDOStatement->execute(NULL)
#1 /var/www/html/omeka-s/vendor/doctrine/orm/lib/Doctrine/ORM/Persisters/Entity/BasicEntityPersister.php(281): Doctrine\DBAL\Statement->execute()
#2 /var/www/html/omeka-s/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(1014): Doctrine\ORM\Persisters\Entity\BasicEntityPersister->executeInserts()
#3 /var/www/html/omeka-s/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(378): Doctrine\ORM\UnitOfWork->executeInserts(Object(Doctrine\ORM\Mapping\ClassMetadata))
#4 /var/www/html/omeka-s/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php(356): Doctrine\ORM\UnitOfWork->commit(NULL)
#5 /var/www/html/omeka-s/application/src/Api/Adapter/AbstractEntityAdapter.php(270): Doctrine\ORM\EntityManager->flush()
#6 /var/www/html/omeka-s/application/src/Api/Manager.php(224): Omeka\Api\Adapter\AbstractEntityAdapter->create(Object(Omeka\Api\Request))
#7 /var/www/html/omeka-s/application/src/Api/Manager.php(78): Omeka\Api\Manager->execute(Object(Omeka\Api\Request))
#8 /var/www/html/omeka-s/modules/CSVImport/src/Job/Import.php(346): Omeka\Api\Manager->create('items', Array)
#9 /var/www/html/omeka-s/modules/CSVImport/src/Job/Import.php(257): CSVImport\Job\Import->create(Array)
#10 /var/www/html/omeka-s/modules/CSVImport/src/Job/Import.php(196): CSVImport\Job\Import->processBatchData(Array)
#11 /var/www/html/omeka-s/application/src/Job/DispatchStrategy/Synchronous.php(37): CSVImport\Job\Import->perform()
#12 /var/www/html/omeka-s/application/src/Job/Dispatcher.php(105): Omeka\Job\DispatchStrategy\Synchronous->send(Object(Omeka\Entity\Job))
#13 /var/www/html/omeka-s/application/data/scripts/perform-job.php(43): Omeka\Job\Dispatcher->send(Object(Omeka\Entity\Job), Object(Omeka\Job\DispatchStrategy\Synchronous))
#14 {main}
Next Doctrine\DBAL\Exception\DriverException: An exception occurred while executing 'INSERT INTO value (type, lang, value, uri, resource_id, property_id, value_resource_id) VALUES (?, ?, ?, ?, ?, ?, ?)' with params ["literal", null, "\x72\x65\x73\x73\x6f\x75\x72\x63\x65\x20\x74\x65\x78\x74\x75\x65\x6c\x6c\x65\xc3", null, 93, 8, null]:
SQLSTATE[HY000]: General error: 1366 Incorrect string value: '\xC3' for column 'value' at row 1 in /var/www/html/omeka-s/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/AbstractMySQLDriver.php:115
Stack trace:
#0 /var/www/html/omeka-s/vendor/doctrine/dbal/lib/Doctrine/DBAL/DBALException.php(128): Doctrine\DBAL\Driver\AbstractMySQLDriver->convertException('An exception oc...', Object(Doctrine\DBAL\Driver\PDOException))
#1 /var/www/html/omeka-s/vendor/doctrine/dbal/lib/Doctrine/DBAL/Statement.php(177): Doctrine\DBAL\DBALException::driverExceptionDuringQuery(Object(Doctrine\DBAL\Driver\PDOMySql\Driver), Object(Doctrine\DBAL\Driver\PDOException), 'INSERT INTO val...', Array)
#2 /var/www/html/omeka-s/vendor/doctrine/orm/lib/Doctrine/ORM/Persisters/Entity/BasicEntityPersister.php(281): Doctrine\DBAL\Statement->execute()
#3 /var/www/html/omeka-s/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(1014): Doctrine\ORM\Persisters\Entity\BasicEntityPersister->executeInserts()
#4 /var/www/html/omeka-s/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(378): Doctrine\ORM\UnitOfWork->executeInserts(Object(Doctrine\ORM\Mapping\ClassMetadata))
#5 /var/www/html/omeka-s/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php(356): Doctrine\ORM\UnitOfWork->commit(NULL)
#6 /var/www/html/omeka-s/application/src/Api/Adapter/AbstractEntityAdapter.php(270): Doctrine\ORM\EntityManager->flush()
#7 /var/www/html/omeka-s/application/src/Api/Manager.php(224): Omeka\Api\Adapter\AbstractEntityAdapter->create(Object(Omeka\Api\Request))
#8 /var/www/html/omeka-s/application/src/Api/Manager.php(78): Omeka\Api\Manager->execute(Object(Omeka\Api\Request))
#9 /var/www/html/omeka-s/modules/CSVImport/src/Job/Import.php(346): Omeka\Api\Manager->create('items', Array)
#10 /var/www/html/omeka-s/modules/CSVImport/src/Job/Import.php(257): CSVImport\Job\Import->create(Array)
#11 /var/www/html/omeka-s/modules/CSVImport/src/Job/Import.php(196): CSVImport\Job\Import->processBatchData(Array)
#12 /var/www/html/omeka-s/application/src/Job/DispatchStrategy/Synchronous.php(37): CSVImport\Job\Import->perform()
#13 /var/www/html/omeka-s/application/src/Job/Dispatcher.php(105): Omeka\Job\DispatchStrategy\Synchronous->send(Object(Omeka\Entity\Job))
#14 /var/www/html/omeka-s/application/data/scripts/perform-job.php(43): Omeka\Job\Dispatcher->send(Object(Omeka\Entity\Job), Object(Omeka\Job\DispatchStrategy\Synchronous))
#15 {main}
2018-08-30T15:42:15+00:00 ERR (3): Doctrine\ORM\ORMException: The EntityManager is closed. in /var/www/html/omeka-s/vendor/doctrine/orm/lib/Doctrine/ORM/ORMException.php:144
Stack trace:
#0 /var/www/html/omeka-s/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php(741): Doctrine\ORM\ORMException::entityManagerClosed()
#1 /var/www/html/omeka-s/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php(354): Doctrine\ORM\EntityManager->errorIfClosed()
#2 /var/www/html/omeka-s/application/src/Api/Adapter/AbstractEntityAdapter.php(325): Doctrine\ORM\EntityManager->flush()
#3 /var/www/html/omeka-s/application/src/Api/Manager.php(227): Omeka\Api\Adapter\AbstractEntityAdapter->batchCreate(Object(Omeka\Api\Request))
#4 /var/www/html/omeka-s/application/src/Api/Manager.php(97): Omeka\Api\Manager->execute(Object(Omeka\Api\Request))
#5 /var/www/html/omeka-s/modules/CSVImport/src/Job/Import.php(369): Omeka\Api\Manager->batchCreate('csvimport_entit...', Array, Array, Array)
#6 /var/www/html/omeka-s/modules/CSVImport/src/Job/Import.php(257): CSVImport\Job\Import->create(Array)
#7 /var/www/html/omeka-s/modules/CSVImport/src/Job/Import.php(196): CSVImport\Job\Import->processBatchData(Array)
#8 /var/www/html/omeka-s/application/src/Job/DispatchStrategy/Synchronous.php(37): CSVImport\Job\Import->perform()
#9 /var/www/html/omeka-s/application/src/Job/Dispatcher.php(105): Omeka\Job\DispatchStrategy\Synchronous->send(Object(Omeka\Entity\Job))
#10 /var/www/html/omeka-s/application/data/scripts/perform-job.php(43): Omeka\Job\Dispatcher->send(Object(Omeka\Entity\Job), Object(Omeka\Job\DispatchStrategy\Synchronous))
#11 {main}
We checked many times that the CSV files are UTF-8 encoded, without BOM. And we tried other special characters, no other caused the same error message.
At first, the ‘value’ field of the VALUE table was set on utf8mb4_unicode_ci.
We changed it for utf8_unicode_ci and still got errors but the error code and log content is slightly different from before :
2018-08-31T09:12:10+00:00 ERR (3): PDOException: SQLSTATE[22001]: String data, right truncated: 1406 Data too long for column 'value' at row 1 in /var/www/html/omeka-s/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php:105
Stack trace:
#0 /var/www/html/omeka-s/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php(105): PDOStatement->execute(NULL)
#1 /var/www/html/omeka-s/vendor/doctrine/dbal/lib/Doctrine/DBAL/Statement.php(168): Doctrine\DBAL\Driver\PDOStatement->execute(NULL)
#2 /var/www/html/omeka-s/vendor/doctrine/orm/lib/Doctrine/ORM/Persisters/Entity/BasicEntityPersister.php(281): Doctrine\DBAL\Statement->execute()
#3 /var/www/html/omeka-s/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(1014): Doctrine\ORM\Persisters\Entity\BasicEntityPersister->executeInserts()
#4 /var/www/html/omeka-s/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(378): Doctrine\ORM\UnitOfWork->executeInserts(Object(Doctrine\ORM\Mapping\ClassMetadata))
#5 /var/www/html/omeka-s/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php(356): Doctrine\ORM\UnitOfWork->commit(NULL)
#6 /var/www/html/omeka-s/application/src/Api/Adapter/AbstractEntityAdapter.php(270): Doctrine\ORM\EntityManager->flush()
#7 /var/www/html/omeka-s/application/src/Api/Manager.php(224): Omeka\Api\Adapter\AbstractEntityAdapter->create(Object(Omeka\Api\Request))
#8 /var/www/html/omeka-s/application/src/Api/Manager.php(78): Omeka\Api\Manager->execute(Object(Omeka\Api\Request))
#9 /var/www/html/omeka-s/modules/CSVImport/src/Job/Import.php(346): Omeka\Api\Manager->create('items', Array)
#10 /var/www/html/omeka-s/modules/CSVImport/src/Job/Import.php(257): CSVImport\Job\Import->create(Array)
#11 /var/www/html/omeka-s/modules/CSVImport/src/Job/Import.php(196): CSVImport\Job\Import->processBatchData(Array)
#12 /var/www/html/omeka-s/application/src/Job/DispatchStrategy/Synchronous.php(37): CSVImport\Job\Import->perform()
#13 /var/www/html/omeka-s/application/src/Job/Dispatcher.php(105): Omeka\Job\DispatchStrategy\Synchronous->send(Object(Omeka\Entity\Job))
#14 /var/www/html/omeka-s/application/data/scripts/perform-job.php(43): Omeka\Job\Dispatcher->send(Object(Omeka\Entity\Job), Object(Omeka\Job\DispatchStrategy\Synchronous))
#15 {main}
Next Doctrine\DBAL\Driver\PDOException: SQLSTATE[22001]: String data, right truncated: 1406 Data too long for column 'value' at row 1 in /var/www/html/omeka-s/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php:107
Stack trace:
#0 /var/www/html/omeka-s/vendor/doctrine/dbal/lib/Doctrine/DBAL/Statement.php(168): Doctrine\DBAL\Driver\PDOStatement->execute(NULL)
#1 /var/www/html/omeka-s/vendor/doctrine/orm/lib/Doctrine/ORM/Persisters/Entity/BasicEntityPersister.php(281): Doctrine\DBAL\Statement->execute()
#2 /var/www/html/omeka-s/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(1014): Doctrine\ORM\Persisters\Entity\BasicEntityPersister->executeInserts()
#3 /var/www/html/omeka-s/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(378): Doctrine\ORM\UnitOfWork->executeInserts(Object(Doctrine\ORM\Mapping\ClassMetadata))
#4 /var/www/html/omeka-s/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php(356): Doctrine\ORM\UnitOfWork->commit(NULL)
#5 /var/www/html/omeka-s/application/src/Api/Adapter/AbstractEntityAdapter.php(270): Doctrine\ORM\EntityManager->flush()
#6 /var/www/html/omeka-s/application/src/Api/Manager.php(224): Omeka\Api\Adapter\AbstractEntityAdapter->create(Object(Omeka\Api\Request))
#7 /var/www/html/omeka-s/application/src/Api/Manager.php(78): Omeka\Api\Manager->execute(Object(Omeka\Api\Request))
#8 /var/www/html/omeka-s/modules/CSVImport/src/Job/Import.php(346): Omeka\Api\Manager->create('items', Array)
#9 /var/www/html/omeka-s/modules/CSVImport/src/Job/Import.php(257): CSVImport\Job\Import->create(Array)
#10 /var/www/html/omeka-s/modules/CSVImport/src/Job/Import.php(196): CSVImport\Job\Import->processBatchData(Array)
#11 /var/www/html/omeka-s/application/src/Job/DispatchStrategy/Synchronous.php(37): CSVImport\Job\Import->perform()
#12 /var/www/html/omeka-s/application/src/Job/Dispatcher.php(105): Omeka\Job\DispatchStrategy\Synchronous->send(Object(Omeka\Entity\Job))
#13 /var/www/html/omeka-s/application/data/scripts/perform-job.php(43): Omeka\Job\Dispatcher->send(Object(Omeka\Entity\Job), Object(Omeka\Job\DispatchStrategy\Synchronous))
#14 {main}
Next Doctrine\DBAL\Exception\DriverException: An exception occurred while executing 'INSERT INTO value (type, lang, value, uri, resource_id, property_id, value_resource_id) VALUES (?, ?, ?, ?, ?, ?, ?)' with params ["literal", null, "\x4d\x65\x6d\x6f\x72\x69\x61\x20\x63\x61\x74\x74\x6f\x6c\x69\x63\x61\x20\x64\x61\x20\x70\x72\x65\x73\x65\x6e\x74\x61\x72\x73\x69\x20\x61\x20\x53\x75\x61\x20\x53\x61\x6e\x74\x69\x74\xc3", null, 139, 1, null]:
SQLSTATE[22001]: String data, right truncated: 1406 Data too long for column 'value' at row 1 in /var/www/html/omeka-s/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/AbstractMySQLDriver.php:115
Stack trace:
#0 /var/www/html/omeka-s/vendor/doctrine/dbal/lib/Doctrine/DBAL/DBALException.php(128): Doctrine\DBAL\Driver\AbstractMySQLDriver->convertException('An exception oc...', Object(Doctrine\DBAL\Driver\PDOException))
#1 /var/www/html/omeka-s/vendor/doctrine/dbal/lib/Doctrine/DBAL/Statement.php(177): Doctrine\DBAL\DBALException::driverExceptionDuringQuery(Object(Doctrine\DBAL\Driver\PDOMySql\Driver), Object(Doctrine\DBAL\Driver\PDOException), 'INSERT INTO val...', Array)
#2 /var/www/html/omeka-s/vendor/doctrine/orm/lib/Doctrine/ORM/Persisters/Entity/BasicEntityPersister.php(281): Doctrine\DBAL\Statement->execute()
#3 /var/www/html/omeka-s/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(1014): Doctrine\ORM\Persisters\Entity\BasicEntityPersister->executeInserts()
#4 /var/www/html/omeka-s/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(378): Doctrine\ORM\UnitOfWork->executeInserts(Object(Doctrine\ORM\Mapping\ClassMetadata))
#5 /var/www/html/omeka-s/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php(356): Doctrine\ORM\UnitOfWork->commit(NULL)
#6 /var/www/html/omeka-s/application/src/Api/Adapter/AbstractEntityAdapter.php(270): Doctrine\ORM\EntityManager->flush()
#7 /var/www/html/omeka-s/application/src/Api/Manager.php(224): Omeka\Api\Adapter\AbstractEntityAdapter->create(Object(Omeka\Api\Request))
#8 /var/www/html/omeka-s/application/src/Api/Manager.php(78): Omeka\Api\Manager->execute(Object(Omeka\Api\Request))
#9 /var/www/html/omeka-s/modules/CSVImport/src/Job/Import.php(346): Omeka\Api\Manager->create('items', Array)
#10 /var/www/html/omeka-s/modules/CSVImport/src/Job/Import.php(257): CSVImport\Job\Import->create(Array)
#11 /var/www/html/omeka-s/modules/CSVImport/src/Job/Import.php(196): CSVImport\Job\Import->processBatchData(Array)
#12 /var/www/html/omeka-s/application/src/Job/DispatchStrategy/Synchronous.php(37): CSVImport\Job\Import->perform()
#13 /var/www/html/omeka-s/application/src/Job/Dispatcher.php(105): Omeka\Job\DispatchStrategy\Synchronous->send(Object(Omeka\Entity\Job))
#14 /var/www/html/omeka-s/application/data/scripts/perform-job.php(43): Omeka\Job\Dispatcher->send(Object(Omeka\Entity\Job), Object(Omeka\Job\DispatchStrategy\Synchronous))
#15 {main}
2018-08-31T09:12:10+00:00 ERR (3): Doctrine\ORM\ORMException: The EntityManager is closed. in /var/www/html/omeka-s/vendor/doctrine/orm/lib/Doctrine/ORM/ORMException.php:144
Stack trace:
#0 /var/www/html/omeka-s/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php(741): Doctrine\ORM\ORMException::entityManagerClosed()
#1 /var/www/html/omeka-s/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php(354): Doctrine\ORM\EntityManager->errorIfClosed()
#2 /var/www/html/omeka-s/application/src/Api/Adapter/AbstractEntityAdapter.php(325): Doctrine\ORM\EntityManager->flush()
#3 /var/www/html/omeka-s/application/src/Api/Manager.php(227): Omeka\Api\Adapter\AbstractEntityAdapter->batchCreate(Object(Omeka\Api\Request))
#4 /var/www/html/omeka-s/application/src/Api/Manager.php(97): Omeka\Api\Manager->execute(Object(Omeka\Api\Request))
#5 /var/www/html/omeka-s/modules/CSVImport/src/Job/Import.php(369): Omeka\Api\Manager->batchCreate('csvimport_entit...', Array, Array, Array)
#6 /var/www/html/omeka-s/modules/CSVImport/src/Job/Import.php(257): CSVImport\Job\Import->create(Array)
#7 /var/www/html/omeka-s/modules/CSVImport/src/Job/Import.php(196): CSVImport\Job\Import->processBatchData(Array)
#8 /var/www/html/omeka-s/application/src/Job/DispatchStrategy/Synchronous.php(37): CSVImport\Job\Import->perform()
#9 /var/www/html/omeka-s/application/src/Job/Dispatcher.php(105): Omeka\Job\DispatchStrategy\Synchronous->send(Object(Omeka\Entity\Job))
#10 /var/www/html/omeka-s/application/data/scripts/perform-job.php(43): Omeka\Job\Dispatcher->send(Object(Omeka\Entity\Job), Object(Omeka\Job\DispatchStrategy\Synchronous))
#11 {main}
We did a few tests with other special characters. They all work well, except for “à” and “À”. And those are definitely causing an error when the character is located left to a value separator. In our case, the | character for multivaluation and the ; for the main separator.
You can find the original CSV file here :
https://filesender.renater.fr/?s=download&token=c696c8e4-7090-b9a4-c366-a39c1a7613b4
We are working on Omeka-S 1.2.0 with the CSVImport 1.1.2 module.
In addition, we tried this on a former installation (Omeka-S 1.1.0 and CSVImport 1.1.1) and the character was simply deleted, without a warning or error message. But this particular character was causing issue on the “old” configuration as well.
Is the original poster still having the mentionned issues ?
Thank in advance for your help and good day to all.