CSV Import module: issues when appending data to resources

[steasdale]

Good afternoon

I have been trying to use the CSV Import feature and have run into a problem with importing CSV files that append data to existing resources. Here is an outline of the process and the problems that I am experiencing.

STAGE 1:

I imported a CSV file that described the identities of about 3000 persons. No problems here… everything worked fine. Here is an example imported item with the relevant properties, which all imported properly.

STAGE 2:

I imported another CSV file that identified the fathers for about 2000 of these 3000 persons.

First, I mapped the column that contained the Omeka Resource ID for these fathers in the “Map to Omeka S Data” tab.

Second, I selected the relevant Resource Template and Class in the “Basic Settings” tab.

Third, I selected the “Append data to resource” Action and selected the relevant column (that contained the UUID) for the Resource identifier column and selected the relevant property for the Resource identifier property .

Again, everything worked out fine and the fathers were imported and displayed as linked items. Here is the example item above after second CSV import with the father linked to the Parent(s) property.

STAGE 3:

This is where I am running into problems.

Again, I mapped the column that contained the Omeka Resource ID for these children in the “Map to Omeka S Data” tab.

Second, I selected the relevant Resource Template and Class in the “Basic Settings” tab.

Third, I selected the “Append data to resource” Action and selected the relevant column (that contained the UUID) for the Resource identifier column and selected the relevant property for the Resource identifier property .

Now, when I look at the item there are two problems.

PROBLEM 1:

If the father had one or two children, then these were imported and displayed as linked items. However, the previous link to the Parent(s) has disappeared. Here is the sample item after the third CSV Import:

As shown, the child relationship (in the Has children property ) has been imported properly, but the father relationship (in the Parent(s) property ) has disappeared.

At first, I thought that there might be a problem in the Resource Template, in that the properties might be incorrectly defined or somehow overlapping. But this is not the case (see below).

Furthermore, there are no issues if I manually add a parent item the Parent(s) property afterwards, as shown below.

This is how it should look after importing and appending the child relationships. However, we do not want to do this manually for 2000+ items, especially since they are already accurately encoded in the CSV.

PROBLEM 2:

Along with the disappearance of the previous link to the Parent(s) property, there is another issue. If the father had three or more children (each defined by a separate row in the CSV), then only the final two of them are imported and displayed as linked items for the Has Children property. For example, the CSV defines three children as shown below.

However, here is the sample item after importing this CSV, showing how only the final two rows were imported properly (as well as noting how the Parent(s) property has disappeared):

Any insights into what may be causing these issues would be greatly appreciated! I have loads of CSV data that needs to imported into Omeka.

Thanks in advance,
Steven Teasdale

[jflatnes]

Can you specify which versions of Omeka S and CSV Import you’re using?

[steasdale]

Here are the versions that I am using

CSV Import: version 2.2.1
Omeka-S: version 3.0.1

Thanks!

[jflatnes]

WIth a very small test case (just two columns, but similarly updating the same one item on all three rows, looking up both the item to update and the items to link to by internal ID, using the “append” mode) I wasn’t able to reproduce either of your problems: all three Resource values were imported successfully the first pass, and all three successfully imported to a different property on the second, with no existing data changed.

Do you have the same issue with a similarly pared-down test case (just the two necessary columns and the data for a single item)?

[steasdale]

Good afternoon

I created a similarly pared-down test case, with five individuals and two columns. The individuals are a man, his father, and his three daughters.

– Man
– Father
– Daughter 1
– Daughter 2
– Daughter 3

The CSV file for the individuals is shown below.

omeka-ID,cidoc-crm:P149_is_identified_by,cidoc-crm:P48_has_preferred_identifier
25916,Person1,Man
25917,Person2,Father
25918,Person3,Daughter1
25919,Person4,Daughter2
25920,Person5,Daughter3

The CSV file for the parent relationship (that is, Person1 —hasParent—> Person2) is shown below.

cidoc-crm:P149_is_identified_by,cidoc-crm:P152_has_parent
Person1,25917

The CSV file for the daughter relationship (that is, Person1 —hasDaughter—> Person3, etc.) is shown below.

cidoc-crm:P149_is_identified_by,cidoc-crm:P108_has_produced
Person1,25918
Person1,25919
Person1,25920

Now, when I import the five individuals from the first CSV (that is, the first pass of the CSV import) everything is fine and they populate my item list appropriately.

Next, when I import the parent relationship from the second CSV (that is, the second pass of the CSV import) everything is again fine and the parent relation appears in the item entry for “Man”

Finally, when I import the parent relationship from the third CSV (the third pass of the CSV import) we have the same problem I described earlier. That is, the Parent relationship is deleted from the “Man” item and only the final two of the three Daughter relationships are imported.

Unfortunately this forum is only allowing me to upload a single image in my reply, so I have only included the final screenshot for the “Man” item.

So, the same issue appears in my test set as well. I’m not sure why this might be happening, as it seems to be “partly” working and there are no errors reported in the log file.

Thanks in advance,
Steven Teasdale

[jflatnes]

I tried again to reproduce this, following more closely to your description, and still am not able to.

After 2 “append” imports all targeting the same item using Resource values like this, both sets are preserved and there’s no issue on either import with doing 3 values.

Where did you get the module from? Just the omeka.org website? Trying to make sure we’re actually using the same version.

[steasdale]

Good afternoon

I had obtained the module from the Omeka website.

I de-installed and installed the CSV-Import module version 2.2.0 and the same issue occurred.
I then reinstalled CSV-Import module version 2.2.1 from the Omeka website and the issue continued.
I then installed the version from the CSV-Import Github and the issue still remains.

I’m at a loss of what is causing this. I even, in desperation, tried modifying the resource template, but the issue still occurs.

[steasdale]

Some more information…

For another step I added some more items

omeka-ID,cidoc-crm:P149_is_identified_by,cidoc-crm:P48_has_preferred_identifier
25916,Person1,Man
25917,Person2,Father
25918,Person3,Daughter1
25919,Person4,Daughter2
25920,Person5,Daughter3
25921,Person6,Daughter4
25930,Person7,Daughter5
25931,Person8,Daughter6

In this case, when I import the CSV for the daughter files shown below

cidoc-crm:P149_is_identified_by,cidoc-crm:P108_has_produced
Person1,25918
Person1,25919
Person1,25920
Person1,25921
Person1,25930
Person1,25331

Then only the final item 25331 (Daughter6) is shown as a daughter for item 25916 (Man).

Oddly enough, if I then re-do the CSV import, then item 25531 (Daughter6) disappears and is replaced by 25330 (Daughter5) when viewing item 25916 (Man).

Might there be some kind of recursion problem in importing a CSV with the same identifier specified for the “Resource identifier column”?

[jflatnes]

You have all the IDs correct, yes? I’m just noting that you’re listing some seemingly incorrect ID numbers in your last post here.

As for issues with specifying the same identifier over and over, that was my suspicion as to what could be going on, but that’s exactly what I’ve been doing in my attempts to reproduce your problem, without success.

The only things I can think of are that it could be a bad interaction with some other module (you could try deactivating your other modules) or that it’s some issue with the specific way you’re mapping these CSVs in the importer, but I can’t think of anything that would cause this.

[jflatnes]

In the realm of workarounds, you can avoid specifying the same item multiple times by using the “multivalue” feature: specifying all the IDs to link to in one cell separated by a character (comma or semicolon are typical), then filling in that character to the mapping settings and choosing the “Multivalue” option for the column. That way you only have one row per item being updated.

Though it wouldn’t point toward whatever is actually happening in your situation here, that may resolve your problem.

[steasdale]

Thanks for your suggestions!

It was indeed a bad interaction with another module. In this case, the Bulk Edit module, version 3.3.12.4. When it’s disabled, everything works great. I tested this three times… . each time the Bulk Edit module is enabled, I run into the importing issues described above.

It didn’t occur to me to disable the other modules to check for bad interactions. Sorry for all the work this caused you.

Is there a way of entering an exception or ticket regarding this issue? I would like to have use of the Bulk Edit module in the future.

Once again, thanks for your assistance.
Steven Teasdale

[jflatnes]

OK, well that’s interesting.

You can file issues with Bulk Edit, CSV Import, or both on their respective Github repositories (should be linked off the module pages). I’m not sure which side is “at fault” here (if either is, really).