Importing 64k tupples into neo4j - csv

Question

I'm trying to import roughly 64 thousand rows into a neo4j graph. During the import I'm converting some attributes to relations as these are being used by other fields as well with a merge.
This is my cypher query:
USING PERIODIC COMMIT 150
LOAD CSV WITH HEADERS FROM "http://example.com/some.csv" as csvline
MERGE (gem:Gemeente { name: csvline.GEMEENTE})
MERGE (cbs:CBS { name: csvline.CBSCODE})
CREATE (obj:Object { id: toInt(csvline.NUMMER),
prop2: toInt(csvline.PROP2)
})
CREATE (obj)-[:IN_GEMEENTE]->(gem)
CREATE (obj)-[:CBS_CODE]->(cbs)
When I manually truncate the csv-file to 10 rows; this cypher runs perfectly. I'm getting a nice graph with the appropriate relationships.
But running the Cypher-script for every row in my csv-file the server just stalls with an error/warning.
Within the dashboard at 7474 I'm just getting a plain simple error, without any information. While in the neo4j shell I'm getting the following error:
Error occurred in server thread; nested exception is:
java.lang.OutOfMemoryError: Java heap space
So it appears I'm running out of memory. So I tried to reduce the commit number; but this has no effect.
Off course I have a indexes on both :Gemeente(naam) and :CBS(naam)
A solution could be to split up the file in 'affordable' chunks; but that's off course a lot of work :) And not a real solution.
How can I resolve this issue?

Answer 1

You're probably running into the "eager" issue. It's discussed in these posts:
http://jexp.de/blog/2014/10/load-cvs-with-success/
http://www.markhneedham.com/blog/2014/10/23/neo4j-cypher-avoiding-the-eager/
It will probably work better like this:
USING PERIODIC COMMIT 150
LOAD CSV WITH HEADERS FROM "http://example.com/some.csv" as csvline
MERGE (gem:Gemeente { name: csvline.GEMEENTE});
USING PERIODIC COMMIT 150
LOAD CSV WITH HEADERS FROM "http://example.com/some.csv" as csvline
MERGE (cbs:CBS { name: csvline.CBSCODE});
USING PERIODIC COMMIT 150
LOAD CSV WITH HEADERS FROM "http://example.com/some.csv" as csvline
CREATE (obj:Object { id: toInt(csvline.NUMMER),
prop2: toInt(csvline.PROP2)
})
MATCH
(gem:Gemeente { name: csvline.GEMEENTE}),
(cbs:CBS { name: csvline.CBSCODE})
CREATE (obj)-[:IN_GEMEENTE]->(gem)
CREATE (obj)-[:CBS_CODE]->(cbs)
You may not need to split it up as much as that, though. Also, since you'd need to load the csv file at least twice, you might want to lave it locally and run the CSV import from disk. The syntax is LOAD CSV WITH HEADERS FROM "file:///path/to/file" as csvline (I had lots of trouble finding an example when I first tried it. Itsfile://` followed by the path. My example is a unix path, but that can also be followed by a windows path, I believe)