I used the following comnands in neo4j, but the system always responds the following error message.
"Couldn't load the external resource at: file:/import/Tokyo_subway_system.csv ()"
Here is my script:
load csv with headers from "file:///Tokyo_subway_system.csv" as csvLine
create (s:Station {id: toInteger(csvLine.id), station_No: csvLine.station_No, station_Name: csvLine.station_Name, station_English: csvLine.station_English, line_Name: csvLine.line_Name ,line_English: csvLine.line_English, latitude: csvLine.latitude, longitade: csvLine.longitade})
Find your $NEO4J_HOME/import/ folder in your server or local directory. Then copy that file Tokyo_subway_system.csv in that directory. If you have multiple versions of neo4j installed, ensure that you are on the right neo4j home directory.
Related
I am trying to load data into Redshift using a Firehose delivery stream.
I am using a jsonpaths file uploaded to S3 at the following location.
s3://my_bucket/jsonpaths.json
This file contains the following jsonpaths config
{
"jsonpaths": [
"$['col_1']",
"$['col_2']",
"$['col_3']",
"$['col_4']"
]
}
To me this config looks ok, but the Firehose Redshift logs keep showing the following error.
"The provided jsonpaths file is not in a supported JSON format."
A similar error is seen even if I run the following copy command directly on the Redshift cluster.
reshift_db=# COPY my_schema.my_table
FROM 's3://my_bucket/data.json'
FORMAT JSON 's3://my_bucket/jsonpaths.json'
CREDENTIALS 'aws_iam_role=<role_arn>'
;
ERROR: Manifest file is not in correct json format
DETAIL:
-----------------------------------------------
error: Manifest file is not in correct json format
code: 8001
context: Manifest file location = s3://my_bucket/jsonpaths.json
query: yyyyy
location: s3_utility.cpp:338
process: padbmaster [pid=xxxxx]
-----------------------------------------------
Can someone help with what is going wrong here?
The problem in my case was a BOM (Byte Order Mark) at the beginning of the jsonpaths file. Some editors can save a file with BOM, and this does not show as characters when seen in the editor. And apparently Redshift does not like BOM at the beginning of the jsonpaths file.
For those of you who want to check if this is the case for your jsonpaths file, you can open the file in a hex editor. For the S3 file this can be done as follows.
# aws s3 cp s3://my_bucket/jsonpaths.json - | hexdump -C
To remove the BOM from the file you can do the following.
# aws s3 cp s3://my_bucket/jsonpaths.json - | dos2unix | aws s3 cp - s3://my_bucket/jsonpaths.json
Almost after 2 days of trying, and after having raised an AWS Support ticket, and having posted this question, it struct me that I should check the file in a hex editor.
I working on Flume to append the data from a local directory to HDFS using Flume Source TAILDIR.
My use case is to do Delta Load If the new line comes in the source file in local dir so that will append in hdfs.
This is my Flume Conf file :
#configure the agent
agent.sources=r1
agent.channels=k1
agent.sinks=c1
agent.sources.r1.type=TAILDIR
agent.sources.r1.positionFile = /home/flume/Documents/taildir_position.json
agent.sources.r1.filegroups=f1
agent.sources.r1.filegroups.f1=/home/flume/Documents/spooldir/
agent.sources.r1.batchSize = 20
agent.sources.r1.writePosInterval=2000
agent.sources.r1.maxBackoffSleep=5000
agent.sources.r1.fileHeader = true
agent.sources.r1.channels=k1
agent.channels.k1.type=memory
agent.channels.k1.capacity=10000
agent.channels.k1.transactionCapacity=1000
agent.sinks.c1.type=hdfs
agent.sinks.c1.channel=k1
agent.sinks.c1.hdfs.path=hdfs://localhost:8020/flume_sink
agent.sinks.c1.hdfs.batchSize = 1000
agent.sinks.c1.hdfs.rollSize = 268435456
agent.sinks.c1.hdfs.writeFormat=Text
while running flume command : flume-ng agent -n agent -c conf -f /home/swechchha/Documents/flumereal.conf
I am getting error
I am getting error to load JSON file.
Here is the code. It crashes at the line 110. Please make sure that flume user has access to that JSON file and that the file is correctly formatted.
The Flume.conf mentioned in Question Statement is having a problem.
TAILDIR SOURCE: Watch the specified files, and tail them in nearly real-time once detected new lines appended to each files. If the new lines are being written, this source will retry reading them in wait for the completion of the write.
While writing filegroups property directory may contain multiple files in this case it should be mentioned like directory path/ .filestype.
agent.sources.r1.filegroups.f1=/home/flume/Documents/spooldir/.*txt.*
Then run flume.conf and check the result it will work fine.
In JMeter I am passing multiple JSON inputs as body, Variable name is defined as JSON_FILE and coming from CSV Data Config
${__FileToString(${__eval(${JSON_FILE})}.json,,)}
CSV Data
designO1015643320
.
.
designO1077673985
designO1088516727
Running load test from Jmeter UI works fine, but running as mvn project is giving error about FileNotFoundException even though .csv file and .json files are in same folder as .jmx file
Error from .jmx.log:
WARN - jmeter.functions.FileToString: Could not read file: designO1015643320.json File 'designO1015643320.json' does not exist java.io.FileNotFoundException: File 'designO1015643320.json' does not exist
Response in .jtl:
httpSample t="4" lt="0" ts="1508530091457" s="false" lb="CreateDesign_PUT" rc="Non HTTP response code: org.apache.jorphan.util.JMeterStopThreadException" rm="Non HTTP response message: End of sequence" tn="Design_APIs 1-1" dt="text" by="1822" ng="1" na="1"/>
JMeter GUI default relative path is the bin folder
Relative paths are resolved relative to the current working directory (which defaults to the bin/ directory).
Maven search in different default path for files src/test/jmeter directory
See guide:
in the src/test/jmeter directory. When running the project, the JMeter Maven plugin searches for tests to run in this directory.
And you can find this path dynamically
I heard Groovy is a new black so I would recommend replacing your __FileToString() function with __groovy() function, the Groovy equivalent of dynamically getting the file path relative to Maven's plugin current working directory would be something like:
${__groovy(new File(org.apache.jmeter.services.FileServer.getFileServer().getBaseDir() + System.getProperty('file.separator') + vars.get('JSON_FILE') + '.json').text,)}
See JavaDoc on FileServer class for more details.
I am using Neo4j3.0.1 and for loading a csv file
LOAD CSV WITH HEADERS FROM 'file:///D:/dummy.csv' as line
CREATE (:myData {line})
But it throws an error :
Couldn't load the external resource at: file:/D:/dummy.csv
Note : I've already tried configuring neo4j.conf which was described here
Suggest any other alternative besides placing csv file into import folder.
Try setting dbms.directories.import to D: in neoj4.conf
dbms.directories.import=D:
and after run
LOAD CSV WITH HEADERS FROM 'file:///dummy.csv' as line
CREATE (:myData {line})
EDIT:
As shown in comments the problem was solved by changing the owner of the CSV file location directory, as described in this answer.
sudo chown neo4j:adm <csv file location>
I'm importing csv using
USING PERIODIC COMMIT
LOAD CSV WITH HEADERS FROM "file:///root/rahul/Neo4jData/after_etl_data.csv" AS row
CREATE (:AFTER_ETL_DATA {user_tweet_id:row.user_tweet_id});
I'm getting the following error
QueryExecutionKernelException: Couldn't load the external resource at:
file:/root/rahul/Neo4jData/after_etl_data.csv
Things I have done
1) Changed the permission of files to
777
2) Chnaged the owner of the file
ie -rwxrwxrwx 1 neo4j adm 553942876 Sep 12 13:54 after_etl_data.csv
3) Added the line
dbms.security.allow_csv_import_from_file_urls=true
in /etc/neo4j/neo4j-server.properties
I'm using neo4j 2.2.5
I dont know how to solve this.
Note - However if I start my shell using
./neo4j-shell -path graph.db -config
/var/lib/neo4j/conf/neo4j.properties
I'm able to insert the data , but since it starts in local mode , I'm unable to view the data in Neo4j UI Interface.