I am using Neo4j3.0.1 and for loading a csv file
LOAD CSV WITH HEADERS FROM 'file:///D:/dummy.csv' as line
CREATE (:myData {line})
But it throws an error :
Couldn't load the external resource at: file:/D:/dummy.csv
Note : I've already tried configuring neo4j.conf which was described here
Suggest any other alternative besides placing csv file into import folder.
Try setting dbms.directories.import to D: in neoj4.conf
dbms.directories.import=D:
and after run
LOAD CSV WITH HEADERS FROM 'file:///dummy.csv' as line
CREATE (:myData {line})
EDIT:
As shown in comments the problem was solved by changing the owner of the CSV file location directory, as described in this answer.
sudo chown neo4j:adm <csv file location>
Related
Load data infile has always been a pain in the ass to set up and there does not seem to be standard way to get it to work but I have always been able to get it to work ….eventually except now.
I am struggling with
Error Code: 29. File '/loadinfile/file.csv' not found (OS errno 13 - Permission denied)
The system is as follows:
MySQL version 8.0.29-0ubuntu0.22.04.2
Ubuntu 22.04
The load file directory is owned by mysql:mysql and same with the file and I have tried chmod 757 and even 777
In the config file I have tried the following variations:
secure_file_priv = "/loadinfile/"
secure_file_priv = '/loadinfile/'
secure_file_priv = ""
secure_file_priv = ''
The following is also set :
[mysqld]
local-infile = 1
[client]
loose-local-infile=1
Changing
LOAD DATA INFILE '/loadinfile/file.csv' INTO TABLE table
to
LOAD DATA LOCAL INFILE '/loadinfile/file.csv' INTO TABLE table
changes the error message to:
Error Code: 2. File '\loadinfile\file.csv' not found (OS errno 2 - No such file or directory)
Error Code: 2. File '\loadinfile\file.csv' not found (OS errno 2 - No such file or directory)
Above message indicates that your directory where the file is located is wrong! You must define the full path:
For example In Mac If your file resides on Desktop,It must be like this :
LOAD DATA LOCAL INFILE '/Users/computer_name/Desktop/filename.file_extension' INTO TABLE table
In windows
LOAD DATA LOCAL INFILE 'C:\Users\computer_name\Desktop\filename.file_extension' INTO TABLE table
I hope It solves your problem.
Error Code: 2. indicates invalid file path. Keep the LOCAL INFILE option and either specify the full path of the CSV file or make sure the CSV file path is correct. give it a try with Full path and see if it solves the problem or not.
From MySQL Docs
If LOCAL is specified, the file must be located on the client host.
The client program reads the file, locating it as follows:
If the file name is an absolute path name, the client program uses it
as given.
If the file name is a relative path name, the client program looks for
the file relative to its invocation directory
Edited
Make sure your mysqld.cnf or your main MySQL config file has the following variables
secure_file_priv = ""
local_infile=1
And your Global variable local_infile is OFF, turn it ON.
you can check that if you run this query
SHOW GLOBAL VARIABLES LIKE 'local_infile';
if it is OFF, turn it ON
And lastly, Make sure you restart your SQL service to reflect the changes. you can do that if you execute
service mysql restart
Hope that solves your problem
I used the following comnands in neo4j, but the system always responds the following error message.
"Couldn't load the external resource at: file:/import/Tokyo_subway_system.csv ()"
Here is my script:
load csv with headers from "file:///Tokyo_subway_system.csv" as csvLine
create (s:Station {id: toInteger(csvLine.id), station_No: csvLine.station_No, station_Name: csvLine.station_Name, station_English: csvLine.station_English, line_Name: csvLine.line_Name ,line_English: csvLine.line_English, latitude: csvLine.latitude, longitade: csvLine.longitade})
Find your $NEO4J_HOME/import/ folder in your server or local directory. Then copy that file Tokyo_subway_system.csv in that directory. If you have multiple versions of neo4j installed, ensure that you are on the right neo4j home directory.
I am trying to load data into Redshift using a Firehose delivery stream.
I am using a jsonpaths file uploaded to S3 at the following location.
s3://my_bucket/jsonpaths.json
This file contains the following jsonpaths config
{
"jsonpaths": [
"$['col_1']",
"$['col_2']",
"$['col_3']",
"$['col_4']"
]
}
To me this config looks ok, but the Firehose Redshift logs keep showing the following error.
"The provided jsonpaths file is not in a supported JSON format."
A similar error is seen even if I run the following copy command directly on the Redshift cluster.
reshift_db=# COPY my_schema.my_table
FROM 's3://my_bucket/data.json'
FORMAT JSON 's3://my_bucket/jsonpaths.json'
CREDENTIALS 'aws_iam_role=<role_arn>'
;
ERROR: Manifest file is not in correct json format
DETAIL:
-----------------------------------------------
error: Manifest file is not in correct json format
code: 8001
context: Manifest file location = s3://my_bucket/jsonpaths.json
query: yyyyy
location: s3_utility.cpp:338
process: padbmaster [pid=xxxxx]
-----------------------------------------------
Can someone help with what is going wrong here?
The problem in my case was a BOM (Byte Order Mark) at the beginning of the jsonpaths file. Some editors can save a file with BOM, and this does not show as characters when seen in the editor. And apparently Redshift does not like BOM at the beginning of the jsonpaths file.
For those of you who want to check if this is the case for your jsonpaths file, you can open the file in a hex editor. For the S3 file this can be done as follows.
# aws s3 cp s3://my_bucket/jsonpaths.json - | hexdump -C
To remove the BOM from the file you can do the following.
# aws s3 cp s3://my_bucket/jsonpaths.json - | dos2unix | aws s3 cp - s3://my_bucket/jsonpaths.json
Almost after 2 days of trying, and after having raised an AWS Support ticket, and having posted this question, it struct me that I should check the file in a hex editor.
I working on Flume to append the data from a local directory to HDFS using Flume Source TAILDIR.
My use case is to do Delta Load If the new line comes in the source file in local dir so that will append in hdfs.
This is my Flume Conf file :
#configure the agent
agent.sources=r1
agent.channels=k1
agent.sinks=c1
agent.sources.r1.type=TAILDIR
agent.sources.r1.positionFile = /home/flume/Documents/taildir_position.json
agent.sources.r1.filegroups=f1
agent.sources.r1.filegroups.f1=/home/flume/Documents/spooldir/
agent.sources.r1.batchSize = 20
agent.sources.r1.writePosInterval=2000
agent.sources.r1.maxBackoffSleep=5000
agent.sources.r1.fileHeader = true
agent.sources.r1.channels=k1
agent.channels.k1.type=memory
agent.channels.k1.capacity=10000
agent.channels.k1.transactionCapacity=1000
agent.sinks.c1.type=hdfs
agent.sinks.c1.channel=k1
agent.sinks.c1.hdfs.path=hdfs://localhost:8020/flume_sink
agent.sinks.c1.hdfs.batchSize = 1000
agent.sinks.c1.hdfs.rollSize = 268435456
agent.sinks.c1.hdfs.writeFormat=Text
while running flume command : flume-ng agent -n agent -c conf -f /home/swechchha/Documents/flumereal.conf
I am getting error
I am getting error to load JSON file.
Here is the code. It crashes at the line 110. Please make sure that flume user has access to that JSON file and that the file is correctly formatted.
The Flume.conf mentioned in Question Statement is having a problem.
TAILDIR SOURCE: Watch the specified files, and tail them in nearly real-time once detected new lines appended to each files. If the new lines are being written, this source will retry reading them in wait for the completion of the write.
While writing filegroups property directory may contain multiple files in this case it should be mentioned like directory path/ .filestype.
agent.sources.r1.filegroups.f1=/home/flume/Documents/spooldir/.*txt.*
Then run flume.conf and check the result it will work fine.
In JMeter I am passing multiple JSON inputs as body, Variable name is defined as JSON_FILE and coming from CSV Data Config
${__FileToString(${__eval(${JSON_FILE})}.json,,)}
CSV Data
designO1015643320
.
.
designO1077673985
designO1088516727
Running load test from Jmeter UI works fine, but running as mvn project is giving error about FileNotFoundException even though .csv file and .json files are in same folder as .jmx file
Error from .jmx.log:
WARN - jmeter.functions.FileToString: Could not read file: designO1015643320.json File 'designO1015643320.json' does not exist java.io.FileNotFoundException: File 'designO1015643320.json' does not exist
Response in .jtl:
httpSample t="4" lt="0" ts="1508530091457" s="false" lb="CreateDesign_PUT" rc="Non HTTP response code: org.apache.jorphan.util.JMeterStopThreadException" rm="Non HTTP response message: End of sequence" tn="Design_APIs 1-1" dt="text" by="1822" ng="1" na="1"/>
JMeter GUI default relative path is the bin folder
Relative paths are resolved relative to the current working directory (which defaults to the bin/ directory).
Maven search in different default path for files src/test/jmeter directory
See guide:
in the src/test/jmeter directory. When running the project, the JMeter Maven plugin searches for tests to run in this directory.
And you can find this path dynamically
I heard Groovy is a new black so I would recommend replacing your __FileToString() function with __groovy() function, the Groovy equivalent of dynamically getting the file path relative to Maven's plugin current working directory would be something like:
${__groovy(new File(org.apache.jmeter.services.FileServer.getFileServer().getBaseDir() + System.getProperty('file.separator') + vars.get('JSON_FILE') + '.json').text,)}
See JavaDoc on FileServer class for more details.