Reading multiple files in Python from Excel csv - csv

I am attempting to read files from my computer using os in python 2.7
import os
path = 'C:\Users\Owner\OneDrive\_Program\Stocks\AMEX'
for filename in os.listdir(path):
print(filename)
data1=os.open(filename,1)
the program can see the files and list them for me so i know they are there. however when i try to open them with an open statement or a read statement (not shown) it does not do it.
OSError: [Errno 2] No such file or directory: 'AMEX_20170608.csv'
I would like to open and read each file and then perform some functions. I have found several similar questions but i am on a windows machine and they do not seem to work.
has anyone encountered this before?
Thanks in advance.

You need to put the path back in before you open the file
import os
path = 'C:/Users/Owner/OneDrive/_Program/Stocks/AMEX'
for filename in os.listdir(path):
print(filename)
data1=os.open(path+'/'+filename,1)

Related

Import .csv file to EEGLAB

I have downloaded and installed EEGLAB plugin in matlab.
I am trying to import .csv file into EEGLab. I am unable to do it because the tool looks for .SET files.
I tried installing some plugins like musemonitor which accepts .csv files as input.
After importing a .csv file, nothing happens.
Kindly let me know what I am missing here? Can EEGLab handle .csv files that are recorded using an Emotiv Headset ?

python3 when saving a file it doesn't save to same directory as the EXE pyinstaller

I have used Pyinstaller to pack my script as onedir EXE, then installed to Program Files folder. i used this line to save a json to a file:
json.dump(data, open("data_file.json",'w'))
on some computers the file saved to where EXE is which is what i want, but others save it to different locations.
so when i try to get the file using json.load it says not file found:
json.load(open("data_file.json"), encoding="utf-8")
How can i solve this issue so the script only save files to the same EXE directory?
The cause of the problem is related to writing permission on windows. so instead of writing app data to the Program Files folder which is not recommended, you need to write the data to APPDATA folder.
i have used the code posted by giuliano-oliveira and it works perfectly cross platform.
get_user_data_dir

Trying to knit Rmarkdown but getting pandoc.utils error or pandoc error 83

I am new to Rmarkdown and trying to learn how to knit a code. when I run the sample code and knit, it gives me the following error:
*
Error running filter C:/Users/bbalamdari/Documents/R/win-library/3.6/rmarkdown/rmd/lua/pagebreak.lua:
...uments/R/win-library/3.6/rmarkdown/rmd/lua/pagebreak.lua:18: module 'pandoc.utils' not found:
no field package.preload['pandoc.utils']
no file 'C:\PROGRA~2\Pandoc\lua\pandoc\utils.lua'
no file 'C:\PROGRA~2\Pandoc\lua\pandoc\utils\init.lua'
no file 'C:\PROGRA~2\Pandoc\pandoc\utils.lua'
no file 'C:\PROGRA~2\Pandoc\pandoc\utils\init.lua'
no file 'C:\PROGRA~2\Pandoc\..\share\lua\5.3\pandoc\utils.lua'
no file 'C:\PROGRA~2\Pandoc\..\share\lua\5.3\pandoc\utils\init.lua'
no file '.\pandoc\utils.lua'
no file '.\pandoc\utils\init.lua'
no file 'C:\PROGRA~2\Pandoc\pandoc\utils.dll'
no file 'C:\PROGRA~2\Pandoc\..\lib\lua\5.3\pandoc\utils.dll'
no file 'C:\PROGRA~2\Pandoc\loadall.dll'
no file '.\pandoc\utils.dll'
no file 'C:\PROGRA~2\Pandoc\pandoc.dll'
no file 'C:\PROGRA~2\Pandoc\..\lib\lua\5.3\pandoc.dll'
no file 'C:\PROGRA~2\Pandoc\loadall.dll'
no file '.\pandoc.dll'
Error: pandoc document conversion failed with error 83
Execution halted
*
I thought perhaps I should install package"Pandoc.utils" but once I called the package in R, it gave this error:
package ‘pandoc.utils’ is not available (for R version 3.6.0)
Is this because I have the latest version of R and it cant install the older version of pandoc? does anyone know what is going on?
It looks like rmarkdown can't find pandoc. RStudio typically includes the pandoc files in your\path\to\Rstudio\bin\pandoc. When you open RStudio, it secretly sets an environment variable named RSTUDIO_PANDOC which rmarkdown uses to find pandoc.
Run the command Sys.getenv("RSTUDIO_PANDOC"), which should return the location of your pandoc installation. If it doesn't, that is likely to be your problem.
Once you find where RStudio is installed, you can set this environment variable with Sys.setenv(RSTUDIO_PANDOC = "your/path/to/Rstudio/bin/pandoc"). After that (assuming this is your problem) you should be able to knit the document as expected.
I had this problem at my Windows 10 desktop, and fixed it.
There is a long command before your error in the R Markdown terminal if you use Rstudio. It may looks like this:
"C:/Users/15510/AppData/Local/Pandoc/pandoc" ...
And it is inconsistent with my Sys.getenv("RSTUDIO_PANDOC"), which is "C:/Program Files/RStudio/bin/pandoc/pandoc"
So I remove my pandoc path(C:/Users/15510/AppData/Local/Pandoc/pandoc) from the system environment in Windows, and then the rmd can run successfully with the terminal saying this:
"C:/Program Files/RStudio/bin/pandoc/pandoc" ...
P.S. I guess you have installed pandoc in your user directory, maybe you can check your system path of the computer.
Unfortunately, none of the above worked for me. Surprisingly, disconnecting from the internet helped. May be this will help someone else too.

Where does Neo4j Desktop look for CSV files - Windows 10?

I'm at a loss, have installed Neo4j Desktop multiple times and cannot get it to load a CSV file. It's hard to know where to begin. None of the online documentation, or examples seem to be even relevant.
Loading from the browser using this code:
USING PERIODIC COMMIT
LOAD CSV WITH HEADERS FROM "file:///GraphElements.nodes.csv" AS row
CREATE (:Type {name: row.Name, uuid: row.UUID});
results in this error:
Neo.ClientError.Statement.ExternalResourceFailed: Couldn't load the
external resource at: file:/GraphElements.nodes.csv
enter image description here
According all the doc I've been able to find Neo4j looks for the file in its import folder, but where's the import folder supposed to be?
Neo4j was installed at: C:\Program Files\Neo4j Desktop
Note: 'Neo4j Desktop.exe' is in this folder, there is no bin folder
I created an import folder immediately below, i.e. C:\Program Files\Neo4j Desktop\import
and put the CSV file there. No luck, same error.
Rummaging around, I find:
C:\Users\CTG\.Neo4jDesktop
with some likely looking subfolders:
C:\Users\CTG\.Neo4jDesktop\neo4jDatabases\database-2c020f68-86a8-439d-937e-b5b46ad7f629
C:\Users\CTG\.Neo4jDesktop\neo4jDatabases\database-0541f039-2dfa-4c4d-85da-2c213b54d0c7
C:\Users\CTG\.Neo4jDesktop\neo4jDatabases\database-35274b1f-3b7b-42d3-83f1-06b881147e90
each of these has a bin folder a couple of levels down.
I've created an import folder under each of these, and copied the CSV file into them. Still get the problem.
Does anybody have any idea what's going on?
FWIW, the Neo4j doc doesn't seem to be up to date vis-a-vis Desktop.
The current Operations manual here only mentions Community and Enterprise editions, nothing about Desktop.
The File Locations reference claims that the default import location is at:
%APPDATA%\Neo4j Community Edition\import
but I cannot find a 'Neo4j Community Edition' folder. And the Community Edition seems to be gone from their site.
At this point Neo4j is only useful to play around with.
Thanks in advance for any clarification or help.
The import folder should be relative to the Neo4j instance in question.
If you Manage an instance in the Desktop app, you will see some buttons near the top. One of them is "Open Folder" with a dropdown menu. The dropdown has the option to open the Import folder for that instance.
This is what worked on Windows 10:
LOAD CSV FROM "file:///c:/Users/joe/Documents/sample.csv/" AS line RETURN count(*);
Go to the configuration file found here:
C:\Users\joe.Neo4jDesktop\neo4jDatabases\database-415cd6e1-0a1b-4973-ab3d-c32280b8a72b\installation-3.4.1\conf
Then change the conf file line 25 by commenting it out
#dbms.directories.import=import

Cypher Neo4j Couldn't load the external resource

In a Windows environment, I'm trying to load a .csv file with statement:
LOAD CSV WITH HEADERS FROM "file:///E:/Neo4j/customers.csv" AS row
It seems not to work properly and returns:
Couldn't load the external resource at:
file:/E:/Neo4j/Customers.csv
Neo.TransientError.Statement.ExternalResourceFailure
What am I doing wrong? thanks in advance
I was getting this error on Community Edition 3.0.1 on Mac OS X 10.10
It appears that the LOAD CSV file:/// looks for files in a predefined directory. One would think that in the argument that one would give the Cypher statement the full path but that is not the case.
The file:/// - for my situation" meant that neo4j would append the given argument you gave to one that was already predefined and then go look for that combined path
The file:/// pre-defined directory directory did not exist entirely
/Users/User/Documents/Neo4j/default.graphdb/import, in my computers directory structure I was missing the "/import" folder, which was not created at install
To fix on my system, I created an "import" directory, put the file to be read in that directory. I executed the Cypher load statement I ONLY put the name of the file to be read in the file argument i.e.
LOAD CSV file:///data.csv
this worked for me.
It appears to be a security configuration. Here's the original answer I found: https://stackoverflow.com/a/37444571/327004
You can add the following setting in conf/neo4j.conf in order to bypass this :
dbms.security.allow_csv_import_from_file_urls=true
Or change the import directory dbms.directories.import=import
You can find the answer in the file
"C:\Users\Jack\AppData\Roaming\Neo4j Community Edition\neo4j.conf"
(above "dbms.directories.import=import")
For version neo4j-community_windows-x64_3_1_1 you have to comment out this line or you have to create the folder \import (which isn´t created through the installation) and add your file into the folder.
There it´s written that due to security reasons they only allow file load from the \Documents\Neo4j\default.graphdb\import folder
After commenting out on # dbms.directories.import=import , you can execute e.g. from
LOAD CSV FROM "file:///C:/Users/Jack/Documents/products.csv" AS row
In neo4j.conf I didn´t have to add/set
dbms.security.allow_csv_import_from_file_urls=true
On (Arch) Linux + neo4j-community-3.4.0-alpha09, edit $NEO4J_HOME/conf
/neo4j.conf:
uncomment or add: dbms.security.allow_csv_import_from_file_urls=true
comment: #dbms.directories.import=import
Restart neo4j (in terminal: neo4j restart), and reload the Neo4j Browser (http://localhost:7474/browser/) if you are using a web browser as your Neo4j interface/GUI.
Then, you should be able to load a csv from outside your $NEO4J_HOME/... directory
E.g.,
LOAD CSV WITH HEADERS FROM "file:///mnt/Vancouver/Programming/data/metabolism/practice/a.csv" AS ...
where my $NEO4J_HOME/ is /mnt/Vancouver/apps/neo4j/neo4j-community-3.4.0-alpha09/
LOAD CSV WITH HEADERS FROM "file:/mnt/Vancouver/Programming/data/metabolism/practice/a.csv" AS ...
also works, but not
LOAD CSV WITH HEADERS FROM "file://mnt/Vancouver/Programming/data/metabolism/practice/a.csv" AS...
or
LOAD CSV WITH HEADERS FROM "/mnt/Vancouver/Programming/data/metabolism/practice/a.csv" AS...
i.e. use ...file:/... or ...file:///...
It's probably an URL issue, try file:c:/path/to/data.csv
See my blog posts:
http://jexp.de/blog/2014/10/load-cvs-with-success/
http://jexp.de/blog/2014/06/load-csv-into-neo4j-quickly-and-successfully/
For the ubuntu system, I placed the file in /usr/lib/neo4j which helped me solved the issue. On every other location, i tried giving full permissions(777) but the problem remains the same. After going through another stackoverflow post, i realized that the file should be kept in neo4j directory.
In the Neo4j desktop select the database you are using, go to the setting and there you will find the solution... just comment the "dbms.directories.import=import" line
# This setting constrains all LOAD CSV import files to be under the import directory. Remove or comment it out to
# allow files to be loaded from anywhere in the filesystem; this introduces possible security problems. See the
# LOAD CSV section of the manual for details.
dbms.directories.import=import ### COMMENT THIS LINE
For macOS Mojave v 10.14.5
Actually, I had to uncomment dbms.directories.import=import from ~/Library/Application Support/Neo4j Desktop/Application/neo4jDatabases/database-e2dd2a9c-d450-4639-861b-1e7e42b56b31/installation-3.5.5/conf/neo4j.conf and restart the service. Then it worked. All files has to be placed in import directory.
Run command LOAD CSV WITH HEADERS FROM 'FILE:/<yourCSV>.csv' as l return l
I am using the Neo4j Desktop and as others have said, the default graph database has a predefined import location. You can find the location by using the UI. If you put the CSV into the import directory, then you can use the relative path directly from you load csv command
Neo4j version is 3.1.1, OS is win10.
For me, LOAD CSV would read from Neo4j_Database_Location/testDB/import/artists.csv.
At first, I put csv file on the path F:\code\java\helloworld\artists.csv, and my cypher sentence is
LOAD CSV FROM 'file:///F:\\code\\java\\helloworld\\artists.csv' AS line
CREATE(:Artist {name:line[1],year:toInt(line[2])})
Then I get the error message returned as follows:
Couldn't load the external resource at: file:/D:/Neo4j/db/testDB/import/code/java/helloworld/artists.csv
It means neo4j itself concat the file path.
"D:/Neo4j/db/testDB/import/" is the Neo4j database location, and the "code/java/helloworld/artists.csv" is the csv file location.
For example, I install Neo4j on the path D:\Neo4j\Neo4j CE 3.1.1, and database loaction is D:\Neo4j\db. I put the CSV file on the path D:\Neo4j\db\testDB\import\artist.csv. If you don't have the file folder "import" on the path, you should creat it by yourself and put your file in the folder "import".
Then, put your csv file in the path, and input cyper sentence:
LOAD CSV from 'file:///artist.csv' as LINE
CREATE(:Artist {name:line[1],year:toInt(line[2])})
In a word, once you put the CSV file in the right path, the problem can be solved.
Related explaination in the LOAD CSV developer-manal
If dbms.directories.import is set to the default value import, using the above URLs in LOAD CSV would read from /import/myfile.csv and import/myproject/myfile.csv respectively.
If it is set to /data/csv, using the above URLs in LOAD CSV would read from /data/csv/myfile.csv and /data/csv/myproject/myfile.csv respectively.
Set the Property "dbms.directories.import=import"
Create folder 'import' explicitly at "/Users/User/Documents/Neo4j/default.graphdb/" because pre-defined directory did not exist entirely
place the csv data set here in the import folder
then run the code like - LOAD CSV FROM "file:///C:/customers.csv" AS row
In addition after you run the line, you can analyze what is going wrong in the code section to get a better understanding
you put your dataset into the import directory in neo4j-community path.
Then re-run your command.
Add your csv file in the import folder of neo4j installation guide to do this.
open neo4j and start graph of ur project
then in open folders tab open import folders
Copy ur csv file in this folder
Copy that part in ur load syntax as file:///C:/neo4j_module_datasets/test.csv since ur neo4j in running
in C drive
Snapshot for your reference
Use the following syntax:
LOAD CSV WITH HEADERS FROM "file:///my_collection.csv" AS row CREATE (n:myCollection) SET n = row
If you are running a docker then, follow these commands before running above query:
docker run \
-p=7474:7474 \
-p=7687:7687 \
-v=$HOME/neo4j/data:/data \
-v=$HOME/neo4j/logs:/logs \
-v=$HOME/local_import_dir:/var/lib/neo4j/import \
neo4j:3.0
Then,
sudo cp my_collection.csv /home/bajju/local_import_dir/
One of the following should solve the LOAD CSV errors (assuming you have dbms.security.allow_csv_import_from_file_urls=true)
If using Linux check for the permissions for the file. Change it using chmod 777 file_name.csv
Check if the file format/format for the contents within the file is correct.
The easiest way (be ware of security) is to serve you directory over http and use the http import
in the command line go the folder where csv files are lcoated
run the following depending on your python env.
Python 2
$ python -m SimpleHTTPServer 8000
Python 3
$ python3 -m http.server 8000
- Now you can load your files from your local host
LOAD CSV FROM 'http://localhost:8000/mycsvfile.csv' AS row
return row
- you can actually expose files on one host and load them where your DB is running by exposing the folder and replacing localhost with your IP