How do I parse a command line parameter and use it as a variable in the script to be used as filename to be saved as. I have tried the below but it is not working
fname:.z.X[2]
.....
...more code...
....
/Save the table into a csv file
`:(fname,".csv") 0:csv 0: table
You need to always remember the left of right evaluation.
In your case you are trying to write the csv delimited table to (fname,".csv"), which is only a string.
Further you want to use `$ to parse to a symbol (not `:), and use hsym to create a file path (prefix with ":")
bash> q script.q filename
q)(hsym `$ .z.x[0],".csv") 0:csv 0: ([]10?10)
`:filename.csv
Related
I have a CSV file with a column contains " as part of the data
The file is initially copied in ASF from SFTP location to ADLS
in the copy activity I tried adding quotes as the Quote character so my data looks like (this is just an example, the actual file is bigger)
ID ^ Name
"1" ^ "Galaxy 7" "
what breaking the Azure Synapse COPY INTO command is the inches sign (") next to Galaxy 7
I even tried adding \ as an escape character in the data source that outputs the CSV into adls but did not help
always getting same error
Bulk load failed due to invalid column value in CSV data file
/dev/switchingdb/avi_1.txt in row 2, column 21.
copy into command
COPY INTO stg.cdm_stg_Switching_DB FROM 'https://prdacrdatalake.dfs.core.windows.net/adf/dev/switchingdb/avi_1.txt'
WITH (
FILE_TYPE = 'CSV'
-- ,COMPRESSION = 'gzip'
,FIELDQUOTE ='"'
,FIRSTROW =2
,FIELDTERMINATOR ='^'
,CREDENTIAL = (IDENTITY = 'Managed Identity')
)
I am trying to import csv file to table in postgres using COPY command. I have problem that one column is of json data type. I tried to escape json data in csv using dollars ($$...$$) docu_4.1.2.2.
This is first line of csv:
3f382d8c-bd27-4092-bd9c-8b50e24df7ec;370038757|PRIMARY_RESIDENTIAL;$${"CustomerData": "{}", "PersonModule": "{}"}$$
This is command used for import:
psql -c "COPY table(id, name, details) FROM '/path/table.csv' DELIMITER ';' ENCODING 'UTF-8' CSV;"
This is error I get:
ERROR: invalid input syntax for type json
DETAIL: Token "$" is invalid.
CONTEXT: JSON data, line 1: $...
COPY table, line 1, column details: "$${CustomerData: {}, PersonModule: {}}$$"
How should I escape/import json value using COPY? Should I give up and use something like pg_loader instead? Thank you
In case of failing with importing the JSON data please give a try to the following setup - this worked for me even for quite complicated data:
COPY "your_schema_name.yor_table_name" (your, column_names, here)
FROM STDIN
WITH CSV DELIMITER E'\t' QUOTE '\b' ESCAPE '\';
--here rows data
\.
How to convert a DBF to CSV?
I need, use this library but it gave error: http://pythonhosted.org/dbf
import dbf
dbf.export('crop1-fx')
print 'Done'
"C:\Users\User\Anaconda2\python.exe"
"C:/Users/User/Desktop/Python/23/dbf/insertValuesDBF.py" Traceback
(most recent call last): File
"C:/Users/User/Desktop/Python/23/dbf/insertValuesDBF.py", line 3, in
dbf.export('crop1-fx') File "C:\Users\User\Anaconda2\lib\site-packages\dbf\ver_2.py", line 7824,
in export
table = source_table(table_or_records[0]) File "C:\Users\User\Anaconda2\lib\site-packages\dbf\ver_2.py", line 7956,
in source_table
table = thingie._meta.table() AttributeError: 'str' object has no attribute '_meta'
Process finished with exit code 1
You almost had it:
import dbf
db = dbf.Table('crop1-fx')
dbf.export(db)
The above will create a crop1-fx.csv file; however, I'm not sure this will work with a 24-digit numeric field in the table.
To convert a .DBF file to .CSV, download dBASE III PLUS or any other dBASE
software available on NET. Please note I am referring to 16 bit platform on
DOS.
Once dBASE is downloaded, go to the DOT prompt and give the following commands:
Type
use <the dbf file in question without the extension .dbf>
You will see the name of the dbf file on the display bar
Then, type "copy to" <file name you want, limited to 8 characters>
"delimited"
Now the data in the dbf file is sent to a TEXT file with data in each field surrounded by " " (double inverted commas) and separated by , (comma)
Now this file can be used to export the data to any other DATABASE SYSTEM which has got provision to convert this .CSV or DELIMITED FILE to the new database.
If only comma-separated file without the " " marks are required a procedure
can be written in dBASE to achieve that also.
I'm trying to import a JSON file into a table. I'm using the solution mentioned here: https://stackoverflow.com/a/33130304/1663462:
create temporary table temp_json (values text) on commit drop;
copy temp_json from 'data.json';
select
values->>'annotations' as annotationstext
from (
select json_array_elements(replace(values,'\','\\')::json) as values
from temp_json
) a;
Json file content is:
{"annotations": "<?xml version=\"1.0\"?>"}
I have verified that this is a valid JSON file.
The json file contains a \" which I presume is responsible for the following error:
CREATE TABLE
COPY 1
psql:insertJson2.sql:13: ERROR: invalid input syntax for type json
DETAIL: Expected "," or "}", but found "1.0".
CONTEXT: JSON data, line 1: {"annotations": "<?xml version="1.0...
Are there any additional characters that need to be escaped?
Because copy command processes escape ('\') characters for text format without any options there are two ways to import such data.
1) Process file using external utility via copy ... from program, for example using sed:
copy temp_json from program 'sed -e ''s/\\/\\\\/g'' data.json';
It will replace all backslashes to doubled backslashes, which will be converted back to single ones by copy.
2) Use csv import:
copy temp_json from 'data.json' with (format csv, quote '|', delimiter E'\t');
Here you should to set quote and delimiter characters such that it does not occur anywhere in your file.
And after that just use direct conversion:
select values::json->>'annotations' as annotationstext from temp_json;
I have a folder containing a number of csv files, e.g. "leeds dz.csv", "leeds gh.csv", "leeds fr.csv". The first part of the file names is constant (i.e. always "leeds").
I want to import each to Stata individually, convert to .dta file and save it. Currently I have this code:
cd "etcetc"
clear
local myfilelist : dir . files"*.csv"
foreach file of local myfilelist {
drop _all
insheet using `file', comma
local outfile = subinstr("`file'",".csv","",.)
save "`outfile'", replace
}
The code works fine if I rename all the .csv files manually to delete the "leeds" part, ie if each .csv is named "dz.csv" instead of "leeds dz.csv" etc.
However, if I do not do this deletion I receive the error "invalid 'dz.csv' "
I'm guessing this has something to do with my 3rd line of code, in particular the "*.csv". But I'm unsure how to adapt the code/ why it won't allow me to import files with a space in the name?
The line
insheet using `file', comma
will be problematic with any filename containing spaces.
Try
insheet using "`file'", comma
The help for insheet is quite explicit on this:
If filename is specified without an extension, .raw is assumed. If your
filename contains embedded spaces, remember to enclose it in double
quotes.