I'm running PostgreSQL 9.6, and I've got a table consisting of lots of columns.
I've got a csv-file containing the following format:
id, insert_time, JSON-object
The JSON-object has the following format:
{ column_nameY: valueX, column_nameY: valueY, ... }
The column_names in the JSON-object matches the columns in my PostgreSQL-table.
Is there a dynamic way to import such file, so I'll get the id, insert_time, and the remaining column values from the JSON object?
The order of columns in the JSON object might not match the order of the columns in the PostgreSQL table.
I am assuming you know how to get that csv file imported into postgresql and that you know the fields inside that json object field.
first we create a table to store the contents of the csv file. Notice the datatype of that JSON field is jsonb.
create table test11 (id int, insert_time timestamp, json_object jsonb )
Know you import the csv file. but for illustration purposes, i will insert sample data into this Table.
insert into test11 (id, insert_time, json_object) values (1, '2017-11-14'::timestamp, '{ "column_nameX": "3", "column_nameY": "4" }'::jsonb);
insert into test11 (id, insert_time, json_object) values (1, '2017-11-14'::timestamp, '{ "column_nameX": "13", "column_nameY": "14" }'::jsonb);
we now select from that table;
Select id, insert_time, json_object->>'column_nameY' as Column_NameY, json_object->>'column_nameX' as Column_NameX from test11
your results should look like this...
id |insert_time |column_namey |column_namex
1 |11/14/2017 |4 |3
1 |11/14/2017 |14 |13
-HTH
Related
I am trying to fetch data from a Snowflake table Employee and insert values to VARIANT column of another snowflake table in JSON format.
EMP_ID
EMP_NAME
SALARY
POSITION
1
ABC
100
ENGINEER
2
DEF
300
MANAGER
3
GHI
500
DIRECTOR
Expected JSON format:
{
"EMP_ID":"1",
"EMP_NAME":"ABC",
"SALARY":"100",
"POSITION":"ENGINEER"
}
{
"EMP_ID":"2",
"EMP_NAME":"DEF",
"SALARY":"300",
"POSITION":"MANAGER"
}
{
"EMP_ID":"3",
"EMP_NAME":"GHI",
"SALARY":"500",
"POSITION":"DIRECTOR"
}
The above JSON formatted data should be loaded to a table EMP_JSON(load_data VARIANT, load_date TIMESTAMP_LTZ).
I tried hardcoding the values. But I would like to fetch the values dynamically.
INSERT INTO EMP_JSON
SELECT PARSE_JSON('{"EMP_ID":"1", "EMP_NAME":"ABC", "SALARY":"100", "POSITION":"ENGINEER"}'), CURRENT_TIMESTAMP AS LOAD_DATE ;
Could you please tell me how to load such JSON value to a variant column in Snowflake?
As it's mentioned, you can use object_construct to generate JSON object. The important point is, you can call object_construct with * parameter to read all columns dynamically (you don't need to specify columns) and produce the JSON:
create table emp( EMP_ID number, EMP_NAME varchar, SALARY number, POSITION varchar );
INSERT INTO emp(EMP_ID,EMP_NAME,SALARY,POSITION) VALUES (1,'ABC',100,'ENGINEER');
INSERT INTO emp(EMP_ID,EMP_NAME,SALARY,POSITION) VALUES (2,'DEF',300,'MANAGER');
INSERT INTO emp(EMP_ID,EMP_NAME,SALARY,POSITION) VALUES (3,'GHI',500,'DIRECTOR');
create table EMP_JSON(load_data VARIANT, load_date TIMESTAMP_LTZ);
insert into EMP_JSON select object_construct(*), current_timestamp from emp;
select * from EMP_JSON;
If you want to select specific columns:
insert into EMP_JSON select select object_construct('EMP_ID',EMP_ID, 'EMP_NAME', EMP_NAME), current_timestamp from emp;
OBJECT_CONSTRUCT https://docs.snowflake.com/en/sql-reference/functions/object_construct.html
#vvazza, use the object_construct function to select the data from table and it returns a JSON object and that can go to variant column.
I have a simple JSON document "example_1.JSON:
{
"fruit": "Apple",
"size": "Large",
"color": "Red"
}
I created a temp table "temp_json" to copy the file into it:
CREATE TEMP TABLE temp_json (mydata text);
I copied the JSON file using the following statement:
COPY temp_json from 'C:\Program Files\PostgreSQL\9.5\data\example_1.JSON';
It is ok with the copy part. When I insert the values from the temp table into my database table "jsontable", the insertion happens with no errors, but it inserts the JSON values in several rows inside my database table!!!
My database table is created as follows:
CREATE TABLE public.jsontable (
id bigint NOT NULL DEFAULT nextval('jsontable_id_seq'::regclass),
jsondata jsonb,
CONSTRAINT jsontable_pkey PRIMARY KEY (id)
);
The insert statement from the temp table to the jsontable:
INSERT INTO jsontable(jsondata) SELECT to_jsonb(mydata::text) FROM temp_json;
But when I select rows from jsontable, I don't get the JSON values in a single row!
SELECT * FROM jsontable;
Any suggestions to solve this problem?
You have two options. Control the data from the source file, i.e. pre-format the contents of the file to have them on a single line before copying.
Another option might be to concatenate the lines using string_agg. To maintain the order, I would suggest you to have a default identity column in your temp table.
create sequence seq_temp_json;
CREATE temp TABLE temp_json
(
id INT DEFAULT NEXTVAL('seq_temp_json'::regclass),
mydata TEXT
);
Now, load the temp table and check the order, json order should be in the ascending order of id.
COPY temp_json(mydata) from 'C:\Program Files\PostgreSQL\9.5\data\example_1.JSON';
knayak=# select * from temp_json;
id | mydata
----+-------------------
1 | {
2 | "fruit": "Apple",
3 | "size": "Large",
4 | "color": "Red"
5 | }
(5 rows)
Load the JSON into main table
INSERT INTO jsontable ( jsondata )
SELECT string_agg( mydata ,e'\n' ORDER BY id)::jsonb
FROM temp_json;
The column now contains the complete JSONB.
knayak=# select * from jsontable;
id | jsondata
----+-----------------------------------------------------
6 | {"size": "Large", "color": "Red", "fruit": "Apple"}
(1 row)
I'm trying to perform a SELECT at a PostgreSQL table with a JSON column. The problem is that that column is not a real JSON.
When I export to CSV the result is the following:
"{""access"":""1.SQnGhZKY4cemiRU+Svteoj2ZPQJ74uqXoWmGg0BmVf8ho3D2c7g1xYogcwz"",""objectId"":""58e3a109b48a6"",""deviceId"":""58e65028b4567"",""deviceBusinessUnits"":[],""crmId"":""443-f01246e5"",""crmBusinesUnits"":[],""crm"":{""gender"":""m"",""birthDate"":"""",""age"":null,""customFields"":{""displayName"":""te4"",""givenName"":""test"",""familyName"":""test"",""email"":""tes#yopil.net"",""mobileNumber"":""68"",""mobileNumberCountryCode"":""34"",""residencyCountry"":""LUX"",""mailingStreetName1"":null,""mailingStreetName2"":null,""mailingSubAdministrativeArea"":null,""mailingMunicipality"":null,""mailingPostalCode"":""208"",""mailingAdministrativeArea"":null,""mailingCountry"":null,""shippingStreetName1"":null,""shippingStreetName2"":null,""shippingSubAdministrativeArea"":null,""shippingMunicipality"":null,""shippingPostalCode"":null,""shippingAdministrativeArea"":null,""shippingCountry"":null},""tags"":[],""businessUnits"":[],""crmProvider"":{""type"":""cid"",""sessionToken"":""7drmcvu""}}}"
My column is named "user" and my query is:
select
mytable.user ->> 'accessToken' as userRaw
from mytable;
But I get:
How could I get the JSON values from this column?
Recently there was a driver upgrade for Cassandra from 2.0.4 to 3.1.0 in my project. I found that the insert statement does not work on the tables that has 'json' as column name. The same insert statement used to work on previous cassandra version.
Sample insert statement:
insert into table_name (name, id, json) values ("AAA", "123", '{"display-order":"1","product-id":"QWERTY"}');
When I tried the query in Datastax DevCenter, I am getting the error as "no viable alternative at input 'json'"
Does that mean that we have to change the insert statements?
Please help!
Edit:
I made a mistake in the example query, json type is not the issue. When I enabled debug I found that the issue is with timestamp column.
I did not mention that in my example as I thought json could be the reason.
This is because of the recent driver update to 3.1.0, as the same query does not work now.
When I tried to fix it, I could not find the timestamp converter function (toTimestamp) or the dateOf() function in 3.1.0 version.
I see only now() but it returns only of type timeuuid.
Is there any other way I can convert the current date to timestamp?
It's not because of json field, your insert format is not correct
Use string as enclosed with single quote :
cassandra#cqlsh:test> insert into test_json (name, id, json) values ('AAA', '123', '{"display-order":"1","product-id":"QWERTY"}');
cassandra#cqlsh:test> SELECT * FROM test_json ;
id | json | name
-----+---------------------------------------------+------
123 | {"display-order":"1","product-id":"QWERTY"} | AAA
Or Double dollar signs to enclose a string with quotes, backslashes, or other characters that would normally need to be escaped
cassandra#cqlsh:test> insert into test_json (name, id, json) values ('AAA', '123', $${"display-order":"1","product-id":"QWERTY"}$$);
cassandra#cqlsh:test> SELECT * FROM test_json ;
id | json | name
-----+---------------------------------------------+------
123 | {"display-order":"1","product-id":"QWERTY"} | AAA
By the way, if you want to insert current timestamp use dateof(now()) or if you use java use new Date()
I'm getting this error when trying to access data in a JSON object, does anybody know what it is causing it?
This is the query:
SELECT id, data FROM cities WHERE data->'location'->>'population' = '270816'
This is the JSON object:
location": {
"population": 270816,
"type": "city"
}
Any help would be really appreciate it. Thanks
I was able to get this SELECT to work in Postgres 9.3.1. Here's an sqlfiddle which illustrates that.
Here is the DDL and INSERT statement I used in the sqlfiddle:
create table cities
(
id serial,
data json
);
insert into cities (data) VALUES
('{
"location": {
"population": 270816,
"type": "city"
}
}'
);
What version of Postgres are you using? How are you inserting the JSON? What's the DDL for your cities table?
It suspect it may be an issue with the way you are inserting the JSON data. Try inserting it similar to the way I am doing in the sqlfiddle above and see if that works for you. i.e. as a pure SQL string, but one with valid JSON inside, into a column defined as json.
Just had what sounds like the same issue on Postgres 9.6.6. Improper string escaping caused mysterious JSONB behavior. Using pgAdmin4,
CREATE TABLE json_test (json_data JSONB);
INSERT INTO json_test (json_data) VALUES ('"{\"id\": \"test1\"}"');
INSERT INTO json_test (json_data) VALUES ('{"id": "test2"}');
SELECT json_data, json_data->>'id' as id FROM json_test;
returns the following pgAdmin4 output showing baffling failure to find id test2. Turns out the pgAdmin4 display is misleading. Situation becomes clear using text display from PSQL:
db=> CREATE TABLE json_test (json_data JSONB);
CREATE TABLE
db=> INSERT INTO json_test (json_data) VALUES ('"{\"id\": \"test1\"}"');
INSERT 0 1
db=> INSERT INTO json_test (json_data) VALUES ('{"id": "test2"}');
INSERT 0 1
db=> SELECT json_data, json_data->>'id' as id FROM json_test;
json_data | id
-----------------------+-------
"{\"id\": \"test1\"}" |
{"id": "test2"} | test2
(2 rows)
Where it is obvious that the first row was inserted as a string which just looks like JSON, not as a nested JSON object.