In sqlalchemy with postgres, how do I create an index on an attribute inside a json type field?
This didn't work:
Index("tablename.jsonfield.mykey.index", "'jsonfield'->>'mykey'")
Related
So I have three databases - an Oracle one, SQL Server one, and a Postgres one. I have a table that has two columns: name, and value, both are texts. The value is a stringified JSON object. I need to update the nested value.
This is what I currently have:
name: 'MobilePlatform',
value:
'{
"iosSupported":true,
"androidSupported":false,
}'
I want to add {"enableTwoFactorAuth": false} into it.
In PostgreSQL you should be able to do this:
UPDATE mytable
SET MobilePlatform = jsonb_set(MobilePlatform::jsonb, '{MobilePlatform,enableTwoFactorAuth}', 'false');
In Postgres, the plain concatenation operator || for jsonb could do it:
UPDATE mytable
SET value = value::jsonb || '{"enableTwoFactorAuth":false}'::jsonb
WHERE name = 'MobilePlatform';
If a top-level key "enableTwoFactorAuth" already exists, it is replaced. So it's an "upsert" really.
Or use jsonb_set() for manipulating nested values.
The cast back to text works implicitly as assignment cast. (Results in standard format; any insignificant whitespace is removed effectively.)
If the content is valid JSON, the storage type should be json to begin with. In Postges, jsonb would be preferable as it's easier to manipulate, but that's not directly portable to the other two RDBMS mentioned.
(Or, possibly, a normalized design without JSON altogether.)
For ORACLE 21
update mytable
set json_col = json_transform(
json_col,
INSERT '$.value.enableTwoFactorAuth' = 'false'
)
where json_exists(json_col, '$?(#.name == "MobilePlatform")')
;
With json_col being JSON or VARCHAR2|CLOB column with IS JSON constraint.
(but must be JSON if you want a multivalue index on json_value.name:
create multivalue index ix_json_col_name on mytable t ( t.json_col.name.string() );
)
Two of the databases you are using support JSON data type, so it doesn't make sense to have them as stringified JSON object in a Text column.
Oracle: https://docs.oracle.com/en/database/oracle/oracle-database/21/adjsn/json-in-oracle-database.html
PostgreSQL: https://www.postgresql.org/docs/current/datatype-json.html
Apart from these, MSSQL Server also provides methods to work with JSON data type.
MS SQL Server: https://learn.microsoft.com/en-us/sql/relational-databases/json/json-data-sql-server?view=sql-server-ver16
Using a JSON type column in any of the above databases would enable you to use their JSON functions to perform the tasks that you are looking for.
If you've to use Text only then you can use replace to add the key-value pair at the end of your JSON
update dataTable set value = REPLACE(value, '}',",\"enableTwoFactorAuth\": false}") where name = 'MobilePlatform'
Here dataTable is the name of table.
The cleaner and less riskier way would be connect to db using the application and use JSON methods such as JSON.parse in Javascript and JSON.loads in Python. This would give you the JSON object (dictionary in case of Python) to work on. You can look for similar methods in other languages as well.
But i would suggest, if possible use JSON columns instead of Text to store the JSON value wherever possible.
I'm working with mongoDB, and I used a wrapper mongo/Postegres.
Now, I can find my tables and data.
I want to do some statistics but I can't reach objects that got json type in Postgres.
My problem is that I got all the object in json but I need to separate the fields.
I used this :
CREATE FOREIGN TABLE rents( _id NAME, status text, "from" json )
SERVER mongo_server
OPTIONS (database 'tr', collection 'rents');
The field "from" is an object.
I found something like this :
enter code here
but nothing happened
The error (why a screenshot??) means that the data are not in valid json format.
As a first step, you could define the column as type text instead of json. Then querying the foreign table will probably work, and you can see what is actually returned and why PostgreSQL thinks that this is not valid JSON.
Maybe you can create a view on top of the foreign table that converts the value to valid JSON for further processing.
I built the structure on cassandra DB to store the time series data of the OS data like services, process and other information. To understand how to works Cassandra about storing JSON data and retrieval the data by CQL queries with condition I prefered to simplify the model. Because in the total model DB I'll have the TYPE more complex than report_object like hashMap of array of hashMap for example:
Type NETSTAT--> Object[n] --> {host:192.168.0.23, protocol: TCP ,LocalAddress : 0.0.0.0}
so the Type NETSTAT will have a list of hashMaps that will contain the fields key -> value.
For simplify I have choosen to show the following schema:
CREATE TYPE report_object (RTIME varchar, RMINORVER int, RUSER varchar, RLANG varchar, RSCRIPT varchar, RMAJORVER int, RHOST varchar, RPATH varchar);
CREATE TABLE test (
REPORTUUID uuid PRIMARY KEY,
report frozen<report_object>);
Inside the table I injectioned the JSON data with the followed query inside java class:
INSERT INTO test JSON '{"REPORTUUID": "9fb21fb9-333e-4017-ab77-0fa6ee1e20e3" ,"REPORT":{"RTIME":"6/MAR/2016 6:0:0 PM","RMINORVER":0,"RUSER":"Administrator","RLANG":"vbs","RSCRIPT":"Main","RMAJORVER":5,"RHOST":"WIN-SAPV9MUEMNS","RPATH":"C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\IXP000.TMP"}}';
I inectioned other data with the query above.
The questions to clarify my concepts are:
- I would like to do the queries with conditions that check inside TYPE defined, is it possible with CQL or is necessary to use spark SQL?
Is design DB model right for the purpose (Because I have passed from RDBMS to DB NoSQL) ?
To be able to query User Defined Type using Cassandra you'll have to create an index first:
CREATE INDEX on test.test(report);
but it allows only a predicate based on a full document:
SELECT * FROM test
WHERE report=fromJson('{"RTIME":"6/MAR/2016 6:0:0 PM","RMINORVER":0,"RUSER":"Administrator","RLANG":"vbs","RSCRIPT":"Main","RMAJORVER":5,"RHOST":"WIN-SAPV9MUEMNS","RPATH":"C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\IXP000.TMP"}');
You'll find more details and explanation in how to filter cassandra query by a field in user defined type
When exposed using Spark these values can be filtered using filter on CassandraTableScanRDD:
val rdd = sc.cassandraTable("test", "test")
rdd.filter(row =>
row.getUDTValue("report").getString("rscript") == "Main")
or where / filter on a DataFrame:
df.where($"report.rscript" === "Main")
Although query like this using Spark a whole table has to be fetched before data can be filtered. While it is not clear what exactly you are trying to achieve but it is rather unlikely this will be an useful structure in general.
I'm trying to use Talend to get JSON data that is stored in MySQL as a VARCHAR datatype and export it into PostgreSQL 9.4 table of the following type:
CREATE TABLE myTable( myJSON as JSONB)
When I try running the job I get the following error:
ERROR: column "json_string" is of type json but expression is of type
character varying
Hint: You will need to rewrite or cast the expression. Position:
54
If I use python or just plain SQL with PostgreSQL insert I can insert a string such as '{"Name":"blah"}' and it understands it.
INSERT INTO myTable(myJSON) VALUES ('{"Name":"blah"}');
Any Idea's how this can be done in Talend?
You can add a type-cast by opening the "Advanced Settings" tab on you "tPostgresqlOutput" component. Consider the following example:
In this case, the input row to "tPostgresqlOutput_1" has one column data. This column is of type String and is mapped to the database column data of type VARCHAR (as by the default suggested by Talend):
Next, open the component settings for tPostgresqlOutput_1 and locate the "Advanced settings" tab:
On this tab, you can replace the existing data column by a new expression:
In the name column, specify the target column name.
In the SQL Expression column, do your type casting. In this case: "?::json"`. Note the usage of the placeholder character?`` which will be replaced with the original value.
In Position, specify Replace. This will replace the value proposed by Talend with your SQL expression (including the type cast).
As Reference Column use the source value.
This should do the trick.
Here is a sample schema for where in i have the input row 'r' which has question_json and choice_json columns which are json strings. From which i know the key what i wanted to extract and here is how i do
you should look at the columns question_value and choice_value. Hope this helps you
In PostgreSQL 9.3 Beta 2 (?), how do I create an index on a JSON field? I tried it using the -> operator used for hstore but got the following error:
CREATE TABLE publishers(id INT, info JSON);
CREATE INDEX ON publishers((info->'name'));
ERROR: data type json has no default operator class for access method
"btree" HINT: You must specify an operator class for the index or
define a default operator class for the data type.
Found:
CREATE TABLE publishers(id INT, info JSON);
CREATE INDEX ON publishers((info->>'name'));
As stated in the comments, the subtle difference here is ->> instead of ->. The former one returns the value as text, the latter as a JSON object.