I have a table in MySql somewhat like this:
create table json_docs (
id int auto_increment primary_key,
doc json not null
)
I also have a json schema that this doc column obeys.
{
"$id": "https://my.org/namespace/doc.schema.json",
"type": "object",
"properties": {...}
}
Is there a way to enforce that schema in MySql during an insert or update?
insert into json_docs (doc) values ('{...}')
I am also using PHP Laravel. If there is a way to do this in the Model layer of Laravel I would also be interested.
Thanks in advance for your help!
MySQL does have json validation support.
https://dev.mysql.com/doc/refman/8.0/en/json-validation-functions.html
MySQL does not have any feature built-in to validate JSON documents against a schema.
Update: This feature now exists in 8.0.17, but prior to that version, the following answer still applies:
You would have to validate your document in your app, independently from inserting it into a database.
See https://github.com/justinrainbow/json-schema for an open-source JSON validator in PHP.
Related
how to convert my mysql table result into json Object in database level
for example ,
SELECT json_array(
group_concat(json_object( name, email))
FROM ....
it will produce the result as
[
{
"name": "something",
"email": "someone#somewhere.net"
},
{
"name": "someone",
"email": "something#someplace.com"
}
]
but what i need is i need to given my own query which may contains functions, subqueries etc.
like in postgres select row_to_json(select name,email,getcode(branch) from .....) then i will get the whole result as json object
in mysql is there any possibilities to do like this?
select jsonArray(select name,email,getcode(branch) from .....)
I only found in official Mysql 8 and 5.7 documentation that it supports casting to JSON type. It includes a JSON_ARRAY function in MySQL 8 and 5.7, and JSON_ARRAYAGG function in MySQL 8. Please see the full JSON functions reference here.
It means that does not exist an easy mysql built-in solution to the problem.
Fortunately, our colleagues started a similar discussion here. Maybe you could find your solution there.
For one searching for well-defined attributes JSON casting, the solution is here.
I have the following problem. We are using Azure SQL Database for the Data processing. Instead of wizard import every time we would like to automatically load the data through API from our accounting platforms. (API Documentation Links: https://hellocashapi.docs.apiary.io/#introduction/authentication , https://www.zoho.com/books/api/v3/)
Basically my task is to get the data through API from these platforms and create the table in our Azure SQL Database and insert this data therein.
Can anyone recommend me the platform to resolve this issue? or please send me the link with the documentation which would show me the way to do that.
Thank you.
If you can put the JSON on a SQL variable like this
DECLARE #json NVARCHAR(MAX) = N'[
{
"Order": {
"Number":"SO43659",
"Date":"2011-05-31T00:00:00"
},
"AccountNumber":"AW29825",
"Item": {
"Price":2024.9940,
"Quantity":1
}
},
{
"Order": {
"Number":"SO43661",
"Date":"2011-06-01T00:00:00"
},
"AccountNumber":"AW73565",
"Item": {
"Price":2024.9940,
"Quantity":3
}
}
]
Then you can create a table using the WITH clause
SELECT * INTO TableName1
FROM OPENJSON ( #json )
WITH (
Number varchar(200) '$.Order.Number',
Date datetime '$.Order.Date',
Customer varchar(200) '$.AccountNumber',
Quantity int '$.Item.Quantity',
[Order] nvarchar(MAX) AS JSON
)
Firstly, not all the API is supported as Source data in Data Factory.
Please reference this document: Azure Data Factory connector overview
Data Factory doesn't support hellocashAPI. That means do that with Data Factory.
Secondly, Data Factory now support supports creating a destination table automatically.
Referecec:Copy Activity in Azure Data Factory supports creating a destination table automatically.
Summary:
Load data faster with new support from the Copy Activity feature of Azure Data Factory. Now, if you’re trying to copy data from an any supported source into SQL database/data warehouse and find that the destination table doesn’t exist, Copy Activity will create it automatically. After the data ingestion, review and adjust the sink table schema as needed.
This feature is supported with:
Azure SQL Database
Azure SQL Database Managed Instance
Azure SQL Data Warehouse
SQL Server
Hope this helps.
Simply speaking I would create table with given name providing only data.
I have some JUnit's with sample data (jsons)
I have to provide schema for above files to create tables for them
I suppose that don't need provide above schemas.
Why? Because in BigQuery console I can create table from query (even such simple like: select 1, 'test') or I can upload json to create table with schema autodetection => probably could also do it programatically
I saw https://chartio.com/resources/tutorials/how-to-create-a-table-from-a-query-in-google-bigquery/#using-the-api and know that could parse jsons with data to queries and use Jobs.insert API to run them but it's over engineered and has some other disadvanteges e.g. boilerplate code.
After some research I found possibly simpler way of creating table on fly, but it doesn't work for me, code below:
Insert insert = bigquery.jobs().insert(projectId,
new Job().setConfiguration(
new JobConfiguration().setLoad(
new JobConfigurationLoad()
.setSourceFormat("NEWLINE_DELIMITED_JSON")
.setDestinationTable(
new TableReference()
.setProjectId(projectId)
.setDatasetId(dataSetId)
.setTableId(tableId)
)
.setCreateDisposition("CREATE_IF_NEEDED")
.setWriteDisposition(writeDisposition)
.setSourceUris(Collections.singletonList(sourceUri))
.setAutodetect(true)
)
));
Job myInsertJob = insert.execute();
JSON file which is used as a source data is pointed by sourceUri, looks like:
[
{
"stringField1": "value1",
"numberField2": "123456789"
}
]
Even if I used setCreateDisposition("CREATE_IF_NEEDED") I still receive error: "Not found: Table ..."
Is there any other method in API or better approach than above to exclude schema?
The code in your question is perfectly fine, and it does create table if it doesn't exist. However, it fails when you use partition id in place of table id, i.e. when destination table id is "table$20170323" which is what you used in your job. In order to write to partition, you will have to create table first.
I have one SQL table in postgress database having multiple JSON type columns.
We want to do some kind of validation on those JSON coulmns before inserting the data.
Our target is to have validation using stored procedure, which call noSQL queries and respond error in case of any validation failure on that JSON column.
Example -
lets say we have 5 to 6 JSON columns in User table,
JSON columns like -
Personal Information, Accounts, OwnerDetails etc, and these coulmns having complex JSON like list of JSON data or JSON within JSON.
Now we want to validate each column one by one and in case of error, return unique error code.
When using the JSON type, we can easily define constraints on the JSON fields. Here is a basic demonstration:
CREATE FUNCTION is_valid_description(descr text) RETURNS BOOLEAN AS
'SELECT length($1) > 10;'
LANGUAGE sql;
CREATE TABLE wizards (
id SERIAL,
data JSON,
CONSTRAINT validate_name CHECK ((data->>'name') IS NOT NULL AND length(data->>'name') > 0),
CONSTRAINT validate_description CHECK ((data->>'description') IS NOT NULL AND (is_valid_description(data->>'description')))
);
We check for constraints on two JSON fields of the data column. In one case, we use a stored procedure written in SQL, but we could write more complex functions with PL/pgSQL, or in Perl or Python.
We may also want to enforce uniqueness on one of the fields:
CREATE UNIQUE INDEX ui_wizards_name ON wizards((data->>'name'));
Validation and uniqueness constraints are then enforced when inserting or updating rows.
-- violates check constraint "validate_description"
INSERT INTO wizards(data) VALUES('{
"name": "Kirikou",
"description": "A witch"
}');
-- passes
INSERT INTO wizards(data) VALUES('{
"name": "Kirikou",
"description": "A witch of African descent"
}');
-- violates unique constraint "ui_wizards_name"
INSERT INTO wizards(data) VALUES('{
"name": "Kirikou",
"description": "The same witch of African descent"
}');
Code to play with on http://sqlfiddle.com/#!15/23974/2.
I can't find any information about JSON schema validation in PostgreSQL, is there any way to implement JSON Schema validation on PostgreSQL JSON data type?
There is another PostgreSQL extension that implements json validation. The usage is almost the same as "Postgres-JSON-schema"
CREATE TABLE example (id serial PRIMARY KEY, data jsonb);
-- do is_jsonb_valid instead of validate_json_schema
ALTER TABLE example ADD CONSTRAINT data_is_valid CHECK (is_jsonb_valid('{"type": "object"}', data));
INSERT INTO example (data) VALUES ('{}');
-- INSERT 0 1
INSERT INTO example (data) VALUES ('1');
-- ERROR: new row for relation "example" violates check constraint "data_is_valid"
-- DETAIL: Failing row contains (2, 1).
I've done some benchmarking validating tweets and it is 20x faster than "Postgres-JSON-schema", mostly because it is written in C instead of SQL.
Disclaimer, I've written this extension.
There is a PostgreSQL extension that implements JSON Schema validation in PL/PgSQL.
It is used like this (taken from the project README file):
CREATE TABLE example (id serial PRIMARY KEY, data jsonb);
ALTER TABLE example ADD CONSTRAINT data_is_valid CHECK (validate_json_schema('{"type": "object"}', data));
INSERT INTO example (data) VALUES ('{}');
-- INSERT 0 1
INSERT INTO example (data) VALUES ('1');
-- ERROR: new row for relation "example" violates check constraint "data_is_valid"
-- DETAIL: Failing row contains (2, 1).
What you need is something to translate JSON Schema constraints into PostgreSQL ones, e.g.:
{
"properties": {
"age": {"minimum": 21}
},
"required": ["age"]
}
to:
SELECT FROM ...
WHERE (elem->>'age' >= 21)
I'm not aware of any existing tools. I know of something similar for MySQL which might be useful for writing your own, but nothing for using the JSON type in PostgreSQL.