Error DeserializeJSON() MySQL json_object - mysql

I am getting back a JSON string from a MySQL 5.7 query in ColdFusion 9.0.1. Here is my query:
SELECT (
SELECT GROUP_CONCAT(
JSON_OBJECT(
'nrtype', nrt.nrtype,
'number', nr.number
)
)
) AS nrJSON
FROM ...
The returned data looks like this:
{"nrtype": "Phone 1", "number": "12345678"},{"nrtype": "E-Mail 1", "number": "some#email.com"}
But as soon as I try to use DeserializeJSON() on it I am getting the following error:
JSON parsing failure at character 44:',' in {"nrtype": "Phone 1", "number": "12345678"},{"nrtype": "E-Mail 1", "number": "some#email.com"}
I am a little confused. What I want to get is a structure created by the DeserializeJSON() function.
What can I do?

That is not valid JSON as the parser is describing. If you wrap that JSON within square brackets '[' and ']' it would be valid (or at least parsable). They will make it an array of structures. Not sure how to make MySQL return the data within those brackets?
I guess you could add the brackets using ColdFusion but I would prefer to have the source do it correctly.
jsonhack = '[' & queryname.nrJSON & ']';
datarecord = DeserializeJSON(jsonhack);
writeDump(datarecord);
I created an example with your data that you can see here - trycf.com gist
From the comments
The solution indeed was [to add the following to the SQL statement]:
CONTACT('[',
GROUP_CONCAT(
JSON_OBJECT(...)
),
']')

If you have columns with some already containing JSON format String, try this : https://stackoverflow.com/a/45278722/2282880
Portion of code with JSON_MERGE() :
...
CONCAT(
'{"elements": [',
GROUP_CONCAT(
JSON_MERGE(
JSON_OBJECT(
'type', T2.`type`,
'data', T2.`data`
),
CONCAT('{"info": ', T2.`info`, '}')
)
),
']}'
) AS `elements`,
...

Related

jinja2.exceptions.TemplateSyntaxError: expected token ':', got '}'

I'm trying to use an xcom_pull inside an SQL phrase executed by a Snowflake operator in Airflow.
I need the task_id name to use a variable since I want to support different tasks.
I tried this syntax but seems it is not being rendered ok.
Anyone has an idea how to do it?
This is the Python code:
for product, val in PRODUCTS_TO_EXTRACT_INC.items():
product_indicator, prefix = val
params['product_prefix'] = prefix
calculate_to_date = SnowflakeOperator(
dag=dag,
task_id=f'calculate_to_date_{prefix}',
snowflake_conn_id = SF_CONNECTION_ID,
warehouse=SF_WAREHOUSE,
database=BI_DB,
schema=STG_SCHEMA,
role=SF_ROLE,
sql= [ """
{SQL_FILE}
""".format(SQL_FILE="{% include '" + QUERIES_DIR + ETL + "/calculate_to_date.sql'" + " %}")
],
params=params
)
This is the SQL code for calculate_to_date.sql:
select '{{{{ (ti.xcom_pull(key="return_value", task_ids=["calculate_from_date_{}"])[0][0]).get("FROM_DATE") }}}}'.format(params.product_prefix) AS TO_DATE
This is the error message:
File "/home/airflow/gcs/dags/Test/queries/fact_subscriptions_events/calculate_to_date.sql", line 11, in template
select '{{{{ (ti.xcom_pull(key="return_value", task_ids=["calculate_from_date_{}"])[0][0]).get("FROM_DATE") }}}}'.format(params.product_prefix)
jinja2.exceptions.TemplateSyntaxError: expected token ':', got '}'
the correct syntax is
select '{{ (ti.xcom_pull(key="return_value", task_ids="calculate_from_date_{}".format(params.product_prefix))[0]).get("FROM_DATE") }}' AS TO_DATE
it works like a charm

How to exclude escape characters from ROOT when using FOR JSON from SQL server

I am trying to create JSON output from SQL Server using FOR JSON. My JSON record is good however I have a number of elements I need in the ROOT section but this is resulting in escape characters which is causing issues.
My select statement is as follows;
SELECT RTRIM(a.ACCNT_CODE) AS ACCOUNT_REF,
'PK1' AS SUN_DB,
RTRIM(a.TREFERENCE) AS REFERENCE,
RTRIM(x.GNRL_DESCR_25) AS ISR_NUMBER,
'' AS CHECK_DIGIT,
'' AS PAID_DATE,
'' AS PAID_FLAG
FROM PK1_A_SALFLDG a
JOIN PK1_A_SALFLDG_LAD x ON a.ACCNT_CODE = x.ACCNT_CODE
AND a.JRNAL_NO = x.JRNAL_NO
AND a.JRNAL_LINE = x.JRNAL_LINE
WHERE LEN(RTRIM(x.GNRL_DESCR_25)) > 15
FOR JSON PATH, INCLUDE_NULL_VALUES, ROOT('FORMATCODE":"CHMCT","TYPE":"PA_B4B_AUX_CHMCT_ISRReference","DATA');
But the ROOT is being output as
{"FORMATCODE\\":\"CHMCT\\",\\"TYPE\\":\\"PA_B4B_AUX_CHMCT_ISRReference\\",\\"DATA":[
How can I exclude the backslashes from the ROOT element?
It seems like you're trying to embed extra JSON data into ROOT. Instead of trying to do that try instead to use a nested JSON query, e.g.:
SELECT
'CHMCT' as [FORMATCODE],
'PA_B4B_AUX_CHMCT_ISRReference' as [TYPE],
(
SELECT
RTRIM(a.ACCNT_CODE) AS ACCOUNT_REF,
'PK1' AS SUN_DB,
RTRIM(a.TREFERENCE) AS REFERENCE,
RTRIM(x.GNRL_DESCR_25) AS ISR_NUMBER,
'' AS CHECK_DIGIT,
'' AS PAID_DATE,
'' AS PAID_FLAG
FROM PK1_A_SALFLDG a
JOIN PK1_A_SALFLDG_LAD x
ON a.ACCNT_CODE = x.ACCNT_CODE
AND a.JRNAL_NO = x.JRNAL_NO
AND a.JRNAL_LINE = x.JRNAL_LINE
WHERE LEN(RTRIM(x.GNRL_DESCR_25)) > 15
FOR JSON PATH, INCLUDE_NULL_VALUES
) DATA
FOR JSON PATH, WITHOUT_ARRAY_WRAPPER;
Which will yield output similar to...
{
"FORMATCODE": "CHMCT",
"TYPE": "PA_B4B_AUX_CHMCT_ISRReference",
"DATA": [
{
"ACCOUNT_REF": "...",
"SUN_DB": "PK1",
"REFERENCE": "...",
"ISR_NUMBER": "...",
"CHECK_DIGIT": "",
"PAID_DATE": "",
"PAID_FLAG": ""
}
]
}

Replacing a character from mysql table to export to JSON file

I have a very long sized table (over 4M of records,i work with MySql) and it has a lot of records with this string: \\"
I'm trying to export this table to mongodb, but when I import the JSON file mongodb throws to me this error:
Failed: error processing document #18: invalid character 't' after object key:value pair
this is my query:
MySQL
SELECT json_object(
"id", id,
"execution_id", execution_id,
"type", type,
"info", info,
"position", position,
"created_at", json_object("$date", DATE_FORMAT(created_at,'%Y-%m-%dT%TZ')),
"updated_at", json_object("$date", DATE_FORMAT(updated_at,'%Y-%m-%dT%TZ'))
)as 'json'
FROM myTable
INTO OUTFILE 'myPath';
I know the problem is the string, my question is: how can I change this certain string to \"? Manually change it´s not an option, and my knowledge about query is limited. Please help. Thank you for reading me .
The column that has this character is "info", here is an example:
{
"id": 30,
"execution_id": 2,
"type": "PHASE",
"info": "{ \\r\\n \\"title\\": \\"Phase\\",
\\r\\n \\"order\\": \\"1\\",
\\r\\n \\"description\\": \\"Example Phase 1\\",
\\r\\n \\"step\\": \\"end\\",
\\r\\n \\"status\\": \\"True\\"\\r\\n}",
"position": 24,
"created_at": {"$date": "2018-01-11T15:01:46Z"},
"updated_at": {"$date": "2018-01-11T15:01:46Z"}
}
You should be able to do this using the MySQL REPLACE() function.
The backslash is a bit of a special case in the MySQL REPLACE() function, so you will need to use \\ to represent each literal \, thus to replace \\ with \ you need to run something like this:
REPLACE(info,'\\\\','\\')
Your full query would look something like this:
SELECT json_object(
"id", id,
"execution_id", execution_id,
"type", type,
"info", REPLACE(info,'\\\\','\\'),
"position", position,
"created_at", json_object("$date", DATE_FORMAT(created_at,'%Y-%m-%dT%TZ')),
"updated_at", json_object("$date", DATE_FORMAT(updated_at,'%Y-%m-%dT%TZ'))
)as 'json'
FROM myTable
INTO OUTFILE 'myPath';

PostgreSQL jsonb string format

I'm using PostgreSQL jsonb and have the following in my database record:
{"tags": "[\"apple\",\" orange\",\" pineapple\",\" fruits\"]",
"filename": "testname.jpg", "title_en": "d1", "title_ja": "1",
"description_en": "d1", "description_ja": "1"}
and both SELECT statements below retrived no results:
SELECT "photo"."id", "photo"."datadoc", "photo"."created_timestamp","photo"."modified_timestamp"
FROM "photo"
WHERE datadoc #> '{"tags":> ["apple"]}';
SELECT "photo"."id", "photo"."datadoc", "photo"."created_timestamp", "photo"."modified_timestamp"
FROM "photo"
WHERE datadoc -> 'tags' ? 'apple';
I wonder it is because of the extra backslash added to the json array string, or the SELECT statement is incorrect.
I'm running "PostgreSQL 10.1, compiled by Visual C++ build 1800, 64-bit" on Windows 10.
PostgreSQL doc is here.
As far as any JSON parser is concerned, the value of your tags key is a string, not an array.
"tags": "[\"apple\",\" orange\",\" pineapple\",\" fruits\"]"
The string itself happens to be another JSON document, like the common case in XML where the contents of a string happen to be an XML or HTML document.
["apple"," orange"," pineapple"," fruits"]
What you need to do is extract that string, then parse it as a new JSON object, and then query that new object.
I can't test it right now, but I think that would look something like this:
(datadoc ->> 'tags') ::jsonb ? 'apple'
That is, "extract the tags value as text, cast that text value as jsonb, then query that new jsonb value.
Hey there i know this is very late answer, but here is the good approach, with data i have.
initital data in db:
"{\"data\":{\"title\":\"test\",\"message\":\"string\",\"image\":\"string\"},\"registration_ids\":[\"s
tring\"],\"isAllUsersNotification\":false}"
to convert it to json
select (notificationData #>> '{}')::jsonb from sent_notification
result:
{"data": {"image": "string", "title": "string", "message": "string"}, "registration_ids": ["string"], "isAllUsersNotification": false}
getting a data object from json
select (notificationData #>> '{}' )::jsonb -> 'data' from sent_notification;
result:
{"image": "string", "title": "string", "message": "string"}
getting a field from above result:
select (notificationData #>> '{}' )::jsonb -> 'data' ->>'title' from sent_notification;
result:
string
performing where operations,
Q: get records where title ='string'
ans:
select * from sent_notification where (notificationData #>> '{}' )::jsonb -> 'data' ->>'title' ='string'

Cant validate JSON when using different language. Error invalid characters found

First time im trying to create a JSON file,
Im trying to create a JSON file with different language other than English , but when i try to validate, it show Error Invalid characters found.
i tried this
{
"data": [
{
"id": "1",
"title": "Oru Velli Thaaram Vaana Veedhiyil",
"lyrics": "ഒരു വെള്ളിത്താരം വനവീഥിയിൽ തെളിയവേ
കുളിരീറൻ കാറ്റും കുഞ്ഞുതരാട്ട് മൂളവേ
ഇരുളിനലകൾ മൂടും ധരയിതിലൊരു ദീപം
കദനഭാരമെല്ലാം നീക്കിടുന്ന സ്നേഹം
പിറന്നു മണ്ണിലുഷസ്സിൻ ശോഭ പോലെ
(ഒരു വെള്ളിത്താരം…
മരുഭൂവിൽ അലയുമ്പോൾ ആ താരം മുൻപേ
മറയാതെ രാജക്കൾക്കതുമാർഗമായി
മരുഭൂവിൽ അലയുമ്പോൾ ആ താരം മുൻപേ
മറയാതെ രാജക്കൾക്കതുമാർഗമായി
പുൽക്കൂടും തേടിത്തേടി ബെത്ലഹേമിലവരണയുമ്പോൾ
ഗീതങ്ങൾ പാടിപ്പാടി വാനദൂതരും അണയുന്നൂ
തിരുസുതനെ കാണുംനേരം പാടുന്നു ഗ്ലോറിയ …
(ഒരു വെള്ളിത്താരം…
ശാരോനിൻ താഴ്വാരം തഴുകുന്ന കാറ്റെ
വരുമോ എൻ നാഥൻറെ അരികിൽ നീ മെല്ലെ
ശാരോനിൻ താഴ്വാരം തഴുകുന്ന കാറ്റെ
വരുമോ എൻ നാഥൻറെ അരികിൽ നീ മെല്ലെ
തഴുകൂ നിൻ വിരലാൽ നെറുകിൽ സ്നേഹനാഥനെ ആലോലം
പാടൂ നൽ ശ്രുതിയാൽ കാതിൽ സാന്ദ്രമാനന്ദ സംഗീതം
ഈ രാവിൽ പാരാകെ പാടുന്നു ഗ്ലോറിയ
(ഒരു വെള്ളിത്താരം…",
},
{
"id": "2",
"title": "Pukootil Vannu Jaathanayi",
"lyrics": "പുൽക്കൂട്ടിൽ വന്നു ജാതനായി
നക്ഷത്രം ഇന്ന് മിന്നി നിന്നു
ക്രിസ്മസ് രാവിൻറെ ഗാനമായി
വിണ്ണിൽ ആനന്ദമേളമായി താരകം ദീപമായ്
കൺകളിൽ തിളങ്ങി നിന്നു (2 )
ദൂതരാ വീണകൾ മീട്ടിടുന്നിതാ
ലോകരാ കീർത്തനം കേട്ടിടുന്നിതാ
ദേവദാരു പൂത്തു പാതിരാവു പെയ്തു
മഞ്ഞുതുള്ളി വീണവീഥി മിന്നിടുന്നു
( പുൽക്കൂട്ടിൽ)
വിദ്വരോ കാഴ്ചകൾ നല്കിടുന്നിതാ
വിന്നതിൽ നോക്കി സംപ്രീതരായിതാ
കീറ്റുശീല തന്നിൽ ദിവ്യശോഭ കണ്ടു
ആട്ടിടയരെത്തി ആർത്തു പാടിടുന്നു
( പുൽക്കൂട്ടിൽ)",
}
]
}
Error shows as INVALID JSON , Invalid characters found.
please help me to resolve this problem .
Problem Coming from
"lyrics": "ഒരു വെള്ളിത്താരം വനവീഥിയിൽ തെളിയവേ
Error Type
Expecting 'STRING', 'NUMBER', 'NULL', 'TRUE', 'FALSE', '{', '[', got 'undefined'
Reason
Line breaks inside your string . Encode with \n .A string is a sequence of zero or more Unicode characters .