Related
I'm trying to store the JSON bytes to PostgreSQL, but there's a problem.
\u0000 cannot be converted to text.
As you can see below, the JSON contains escape sequences such as \u0000, which it seems PostgreSQL is interpreting as unicode characters, not JSON strings.
err := raws.SaveRawData(data, url)
// if there is "\u0000" in the bytes
if err.Error() == "ERROR: unsupported Unicode escape sequence (SQLSTATE 22P05)" {
// try to remove \u0000, but not work
data = bytes.Trim(data, "\u0000")
e := raws.SaveRawData(data, url) // save data again
if e != nil {
return e // return the same error
}
return nil
}
Origin API data can be access form Here. There is \u0000 in it:
{
"code": 0,
"message": "0",
"ttl": 1,
"data": {
"bvid": "BV1jb411C7m3",
"aid": 42443484,
"videos": 1,
"tid": 172,
"tname": "手机游戏",
"copyright": 1,
"pic": "http://i0.hdslb.com/bfs/archive/c76ee4798bf2ba0efc8449bcb3577d508321c6c5.jpg",
"title": "冰塔:我连你的大招都敢硬抗,所以告诉我谁才是生物女王?!单s冰塔怒砍档案女王巴德尔,谁,才是生物一姐?(手动滑稽)",
"pubdate": 1549100438,
"ctime": 1549100438,
"desc": "bgm:逮虾户\n今天先水一期冰塔的,明天再水\\u0000绿塔的,后天就可以下红莲啦,计划通嘿嘿嘿(º﹃º )",
"desc_v2": [
{
"raw_text": "bgm:逮虾户\n今天先水一期冰塔的,明天再水\\u0000绿塔的,后天就可以下红莲啦,计划通嘿嘿嘿(º﹃º )",
"type": 1,
"biz_id": 0
}
],
"state": 0,
"duration": 265,
"rights": {
"bp": 0,
"elec": 0,
"download": 1,
"movie": 0,
"pay": 0,
"hd5": 0,
"no_reprint": 1,
"autoplay": 1,
"ugc_pay": 0,
"is_cooperation": 0,
"ugc_pay_preview": 0,
"no_background": 0,
"clean_mode": 0,
"is_stein_gate": 0
},
"owner": {
"mid": 39699039,
"name": "明眸-雅望",
"face": "http://i0.hdslb.com/bfs/face/240f74f8706955119575ea6c6cb1d31892f93800.jpg"
},
"stat": {
"aid": 42443484,
"view": 1107,
"danmaku": 7,
"reply": 22,
"favorite": 5,
"coin": 4,
"share": 0,
"now_rank": 0,
"his_rank": 0,
"like": 10,
"dislike": 0,
"evaluation": "",
"argue_msg": ""
},
"dynamic": "#崩坏3#",
"cid": 74479750,
"dimension": {
"width": 1280,
"height": 720,
"rotate": 0
},
"no_cache": false,
"pages": [
{
"cid": 74479750,
"page": 1,
"from": "vupload",
"part": "冰塔:我连你的大招都敢硬抗,所以告诉我谁才是生物女王?!单s冰塔怒砍档案女王巴德尔,谁,才是生物一姐?(手动滑稽)",
"duration": 265,
"vid": "",
"weblink": "",
"dimension": {
"width": 1280,
"height": 720,
"rotate": 0
}
}
],
"subtitle": {
"allow_submit": false,
"list": []
},
"user_garb": {
"url_image_ani_cut": ""
}
}
}
The struct for save is:
type RawJSONData struct {
ID uint64 `gorm:"primarykey" json:"id"`
CreatedAt time.Time `json:"-"`
DeletedAt gorm.DeletedAt `json:"-" gorm:"index"`
Data datatypes.JSON `json:"data"`
URL string `gorm:"index" json:"url"`
}
datatypes.JSON is from gorm.io/datatypes. It seems just is json.RawMessage, it is (extend from?) a []byte.
I use PostgreSQL's JSONB type for storage this data.
Table:
create table raw_json_data
(
id bigserial not null constraint raw_json_data_pke primary key,
created_at timestamp with time zone,
deleted_at timestamp with time zone,
data jsonb,
url text
);
The Unicode escape sequence \u0000 is simply not supported in Postgres TEXT and JSONB columns:
The jsonb type also rejects \u0000 (because that cannot be represented in PostgreSQL's text type)
You can change the column type to JSON:
create table Foo (test JSON);
insert into Foo (test) values ('{"text": "明天再水\u0000绿塔的"}');
-- works
The json data type stores an exact copy of the input text
This has the advantage of keeping the data identical to what you received from the API, in case the escape sequence has some meaning that you need to preserve.
It'll also allow you to query using Postgres JSON operators (e.g. ->>), albeit converting a JSON field with \u0000 to text will still fail:
select test->>'text' from Foo
-- ERROR: unsupported Unicode escape sequence
Columns of type BYTEA also accept any byte sequence without having to manipulate the data. In Gorm, use type:bytea tag:
type RawJSONData struct {
// ... other fields
Data string `gorm:"type:bytea" json:"data"`
}
If any of the above is not acceptable for you, then you must sanitize the input string...
In the below json file I want to access "934934507945312256", "934934503604174848",.... and then the keys inside them.
But after using UNWIND clause I am unable to access the data of these keys(quote_count,reply_count,etc.) as these keys("934934507945312256" ,"934934503604174848",...) are randomly generated.
{
"934934507945312256": {
"quote_count": 0,
"reply_count": 0,
"hashtags": null,
"datetime": "2017-11-26 23:58:51",
"date": "2017-11-26",
"like_count": 0,
"verified": "False",
"sentiment": 0,
"author": "Judy💯The Resistance",
"location": "Hollywood, California USA🇺🇸",
"tid": "934934507945312256",
"retweet_count": 0,
"type": "retweet",
"media_list": null,
"quoted_source_id": null,
"url_list": null,
"tweet_text": "RT #kylegriffin1: Reminder: The Senate Judiciary Committee gave Jared Kushner a November 27 deadline to turn over the missing records… ",
"author_profile_image": "https://pbs.twimg.com/profi...",
"author_screen_name": "jgirl66",
"author_id": "23737528",
"lang": "en",
"keywords_processed_list": [
"reminder",
"senate judiciary committee",
"kushner november",
"deadline"
],
"retweet_source_id": "934872065471115264",
"mentions": [
"kylegriffin1"
],
"replyto_source_id": null
},
"934934503604174848": {
"quote_count": 0,
"reply_count": 2,
"hashtags": [
"MissUniverse",
"Thailand"
],
"datetime": "2017-11-26 23:58:50",
"date": "2017-11-26",
"like_count": 38,
"verified": "False",
"sentiment": 0,
"author": "P'Hmee7.5",
"location": "Bangkok, Thailand",
"tid": "934934503604174848",
"retweet_count": 105,
"type": "Tweet",
"media_list": null,
"quoted_source_id": null,
"url_list": null,
"tweet_text": "รอโหวต มรญ #MissUniverse #Thailand",
"author_profile_image": "
Thumbnail
",
"author_screen_name": "Peehmee75",
"author_id": "700720806972624897",
"lang": "th",
"keywords_processed_list": null,
"retweet_source_id": null,
"mentions": null,
"replyto_source_id": null
},
"934934336381636608": {
"quote_count": 0,
"reply_count": 0,
"hashtags": null,
"datetime": "2017-11-26 23:58:10",
"date": "2017-11-26",
"like_count": 0,
"verified": "False",
"sentiment": 0,
"author": "selfresqingprncess",
"location": "Maine, USA",
"tid": "934934336381636608",
"retweet_count": 0,
"type": "retweet",
"media_list": null,
"quoted_source_id": null,
"url_list": null,
"tweet_text": "RT #kylegriffin1: Reminder: The Senate Judiciary Committee gave Jared Kushner a November 27 deadline to turn over the missing records… ",
"author_profile_image": "https://pbs.twimg.com/profi...",
"author_screen_name": "slfresqngprncss",
"author_id": "100536014",
"lang": "en",
"keywords_processed_list": [
"reminder",
"keywords_processed_list": [
"reminder",
"senate judiciary committee",
"kushner november",
"deadline"
],
"retweet_source_id": "934872065471115264",
"mentions": [
"kylegriffin1"
],
"replyto_source_id": null
}
}
There's I have tried :-
query = """
with {json} as data UNWIND data as doc FOREACH( l in doc| MERGE (label1:Label1 {author:l.author}))
"""
But I am getting error:- Cannot merge node using null property value for author.
Your JSON file is malformed.
A JSON string cannot contain control characters (like carriage returns or linefeeds). And your JSON contains an extraneous section (that is probably a copy/paste error):
"keywords_processed_list": [
"reminder",
Furthermore, UNWIND can only be used with lists. You cannot use it to get the properties from a map. Once you have fixed your JSON errors, try the following query (I assume that json is a parameter whose value is your JSON data):
UNWIND KEYS($json) AS k
MERGE(label1:Label1 {author: $json[k].author})
I am new in this stuff,
I have input json:
{
"2017-09-15": {
"key": "stops",
"node": {
"lastUpdate": "2017-09-15 05:15:26",
"stops": [
{
"stopId": 11000,
"stopCode": null,
"stopName": null,
"stopShortName": "1000",
"stopDesc": "Tom Tail",
"subName": "1000",
"zoneId": null,
"zoneName": null,
"stopUrl": "",
"locationType": null,
"parentStation": null,
"stopTimezone": "",
"wheelchairBoarding": null,
"virtual": null,
"nonpassenger": null,
"depot": null,
"ticketZoneBorder": null,
"onDemand": null,
"activationDate": "2017-09-14"
},
{
"stopId": 1100,
"stopCode": "04",
"stopName": "One Three",
"stopShortName": "1100",
"stopDesc": "Big Wall",
"subName": "04",
"zoneId": 1,
"zoneName": "Demo2",
"stopUrl": "",
"locationType": null,
"parentStation": null,
"stopTimezone": "",
"wheelchairBoarding": null,
"virtual": 0,
"nonpassenger": 0,
"depot": 0,
"ticketZoneBorder": 0,
"onDemand": 0,
"activationDate": "2017-09-24"
}
]
}
}
}
I can get all stops by filter:
jq -r '.[].node.stops' json
but how to search for a specific stop, for example basing on stopDescr - exact name or containing a string?
thx.
You're on the right track ...
.[].node.stops[] | select(.stopDesc == "Big Wall")
unique
Regarding the supplementary question:
1) sort and unique both expect arrays as input
2) calling unique obviates the need to call sort
3) unique produces an array, so you might want to use unique[]
4) One way to convert a stream of (JSON) values to an array is to wrap the expression producing the stream in square brackets.
Here is an example:
[.[].node.stops[] | select(.stopDesc == "Big Wall") | .zoneName]
| unique
LR script throws warning and fails for the JSON Response.
Below is the script
char *request_json_base;
char *request_json;
web_add_header("Content-Type", "application/json; charset=utf-8");
request_json_base= "{\"GLDefaultId\":0,\"ProjectId\":{ContactProjectId},\"CoaMainId\":10,\"UserId\":{UserId},\"Notes\":\"\",\"PRO_GLDefaultValue\":[{\"GLDefaultValueId\":0,\"GLDefaultId\":0,\"SegmentId\":16,\"Alias\":\"ACT\",\"SegmentType\":\"DETAIL\",\"FunctionTypeId\":0,\"Value\":\"&&&\",\"Mask\":\"&&&\",\"IsRequired\":true,\"LastModified\":null,\"IsCoaRequired\":true},{\"GLDefaultValueId\":0,\"GLDefaultId\":0,\"SegmentId\":17,\"Alias\":\"ST\",\"SegmentType\":\"SET\",\"FunctionTypeId\":0,\"Value\":\"&&\",\"Mask\":\"&&\",\"IsRequired\":false,\"LastModified\":null,\"IsCoaRequired\":false}]}",
request_json = lr_eval_string(request_json_base);
lr_save_string(request_json, "REQUEST_JSON_PARAM");
web_custom_request("PRO_GLDefault",
"URL=https://{URL}/PayrollWebApi/V1/PRO_GLDefault?userId={UserId}",
"Method=POST",
"TargetFrame=",
"Resource=0",
"RecContentType=application/json",
"Referer=",
"Snapshot=t299.inf",
"Mode=HTML",
"EncType=application/json",
"Body={REQUEST_JSON_PARAM}",
LAST);
Below is Replay Log :
Action3_SUNP_ProjectSetup_RatesandSetupInfo.c(81): Warning: The string '"GLDefaultValueId":0,"GLDefaultId":0,"SegmentId":16,"Alias":"ACT","SegmentType":"DETAIL","FunctionTypeId":0,"Value":"&&&","Mask":"&&&","IsRequired":true,"LastModified":null,"IsCoaRequired":true' with parameter delimiters is not a parameter.
Action3_SUNP_ProjectSetup_RatesandSetupInfo.c(81): Warning: The string '"GLDefaultValueId":0,"GLDefaultId":0,"SegmentId":17,"Alias":"ST","SegmentType":"SET","FunctionTypeId":0,"Value":"&&","Mask":"&&","IsRequired":false,"LastModified":null,"IsCoaRequired":false' with parameter delimiters is not a parameter.
Action3_SUNP_ProjectSetup_RatesandSetupInfo.c(83): Notify: Saving Parameter "REQUEST_JSON_PARAM = {"GLDefaultId":0,"ProjectId":1810,"CoaMainId":10,"UserId":67,"Notes":"","PRO_GLDefaultValue":[{"GLDefaultValueId":0,"GLDefaultId":0,"SegmentId":16,"Alias":"ACT","SegmentType":"DETAIL","FunctionTypeId":0,"Value":"&&&","Mask":"&&&","IsRequired":true,"LastModified":null,"IsCoaRequired":true},{"GLDefaultValueId":0,"GLDefaultId":0,"SegmentId":17,"Alias":"ST","SegmentType":"SET","FunctionTypeId":0,"Value":"&&","Mask":"&&","IsRequired":false,"LastModified":null,"IsCoaRequired":false}]}".
Action3_SUNP_ProjectSetup_RatesandSetupInfo.c(85): {"$id":"1","Message":"An error has occurred."}
Below is Body of Original Request call:
"Body={\"GLDefaultId\":0,\"ProjectId\":{ContactProjectId},\"CoaMainId\":10,\"UserId\":{UserId},\"Notes\":\"\",\"PRO_GLDefaultValue\":[{\"GLDefaultValueId\":0,\"GLDefaultId\":0,\"SegmentId\":16,\"Alias\":\"ACT\",\"SegmentType\":\"DETAIL\",\"FunctionTypeId\":0,\"Value\":\"&&&\",\"Mask\":\"&&&\",\"IsRequired\":true,\"LastModified\":null,\"IsCoaRequired\":true},{\"GLDefaultValueId\":0,\"GLDefaultId\":0,\"SegmentId\":17,\"Alias\":\"ST\",\"SegmentType\":\"SET\",\"FunctionTypeId\":0,\"Value\":\"&&\",\"Mask\":\"&&\",\"IsRequired\":false,\"LastModified\":null,\"IsCoaRequired\":false}]}",
LAST);
Below is the Successful Response during the recording:
Request :
{
"GLDefaultId": 0,
"ProjectId": 1806,
"CoaMainId": 10,
"UserId": 2,
"Notes": "",
"PRO_GLDefaultValue": [
{
"GLDefaultValueId": 0,
"GLDefaultId": 0,
"SegmentId": 16,
"Alias": "ACT",
"SegmentType": "DETAIL",
"FunctionTypeId": 0,
"Value": "&&&",
"Mask": "&&&",
"IsRequired": true,
"LastModified": null,
"IsCoaRequired": true
},
{
"GLDefaultValueId": 0,
"GLDefaultId": 0,
"SegmentId": 17,
"Alias": "ST",
"SegmentType": "SET",
"FunctionTypeId": 0,
"Value": "&&",
"Mask": "&&",
"IsRequired": false,
"LastModified": null,
"IsCoaRequired": false
}
]
}
Response:
{
"$id": "1",
"GLDefaultId": 1,
"ProjectId": 1806,
"CoaMainId": 10,
"Notes": "",
"PRO_GLDefaultValue": [
{
"$id": "2",
"GLDefaultValueId": 1,
"GLDefaultId": 1,
"Alias": "ACT",
"FunctionTypeId": 0,
"Value": "&&&",
"SegmentId": 16,
"IsRequired": true,
"ModifiedOn": "7/1/2017 4:48:06 PM",
"ModifiedBy": 2,
"PRO_GLDefault": {
"$ref": "1"
},
"LastModified": null
},
{
"$id": "3",
"GLDefaultValueId": 2,
"GLDefaultId": 1,
"Alias": "ST",
"FunctionTypeId": 0,
"Value": "&&",
"SegmentId": 17,
"IsRequired": false,
"ModifiedOn": "7/1/2017 4:48:06 PM",
"ModifiedBy": 2,
"PRO_GLDefault": {
"$ref": "1"
},
"LastModified": null
}
]
}
Please advise me how to overcome this
Thanks,
Raj
Warning is just that, a warning, not an error, and expected if you have curly braces in your post
What is line 85, where the error is generated?
See these two lines:
request_json_base= "{\"GLDefaultId\":0,\"ProjectId\":{ContactProjectId},\"CoaMainId\":10,\"UserId\":{UserId},\"Notes\":\"\",\"PRO_GLDefaultValue\":[{\"GLDefaultValueId\":0,\"GLDefaultId\":0,\"SegmentId\":16,\"Alias\":\"ACT\",\"SegmentType\":\"DETAIL\",\"FunctionTypeId\":0,\"Value\":\"&&&\",\"Mask\":\"&&&\",\"IsRequired\":true,\"LastModified\":null,\"IsCoaRequired\":true},{\"GLDefaultValueId\":0,\"GLDefaultId\":0,\"SegmentId\":17,\"Alias\":\"ST\",\"SegmentType\":\"SET\",\"FunctionTypeId\":0,\"Value\":\"&&\",\"Mask\":\"&&\",\"IsRequired\":false,\"LastModified\":null,\"IsCoaRequired\":false}]}",
request_json = lr_eval_string(request_json_base);
They are not doing what you think they are doing. You are neither allocating space for this data ( see malloc() in combination with a char * variable), nor are you assigning the value to the variable correctly for the C language (see strcpy() )
Another reference here on Stackoverflow
How to correctly assign a new string value?
I am passing a string like this whithin the service header.
FormData = "{\"LeaveEntryCode\":0,\"RequestId\":0,\"EmployeeCode\":17227,\"LeaveYear\":2016,\"LeaveTypeCode\":1,\"BaseType\":\"ess\",\"StartDate\":\"2016-08-2T00:00:00\",\"EndDate\":\"2016-08-2T00:00:00\",\"NoOfDays\":1.0,\"StartDateSession\":\"full\",\"EndDateSession\":\"half\",\"PreApproved\":false,\"ForDate\":\"1901-01-01T00:00:00\",\"Remarks\":\"Test\",\"RequestStatus\":\"P\",\"Deleted\":false,\"Status\":false,\"CreatedBy\":0,\"CreatedDate\":\"0001-01-01T00:00:00\",\"UpdatedBy\":0,\"UpdatedDate\":\"0001-01-01T00:00:00\",\"DeletedBy\":0,\"DeletedDate\":\"0001-01-01T00:00:00\",\"ModuleId\":2,\"ObjectId\":20,\"StartDateString\":\"08/2/2016\",\"EndDateString\":\"08/2/2016\",\"LeaveDayList\":[\"08/02/2016-FH,08/02/2016-SH\"],\"SystemLeaveTypeCode\":\"ANN\",\"LeaveTypeName\":\"ANNUAL\",\"Employee\":\"\",\"LieuDayList\":null,\"LeaveReasonCode\":0,\"CoveringPersonCode\":0,\"AttachedDocument\":null,\"BaseLeaveType\":\"ANN\",\"CoveringPersonName\":null,\"LeaveReasonName\":null,\"DocumentSource\":\"LEAVE\"}";
This FormData is the header name.
But I am getting this error in postman
"Message": "An error has occurred.",
"ExceptionMessage": "Invalid property identifier character: \\. Path '', line 1, position 1.",
"ExceptionType": "Newtonsoft.Json.JsonReaderException",
"StackTrace": " at Newtonsoft.Json.JsonTextReader.Pars
What is the reson for this?
Please help me.
Thanks
Select type as raw and content as Json and copy below json in the body.
{
"LeaveEntryCode": 0,
"RequestId": 0,
"EmployeeCode": 17227,
"LeaveYear": 2016,
"LeaveTypeCode": 1,
"BaseType": "ess",
"StartDate": "2016-08-2T00:00:00",
"EndDate": "2016-08-2T00:00:00",
"NoOfDays": 1.0,
"StartDateSession": "full",
"EndDateSession": "half",
"PreApproved": false,
"ForDate": "1901-01-01T00:00:00",
"Remarks": "Test",
"RequestStatus": "P",
"Deleted": false,
"Status": false,
"CreatedBy": 0,
"CreatedDate": "0001-01-01T00:00:00",
"UpdatedBy": 0,
"UpdatedDate": "0001-01-01T00:00:00",
"DeletedBy": 0,
"DeletedDate": "0001-01-01T00:00:00",
"ModuleId": 2,
"ObjectId": 20,
"StartDateString": "08/2/2016",
"EndDateString": "08/2/2016",
"LeaveDayList": ["08/02/2016-FH,08/02/2016-SH"],
"SystemLeaveTypeCode": "ANN",
"LeaveTypeName": "ANNUAL",
"Employee": "",
"LieuDayList": null,
"LeaveReasonCode": 0,
"CoveringPersonCode": 0,
"AttachedDocument": null,
"BaseLeaveType": "ANN",
"CoveringPersonName": null,
"LeaveReasonName": null,
"DocumentSource": "LEAVE"
}