Related
I'm trying to store the JSON bytes to PostgreSQL, but there's a problem.
\u0000 cannot be converted to text.
As you can see below, the JSON contains escape sequences such as \u0000, which it seems PostgreSQL is interpreting as unicode characters, not JSON strings.
err := raws.SaveRawData(data, url)
// if there is "\u0000" in the bytes
if err.Error() == "ERROR: unsupported Unicode escape sequence (SQLSTATE 22P05)" {
// try to remove \u0000, but not work
data = bytes.Trim(data, "\u0000")
e := raws.SaveRawData(data, url) // save data again
if e != nil {
return e // return the same error
}
return nil
}
Origin API data can be access form Here. There is \u0000 in it:
{
"code": 0,
"message": "0",
"ttl": 1,
"data": {
"bvid": "BV1jb411C7m3",
"aid": 42443484,
"videos": 1,
"tid": 172,
"tname": "手机游戏",
"copyright": 1,
"pic": "http://i0.hdslb.com/bfs/archive/c76ee4798bf2ba0efc8449bcb3577d508321c6c5.jpg",
"title": "冰塔：我连你的大招都敢硬抗，所以告诉我谁才是生物女王？！单s冰塔怒砍档案女王巴德尔，谁，才是生物一姐？（手动滑稽）",
"pubdate": 1549100438,
"ctime": 1549100438,
"desc": "bgm：逮虾户\n今天先水一期冰塔的，明天再水\\u0000绿塔的，后天就可以下红莲啦，计划通嘿嘿嘿(º﹃º )",
"desc_v2": [
{
"raw_text": "bgm：逮虾户\n今天先水一期冰塔的，明天再水\\u0000绿塔的，后天就可以下红莲啦，计划通嘿嘿嘿(º﹃º )",
"type": 1,
"biz_id": 0
}
],
"state": 0,
"duration": 265,
"rights": {
"bp": 0,
"elec": 0,
"download": 1,
"movie": 0,
"pay": 0,
"hd5": 0,
"no_reprint": 1,
"autoplay": 1,
"ugc_pay": 0,
"is_cooperation": 0,
"ugc_pay_preview": 0,
"no_background": 0,
"clean_mode": 0,
"is_stein_gate": 0
},
"owner": {
"mid": 39699039,
"name": "明眸-雅望",
"face": "http://i0.hdslb.com/bfs/face/240f74f8706955119575ea6c6cb1d31892f93800.jpg"
},
"stat": {
"aid": 42443484,
"view": 1107,
"danmaku": 7,
"reply": 22,
"favorite": 5,
"coin": 4,
"share": 0,
"now_rank": 0,
"his_rank": 0,
"like": 10,
"dislike": 0,
"evaluation": "",
"argue_msg": ""
},
"dynamic": "#崩坏3#",
"cid": 74479750,
"dimension": {
"width": 1280,
"height": 720,
"rotate": 0
},
"no_cache": false,
"pages": [
{
"cid": 74479750,
"page": 1,
"from": "vupload",
"part": "冰塔：我连你的大招都敢硬抗，所以告诉我谁才是生物女王？！单s冰塔怒砍档案女王巴德尔，谁，才是生物一姐？（手动滑稽）",
"duration": 265,
"vid": "",
"weblink": "",
"dimension": {
"width": 1280,
"height": 720,
"rotate": 0
}
}
],
"subtitle": {
"allow_submit": false,
"list": []
},
"user_garb": {
"url_image_ani_cut": ""
}
}
}
The struct for save is:
type RawJSONData struct {
ID uint64 `gorm:"primarykey" json:"id"`
CreatedAt time.Time `json:"-"`
DeletedAt gorm.DeletedAt `json:"-" gorm:"index"`
Data datatypes.JSON `json:"data"`
URL string `gorm:"index" json:"url"`
}
datatypes.JSON is from gorm.io/datatypes. It seems just is json.RawMessage, it is (extend from?) a []byte.
I use PostgreSQL's JSONB type for storage this data.
Table:
create table raw_json_data
(
id bigserial not null constraint raw_json_data_pke primary key,
created_at timestamp with time zone,
deleted_at timestamp with time zone,
data jsonb,
url text
);
The Unicode escape sequence \u0000 is simply not supported in Postgres TEXT and JSONB columns:
The jsonb type also rejects \u0000 (because that cannot be represented in PostgreSQL's text type)
You can change the column type to JSON:
create table Foo (test JSON);
insert into Foo (test) values ('{"text": "明天再水\u0000绿塔的"}');
-- works
The json data type stores an exact copy of the input text
This has the advantage of keeping the data identical to what you received from the API, in case the escape sequence has some meaning that you need to preserve.
It'll also allow you to query using Postgres JSON operators (e.g. ->>), albeit converting a JSON field with \u0000 to text will still fail:
select test->>'text' from Foo
-- ERROR: unsupported Unicode escape sequence
Columns of type BYTEA also accept any byte sequence without having to manipulate the data. In Gorm, use type:bytea tag:
type RawJSONData struct {
// ... other fields
Data string `gorm:"type:bytea" json:"data"`
}
If any of the above is not acceptable for you, then you must sanitize the input string...
I'm trying to add a "#" at the beginning to each key of a Json object (got it from RabbitMQ api calls)
here is my attempt :
#!/bin/python
# Libraries import
import requests
import json
import sys
import os
# Define URLs
overview="/api/overview"
nodes="/api/nodes"
queues="/api/queues"
# Get credentials from file
with open('/credentials') as json_file:
data = json.load(json_file)
user = data['user']
passwd = data['pass']
# Test which URL we want to call
if ''.join(sys.argv[1]) == "overview":
commande=overview
if ''.join(sys.argv[1]) == "queues":
commande=queues
if ''.join(sys.argv[1]) == "nodes":
commande=nodes
def append(mydict):
return dict(map(lambda (key, value): ("#"+str(key), value), mydict.items()))
def transform(multileveldict):
new = append(multileveldict)
for key, value in new.items():
if isinstance(value, dict):
new[key] = transform(value)
return new
def upper_keys(x):
if isinstance(x, list):
return [upper_keys(v) for v in x]
elif isinstance(x, dict):
return dict((k.upper(), upper_keys(v)) for k, v in x.iteritems())
else:
return x
# Main
response = requests.get("http://localhost:15672" + commande, auth=(user, passwd))
if(response.ok):
json_data = json.loads(response.content)
json = json.dumps(upper_keys(json_data), indent=4)
print(json)
Here is the JSON that I get in "response.content" :
[
{
"NODE": "rabbit#server567",
"EXCLUSIVE": false,
"NAME": "test-01",
"SYNCHRONISED_SLAVE_NODES": [],
"SLAVE_NODES": [],
"AUTO_DELETE": false,
"VHOST": "/",
"ARGUMENTS": {},
"TYPE": "classic",
"DURABLE": false
},
{
"NODE": "rabbit#server567",
"EXCLUSIVE": false,
"NAME": "test-02",
"SYNCHRONISED_SLAVE_NODES": [],
"SLAVE_NODES": [],
"AUTO_DELETE": false,
"VHOST": "/",
"ARGUMENTS": {},
"TYPE": "classic",
"DURABLE": false
},
{
"NODE": "rabbit#server567",
"EXCLUSIVE": false,
"NAME": "test-03",
"SYNCHRONISED_SLAVE_NODES": [],
"SLAVE_NODES": [],
"AUTO_DELETE": false,
"VHOST": "/",
"ARGUMENTS": {},
"TYPE": "classic",
"DURABLE": false
},
{
"MESSAGES_UNACKNOWLEDGED_RAM": 0,
"RECOVERABLE_SLAVES": null,
"CONSUMERS": 0,
"REDUCTIONS": 9700519,
"AUTO_DELETE": false,
"MESSAGE_BYTES_PAGED_OUT": 0,
"MESSAGE_BYTES_UNACKNOWLEDGED": 0,
"REDUCTIONS_DETAILS": {
"RATE": 0.0
},
"MESSAGE_BYTES": 0,
"MESSAGES_UNACKNOWLEDGED": 0,
"CONSUMER_UTILISATION": null,
"EXCLUSIVE": false,
"VHOST": "/",
"GARBAGE_COLLECTION": {
"MAX_HEAP_SIZE": 0,
"MIN_HEAP_SIZE": 233,
"FULLSWEEP_AFTER": 65535,
"MINOR_GCS": 15635,
"MIN_BIN_VHEAP_SIZE": 46422
},
"MESSAGES_DETAILS": {
"RATE": 0.0
},
"SLAVE_NODES": [
"rabbit#server567"
],
"MESSAGE_BYTES_PERSISTENT": 0,
"POLICY": "ha-all",
"MESSAGES_PAGED_OUT": 0,
"NODE": "rabbit#server566",
"HEAD_MESSAGE_TIMESTAMP": null,
"DURABLE": false,
"MESSAGES_READY_RAM": 0,
"STATE": "running",
"ARGUMENTS": {},
"EFFECTIVE_POLICY_DEFINITION": {
"HA-MODE": "all"
},
"MESSAGES_READY": 0,
"MESSAGES_RAM": 0,
"MESSAGE_BYTES_READY": 0,
"SINGLE_ACTIVE_CONSUMER_TAG": null,
"NAME": "test-04",
"MESSAGES_PERSISTENT": 0,
"BACKING_QUEUE_STATUS": {
"MIRROR_SENDERS": 0,
"Q1": 0,
"Q3": 0,
"Q2": 0,
"Q4": 0,
"AVG_ACK_EGRESS_RATE": 0.0,
"MIRROR_SEEN": 0,
"LEN": 0,
"TARGET_RAM_COUNT": "infinity",
"MODE": "default",
"NEXT_SEQ_ID": 0,
"DELTA": [
"delta",
"undefined",
0,
0,
"undefined"
],
"AVG_ACK_INGRESS_RATE": 0.0,
"AVG_EGRESS_RATE": 0.0,
"AVG_INGRESS_RATE": 0.0
},
"MESSAGES": 0,
"IDLE_SINCE": "2020-10-16 13:50:50",
"OPERATOR_POLICY": null,
"SYNCHRONISED_SLAVE_NODES": [
"rabbit#server567"
],
"MEMORY": 10556,
"EXCLUSIVE_CONSUMER_TAG": null,
"MESSAGES_READY_DETAILS": {
"RATE": 0.0
},
"TYPE": "classic",
"MESSAGES_UNACKNOWLEDGED_DETAILS": {
"RATE": 0.0
},
"MESSAGE_BYTES_RAM": 0
}
]
Here, I made every key in uppercase and can display it has JSON but can't find anything to add this "#" to the beginning of each key
PS : I'm new to Python development
Thank you very much
Since you mentioned that you have successfully converted every keys in a dictionary into upper case keys, why don't you reuse the method and change the part where you do upper case into prepending "#"
# the one you provided
def upper_keys(x):
if isinstance(x, list):
return [upper_keys(v) for v in x]
elif isinstance(x, dict):
return dict((k.upper(), upper_keys(v)) for k, v in x.iteritems())
else:
return x
# the modified method
def prepend_hash_keys(x):
if isinstance(x, list):
return [prepend_hash_keys(v) for v in x]
elif isinstance(x, dict):
# this part from k.upper() to "#" + k
return dict(("#" + k, prepend_hash_keys(v)) for k, v in x.iteritems())
else:
return x
Your transform function actually works fine (for Python 2), you just forgot to actually call it! Instead, you call only upper_keys, but not transform:
json = json.dumps(upper_keys(json_data), indent=4) # where's transform?
If you use both one after the other (order does not matter) it should work:
json = {"nested": {"dict": {"with": {"lowercase": "keys"}}}}
print(transform(upper_keys(json)))
# {'#NESTED': {'#DICT': {'#WITH': {'#LOWERCASE': 'keys'}}}}
However, both transform and upper_keys can be simplified a lot using dictionary comprehensions (also available in Python 2), and you can combine both in one function:
def transform_upper(d):
if isinstance(d, dict):
return {"#" + k.upper(): transform_upper(v) for k, v in d.items()}
else:
return d
print(transform_upper(json))
# {'#NESTED': {'#DICT': {'#WITH': {'#LOWERCASE': 'keys'}}}}
From the look of it you already tried something like that in append() function.
If you modify that a bit to have something like this, it may do what you are looking for:
mydict = {
'name':1,
'surname':2
}
def append(mydict):
new_dict = {}
for key, val in mydict.items():
new_dict['#'+key]=val
return new_dict
print(append(mydict))
I'm trying to add compliance policies to Intune using the Graph API. However, when I try to use it it says "Provided JSON isn't in valid JSON format" and "Invalid JSON primitive: True".
The code I'm using comes from the Microsoft Graph samples (found here)
I think it's probably something really simple, but I don't have much experience with Powershell. I can't seem to find it though.
$JSON_Android = #"
{
"#odata.type": "#microsoft.graph.androidCompliancePolicy",
"displayName": "Google_Android_Device_Compliance_Powershell",
"passwordRequired": true,
"passwordMinimumLength": 5,
"passwordRequiredType": "numeric",
"passwordMinutesOfInactivityBeforeLock": 15,
"passwordExpirationDays": null,
"passwordPreviousPasswordBlockCount": 5,
"securityPreventsInstallAppsFromUnknownSource": true,
"securityDisableUsbDebugging": false,
"requireAppVerify": false,
"deviceThreatProtectionEnabled": false,
"deviceThreatProtectionRequiredSecurityLevel": "unavailable",
"securityBlockJailbrokenDevices": true,
"osMinimumVersion": null,
"osMaximumVersion": null,
"minAndroidSecurityPatchLevel": null,
"storageRequireEncryption": true
"securityRequireSafetyNetAttestationBasicIntegrity": false,
"securityRequireSafetyNetAttestationCertifiedDevice": false,
"securityRequireGooglePlayServices": false,
"securityRequireUpToDateSecurityProviders": false,
"securityRequireCompanyPortalAppIntegrity": true
}
"#
####################################################
$JSON_iOS = #"
{
"#odata.type": "microsoft.graph.iosCompliancePolicy",
"displayName": "Apple_iOS_Device_Compliance_Powershell",
"passCodeBlockSimple": true,
"passcodeExpirationDays": null,
"passcodeMinimumLength": 5,
"passcodeMinutesOfInactivityBeforeLock" 15,
"passcodePreviousPasscodeBlockCount": 5,
"passcodeMinimumCharacterSetCount": null,
"passcodeRequiredType": "numeric",
"passcodeRequired": true,
"osMinimumVersion": null,
"osMaximumVersion": null,
"securityBlockJailbrokenDevices": true,
"deviceThreatProtectionEnabled": false,
"deviceThreatProtectionRequiredSecurityLevel": "unavailable",
"managedEmailProfileRequired": false
}
"#
####################################################
$JSON_Windows10 = #"
{
"#odata.type": "microsoft.graph.windows10MobileCompliancePolicy",
"displayName": "Microsoft_Windows_Device_Compliance_Powershell"
"passwordRequired": true,
"passwordBlockSimple": false,
"passwordMinimumLength": 5,
"passwordMinimumCharacterSetCount": null,
"passwordRequiredType": "numeric",
"passwordRequiredToUnlockFromIdle": false,
"passwordExpirationDays": null,
"passwordPreviousPasswordBlockCount": 5,
"passwordMinutesOfInactivityBeforeLock": 15,
"requireHealthyDeviceReport": false,
"osMinimumVersion": null,
"osMaximumVersion": null,
"mobileOsMinimumVersion": null,
"mobileOsMaximumVersion": null,
"earlyLaunchAntiMalwareDriverEnabled": false,
"bitLockerEnabled": false,
"secureBootEnabled": false,
"codeIntegrityEnabled": false,
"storageRequireEncryption": true,
"activeFirewallRequired": false,
"uacRequired": false,
"validOperatingSystemBuildRanges": false
}
"#
Your True and False values should be true and false respectively; note the case difference.
"storageRequireEncryption": true
In the first JSON, there should be a comma after that.
I'm using VBA-web (https://vba-tools.github.io/VBA-Web/) to access trello api, to get cards in a list
My function looks like that:
Public Function CountCardsinList(ListId As String) As Integer
WebHelpers.EnableLogging = False
Dim TrelloClient As New WebClient
TrelloClient.BaseUrl = "https://api.trello.com/1/"
Dim Request As New WebRequest
Request.Format = WebFormat.Json
Request.ResponseFormat = Json
Request.Resource = "lists/{ListId}/cards"
Request.AddUrlSegment "ListId", ListId
Request.AddQuerystringParam "key", TrelloAPIKey
Request.AddQuerystringParam "token", TrelloAPIToken
Request.AddQuerystringParam "filter", "open"
Dim Response As WebResponse
Set Response = TrelloClient.Execute(Request)
If Response.StatusCode = WebStatusCode.Ok Then
Debug.Print Response.Content '
'Response.Data("idList").Count
Debug.Print "CountCardsinList =>>> " & Response.Content
CountCardsinList = Response.Data("idList").Count
Else
MsgBox Response.StatusDescription, vbCritical, "Error " & Response.StatusCode
CountCardsinList = ""
End If
Debug.Print "CountCardsinList =>>> " & Response.Content
'Set CountCardsinList = Request
End Function
I receive correct JSON reply from the api:
[{
"id": "584e798dd570ae187b293e5b",
"checkItemStates": null,
"closed": false,
"dateLastActivity": "2016-12-30T09:24:57.531Z",
"desc": "",
"descData": null,
"idBoard": "57873ba94794058756fa0a96",
"idList": "57873bb3a725f734089702b2",
"idMembersVoted": [],
"idShort": 90,
"idAttachmentCover": null,
"manualCoverAttachment": false,
"idLabels": ["57873ba984e677fd3683bef8"],
"name": "card name / other stuff",
"pos": 1999.9923706054688,
"shortLink": "izoqvWJk",
"badges": {
"votes": 0,
"viewingMemberVoted": false,
"subscribed": false,
"fogbugz": "",
"checkItems": 0,
"checkItemsChecked": 0,
"comments": 0,
"attachments": 0,
"description": false,
"due": "2016-12-26T11:00:00.000Z",
"dueComplete": false
},
"dueComplete": false,
"due": "2016-12-26T11:00:00.000Z",
"idChecklists": [],
"idMembers": ["54f0cc079bf18f2798dda8bd"],
"labels": [{
"id": "57873ba984e677fd3683bef8",
"idBoard": "57873ba94794058756fa0a96",
"name": "Urgent",
"color": "red",
"uses": 14
}],
"shortUrl": "https://trello.com/c/vfvfdvdfv",
"subscribed": false,
"url": "https://trello.com/c/fdvfdvdfv/cfvdfv"
},
{
"id": "5832c2fa7f55fe5637d972ea",
"checkItemStates": null,
"closed": false,
"dateLastActivity": "2016-12-30T09:25:09.222Z",
"desc": "",
"descData": null,
"idBoard": "57873ba94794058756fa0a96",
"idList": "57873bb3a725f734089702b2",
"idMembersVoted": [],
"idShort": 80,
"idAttachmentCover": null,
"manualCoverAttachment": false,
"idLabels": ["57873ba984e677fd3683bef6"],
"name": "other card name",
"pos": 2023.9922790527344,
"shortLink": "XhUPgcsD",
"badges": {
"votes": 0,
"viewingMemberVoted": false,
"subscribed": false,
"fogbugz": "",
"checkItems": 0,
"checkItemsChecked": 0,
"comments": 0,
"attachments": 0,
"description": false,
"due": "2016-12-30T15:00:00.000Z",
"dueComplete": false
},
"dueComplete": false,
"due": "2016-12-30T15:00:00.000Z",
"idChecklists": [],
"idMembers": ["54fdbe1a8ecdf184596c7c07"],
"labels": [{
"id": "57873ba984e677fd3683bef6",
"idBoard": "57873ba94794058756fa0a96",
"name": "Medium",
"color": "yellow",
"uses": 1
}],
"shortUrl": "https://trello.com/c/XhdfvdfvUPgcsD",
"subscribed": false,
"url": "https://trello.com/c/XhUPgcsfdvdffvD/
"
But I cannot correctly count idList -> and I'm trying to get number of cards in a list, by using Response.Data("idList").Count
Any information how to do it proper way? or which is the best way to parse JSON data?
General:
Your JSON isn't properly closed. I added }] to the end to close and placed in cell A1 of activesheet (as I don't have the API info). I then read that in from the cell as if it were response text.
Process:
I then used JSONConverter to parse this string from the sheet. This requires you to also add a reference to Microsoft Scripting Runtime via VBE > Tools > References.
The returned object is a collection of dictionaries. I test each dictionary for the existence of an idList key and if present add 1 to the variable itemCount, which keeps track of how many idLists there are.
Code:
Public Sub GetInfoFromSheet()
Dim jsonStr As String, item As Object, json As Object, itemCount As Long
jsonStr = [A1]
Set json = JsonConverter.ParseJson(jsonStr)
For Each item In json 'collection
If item.Exists("idList") Then itemCount = itemCount + 1
Next item
Debug.Print "idList count: " & itemCount
End Sub
I get below JSON as API response. I want to assert if these 5 key values are present
"
"DayClosingDate": "/Date(-62135596800000)/",
"StartTime": "/Date(-62135596800000)/",
"EndTime": "/Date(-62135596800000)/",
"OpeningBalance": 0,
"CashAmount": 0,
"CreditCardAmount": 0,
"
What should be the best assertion?
{
"d": {
"__type": "WSOffline.Response",
"Status": "0",
"Message": "Success",
"Data": [
{
"DayClosingID": 26030,
"UID": 0,
"StoreID": 0,
"CashRegisterID": 0,
"DayClosingDate": "/Date(-62135596800000)/",
"StartTime": "/Date(-62135596800000)/",
"EndTime": "/Date(-62135596800000)/",
"OpeningBalance": 0,
"CashAmount": 0,
"CreditCardAmount": 0,
"OtherAmount": 0,
"BalanceAmount": 0,
"Remarks": null,
"CreatedByEmployeeID": 0,
"CreatedDatetime": "/Date(-62135596800000)/",
"ClosingDoneByEmployeeID": 0,
"ClosingDoneDateTime": "/Date(-62135596800000)/",
"PaymentDetails": [],
"StartTimeText": null,
"EndTimeText": null,
"CashRegisterName": null,
"DayClosingDateText": null,
"InvoiceName": null,
"InvoiceDate": null,
"StoreName": null,
"CreatedByEmployee": null,
"ClosingDoneByEmployee": null,
"DayClosingDone": null
}
]
}
}
Depends on if the values are dynamic. Are they?
If not, you can easily put the following line in a Response Assertion, with matching rule as 'contains'.
"DayClosingDate": "/Date(-62135596800000)/"
You can do this separately for each assertion you need. You can also opt for the jp#gc - JSON Path Assertion plugin depending on your needs.
If you just want to verify param name, then
Assertion: JSONpath Existence Match
This will be your jsonpath expression:
$..DayClosingDate
Result: will be true if it is present.
Or if you want to verify param value, then
Assertion: JSONpath Match
This will be your jsonpath expression:
$..DayClosingDate
Result: /Date(-62135596800000)/