Related
I'm trying to retrieve the data from this dictionary and for some reason I cannot seem to acquire it. I'm new to parsing JSON so apologies if this is rough.
let temp = json["list"].arrayValue.map({$0["main"].dictionaryValue})
print(temp[0])
Here I am setting a value equal to the dictionary from the JSON. However, I know I need to add the key's value that I'm searching for. To be clear, I am searching for the "temp" key which in the example is equal to 28.19999...
Here is an example of the JSON:
"list" : [
{
"dt" : 1641524400,
"main" : {
"humidity" : 68,
"sea_level" : 1014,
"temp_max" : 29.260000000000002,
"feels_like" : 28.199999999999999,
"temp_min" : 28.199999999999999,
"grnd_level" : 1004,
"temp" : 28.199999999999999,
"temp_kf" : -0.58999999999999997,
"pressure" : 1014
},{
"dt" : 1641546000,
"main" : {
"pressure" : 1009,
"feels_like" : 20.93,
"temp_max" : 27.100000000000001,
"temp" : 27.100000000000001,
"humidity" : 83,
"grnd_level" : 999,
"sea_level" : 1009,
"temp_min" : 27.100000000000001,
"temp_kf" : 0
},
"sys" : {
"pod" : "n"
},
"pop" : 0.41999999999999998,
"wind" : {
"deg" : 354,
"speed" : 5.4100000000000001,
"gust" : 10.58
},
"visibility" : 6695,
"weather" : [
{
"main" : "Snow",
"id" : 600,
"description" : "light snow",
"icon" : "13n"
}
],
"snow" : {
"3h" : 0.26000000000000001
},
"clouds" : {
"all" : 100
},
"dt_txt" : "2022-01-07 09:00:00"
},
{
"dt" : 1641556800,
"main" : {
"temp_min" : 26.82,
"humidity" : 90,
"pressure" : 1008,
"temp_kf" : 0,
"temp" : 26.82,
"feels_like" : 18.879999999999999,
"sea_level" : 1008,
"temp_max" : 26.82,
"grnd_level" : 998
},
"sys" : {
"pod" : "n"
},
"pop" : 0.97999999999999998,
"wind" : {
"deg" : 310,
"gust" : 14.359999999999999,
"speed" : 7.5199999999999996
}]
Found my answer:
let temp = json["list"].arrayValue.map({$0["main"]["temp"].stringValue})
Hello can someone help me extract the value of user parameter which is "testuser1"
I tried to use this JSON Path expression $..data I was able to extract the entire response but unable to extract user parameter. Thanks in advance
{
"data": "{ "took" : 13, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 1.0, "hits" : [ { "_index" : "bushidodb_history_network_eval_ea9656ef-0a9b-474b-8026-2f83e2eb9df1_2021-april-10", "_type" : "network", "_id" : "6e2e58be-0ccf-3fb4-8239-1d4f2af322e21618059082000", "_score" : 1.0, "_source" : { "misMatches" : [ "protocol", "state", "command" ], "instance" : "e3032804-4b6d-3735-ac22-c827950395b4|0.0.0.0|10.179.155.155|53|UDP", "protocol" : "UDP", "localAddress" : "0.0.0.0", "localPort" : "12345", "foreignAddress" : "10.179.155.155", "foreignPort" : "53", "command" : "ping yahoo.com ", "user" : "testuser1", "pid" : "10060", "state" : "OUTGOINGFQ", "rate" : 216.0, "originalLocalAddress" : "192.168.100.229", "exe" : "/bin/ping", "md5" : "f9ad63ce8592af407a7be43b7d5de075", "dir" : "", "agentId" : "abcd-dcd123", "year" : "2021", "month" : "APRIL", "day" : "10", "hour" : "12", "time" : "1618059082000", "isMerged" : false, "timestamp" : "Apr 10, 2021 12:51:22 PM", "metricKey" : "6e2e58be-0ccf-3fb4-8239-1d4f2af322e2", "isCompliant" : false }, "sort" : [ 1618059082000 ] } ] }, "aggregations" : { "count_over_time" : { "buckets" : [ { "key_as_string" : "2021-04-10T08:00:00.000-0400", "key" : 1618056000000, "doc_count" : 1 } ] } }}",
"success": true,
"message": {
"code": "S",
"message": "Get Eval results Count Success"
}
}
Actual Response:
Images
What you posted doesn't look like a valid JSON to me.
If in reality you're getting what's in your image, to wit:
{
"data": "{ \"took\" : 13, \"timed_out\" : false, \"_shards\" : { \"total\" : 5, \"successful\" : 5, \"skipped\" : 0, \"failed\" : 0 }, \"hits\" : { \"total\" : 1, \"max_score\" : 1.0, \"hits\" : [ { \"_index\" : \"bushidodb_history_network_eval_ea9656ef-0a9b-474b-8026-2f83e2eb9df1_2021-april-10\", \"_type\" : \"network\", \"_id\" : \"6e2e58be-0ccf-3fb4-8239-1d4f2af322e21618059082000\", \"_score\" : 1.0, \"_source\" : { \"misMatches\" : [ \"protocol\", \"state\", \"command\" ], \"instance\" : \"e3032804-4b6d-3735-ac22-c827950395b4|0.0.0.0|10.179.155.155|53|UDP\", \"protocol\" : \"UDP\", \"localAddress\" : \"0.0.0.0\", \"localPort\" : \"12345\", \"foreignAddress\" : \"10.179.155.155\", \"foreignPort\" : \"53\", \"command\" : \"pingyahoo.com\", \"user\" : \"testuser1\", \"pid\" : \"10060\", \"state\" : \"OUTGOINGFQ\", \"rate\" : 216.0, \"originalLocalAddress\" : \"192.168.100.229\", \"exe\" : \"/bin/ping\", \"md5\" : \"f9ad63ce8592af407a7be43b7d5de075\", \"dir\" : \"\", \"agentId\" : \"abcd-dcd123\", \"year\" : \"2021\", \"month\" : \"APRIL\", \"day\" : \"10\", \"hour\" : \"12\", \"time\" : \"1618059082000\", \"isMerged\" : false, \"timestamp\" : \"Apr10, 202112: 51: 22PM\", \"metricKey\" : \"6e2e58be-0ccf-3fb4-8239-1d4f2af322e2\", \"isCompliant\" : false }, \"sort\" : [ 1618059082000 ] } ] }, \"aggregations\" : { \"count_over_time\" : { \"buckets\" : [ { \"key_as_string\" : \"2021-04-10T08: 00: 00.000-0400\", \"key\" : 1618056000000, \"doc_count\" : 1 } ] } }}",
"success": true,
"message": {
"code": "S",
"message": "Get Eval results Count Success"
}
}
the easiest way is just using 2 JSON Extractors:
Extract data attribute value into a JMeter Variable from the response
Extract user attribute value into a JMeter variable from ${data} JMeter Variable:
Demo:
If the response looks like exactly you posted you won't be able to use JSON Extractors and will have to treat it as normal text so your choice is limited to Regular Expression Extractor, example regular expression:
"user"\s*:\s*"(\w+)"
Add Regular Expression extractor with the corresponding request and extract it. Use the below expression.
Expression: "user" : "(.*?)"
Ref: https://jmeter.apache.org/usermanual/regular_expressions.html
Regular Expression Extractor Sample
I'm new to json and want to extract data (blocks) from a specific json. I've tried to find information on how to do this with jq but so far I cannot seem to get what I want.
My json:
{
"now" : 1589987097.9,
"aircraft" : [
{
"mlat" : [],
"rssi" : -26.2,
"track" : 319,
"speed" : 354,
"messages" : 16,
"seen" : 0.7,
"altitude" : 38000,
"vert_rate" : 0,
"hex" : "44b5b4",
"tisb" : []
},
{
"squawk" : "6220",
"altitude" : 675,
"seen" : 1.1,
"messages" : 7220,
"tisb" : [],
"hex" : "484a95",
"mlat" : [],
"rssi" : -22
},
{
"hex" : "484846",
"tisb" : [],
"messages" : 20,
"speed" : 89,
"seen" : 0.4,
"squawk" : "7000",
"altitude" : 500,
"rssi" : -23.7,
"track" : 185,
"mlat" : []
},
{
"category" : "B1",
"mlat" : [],
"rssi" : -24.3,
"flight" : "ZSGBX ",
"altitude" : 3050,
"squawk" : "7000",
"seen" : 16.8,
"messages" : 37,
"tisb" : [],
"hex" : "00901a"
}
],
"messages" : 35857757
}
I would like to reformat this json to only include 'blocks' that contain specific hex values.
So for example, I want I would like my output to contain 44b5b4 and 00901a:
{
"now" : 1589987097.9,
"aircraft" : [
{
"mlat" : [],
"rssi" : -26.2,
"track" : 319,
"speed" : 354,
"messages" : 16,
"seen" : 0.7,
"altitude" : 38000,
"vert_rate" : 0,
"hex" : "44b5b4",
"tisb" : []
},
{
"category" : "B1",
"mlat" : [],
"rssi" : -24.3,
"flight" : "ZSGBX ",
"altitude" : 3050,
"squawk" : "7000",
"seen" : 16.8,
"messages" : 37,
"tisb" : [],
"hex" : "00901a"
}
],
"messages" : 35857757
}
Can someone tell me how to remove all items not having those 2 hex identifiers but still keep the same json structure?
Thanks a lot!
Do a select() on the array aircraft, matching only the required hex values. The map() function takes input the entire array and the result of the select operation i.e. filtering of objects based on the .hex value is updated back |= to the original array and the rest of the fields are kept intact.
jq '.aircraft |= map(select(.hex == "44b5b4" or .hex == "00901a"))' json
Select blocks whose hex matches one of the specific values and update aircraft to leave only those.
.aircraft |= map(select(.hex | IN("44b5b4", "00901a")))
Online demo
I'm trying to scrape Amazon's Goldbox page by trying to extract the JSON object responsible for the deal details (dealdetails).
I've tried to extract all the JSON within the 40th script tag, however I came out with 15000 lines of code
The JSON within the page is like this:
<script type="text/javascript">(function(f) {var _np=(window.P._namespace("GoldboxMobileMason"));if(_np.guardFatal){_np.guardFatal(f)(_np);}else{f(_np);}}(function(P) {
window.gb = window.gb || {};
{
"GDS" : {
"baseRetryInterval" : 4000,
"maxRetries" : 0,
"ajaxTimeout" : 10000
}
},
{
"GD" : {
"baseRetryInterval" : 4000,
"maxRetries" : 1,
"ajaxTimeout" : 10000
}
},
{
"WD" : {
"baseRetryInterval" : 4000,
"maxRetries" : 0,
"ajaxTimeout" : 10000
}
}
"dealDetails" : {
"3b009cf9" : {
"egressUrl" : "https://www.amazon.com/Meredith-Martha-Stewart-Living/dp/B002PXW0EO",
"maxDealPrice" : "5.49",
"offerID" : 000
"maxPrevPrice" : "5.49",
"minBAmount" : "49.9",
"itemType" : "SINGLE_ITEM",
"minPercentOff" : 89,
"items" : [
]
},
"f87c994b" : {
"egressUrl" : "https://www.amazon.com/s/?url=search-
"reviewAsin" : "B073VYKTZN",
"maxListPrice" : "159.99",
"isMAP" : "0",
"displayPriority" : "0",
"isEligibleForFreeShipping" : "0",
"isPrimeEligible" : "1",
"dealID" : "f87c994b",
"description" : "Save 50% on JUVEA All Natural Talalay Latex Pillows",
"minBAmount" : "99.99",
"currencyCode" : "USD",
"minListPrice" : "129.99",
"merchantID" : "A21VHZ1TV3ZUZI",
"score" : "0",
"bKind" : "OP",
"msToFeatureEnd" : "0",
},
"responseMetadata" : {
"continueRetries" : "1",
"baseRetryInterval" : "12000"
}
};
window.gb.controller.registerWidget(widgetToRegister);
});
}));</script>
I tried using Regex but I think I'm doing it wrong:
page = requests.get(primary_url, auth=('user', 'pass'), headers=headers)
soup = BeautifulSoup(page.text, 'lxml')
data = soup.select("[type='text/javascript']")[40]
raw = "dealdetails" + "\n".join(str(data.find("script")).split("\n")[4:-3])
print(raw)
json_obj = json.loads(raw)
The end result must be:
"dealDetails" : {
"3b009cf9" : {
"egressUrl" : "https://www.amazon.com/Meredith-Martha-Stewart-Living/dp/B002PXW0EO",
"maxDealPrice" : "5.49",
"offerID" : 000
"maxPrevPrice" : "5.49",
"minBAmount" : "49.9",
"itemType" : "SINGLE_ITEM",
"minPercentOff" : 89,
"items" : [
]
},
"f87c994b" : {
"egressUrl" : "https://www.amazon.com/s/?url=search-
"reviewAsin" : "B073VYKTZN",
"maxListPrice" : "159.99",
"isMAP" : "0",
"displayPriority" : "0",
"isEligibleForFreeShipping" : "0",
"isPrimeEligible" : "1",
"dealID" : "f87c994b",
"description" : "Save 50% on JUVEA All Natural Talalay Latex Pillows",
"minBAmount" : "99.99",
"currencyCode" : "USD",
"minListPrice" : "129.99",
"merchantID" : "A21VHZ1TV3ZUZI",
"score" : "0",
"bKind" : "OP",
"msToFeatureEnd" : "0",
},
"responseMetadata" : {
"continueRetries" : "1",
"baseRetryInterval" : "12000"
}
};
My best guess is:
re.search(r'^{.*?^}', script_content, re.MULTILINE | re.DOTALL)[0]
but if the indenting is different you will need to adjust it.
fixed_str = [your json above, fixed into valid json format]
target = fixed_str.replace("dealDetails",'xxx{ "dealDetails').split("xxx") #this splits the script tag by first removing preceding irrelevant stuff
final = target[1].replace("}\n};","}}\n}xxx").split('xxx') #this splits it again by dropping trailing irrelevant stuff
json_obj = json.loads(final[0])
json_obj
And, if all works well :), it should get you your desired end result...
i'm trying to set all the parsed information in json in a dict variable but it returns an empty dict. when i get the array value, everything works completely fine.
here is my code:
let dic = json.arrayValue
for each in dic {
let data = each["data"].dictionaryValue
print (data)
let date = each["date"].stringValue
print (date)
}
parsing date works fine too. and note that my json file is not empty. because when i get the arrayValue everything is fine. here is the output when i print each["data"].arrayValue:
[{
"factoryPrice" : 0,
"size" : 25,
"t5" : 0,
"t3" : 0,
"type" : 1,
"bongahPrice" : 2435,
"sherkat" : "",
"priceConfirmed" : 1,
"id" : 1658,
"factory" : 9,
"exist" : true,
"t1" : 0,
"provice" : 1,
"properties" : {
"طول" : "12 متری",
"info" : "",
"استاندارد" : "A2",
"standard" : "A3",
"رنگ" : "مشکی",
"نوع" : "آجدار"
},
"factoryName" : "نیشابور",
"city" : 306,
"name" : "",
"phoneNumber" : "09338810407",
"createdAt" : "2018-02-16 12:52:50",
"ownerId" : 282,
"shomareSabt" : "",
"t4" : 0,
"profileType" : 0,
"t2" : 0,
"modirName" : "آرزومند",
"bongahName" : "میلگرد تهران",
"updatedAt" : "11:36",
"weight" : 22,
"group" : 57,
"bongahAddress" : "بازاراهن شادباد بوستان بلوک B",
"bongahPhone" : "02166139083"
}]
and this only one of the arrays. i get multiple arrays in response.
so what should i do?
The JSON you have has an array as its top level element. This is clear by the first character: [.
You can't directly get a dictionary value for your JSON because it represents an array. That's the reason trying to get dictionaryValue returns nil.