I have the following JSON which parses OK:
It is just some term code data that I want to send.
[{
"TermCode": 2164,
"ACAD_LEVEL_BOT": "GR",
"ACAD_LEVEL_EOT": "GR",
"ACAD_CAREER": "GRAD",
"UA_PRIM_MAJ_PLN": "DPMDP",
"UA_DEGR_DT": "1900-01-01",
"UA_PRIM_MJ_PLN_DES": "Development Practice",
"UA_PRIM_MIN_PLN_D": "-",
"CUM_GPA": 3.707,
"CUR_GPA": 4,
"TOT_CUMULATIVE": 300,
"UNT_PASSD_PRGRSS": 10,
"UNT_TAKEN_PRGRSS": 10,
"UA_PRIM_MJ_PLN_OWN": "School of Anthropology",
"ADMIT_TYPE": "GRD",
"ADMIT_TERM": "2144",
"UA_FA_ST_RES_DESCR": "Resident",
"VISA_PERMIT_TYPE": "-",
"UA_SALT_ROSTER": "-",
"UA_SALT_STATUS": "-",
"DEGR_CHKOUT_STAT": "-",
"EXP_GRAD_TERM": "-",
"UNT_TRNSFR": 0,
"ACAD_PROG_LD": "Graduate Degree Seeking",
"TOT_TRNSFR": 0,
"TUITION_RES": "RES",
"UA_CITIZEN_COUNTRY": "United States",
"UA_CITIZEN_ST_DESC": "Citizen",
"ACADEMIC_LOAD": "F",
"UNT_AUDIT": 0,
"CAMPUS": "MAIN",
"ACADEMIC_YEAR": "2016-2017",
"UA_DEGREE_LEVEL": "Masters",
"ACAD_PROG": "GDEG",
"UA_SEC_MAJ_PLN": "-",
"ACAD_ORG": "0410",
"EFF_START_DT": "2016-12-18 20:37:08",
"CURRENT_IND": "N",
"PROG_STATUS": "AC",
"PROG_ACTION": "MATR"
}]
I want to use this array as a field named term data so I try:
[{
termdata: [{
"TermCode": 2164,
"ACAD_LEVEL_BOT": "GR",
"ACAD_LEVEL_EOT": "GR",
"ACAD_CAREER": "GRAD",
"UA_PRIM_MAJ_PLN": "DPMDP",
"UA_DEGR_DT": "1900-01-01",
"UA_PRIM_MJ_PLN_DES": "Development Practice",
"UA_PRIM_MIN_PLN_D": "-",
"CUM_GPA": 3.707,
"CUR_GPA": 4,
"TOT_CUMULATIVE": 300,
"UNT_PASSD_PRGRSS": 10,
"UNT_TAKEN_PRGRSS": 10,
"UA_PRIM_MJ_PLN_OWN": "School of Anthropology",
"ADMIT_TYPE": "GRD",
"ADMIT_TERM": "2144",
"UA_FA_ST_RES_DESCR": "Resident",
"VISA_PERMIT_TYPE": "-",
"UA_SALT_ROSTER": "-",
"UA_SALT_STATUS": "-",
"DEGR_CHKOUT_STAT": "-",
"EXP_GRAD_TERM": "-",
"UNT_TRNSFR": 0,
"ACAD_PROG_LD": "Graduate Degree Seeking",
"TOT_TRNSFR": 0,
"TUITION_RES": "RES",
"UA_CITIZEN_COUNTRY": "United States",
"UA_CITIZEN_ST_DESC": "Citizen",
"ACADEMIC_LOAD": "F",
"UNT_AUDIT": 0,
"CAMPUS": "MAIN",
"ACADEMIC_YEAR": "2016-2017",
"UA_DEGREE_LEVEL": "Masters",
"ACAD_PROG": "GDEG",
"UA_SEC_MAJ_PLN": "-",
"ACAD_ORG": "0410",
"EFF_START_DT": "2016-12-18 20:37:08",
"CURRENT_IND": "N",
"PROG_STATUS": "AC",
"PROG_ACTION": "MATR"
}] }]
However, this doesn't parse. What is wrong with my syntax?
There is an error in Stack Overflow that is not letting me end my question here so I have to add more description please down down arrow me I didn't have a choice. So my next step would be to have json such as:
[{ termdata:jsonarray,coursedata:jsonarray,admitdata:jsonarray}]
Any thoughts on that would be appreciated also.
To debug small snippets such as yours, you could use https://jsonlint.com,
which gives helpful error messages making it easy to pinpoint the problem.
As an alternative that also works for large files, you might like to consider jq. With your snippet in the file "input.txt", the invocation:
jq empty input.txt
produces the error message:
parse error: Invalid literal at line 2, column 9
This amounts to saying that the first key was not properly quoted.
(Here, empty has the effect of suppressing normal output.)
Let jq do the walking
Better yet, you can save yourself the hassle by letting jq do the work:
jq '[{termdata:.}]' original.json
This will produce the desired output. Unless specifically instructed otherwise, jq takes JSON (or a JSON stream) and produces JSON (or a JSON stream).
You could also use jq to do more sophisticated transformations with multiple input files.
Related
I have two requests that return response with similar JSON structure. When I try to use JSON extractor on one, it works properly but when I try to extract value in the same way from the second one, it doesn't work. But let's cut to the chase.
My first response looks like this:
{
"values": [
{
"id": 1,
"name": "Fendi",
"logoId": null,
"belongsToUser": true
},
{
"id": 2,
"name": "Jean Paul Gaultier",
"logoId": null,
"belongsToUser": true
},
{
"id": 3,
"name": "Nike",
"logoId": null,
"belongsToUser": false
},
{
"id": 4,
"name": "Adidas",
"logoId": null,
"belongsToUser": true
}
]
}
And I try to extract ID of the object that "belongsToUser": false in this JSON Extractor:
JSON path expression: values[?(#.belongsToUser == false)].id
Match No.: 0
Default Values: null
And it works perfecty fine.
However, when I try this way on my second response, it doesn't work.
The response looks like this:
{
"values": [
{
"id": 12,
"brandName": "Fendi",
"productCategoryName": "Shoes",
"size": "38",
"colorNames": [
"color_green"
],
"date": 1536537600000,
"imageId": null,
"title": "Money",
"numberOfOffers": 0,
"status": "ONGOING"
},
{
"id": 13,
"brandName": "Fendi",
"productCategoryName": "Shoes",
"size": "38",
"colorNames": [
"color_green"
],
"date": 1536537600000,
"imageId": null,
"title": "Exchange",
"numberOfOffers": 0,
"status": "ONGOING"
}
]
}
I try to get id of object that has title variable = "Money" with JSON extractor:
JSON path expression: values[?(#.title == 'Money')].id
Match No.: 0
Default Values: null
But it doesn't find id value and sets my JMeter variable to null.
I also tried to leave Money unquoted or in double quotes and tried different JSON path expresions, like
$.values[?(#.title == 'Money')].id
$..[?(#.title == 'Money')].id
$.[?(#.title == 'Money')].id
But none of these seems to work. Do you have any idea how my JSON path expression shoud look to work properly?
And why doesn't it work in second case when it works in first? Is it because objets in second response have inside array?
I have used your code and it is giving the correct results. Please check the below images.
I have tried with version 3.1 also and it is working fine.
Hope this helps.
Check the below image for different types of options in view result tree.
The $..[?(#.title == 'Money')].id expression should work just fine:
Most probably your JMeter installation is corrupt and you experience some form of jar hell due to some clashing library in JMeter Classpath (it might be caused by presence of deprecated JSON Plugins or similar) . So I would recommend obtaining clean latest version of JMeter and trying out your test on it. If you're using any plugins - install them using JMeter Plugin Manager
If you are not in position to re-install JMeter you can try to get to the bottom of the issue by looking into jmeter.log file. If there are no suspicious entries - add the next line to log4j2.xml file:
<Logger name="org.apache.jmeter.extractor.json" level="debug" />
What started as my personal initiative, ended up being a quiet interesting ( may I say, challenging to some degree) project. My company decided to phase out one product and replace it with new one, which instead of storing data in mdb files, uses JSON files. So I took the initiative to create a converter that will read already created mdb files and convert them into the new format JSON.
However, now I'm at wits-ends with this one:
I can read mdb files, run query to extract specific data.
By placing the targetobj inside the FOR LOOP, I managed to extract data for each row and fed into a dict(targetobj)
for val in rows:
targetobj={"connection_props": {"port": 7800, "service": "", "host": val.Hostname, "pwd": "", "username": ""},
"group_list": val.Groups, "cpu_core_cnt": 2, "target_name": "somename", "target_type": "somethingsamething",
"os": val.OS, "rule_list": [], "user_list": val.Users}
if I print targetobj to console, I can clearly get all extracted values for each row.
Now, my quest is to have the obtained results ( for each row), inserted into the main_dict under the key targets:[]. ( Please see sample of JSON file for illustration)
main_dict = {"changed_time": 0, "year": 0, "description": 'blahblahblah', 'targets':[RESULTS FROM TARGETOBJ SHOULD BE ADDED HERE],"enabled": False}
so for example my Json file should have structure such as:
{"changed_time":1234556,
"year":0,
"description":"blahblahblah",
"targets":[
{"group_list":["QA"],
"cpu_core_cnt":1,
"target_name":"NewTarget",
"os":"unix",
"target_type":"",
"rule_list":[],
"user_list":[""],"connection_props":"port":someport,"service":"","host":"host1","pwd":"","username":""}
},
{"group_list":[],
"cpu_core_cnt":2,
"target_name":"",
"os":"unix",
"target_type":"",
"rule_list":[],
"user_list":["Web2user"],
"connection_props":{"port":anotherport,"service":"","host":"host2","pwd":"","username":""}}
],
"enabled":false}
So far I've been tweaking here and there, to have the results written as intended, however each time,I'm getting only the last row values written.
ie.: putting the targetobj as a variable inside the targets:[]
{"changed_time": 0, "year": 0, "description": 'ConvertedConfigFile', 'targets':[targetobj],
I know I'm missing something, I just need to find what and where.
Any help would be highly appreciated.
thank you
Just create your main_dict first and append to it in your loop, i.e.:
main_dict = {"changed_time": 0,
"year": 0,
"description": "blahblahblah",
"targets": [], # a new list for the target objects
"enabled": False}
for val in rows:
main_dict["targets"].append({ # append this dict to the targets list of main_dict
"connection_props": {
"port": 7800,
"service": "",
"host": val.Hostname,
"pwd": "",
"username": ""},
"group_list": val.Groups,
"cpu_core_cnt": 2,
"target_name": "somename",
"target_type": "somethingsamething",
"os": val.OS,
"rule_list": [],
"user_list": val.Users
})
I'd like to be able to selectively remove elements from a json schema. Imagine a json object that contains a larger but similar array of users like this
[{
"users": [{
"firstName": "Nancy",
"socialSecurityNumber": "123-45-6789",
"sex": "Female",
"id": "1234",
"race": "Smith",
"lastName": "Logan"
}, {
"firstName": "Charles",
"socialSecurityNumber": "321-54-9876",
"sex": "Male",
"id": "3456",
"race": "White",
"lastName": "Clifford"
}],
I'd like to strip the socialSecurityNumber element from the json schema using a regex expression. What would a regex expression to remove
"socialSecurityNumber": "whatever value",
look like where the value of the data pair could be any string?
I cannot be certain of the position of the data pair and whether it would have a trailing comma.
Try replacing the following regular expression with empty:
"socialSecurityNumber": "(\d|\-)",
It can go wrong if this info is split in 2 lines, or if the SSN is the last user field, because there will be no comma after it.
Anyway, after the replacing operation, check if there are any string
"socialSecurityNumber"
to confirm this can be used. If there are still strings that weren't replaced, then you will need a JSON parser to correctly eliminate this information.
I'm experiencing issues when loading JSON which are dependent on formatting of input JSON file.
According to Spark documentation on JSON Datasets, each line on input file must be a valid JSON Object. re:
"Note that the file that is offered as a json file is not a typical JSON file. Each line must contain a separate, self-contained valid JSON object. As a consequence, a regular multi-line JSON file will most often fail."
So, if I have an input JSON file such as:
{
"Year": "2013",
"First Name": "DAVID",
"County": "KINGS",
"Sex": "M",
"Count": "272"
},
{
"Year": "2013",
"First Name": "JAYDEN",
"County": "KINGS",
"Sex": "M",
"Count": "268"
}
Are there any existing tools or scripts to convert to:
{"Year": "2013","First Name": "DAVID","County": "KINGS","Sex": "M","Count":"272"},
{"Year": "2013","First Name": "JAYDEN","County": "KINGS","Sex": "M","Count": "268"}
where the JSON conforms to "Each line must contain a separate, self-contained valid JSON object"
If I format to this style above, things work as expected. But, I made these mods manually over a few rows. I cannot do this for entire data set, so looking for an existing script or tool.
OR
I could load to JDBC available database if that's a better option. Thoughts?
Thanks in advance
You can simply load the JSON files into an RDD first using sc.wholeTextFiles() and remove the file name column, then run the SQLContext read on the RDD contents.
e.g.
val jsonRdd = sc.wholeTextFiles("samplefile.json").map(x => x._2)
val jsonDf = sqlContext.read.json(jsonRdd)
What if you make it an array by adding square brackets. like this;
[
{
"Year": "2013",
"FName": "DAVID",
"County": "KINGS",
"Sex": "M",
"Count": "272"
},
{
"Year": "2013",
"FName": "JAYDEN",
"County": "KINGS",
"Sex": "M",
"Count": "268"
}
]
If I take your file and add the brackets I can iterate through it with Node.js and output a file that looks like what you want. The caveat in node.js being I cant have variable First Name-- I had to change it to FName.
I want to be able to access deeper elements stored in a json in the field json, stored in a postgresql database. For example, I would like to be able to access the elements that traverse the path states->events->time from the json provided below. Here is the postgreSQL query I'm using:
SELECT
data#>> '{userId}' as user,
data#>> '{region}' as region,
data#>>'{priorTimeSpentInApp}' as priotTimeSpentInApp,
data#>>'{userAttributes, "Total Friends"}' as totalFriends
from game_json
WHERE game_name LIKE 'myNewGame'
LIMIT 1000
and here is an example record from the json field
{
"region": "oh",
"deviceModel": "inHouseDevice",
"states": [
{
"events": [
{
"time": 1430247045.176,
"name": "Session Start",
"value": 0,
"parameters": {
"Balance": "40"
},
"info": ""
},
{
"time": 1430247293.501,
"name": "Mission1",
"value": 1,
"parameters": {
"Result": "Win ",
"Replay": "no",
"Attempt Number": "1"
},
"info": ""
}
]
}
],
"priorTimeSpentInApp": 28989.41467999999,
"country": "CA",
"city": "vancouver",
"isDeveloper": true,
"time": 1430247044.414,
"duration": 411.53,
"timezone": "America/Cleveland",
"priorSessions": 47,
"experiments": [],
"systemVersion": "3.8.1",
"appVersion": "14312",
"userId": "ef617d7ad4c6982e2cb7f6902801eb8a",
"isSession": true,
"firstRun": 1429572011.15,
"priorEvents": 69,
"userAttributes": {
"Total Friends": "0",
"Device Type": "Tablet",
"Social Connection": "None",
"Item Slots Owned": "12",
"Total Levels Played": "0",
"Retention Cohort": "Day 0",
"Player Progression": "0",
"Characters Owned": "1"
},
"deviceId": "ef617d7ad4c6982e2cb7f6902801eb8a"
}
That SQL query works, except that it doesn't give me any return values for totalFriends (e.g. data#>>'{userAttributes, "Total Friends"}' as totalFriends). I assume that part of the problem is that events falls within a square bracket (I don't know what that indicates in the json format) as opposed to a curly brace, but I'm also unable to extract values from the userAttributes key.
I would appreciate it if anyone could help me.
I'm sorry if this question has been asked elsewhere. I'm so new to postgresql and even json that I'm having trouble coming up with the proper terminology to find the answers to this (and related) questions.
You should definitely familiarize yourself with the basics of json
and json functions and operators in Postgres.
In the second source pay attention to the operators -> and ->>.
General rule: use -> to get a json object, ->> to get a json value as text.
Using these operators you can rewrite your query in the way which returns correct value of 'Total Friends':
select
data->>'userId' as user,
data->>'region' as region,
data->>'priorTimeSpentInApp' as priotTimeSpentInApp,
data->'userAttributes'->>'Total Friends' as totalFriends
from game_json
where game_name like 'myNewGame';
Json objects in square brackets are elements of a json array.
Json arrays may have many elements.
The elements are accessed by an index.
Json arrays are indexed from 0 (the first element of an array has an index 0).
Example:
select
data->'states'->0->'events'->1->>'name'
from game_json
where game_name like 'myNewGame';
-- returns "Mission1"
select
data->'states'->0->'events'->1->>'name'
from game_json
where game_name like 'myNewGame';
This did help me