Map Reduce to parse JSON data in hadoop 2.2 - json

Hello I have a JSON in the following format.I need to parse this in the map function to get the gender information of all the records.
[
{
"SeasonTicket" : false,
"name" : "Vinson Foreman",
"gender" : "male",
"age" : 50,
"email" : "vinsonforeman#cyclonica.com",
"annualSalary" : "$98,501.00",
"id" : 0
},
{
"SeasonTicket": true,
"name": "Genevieve Compton",
"gender": "female",
"age": 28,
"email": "genevievecompton#cyclonica.com",
"annualSalary": "$46,881.00",
"id": 1
},
{
"SeasonTicket": false,
"name": "Christian Crawford",
"gender": "male",
"age": 53,
"email": "christiancrawford#cyclonica.com",
"annualSalary": "$53,488.00",
"id": 2
}
]
I have tried using JSONparser but am not able to get through the JSON structure.I have been advised to use JAQL and pig but cannot do so.
Any help would be appreciated.

What I understand is that you have a huge file with an array of JSONs. Of this, you need to read the same to a mapper and emit say <id : gender>. The challenge is that JSON falls across to multiple lines.
In this is the case, I would suggest you to change the default delimiter to "}" instead of "\n".
In this case, you will be able to get parts of the JSON into the map method as value. You can discard the key ie. byte offset and do slight re-fractor on the value like removing off unwanted [ ] or , and adding chars like "}" and then parse the remaining string.
This solution works because there is no nesting within JSON and } is a valid JSON end delimiter as per the given example.
For changing the default delimiter, just set the property textinputformat.record.delimiter to "}"
Please check out this example.
Also check this jira.

Related

How to escape "#" when reading from a JSON response?

Let's say I have below sample JSON response from which I want to extract value for "#type":
{
"firstName": "John",
"lastName" : "doe",
"age" : 26,
"phoneNumbers": [
{
"#type" : "iPhone",
"number": "0123-4567-8888"
},
{
"#type" : "home",
"number": "0123-4567-8910"
}
]
}
Validated using:- http://jsonpath.com/
This works for "number":
$.phoneNumbers.[number]
But cannot get value for "#type":
$.phoneNumbers.[#type]
Tried multiple ways but no luck.
Thanks!
Edits:- added another value in the array for "home", now indexing logic [0,1] doesn't work. Even tried with [:] to fetch all values, but no luck.
You can read in this documentation:
Please note, that the return value of jsonPath is an array, which is
also a valid JSON structure.
So basically it always returns an array.
var phoneTypes = jsonPath(json,"$.phoneNumbers.[#type]");
result
["iPhone","home"]
if you want one phone you have to use phoneTypes[0] for iPhone
but I higly recommend you to fix your json using this code
var fixedJson= JSON.parse(JSON.stringify(json).replaceAll("\"#type\"","\"type\"" ));
in this case you can use the real search
var homePhone = jsonPath(fixedJson,"$.phoneNumbers[?(#.type =='home')]")[0].number;
output
0123-4567-8910

How to add new object in a json file using RobotFrameWork

I am trying to add a new bloc to my JSON
I have this JSON that I got after a GET :
{
"name": "John",
"age": 30,
"Deleted": false
}
What I want to do is to add a block trips to this Json using RobotFrameWOrk to get this result:
{
"name": "John",
"age": 30,
"trips": [
{
"dateTime": "2020-01-24T15:28:29.7073727Z",
"FirstName": "John",
"Comment": "TestMe"
}
],
"Deleted": false
}
My questions are:
The object Trips doesn't exist I have to create it manually
and then I should add this object to my JSON after the age and before Deleted
${JsonFinall}= Add String ${FirstJson} ${BlockTrips}
Log ${JsonFinall}
I imagine it would be something like that but I am blocked on the first step I don't know how to create and fill the object trips?
Do you think that I have to work with Regex?
Thanks for your help
***********************EDIT**********
I tried with add object to json : `# Bloc ActionR
${jsonFinall}= Add Object To Json ${JsonAvecAR} Course/AR.txt`
the file AR.txt is the file where I put my object trips :
"trips": [
{
"dateTime": "2020-01-24T15:28:29.7073727Z",
"FirstName": "Alicia",
"Comment": "TestMe"
}
],[![enter image description here][1]][1]

JMeter - JSON Extractor extracts correct value in one case but null in other case

I have two requests that return response with similar JSON structure. When I try to use JSON extractor on one, it works properly but when I try to extract value in the same way from the second one, it doesn't work. But let's cut to the chase.
My first response looks like this:
{
"values": [
{
"id": 1,
"name": "Fendi",
"logoId": null,
"belongsToUser": true
},
{
"id": 2,
"name": "Jean Paul Gaultier",
"logoId": null,
"belongsToUser": true
},
{
"id": 3,
"name": "Nike",
"logoId": null,
"belongsToUser": false
},
{
"id": 4,
"name": "Adidas",
"logoId": null,
"belongsToUser": true
}
]
}
And I try to extract ID of the object that "belongsToUser": false in this JSON Extractor:
JSON path expression: values[?(#.belongsToUser == false)].id
Match No.: 0
Default Values: null
And it works perfecty fine.
However, when I try this way on my second response, it doesn't work.
The response looks like this:
{
"values": [
{
"id": 12,
"brandName": "Fendi",
"productCategoryName": "Shoes",
"size": "38",
"colorNames": [
"color_green"
],
"date": 1536537600000,
"imageId": null,
"title": "Money",
"numberOfOffers": 0,
"status": "ONGOING"
},
{
"id": 13,
"brandName": "Fendi",
"productCategoryName": "Shoes",
"size": "38",
"colorNames": [
"color_green"
],
"date": 1536537600000,
"imageId": null,
"title": "Exchange",
"numberOfOffers": 0,
"status": "ONGOING"
}
]
}
I try to get id of object that has title variable = "Money" with JSON extractor:
JSON path expression: values[?(#.title == 'Money')].id
Match No.: 0
Default Values: null
But it doesn't find id value and sets my JMeter variable to null.
I also tried to leave Money unquoted or in double quotes and tried different JSON path expresions, like
$.values[?(#.title == 'Money')].id
$..[?(#.title == 'Money')].id
$.[?(#.title == 'Money')].id
But none of these seems to work. Do you have any idea how my JSON path expression shoud look to work properly?
And why doesn't it work in second case when it works in first? Is it because objets in second response have inside array?
I have used your code and it is giving the correct results. Please check the below images.
I have tried with version 3.1 also and it is working fine.
Hope this helps.
Check the below image for different types of options in view result tree.
The $..[?(#.title == 'Money')].id expression should work just fine:
Most probably your JMeter installation is corrupt and you experience some form of jar hell due to some clashing library in JMeter Classpath (it might be caused by presence of deprecated JSON Plugins or similar) . So I would recommend obtaining clean latest version of JMeter and trying out your test on it. If you're using any plugins - install them using JMeter Plugin Manager
If you are not in position to re-install JMeter you can try to get to the bottom of the issue by looking into jmeter.log file. If there are no suspicious entries - add the next line to log4j2.xml file:
<Logger name="org.apache.jmeter.extractor.json" level="debug" />

A regex expression that can remove data from a json object

I'd like to be able to selectively remove elements from a json schema. Imagine a json object that contains a larger but similar array of users like this
[{
"users": [{
"firstName": "Nancy",
"socialSecurityNumber": "123-45-6789",
"sex": "Female",
"id": "1234",
"race": "Smith",
"lastName": "Logan"
}, {
"firstName": "Charles",
"socialSecurityNumber": "321-54-9876",
"sex": "Male",
"id": "3456",
"race": "White",
"lastName": "Clifford"
}],
I'd like to strip the socialSecurityNumber element from the json schema using a regex expression. What would a regex expression to remove
"socialSecurityNumber": "whatever value",
look like where the value of the data pair could be any string?
I cannot be certain of the position of the data pair and whether it would have a trailing comma.
Try replacing the following regular expression with empty:
"socialSecurityNumber": "(\d|\-)",
It can go wrong if this info is split in 2 lines, or if the SSN is the last user field, because there will be no comma after it.
Anyway, after the replacing operation, check if there are any string
"socialSecurityNumber"
to confirm this can be used. If there are still strings that weren't replaced, then you will need a JSON parser to correctly eliminate this information.

jqGrid JSON notation on objects

there!
I´ve one column in my jqGrid that is empty.
But i checked the object on chrome console and thats fine.
colModel definition
colModel:[
{name:'id',index:'id', width:55,editable:false,editoptions:{readonly:true,size:10},hidden:true},
{name:'firstName',index:'firstName', width:100,searchoptions: { sopt: ['eq', 'ne', 'cn']}},
{name:'lastName',index:'lastName', width:100,editable:true, editrules:{required:true}, editoptions:{size:10}},
{name:'books[0].nome',index:'books[0].nome', width:100,editable:true, editrules:{required:true}, editoptions:{size:10}},
{"formatter":"myfunction", formatoptions:{baseLinkUrl:'/demo/{firstName}|view-icon'}}
]
JSON response
{
"total": "10",
"page": "1",
"records": "3",
"rows": [
{
"id": 1,
"firstName": "John",
"lastName": "Smith",
"books": [{"nome": "HeadFirst"}]
},
{
"id": 2,
"firstName": "Jane",
"lastName": "Adams",
"books": [{"nome": "DalaiLama"}]
},
{
"id": 35,
"firstName": "Jeff",
"lastName": "Mayer",
"books": [{"nome": "Bobymarley"}]
}
]
}
chrome console inspect object
rowdata.books[0].nome
"HeadFirst"
Any one know where theres are possibles trick?
Tks!
You should use as the value of name property of colModel only the names which can be used as property name in JavaScript and as CSS id names. So the usage of name:'books[0].nome' is not good idea.
To solve your problem you can use jsonmap. For example you can use dotted name conversion:
{name: 'nome', jsonmap: 'books.0.nome', ...
In more complex cases you can use functions as the value of jsonmap. For example
{name: 'nome', jsonmap: function (item) {
return item.books[0].nome;
}, ...
You can find some more code examples about the usage of jsonmap in other old answers: here, here, here, here, here.
name is intended to be a unique name for the row, not a reference to a JSON object. From the jqGrid colModel options documentation:
Set the unique name in the grid for the column. This property is required. As well as other words used as property/event names, the reserved words (which cannot be used for names) include subgrid, cb and rn.
You can also observe how .name is used within grid.base.js - for example:
var nm = {},
...
nm = $t.p.colModel[i].name;
...
res[nm] = $.unformat.call($t,this,{rowId:ind.id, colModel:$t.p.colModel[i]},i);
Anyway, to get back to your question I think you will have better luck by passing down the book name directly - as strings and not objects - and referencing it by name as something like bookName.