Parsing Git Json with Regular Express - json

I am taking a Github json file and parsing it with Java's regular expression library JsonPath. I am having a problem parsing arrays that do not have labels.
I need to send a email every time a particular file is changed in our repository.
Here is the Git Json:
{
"trigger": "push",
"payload": {
"type": "GitPush",
"before": "xxxxxxxx",
"after": "yyyyyyyy",
"branch": "branch-name",
"ref": "refs/heads/branch-name",
"repository": {
"id": 42,
"name": "repo",
"title": "repo",
"type": "GitRepository"
},
"beanstalk_user": {
"type": "Owner",
"id": 42,
"login": "username",
"email": "user#example.org",
"name": "Name Surname"
},
"commits": [
{
"type": "GitCommit",
"id": "ffffffff",
"message": "Important changes.",
"branch": "branch-name",
"author": {
"name": "Name Surname",
"email": "user#example.org"
},
"beanstalk_user": {
"type": "Owner",
"id": 42,
"login": "username",
"email": "user#example.org",
"name": "Name Surname"
},
"changed_files": {
"added": [
"NEWFILE",
],
"deleted": [
"Gemfile",
"NEWFILE"
],
"modified": [
"README.md",
"NEWFILE"
],
"copied": [
]
},
"changeset_url": "https://subdomain.github.com/repository-name/changesets/ffffffff",
"committed_at": "2014/08/18 13:30:29 +0000",
"parents": [
"afafafaf"
]
}
]
}
}
This is the expression I am using: to get the commits
$..changed_files
This return the whole changed files part but I can not explicitly choose the name "NEWFILE"
I tried
$..changed_files.*[?(#.added == "NEWFILE")]
$..changed_files.*[?(#.*== "NEWFILE")]
It just returns a empty array.
I just want it to return Newfile and what type of change. Any Ideas?

You can use the following JsonPath to retrieve the commits which list "NEWFILE" as an added file :
$.payload.commits[?(#.changed_files.added.indexOf("NEWFILE") != -1)]

Related

Update a Contentful post using PUT request via Postman

I'm trying to update a Contentful entry using Postman.
What I did:
In Contentful space, I created a test post to play with.
Went to Settings - API Keys - Content management tokens and generated a Personal access token
Created a GET request in Postman, passing space ID, master environment, and ID of the test post:
https://cdn.contentful.com/spaces/{spaceID}i/environments/master/entries?sys.id={postID}
I also sent authorisation header with content delivery token.
The GET request goes successfully and I'm able to copy the JSON object-response.
{
"sys": {
"type": "Array"
},
"total": 1,
"skip": 0,
"limit": 100,
"items": [
{
"metadata": {
"tags": []
},
"sys": {
"space": {
"sys": {
"type": "Link",
"linkType": "Space",
"id": "d9r4mg123x4v"
}
},
"id": "2Fwow39hxxx1bvMkjpsyV9",
"type": "Entry",
"createdAt": "2021-11-10T14:00:11.935Z",
"updatedAt": "2021-11-10T14:06:51.393Z",
"environment": {
"sys": {
"id": "master",
"type": "Link",
"linkType": "Environment"
}
},
"revision": 3,
"contentType": {
"sys": {
"type": "Link",
"linkType": "ContentType",
"id": "hotelInfo"
}
},
"locale": "en-US"
},
"fields": {
"name": "Test entry",
"slug": "test-entry",
"address": "Lviv",
"cityName": "Lviv",
"phone": "+380931231212",
"coordinates": {
"lon": -115.302,
"lat": 36.18709
},
"dog": "100",
"cat": "100",
"delivery": "100",
"photo": [
{
"sys": {
"type": "Link",
"linkType": "Asset",
"id": "2hSnYhQDJzU99NvlsYdk3k"
}
}
],
"additionalInfo": {
"data": {},
"content": [
{
"data": {},
"content": [
{
"data": {},
"marks": [],
"value": "Test",
"nodeType": "text"
}
],
"nodeType": "paragraph"
}
],
"nodeType": "document"
},
"featuredHotel": true,
"phoneClicks": 1
}
}
],
"includes": {
"Asset": [
{
"metadata": {
"tags": []
},
"sys": {
"space": {
"sys": {
"type": "Link",
"linkType": "Space",
"id": "d9r4mg123x4v"
}
},
"id": "2hSnYhQDJzU99NvlsYdk3k",
"type": "Asset",
"createdAt": "2021-11-10T13:59:59.954Z",
"updatedAt": "2021-11-10T13:59:59.954Z",
"environment": {
"sys": {
"id": "master",
"type": "Link",
"linkType": "Environment"
}
},
"revision": 1,
"locale": "en-US"
},
"fields": {
"title": "JS",
"description": "Lorem Ipsum",
"file": {
"url": "//images.ctfassets.net/d9r4mg123x4v/2hSnYhQDJzU99NvlsYdk3k/6fbabc7be7f4b28dc8b7deadd9892205/JS.png",
"details": {
"size": 23078,
"image": {
"width": 1024,
"height": 1024
}
},
"fileName": "JS.png",
"contentType": "image/png"
}
}
}
]
}
}
Now I want to create PUT request to send the updated JSON to the Contentful.
I paste the JSON I got as a response from GET request.
I change one of the values:
"name": "Test entry"
to
"name": "Test entry 123"
I send PUT request to https://api.contentful.com/spaces/{spaceID}/environments/master/entries/{postID}
The Authorisation header contains the Personal access token I generated before.
The X-Contentful-Version header contains the version of the post, can be found in post details
When I send this request, I get JSON response with an empty "fields": {}
{
"metadata": {
"tags": []
},
"sys": {
"space": {
"sys": {
"type": "Link",
"linkType": "Space",
"id": "d9r4mg123x4v"
}
},
"id": "2Fwow39hxxx1bvMkjpsyV9",
"type": "Entry",
"createdAt": "2021-11-10T13:57:10.882Z",
"updatedAt": "2021-11-11T10:58:39.480Z",
"environment": {
"sys": {
"id": "master",
"type": "Link",
"linkType": "Environment"
}
},
"publishedVersion": 13,
"publishedAt": "2021-11-10T14:06:51.393Z",
"firstPublishedAt": "2021-11-10T14:00:11.935Z",
"createdBy": {
"sys": {
"type": "Link",
"linkType": "User",
"id": "4123123123zOn3MkhuVB"
}
},
"updatedBy": {
"sys": {
"type": "Link",
"linkType": "User",
"id": "4123123123zOn3MkhuVB"
}
},
"publishedCounter": 3,
"version": 23,
"publishedBy": {
"sys": {
"type": "Link",
"linkType": "User",
"id": "4123123123zOn3MkhuVB"
}
},
"contentType": {
"sys": {
"type": "Link",
"linkType": "ContentType",
"id": "hotelInfo"
}
}
},
"fields": {}
}
And in Contentful Admin area, all the fields of the post become empty.
Contentful documentation says:
Contentful doesn't merge changes made to content, so when updating
content, you need to send the entire body of an entry. If you update
content with a subset of properties, you will lose all existing
properties not included in that update.
You should always update resources in the following order:
Fetch current resource.
Make changes to the current resource.
Update the resource by passing the changed resource along with current version number.
This way no unseen changes are overridden and unexpected conflicts are
unlikely to occur.
Note: You can't update any of the sys property fields, including
sys.id.
...so, I guess, I'm doing everything right - taking the post, editing data and sending updated post back.
I tried editing my JSON data to send it without sys fields, but no luck.
I'm stuck, anyone has any ideas what should I proceed with?
Thanks to #whitep4nth3r I was able to solve the problem.
I needed to GET data from the same source I'm trying to PUT it to.
The Authorisation header needed to be replaced with the Personal access token used for PUT request.

Json Path Read from a Kafka Message

I have a kafka message like below, where im trying to read the data from the json path. However im having a challenge when reading some of the attributes from the json path. here is the sample message.
sample1:
{
"header": {
"bu": "google",
"id": "12345",
"bum": "google",
"originTimestamp": "2021-10-09T15:17:09.842+00:00",
"batchSize": "0",
"jobType": "Batch"
},
"payload": {
"derivationdetails": {
"Id": "6783jhvvh897u31y283y",
"itemid": "1234567",
"batchid": 107,
"attributes": {
"itemid": "1234567",
"lineNbr": "1498",
"cat": "5929",
"Id": "6783jhvvh897u31y283y",
"indicator": "false",
"subcat": "3514"
},
"Exception": {
"values": [
{
"type": "PICK",
"value": "blocked",
"Reason": [
"RULE"
],
"rules": [
"439"
]
}
],
"rulesBagInfo": [
{
"Idtype": "XXXX",
"uniqueid": "7889423rbhevfhjaufdyeuiryeukjbdafvjd",
"rulesMatch": [
"439"
]
}
]
}
}
}
}
sample 2: Same message but see the difference in "Payload"
{
"header": {
"bu": "google",
"id": "12345",
"bum": "google",
"originTimestamp": "2021-10-09T15:17:09.842+00:00",
"batchSize": "0",
"jobType": "Batch"
},
"payload": {
"Id": "6783jhvvh897u31y283y",
"itemid": "1234567",
"batchid": 107,
"attributes": {
"itemid": "1234567",
"lineNbr": "1498",
"cat": "5929",
"Id": "6783jhvvh897u31y283y",
"indicator": "false",
"subcat": "3514"
},
"Exception": {
"values": [
{
"type": "PICK",
"value": "blocked",
"Reason": [
"RULE"
],
"rules": [
"439"
]
}
],
"rulesBagInfo": [
{
"Idtype": "XXXX",
"uniqueid": "7889423rbhevfhjaufdyeuiryeukjbdafvjd",
"rulesMatch": [
"439"
]
}
]
}
}
}
If you observe, sometimes the message has "derivationdetails", and sometimes it doesn't. But irrespective of its existence, i need to read the values of id,itemid and batchid. I tried using
$.payload[*].id
$.payload[*].itemid
$.payload[*].batchid
But i see that for batchid is returning null even though it has a value in the message, and the attributes under "attributes" return null if im using the above. For fields under "attributes" im using this(example):
$.payload.attributes.itemId
And, completely blank on how to read the below part.
"Exception": {
"values": [
{
"type": "PICK",
"value": "blocked",
"Reason": [
"RULE"
],
"rules": [
"439"
]
}
],
"rulesBagInfo": [
{
"Idtype": "XXXX",
"uniqueid": "7889423rbhevfhjaufdyeuiryeukjbdafvjd",
"rulesMatch": [
"439"
]
Im new to this and need some suggestions on how to read the attributes properly. Any help would be much appreciated.Thanks
Use ..(recursive descent, Deep scan. JSONPath borrows this syntax from E4X.) to get the values. But It will return a list if there are multiple entries with same key nested in deep.
Below jsonpath expressions will return a list with one item each for both sample1 and sample2
$.payload..attributes.Id
$.payload..attributes.itemid
$.payload..batchid
$.payload..Exception

JSONpath to get a value by filter on another value

I have below JSON and i require to validate the status based on given id in my automation script. For that JSON path require
[
[
{
"id": 9905130204,
"category": {
"id": 0,
"name": "string"
},
"name": "doggie",
"photoUrls": [
"string"
],
"tags": [
{
"id": 0,
"name": "string"
}
],
"status": "available"
},
{
"id": 9905130203,
"name": "Jeffs Doggie11/6/2019 2:44:53 PM",
"photoUrls": [
"string"
],
"tags": [],
"status": "available"
},
{
"id": 9905130217,
"name": "Goot Doggie",
"photoUrls": [
"https://media.karousell.com/media/photos/products/2017/09/14/doggi_door_stopper_1505372529_5cdd1eba0"
],
"tags": [],
"status": "available"
}
]
]
I want to extract "status": "available" based on "id": 9905130217. No clue how to do that, please help.
You can use the following jsonpath expression to find the status where id = n:
$.[?(#.id == 'n')].status
So, for your specific case:
$.[?(#.id == '9905130217')].status
Note, that this assumes id is unique.

Parsing JSON data using GSON api

I have a JSON data and able to get the JSON data into a string.
I wish to get a json array from the root object one by one (using a loop if many are there)
The returned object must be of type "com.google.gson.JsonObject" as another method takes this object as input to get me the attribute values.
my json looks like this:
I would be needing the json array in "Resources" -
{
"totalResults": 2,
"startIndex": 1,
"schemas": [
"urn:ietf:params:scim:api:messages:2.0:ListResponse"
],
"Resources": [
{
"owner": {},
"application": {
"displayName": "Non Auth App",
"value": "4028b88168fa3d38016900d1759405a6",
"$ref": "http://localhost:8080/identityiq/scim/v2/Applications/4028b88168fa3d38016900d1759405a6"
},
"meta": {
"created": "2019-03-22T18:40:30.505+05:30",
"location": "http://localhost:8080/identityiq/scim/v2/Entitlements/297eff8a699f8b3a0169a5863769008b",
"version": "W/\"1553260230505\"",
"resourceType": "Entitlement"
},
"schemas": [
"urn:ietf:params:scim:schemas:sailpoint:1.0:Entitlement"
],
"displayableName": "reader",
"aggregated": false,
"id": "297eff8a699f8b3a0169a5863769008b",
"requestable": true,
"attribute": "Privileges",
"type": "Entitlement",
"descriptions": [],
"value": "reader"
},
{
"owner": {},
"application": {
"displayName": "Non Auth App",
"value": "4028b88168fa3d38016900d1759405a6",
"$ref": "http://localhost:8080/identityiq/scim/v2/Applications/4028b88168fa3d38016900d1759405a6"
},
"meta": {
"created": "2019-03-18T19:11:47.912+05:30",
"location": "http://localhost:8080/identityiq/scim/v2/Entitlements/297eff8a6990fc49016991096d08001e",
"version": "W/\"1552916507912\"",
"resourceType": "Entitlement"
},
"schemas": [
"urn:ietf:params:scim:schemas:sailpoint:1.0:Entitlement"
],
"displayableName": "admin",
"aggregated": false,
"id": "297eff8a6990fc49016991096d08001e",
"requestable": true,
"attribute": "Privileges",
"type": "Entitlement",
"descriptions": [],
"value": "admin"
}
]
}
Any help is appreciated.

How to Index & Search Nested Json in Solr 4.9.0

I want to index & search nested json in solr. Here is my json code
{
"id": "44444",
"headline": "testing US",
"generaltags": [
{
"type": "person",
"name": "Jayalalitha",
"relevance": "0.334",
"count": 1
},
{
"type": "person",
"name": "Kumar",
"relevance": "0.234",
"count": 1
}
],
"socialtags": {
"type": "SocialTag",
"name": "US",
"importance": 2
},
"topic": {
"type": "Topic",
"name": "US",
"score": "0.936"
}
}
When I try to Index, I'm getting the error "Error parsing JSON field value. Unexpected OBJECT_START"
When we tried to use Multivalued Field & index, we couldn't able to search using the multivalued field? Its returning "Undefined Field"
Also Please advice if I need to do any changes in schema.xml file?
You are nesting child documents within your document. You need to use the proper syntax for nested child documents in JSON:
[
{
"id": "1",
"title": "Solr adds block join support",
"content_type": "parentDocument",
"_childDocuments_": [
{
"id": "2",
"comments": "SolrCloud supports it too!"
}
]
},
{
"id": "3",
"title": "Lucene and Solr 4.5 is out",
"content_type": "parentDocument",
"_childDocuments_": [
{
"id": "4",
"comments": "Lots of new features"
}
]
}
]
Have a look at this article which describes JSON child documents and block joins.
Using the format mentioned by #qux you will face "Expected: OBJECT_START but got ARRAY_START at [16]",
"code": 400
as when JSON starting with [....] will parsed as a JSON array
{
"id": "44444",
"headline": "testing US",
"generaltags": [
{
"type": "person",
"name": "Jayalalitha",
"relevance": "0.334",
"count": 1
},
{
"type": "person",
"name": "Kumar",
"relevance": "0.234",
"count": 1
}
],
"socialtags": {
"type": "SocialTag",
"name": "US",
"importance": 2
},
"topic": {
"type": "Topic",
"name": "US",
"score": "0.936"
}
}
The above format is correct.
Regarding searching. Kindly use the index to search for the elements of the JSON array.
The workaround for this can be keeping the whole JSON object inside other JSON object and the indexing it
I was suggesting to keep the whole data inside another JSON object. You can try the following way
{
"data": [
{
"id": "44444",
"headline": "testing US",
"generaltags": [
{
"type": "person",
"name": "Jayalalitha",
"relevance": "0.334",
"count": 1
},
{
"type": "person",
"name": "Kumar",
"relevance": "0.234",
"count": 1
}
],
"socialtags": {
"type": "SocialTag",
"name": "US",
"importance": 2
},
"topic": {
"type": "Topic",
"name": "US",
"score": "0.936"
}
}
]
}
see the syntax in http://yonik.com/solr-nested-objects/
$ curl http://localhost:8983/solr/demo/update?commitWithin=3000 -d '
[
{id : book1, type_s:book, title_t : "The Way of Kings", author_s : "Brandon Sanderson",
cat_s:fantasy, pubyear_i:2010, publisher_s:Tor,
_childDocuments_ : [
{ id: book1_c1, type_s:review, review_dt:"2015-01-03T14:30:00Z",
stars_i:5, author_s:yonik,
comment_t:"A great start to what looks like an epic series!"
}
,
{ id: book1_c2, type_s:review, review_dt:"2014-03-15T12:00:00Z",
stars_i:3, author_s:dan,
comment_t:"This book was too long."
}
]
}
]'
supported from solr 5.3