using jq to return multiple elements? - json

I'm trying to use jq to return multiple elements from a JSON object. The data is coming from an AWS cli lookup, this is just a tiny section of the output:
{
"ReservedInstancesOfferings": [{
"OfferingClass": "convertible",
"OfferingType": "No Upfront",
"AvailabilityZone": "ap-southeast-2a",
"InstanceTenancy": "default",
"PricingDetails": [],
"ProductDescription": "Linux/UNIX",
"UsagePrice": 0.0,
"RecurringCharges": [{
"Amount": 0.167,
"Frequency": "Hourly"
}],
"Marketplace": false,
"CurrencyCode": "USD",
"FixedPrice": 0.0,
"Duration": 94608000,
"Scope": "Availability Zone",
"ReservedInstancesOfferingId": "1fc8c02b-bcc0-42b8-82ef-47c6f6d1c1b5",
"InstanceType": "c4.xlarge"
},
{
"OfferingClass": "convertible",
"OfferingType": "No Upfront",
"AvailabilityZone": "ap-southeast-2a",
"InstanceTenancy": "dedicated",
"PricingDetails": [],
"ProductDescription": "Red Hat Enterprise Linux",
"UsagePrice": 0.0,
"RecurringCharges": [{
"Amount": 0.243,
"Frequency": "Hourly"
}],
"Marketplace": false,
"CurrencyCode": "USD",
"FixedPrice": 0.0,
"Duration": 94608000,
"Scope": "Availability Zone",
"ReservedInstancesOfferingId": "24aaceee-f54e-4882-aba3-ce710a5036c9",
"InstanceType": "c4.xlarge"
}
]
}
I'm trying to extract the fields:
.ReservedInstancesOfferings[].ProductDescription
.ReservedInstancesOfferings[].InstanceType
.ReservedInstancesOfferings[].RecurringCharges[].Amount
I would like to output to appear like this:
"Red Hat Enterprise Linux"
"c4.xlarge"
0.183
As you can see above I understand how to extract all the elements individually, but I can't work out how to put them all together to attain the desired output. I've trued separating the queries with a comma, but that doesn't provide the desired output.
Any help would be very much appreciated.
Thank you!
EDIT:
Oh oh, I got it! Full answer is below. Boy oh boy jq is powerful!
Hopefully this helps someone :)

OK, so I managed to get what I was after. I still have a little tidying up to do as some of the input fields aren't completely uniform, but it's close enough to move forward with:
aws ec2 describe-reserved-instances-offerings --availability-zone "ap-southeast-2a" --instance-type "c4.xlarge" |
jq '.ReservedInstancesOfferings[] | [.ProductDescription, .InstanceType, .RecurringCharges[].Amount]'
Output:
[
"Linux/UNIX",
"c4.xlarge",
0.167
]
[
"Red Hat Enterprise Linux",
"c4.xlarge",
0.243
]
[
"Red Hat Enterprise Linux",
"c4.xlarge",
0.227
]
[
"Linux/UNIX",
"c4.xlarge",
0.183
]
[
"SUSE Linux",
"c4.xlarge",
0.2
]

Related

How can I see logs of the JSON post bodies sent by zapier to my CRM (Current RMS) via the Webhook zap during setup and testing?

I'm trying to send new users / new customres of my WooCommerce store into the rental management app current-rms.com as new Organisations / new contacts. Since Current RMS does not have a native Zap, I am trying to use the generic Webhook zap that Zapier maintains.
Specifically, I'd like to see the sent JSON body in Zapier posts that I make during the setup and testing of the Zap after clicking "Make a Zap!". The Task History is not detailed enough nor does it show hits during test and setup, since it's not live yet.
My trigger is a WooCommerce New Customer. This is working with Zapier WooCommerce Plugin and webhooks OK.
My action is the generic Zapier "Webhooks" Zap. The label "instant" appears next to it in the list at /app/zaps and it is "off".
One version uses JSON PAYLOAD as the action.
Another version uses CUSTOM PAYLOAD as the action.
Wrap request in array is YES.
Unflatten is YES.
My API key and subdomain are in the app URL as query strings and working OK.
When I hit test I get:
We had trouble sending your test through.
The app returned "Invalid JSON - missing or invalid entry for 'member'". This usually happens when your Zap is missing a required field or a field value isn't in a recognized format.
We made a request to api.current-rms.com and received (400) Bad Request.
Official docs are at: https://api.current-rms.com/doc#members-members-post
Logging available at Current RMS side
Part of the authentication of Current RMS involves knowing the domain of the account you are trying to access, in my case its therockfactory due to it being an account for my company https://therockfactory.net/
https://api.current-rms.com/api/v1/members?apikey=APIKEYCENSORED&subdomain=therockfactory
which returns the following when I use the correct API key:
{"webhook_logs":[],"meta":{"total_row_count":0,"row_count":0,"page":1,"per_page":20}}
Maybe if I could see the actual hit that Zapier is posting to Current I could wrap my confused brain around it better? What me worry.
The hit should look somewhat similar to this example, but I've not been able to locate it so far... (in Zapier)
Headers
Content-Type: application/json
Body
{
"member": {
"name": "Chris Bralton",
"description": "Pictures and leaned back was strewn at one would rather more. People don't want of his own means of one hand! Unless it from our pioneer has he fallen tree but that ever stronger and a. Hid among us against the full of verdure through by my eyes.",
"active": true,
"bookable": false,
"location_type": 0,
"locale": "en-GB",
"membership_type": "Contact",
"lawful_basis_type_id": 10001,
"sale_tax_class_id": 1,
"purchase_tax_class_id": 1,
"tag_list": [
"[\"Red\", \"Blue\", \"Green\"]"
],
"custom_fields": {},
"membership": {},
"primary_address": {
"name": "Chris Branson",
"street": "16 The Triangle",
"postcode": "NG2 1AE",
"city": "Nottingham",
"county": "Nottinghamshire",
"country_id": "1",
"country_name": "United Kingdom",
"type_id": 3001,
"address_type_name": "Primary",
"created_at": "2015-06-29T10:00:00.000Z",
"updated_at": "2015-06-29T10:30:00.000Z"
},
"emails": [
{
"address": "abigail.parker#ggmail.co.uk",
"type_id": 4001,
"email_type_name": "Work",
"id": 1
}
],
"phones": [
{
"number": "+44 115 9793399",
"type_id": 6001,
"phone_type_name": "Work",
"id": 1
}
],
"links": [
{
"address": "www.facebook.com/profile.php?id=566828251",
"type_id": 5002,
"link_type_name": "Facebook",
"id": 1
}
],
"addresses": [
{
"name": "Chris Branson",
"street": "16 The Triangle",
"postcode": "NG2 1AE",
"city": "Nottingham",
"county": "Nottinghamshire",
"country_id": "1",
"country_name": "United Kingdom",
"type_id": 3002,
"address_type_name": "Billing",
"created_at": "2017-06-29T10:00:00.000Z",
"updated_at": "2017-06-29T10:30:00.000Z",
"id": 1
}
],
"service_stock_levels": [
{
"item_id": 10,
"store_id": 1,
"member_id": 1,
"asset_number": "Chris Bralton",
"serial_number": "",
"location": "",
"stock_type": 3,
"stock_category": 60,
"quantity_held": "1.0",
"quantity_allocated": "0.0",
"quantity_unavailable": "0.0",
"quantity_on_order": "0.0",
"starts_at": "",
"ends_at": "",
"icon": {
"iconable_id": 85,
"id": 1,
"image_file_name": "abigail.jpeg",
"url": "https://s3.amazonaws.com/current-rms-development/64a0ccd0-5fbd-012f-2201-60f847290680/icons/46/original/abigail.jpeg",
"thumb_url": "https://s3.amazonaws.com/current-rms-development/64a0ccd0-5fbd-012f-2201-60f847290680/icons/46/thumb/abigail.jpeg",
"created_at": "2015-06-29T10:00:00.000Z",
"updated_at": "2015-06-29T10:30:00.000Z",
"iconable_type": "StockLevel"
},
"custom_fields": {},
"id": 487,
"item_name": "Sound Engineer",
"store_name": "Nottingham",
"stock_type_name": "Service",
"stock_category_name": "Resource"
}
],
"day_cost": "",
"hour_cost": "",
"distance_cost": "",
"flat_rate_cost": "",
"icon": {
"image": ""
},
"child_members": [
{
"relatable_id": 317,
"relatable_type": "Member",
"related_id": 25,
"related_type": "Member"
}
],
"parent_members": [
{
"relatable_id": 317,
"relatable_type": "Member",
"related_id": 25,
"related_type": "Member"
}
]
}
}
UPDATE: After reading my chosen answer I was able to see what Zapier was sending:
[
{
"member[emails_attributes][0][address]": "test#test.co.nz",
"member[membership_type]": "Organisation",
"member[name]": "Testafari Testing"
}
]
You can send your webhook to a tool like this one to inspect the payloads that are being sent from anywhere on the internet: https://requestbin.com/
You can find more help in regards to using Webhooks by Zapier and other ideas on how you can troubleshoot issues stemming from its use: https://zapier.com/apps/webhook/help#inspect-the-requests

NYC Open data DOB Missing Information

I am facing some issue in NYC department of building API.
help me if you know any other API giving the same information
I have used this API but didn't work for me
https://data.cityofnewyork.us/resource/83x8-shf7.json
Missing fields
permitee detailed address
https://data.cityofnewyork.us/resource/83x8-shf7.json?$where=filing_date BETWEEN '2018-05-01T06:00:00' AND '2018-05-30T10:00:00'
Also i am not able get expected data using filters for "filing_date" from same api
expected data should return all data between 2018-05-01 and 2018-05-30 for this API But i am getting only few results.
I am getting this data
[
{
"bin__": "3118313",
"bldg_type": "1",
"block": "05143",
"borough": "BROOKLYN",
"city": "BROOKLYN",
"community_board": "314",
"dobrundate": "2018-05-03T00:00:00.000",
"expiration_date": "2018-06-11T00:00:00.000",
"filing_date": "2018-05-02T00:00:00.000",
"filing_status": "INITIAL",
"gis_census_tract": "1522",
"gis_council_district": "40",
"gis_latitude": "40.641731",
"gis_longitude": "-73.966432",
"gis_nta_name": "Flatbush",
"house__": "328",
"issuance_date": "2018-05-02T00:00:00.000",
"job__": "321679046",
"job_doc___": "01",
"job_start_date": "2018-05-02T00:00:00.000",
"job_type": "A2",
"lot": "00068",
"non_profit": "N",
"owner_s_business_name": "N/A",
"owner_s_business_type": "INDIVIDUAL",
"owner_s_first_name": "MATTHEW",
"owner_s_house__": "328",
"owner_s_house_street_name": "ARGYLE ROAD",
"owner_s_last_name": "LIMA",
"owner_s_phone__": "3475968096",
"owner_s_zip_code": "11218",
"permit_sequence__": "01",
"permit_si_no": "3452932",
"permit_status": "ISSUED",
"permit_subtype": "OT",
"permit_type": "EW",
"permittee_s_business_name": "BMB BUILDER INC",
"permittee_s_first_name": "YUAN HANG",
"permittee_s_last_name": "XIAO",
"permittee_s_license__": "0612790",
"permittee_s_license_type": "GC",
"permittee_s_phone__": "9175776544",
"residential": "YES",
"self_cert": "N",
"site_fill": "NOT APPLICABLE",
"state": "NY",
"street_name": "ARGYLE ROAD",
"superintendent_business_name": "BMB BUILDER INC",
"superintendent_first___last_name": "YUAN HANG XIAO",
"work_type": "OT",
"zip_code": "11218"
}]
Expected Data should be
[{
"bin__": "1090379",
"bldg_type": "2",
"block": "00760",
"borough": "MANHATTAN",
"city": "GREAT NECK",
"community_board": "104",
"dobrundate": "2018-05-02T00:00:00.000",
"expiration_date": "2018-10-28T00:00:00.000",
"filing_date": "2018-05-01T00:00:00.000",
"filing_status": "RENEWAL",
"gis_census_tract": "111",
"gis_council_district": "3",
"gis_latitude": "40.753978",
"gis_longitude": "-73.993673",
"gis_nta_name": "Hudson Yards-Chelsea-Flatiron-Union Square",
"house__": "337",
"issuance_date": "2018-05-01T00:00:00.000",
"job__": "121187606",
"job_doc___": "01",
"job_start_date": "2016-02-19T00:00:00.000",
"job_type": "NB",
"lot": "00020",
"non_profit": "N",
"owner_s_business_name": "HKONY WEST 36 LLC",
"owner_s_business_type": "PARTNERSHIP",
"owner_s_first_name": "SAM",
"owner_s_house__": "420",
"owner_s_house_street_name": "GREAT NECK ROAD",
"owner_s_last_name": "CHANG",
"owner_s_phone__": "9178380886",
"owner_s_zip_code": "11021",
"permit_sequence__": "07",
"permit_si_no": "3451790",
"permit_status": "ISSUED",
"permit_type": "NB",
"permittee_s_business_name": "OMNIBUILD CONSTRUCTION IN",
"permittee_s_first_name": "PETER",
"permittee_s_last_name": "SERPICO",
"permittee_s_license__": "0608390",
"permittee_s_license_type": "GC",
"permittee_s_phone__": "2124191930",
"self_cert": "N",
"site_fill": "ON-SITE",
"site_safety_mgr_s_first_name": "ROBERT",
"site_safety_mgr_s_last_name": "FILIPPONE",
"special_district_1": "GC",
"state": "NY",
"street_name": "W 36 ST",
"zip_code": "10018"
}]
Combing through the JSON, it appears that these columns are not matching: permit_subtype, superintendent_business_name, superintendent_first___last_name, site_safety_mgr_s_first_name, site_safety_mgr_s_last_name, and special_district_1.
Looking at the original data sources, the columns that do not match are instances where the field is blank for that field. That is, bin__ = 1090379 does not have a permit_subtype, so it is omitted in the JSON (which is standard practice).
It will, however, be included in the CSV output since that format must include all columns: https://data.cityofnewyork.us/resource/83x8-shf7.csv?$where=filing_date%20BETWEEN%20%272018-05-01T06:00:00%27%20AND%20%272018-05-30T10:00:00%27.
This answer took a bit of digging because it wasn't immediately obvious which columns were different between the two examples. It's always helpful to over-explain to make it easier to track-down the issue.
Likewise, per the filing_date question, please include the query you're attempting to use.

Mapquest API Route Matrix time and day options do nothing

I am writing a simple application in Java to get time and drive distance data from the MapQuest API using their Route Matrix. I can get regular data just fine, but it appears that the date and time parameters (explained here) do not change anything.
My two request bodies are:
{"locations": ["55455","55113"],"options": {"allToAll": true,
"dateType": 0,"date": "12/17/2016","timeType": 2,"localTime": "12:00"}}
and
{"locations": ["55455","55113"],"options": {"allToAll": true,
"dateType": 0,"date": "12/13/2016","timeType": 2,"localTime": "16:30"}}
I get the same response for both requests:
{
"allToAll": true,
"time": [
[
0,
573
],
[
562,
0
]
],
"distance": [
[
0,
7.485
],
[
6.764,
0
]
],
"locations": [
{
"latLng": {
"lng": -93.234543,
"lat": 44.975052
},
"adminArea4": "Hennepin County",
"adminArea5Type": "City",
"adminArea4Type": "County",
"adminArea5": "Minneapolis",
"street": "",
"adminArea1": "US",
"adminArea3": "MN",
"type": "s",
"displayLatLng": {
"lng": -93.234543,
"lat": 44.975052
},
"linkId": 286290952,
"postalCode": "55455",
"dragPoint": false,
"sideOfStreet": "N",
"adminArea1Type": "Country",
"geocodeQuality": "ZIP",
"geocodeQualityCode": "Z1XAA",
"adminArea3Type": "State"
},
{
"latLng": {
"lng": -93.15676,
"lat": 45.012234
},
"adminArea4": "Ramsey County",
"adminArea5Type": "City",
"adminArea4Type": "County",
"adminArea5": "St Paul",
"street": "",
"adminArea1": "US",
"adminArea3": "MN",
"type": "s",
"displayLatLng": {
"lng": -93.15676,
"lat": 45.012234
},
"linkId": 286290770,
"postalCode": "55113",
"dragPoint": false,
"sideOfStreet": "N",
"adminArea1Type": "Country",
"geocodeQuality": "ZIP",
"geocodeQualityCode": "Z1XAA",
"adminArea3Type": "State"
}
],
"manyToOne": false,
"info": {
"copyright": {
"text": "© 2016 MapQuest, Inc.",
"imageUrl": "http://api.mqcdn.com/res/mqlogo.gif",
"imageAltText": "© 2016 MapQuest, Inc."
},
"statuscode": 0,
"messages": []
}
}
So for some reason, Saturday at noon has the same travel times as Tuesday at 16:30. Is there something wrong with my request, or does MapQuest not actually have this capability for the Route Matrix? The API documentation leads me to believe that the date/time options should work on any package in the Directions API, so could someone clear this up? Thanks in advance.
EDIT: I've made sure to try a route that I know for sure has HOV lanes that are restricted during the rush hour and not on the Saturday. I also tried enabling MapQuest's useTraffic parameter. Neither of these had any effect on the data. MapQuest states that useTraffic depends on availability of information, so I think I can only assume that it can't find information (which is strange, since I know that there is tons of data for this area). If anyone has any other ideas or explanations I'd be happy to hear.
MapQuest's Transit & Date/Time Routing "gives the user the ability to specify the time and date of a route, thus compensating for lanes that are HOV at specific times, timed-turned restrictions, and seasonal or specific day of the week closures." It does not account for specific time and date historic traffic. It works great for timed traffic restrictions though.

Convert to dataframe from JSON in R

I am facing issues with conversion of JSON to dataframe. I tried using libraries: jsonlite, RJSONIO,rjson.
I keep getting 'invalid character in the string' or unclosed string.
I am getting this data from a standard API so should be able to parse this json. Also, JSON editors can parse this data just fine.
My question is:
Is there a standard way using which I can make sure that my dataframe gets created and ignore above errors?
My best guess was to convert this data to JSON format using toJSON function from either of the libraries but if I use
newdata <- fromJSON(toJSON(data))
it somehow never gets converted to dataframe. Why is that?
If I instead use
newdata <- fromJSON(data)
I get a valid dataframe but sometimes because of above errors, it doesn't work which is what I am trying to know. How do you deal with this?
I have tried using this too freshDeskTicketsToDF <-
jsonlite::fromJSON(paste(readLines(textConnection(freshDeskTickets)), collapse=""))
It seemed to solve the problem but somewhere I got unclosed string with this method which I otherwise did not.
Are there better ways to deal with this in R?
Also, why is it that using toJSON on data passed to fromJSON never gets converted to a dataframe?
If I decide to take off html tags from the values assisgned to keys in JSON data. How does that work? Can I do that?
Edit: It looks like I get this error when I have <html tags> in my "string data" but I have them all across my JSON data and I don't get it every time.
How to deal with problems like this?
Note: this issue not specific to the data that I have. What I am looking for is ways to deal with problems like these and not one specific solution to a single problem.
I just realized thattoJSON converts R objects to JSON and not JSON to valid JSON. Is there a way to do it instead?
Sample data:
[
{
"cc_emails": [
],
"fwd_emails": [
],
"reply_cc_emails": [
],
"fr_escalated": false,
"spam": false,
"email_config_id": 1000062780,
"group_id": 1000179078,
"priority": 1,
"requester_id": 1022205968,
"responder_id": 1018353725,
"source": 1,
"company_id": null,
"status": 5,
"subject": "Order number-100403891",
"to_emails": [
"contact#stalkbuylove.com"
],
"product_id": null,
"id": 174093,
"type": "Order Status query",
"due_by": "2016-09-02T08:57:30Z",
"fr_due_by": "2016-09-02T02:57:30Z",
"is_escalated": true,
"description": "<div dir=\"ltr\">Hi Team,<div><br></div>\n<div>I have ordered an item from your website, order number-100403891. I had called on August 30 2016 to postpone the delivery date. The guy i spoke from your end had confirmed that he will hold and push the delivery date to September 5 or 6 or 7 2016. And he confirmed the same.</div>\n<div>However, the guy I spoke to<b> did not do it</b>. </div>\n<div>I got to know it from ABHINAV from your customer care team who I spoke to on August 1st at 13:10. Hence I have put a request again and he said he will talk to some guys and give me the desired dates for delivery which is 5,6,7 of September 2016. </div>\n<div>Please let me know the concern on this and hope for a quick turn around.</div>\n<div><br></div>\n<div>Thank you,</div>\n<div>Hari,</div>\n<div>+91-9538199699.</div>\n</div>\n",
"description_text": "Hi Team,\r\n\r\nI have ordered an item from your website, order number-100403891. I had\r\ncalled on August 30 2016 to postpone the delivery date. The guy i spoke\r\nfrom your end had confirmed that he will hold and push the delivery date to\r\nSeptember 5 or 6 or 7 2016. And he confirmed the same.\r\nHowever, the guy I spoke to* did not do it*.\r\nI got to know it from ABHINAV from your customer care team who I spoke to\r\non August 1st at 13:10. Hence I have put a request again and he said he\r\nwill talk to some guys and give me the desired dates for delivery which is\r\n5,6,7 of September 2016.\r\nPlease let me know the concern on this and hope for a quick turn around.\r\n\r\nThank you,\r\nHari,\r\n+91-9538199699.\n",
"custom_fields": {
},
"created_at": "2016-09-01T07:51:18Z",
"updated_at": "2016-09-11T11:00:33Z"
},
{
"cc_emails": [
],
"fwd_emails": [
],
"reply_cc_emails": [
],
"fr_escalated": false,
"spam": false,
"email_config_id": 1000062780,
"group_id": 1000179078,
"priority": 1,
"requester_id": 1022148025,
"responder_id": 1021145209,
"source": 1,
"company_id": null,
"status": 5,
"subject": "Defect in d piece",
"to_emails": [
"contact#stalkbuylove.com"
],
"product_id": null,
"id": 174092,
"type": "Return",
"due_by": "2016-09-01T15:51:00Z",
"fr_due_by": "2016-09-01T09:51:00Z",
"is_escalated": false,
"description": "<div><br></div>\n<div><br></div>\n<div><br></div>\n<div><div style=\"font-size:75%;color:#575757\">Sent from Samsung Mobile</div></div>",
"description_text": "\n\n\nSent from Samsung Mobile",
"custom_fields": {
},
"created_at": "2016-09-01T07:51:00Z",
"updated_at": "2016-09-06T09:00:14Z"
},
{
"cc_emails": [
],
"fwd_emails": [
],
"reply_cc_emails": [
],
"fr_escalated": false,
"spam": false,
"email_config_id": 1000062780,
"group_id": 1000179078,
"priority": 1,
"requester_id": 1022205895,
"responder_id": 1018353725,
"source": 1,
"company_id": null,
"status": 5,
"subject": "Re: StalkBuyLove Return Request for order: 100404435",
"to_emails": [
"StalkBuyLove <contact#stalkbuylove.com>"
],
"product_id": null,
"id": 174088,
"type": "Refund query",
"due_by": "2016-09-01T15:43:56Z",
"fr_due_by": "2016-09-01T09:43:56Z",
"is_escalated": true,
"description": "<div>Hi. Can u deposit the amount if i giv u my account number. Right away i cant choose any other product frim ur site. <br><br>Sent from my iPhone</div>\n<div>\n<br>On Sep 1, 2016, at 12:38 PM, StalkBuyLove <contact#stalkbuylove.com> wrote:<br><br>\n</div>\n<blockquote><div>\n<div><img title=\"StalkBuyLove\" alt=\"Stalkbuylove\" src=\"http://www.stalkbuylove.com/launcher_icons/Newlogo_Stalkbuylove_240x50.png\"></div>\n<div>Hello <b>Anamica Aggarwal</b>,</div>\n<div>We have initiated a return request for order: <b>100404435</b> with the following products:</div>\n<table style=\"width:80%\">\r\n <tbody>\n<tr style=\"background-color:#B0C4DE\">\r\n <th>Item Name</th>\r\n <th>Sku</th>\r\n </tr>\n<tr>\r\n <td style=\"text-align:center\">Articuno Top</td>\r\n <td style=\"text-align:center\">IN1627MTOTOPPCH-198-18</td>\r\n </tr>\n</tbody>\n</table>\n<div>Lots of love,</div>\n<div>Team SBL</div>\n<img src=\"http://mandrillapp.com/track/open.php?u=30069003&id=bff0a5daee4a47fe9c6b04d2680c3c39\" height=\"1\" width=\"1\">\r\n</div></blockquote>",
"description_text": "Hi. Can u deposit the amount if i giv u my account number. Right away i cant choose any other product frim ur site. \n\nSent from my iPhone\n\n> On Sep 1, 2016, at 12:38 PM, StalkBuyLove <contact#stalkbuylove.com> wrote:\n> \n> \n> Hello Anamica Aggarwal,\n> \n> We have initiated a return request for order: 100404435 with the following products:\n> \n> Item Name\tSku\n> Articuno Top\tIN1627MTOTOPPCH-198-18\n> Lots of love,\n> \n> Team SBL\n> \n",
"custom_fields": {
},
"created_at": "2016-09-01T07:43:56Z",
"updated_at": "2016-09-11T11:00:32Z"
},
{
"cc_emails": [
],
"fwd_emails": [
],
"reply_cc_emails": [
],
"fr_escalated": false,
"spam": false,
"email_config_id": 1000062780,
"group_id": 1000179078,
"priority": 1,
"requester_id": 1022205881,
"responder_id": 1021145209,
"source": 1,
"company_id": null,
"status": 5,
"subject": "Details for order",
"to_emails": [
"contact#stalkbuylove.com"
],
"product_id": null,
"id": 174086,
"type": "Order Status query",
"due_by": "2016-09-01T15:42:50Z",
"fr_due_by": "2016-09-01T09:42:50Z",
"is_escalated": false,
"description": "<div><span></span></div>\n<div>\n<span>Hey can i get details of my order </span><br><span>How much more time will it take to get delivered? </span><br><span>Order no-</span><h2 style=\"font-weight: normal; margin: 0px;\"><font><span style=\"background-color: rgba(255, 255, 255, 0);\">100403837</span></font></h2>\n<span></span><br><span>Sent from my iPhone</span><br>\n</div>",
"description_text": "Hey can i get details of my order \r\nHow much more time will it take to get delivered? \r\nOrder no-\r\n100403837\r\n\r\nSent from my iPhone\n",
"custom_fields": {
},
"created_at": "2016-09-01T07:42:50Z",
"updated_at": "2016-09-06T09:00:13Z"
},
{
"cc_emails": [
],
"fwd_emails": [
],
"reply_cc_emails": [
],
"fr_escalated": true,
"spam": false,
"email_config_id": 1000062780,
"group_id": 1000179078,
"priority": 1,
"requester_id": 1022204690,
"responder_id": 1021145209,
"source": 1,
"company_id": null,
"status": 5,
"subject": "Refund",
"to_emails": [
"contact#stalkbuylove.com"
],
"product_id": null,
"id": 174080,
"type": "Refund query",
"due_by": "2016-09-01T15:36:26Z",
"fr_due_by": "2016-09-01T09:36:26Z",
"is_escalated": true,
"description": "<div>\r<br>Bank statement as asked for refund! Please intiate the proccedings asap!<br>\n</div>",
"description_text": "\r\nBank statement as asked for refund! Please intiate the proccedings asap!\n",
"custom_fields": {
},
"created_at": "2016-09-01T07:36:26Z",
"updated_at": "2016-09-07T08:00:19Z"
}
]
library(jsonlite)
df <- stream_in(file("~/data/sample.json"))
This stream_in function directly convert into datafram

Delete or parse data in JSON file with JQ or any Linux Tool

I have this JSON file:
{
"vulnerable_configuration_cpe_2_2": [
"cpe:/o:apple:apple_tv:9.1",
"cpe:/o:apple:watchos:2.1",
"cpe:/o:apple:iphone_os:9.2",
"cpe:/o:apple:mac_os_x:10.11.2"
],
"vulnerable_configuration": [
{
"title": "cpe:2.3:o:apple:apple_tv:9.1",
"id": "cpe:2.3:o:apple:apple_tv:9.1"
},
{
"title": "cpe:2.3:o:apple:watchos:2.1",
"id": "cpe:2.3:o:apple:watchos:2.1"
},
{
"title": "cpe:2.3:o:apple:iphone_os:9.2",
"id": "cpe:2.3:o:apple:iphone_os:9.2"
},
{
"title": "cpe:2.3:o:apple:mac_os_x:10.11.2",
"id": "cpe:2.3:o:apple:mac_os_x:10.11.2"
}
],
"summary": "The Disk Images component in Apple iOS before 9.2.1, OS X before 10.11.3, and tvOS before 9.1.1 allows local users to gain privileges or cause a denial of service (memory corruption) via unspecified vectors.",
"references": [
"https://support.apple.com/HT206168",
"https://support.apple.com/HT205732",
"https://support.apple.com/HT205731",
"https://support.apple.com/HT205729",
"http://lists.apple.com/archives/security-announce/2016/Mar/msg00001.html",
"http://lists.apple.com/archives/security-announce/2016/Jan/msg00005.html",
"http://lists.apple.com/archives/security-announce/2016/Jan/msg00003.html",
"http://lists.apple.com/archives/security-announce/2016/Jan/msg00002.html"
],
"Modified": "2016-03-29T19:01:26.810-04:00",
"Published": "2016-02-01T06:59:01.090-05:00",
"access": {
"vector": "LOCAL",
"complexity": "LOW",
"authentication": "NONE"
},
"cvss": 7.2,
"cvss-time": "2016-03-29T09:43:55.537-04:00",
"cwe": "CWE-119",
"id": "CVE-2016-1717",
"impact": {
"integrity": "COMPLETE",
"confidentiality": "COMPLETE",
"availability": "COMPLETE"
}
I want to delete the following data using JQ or any Linux Tool
access (including vector, complexity, authentication)
references: only first or 1 reference
cvss-time
vulnerable_configuration_cpe_2_2 (Incluide cpe:/o:apple:apple_tv:9.1,
etc)
The result, I want it to be this:
{
"vulnerable_configuration": [
{
"title": "cpe:2.3:o:apple:apple_tv:9.1",
"id": "cpe:2.3:o:apple:apple_tv:9.1"
},
{
"title": "cpe:2.3:o:apple:watchos:2.1",
"id": "cpe:2.3:o:apple:watchos:2.1"
},
{
"title": "cpe:2.3:o:apple:iphone_os:9.2",
"id": "cpe:2.3:o:apple:iphone_os:9.2"
},
{
"title": "cpe:2.3:o:apple:mac_os_x:10.11.2",
"id": "cpe:2.3:o:apple:mac_os_x:10.11.2"
}
],
"summary": "The Disk Images component in Apple iOS before 9.2.1, OS X before 10.11.3, and tvOS before 9.1.1 allows local users to gain privileges or cause a denial of service (memory corruption) via unspecified vectors.",
"references": [
"https://support.apple.com/HT206168",
],
"Modified": "2016-03-29T19:01:26.810-04:00",
"Published": "2016-02-01T06:59:01.090-05:00",
"cvss": 7.2,
"cwe": "CWE-119",
"id": "CVE-2016-1717",
"impact": {
"integrity": "COMPLETE",
"confidentiality": "COMPLETE",
"availability": "COMPLETE"
}
I try
cat file.json | jq ('del(.cvss-time)' and 'access')
but no work
What is the command JQ I have to use? Or what tool can I use?
Tanks!
The following filter for jq >= 1.5 meets your requirements:
del(.access)
| .references |= [.[0]]
| del(."cvss-time")
| del(.vulnerable_configuration_cpe_2_2)
If you are using jq 1.4, use del(.["cvss-time"]) instead of del(."cvss-time").