JQ Array to new fields - json

I have a sample json data like:
{
"phone_number": "780-414-2085",
"city": "Edmonton",
"updated": "2015-10-19T00:03:10",
"name": "Sir William Place ",
"url": "http://www.bwalk.com/en-CA/Rent/Details/Alberta/Edmonton/Sir-William-Place",
"last_visited": "2015-10-19T00:03:10",
"rooms": [{
"available": "Available",
"bathrooms": ["1"],
"suite_type": "1 Bedroom",
"square_feet": ["594", "649"],
"deposit": ["$499"],
"price_range": ["$1059", "$1209"]
}, {
"available": "Available",
"bathrooms": ["1"],
"suite_type": "1 Bedroom + Den",
"square_feet": ["771"],
"deposit": ["$499"],
"price_range": ["$1169", "$1249"]
}, {
"available": "Available",
"bathrooms": ["1", "2"],
"suite_type": "2 Bedroom",
"square_feet": ["894", "970"],
"deposit": ["$499"],
"price_range": ["$1344", "$1494"]
}, {
"available": "Available",
"bathrooms": ["2"],
"deal": ["October FREE and $299 Security Deposit on 12 month leases "],
"suite_type": "2 Bedroom Bi-level",
"square_feet": ["894"],
"deposit": ["$499"],
"price_range": ["$1344", "$1394"]
}, {
"available": "Waiting List",
"bathrooms": ["1"],
"suite_type": "Bachelor",
"square_feet": ["540"],
"deposit": ["$499"],
"price_range": ["$1004", "$1054"]
}],
"address": "8830-85 St., Edmonton, Alberta, T6C 3C3",
"zip_code": "T6C 3C3"
}
And I am running a jq expression like:
'{phone_number, city, updated, name, address, zip_code, url, last_visited} + (.rooms[] | {suite_type, price_range_start: .price_range[0], price_range_end: .price_range[1]} + {available, square_foot_start:.square_feet[0], square_foot_end:.square_feet[1], deposit:.deposit[0], bathrooms:.bathrooms[0]})'
This gives me an ok output but repeats the same names because I just list the rooms array. I want to be able to set each item in the rooms array to something like room1, room2, room3 etc. But also to keep it in one entry, so for example with the sample here it ends up 5 entries because there is 5 rooms and name for instance gets repeated 5 times because the way I have it set now. I think I need to map the rooms to something but not sure how to do that.
Can someone advise on how to do this?

You can update the elements in the array whilst keeping the other elements as is like this:
'.rooms[] |= {suite_type, price_range_start: .price_range[0],
price_range_end: .price_range[1]} + {available,
square_foot_start:.square_feet[0], square_foot_end:.square_feet[1],
deposit:.deposit[0], bathrooms:.bathrooms[0]}'

Here is a solution which uses functions.
def common_columns:
"phone_number", "city", "updated", "name", "address", "zip_code", "url", "last_visited"
;
def common:
.phone_number, .city, .updated, .name, .address, .zip_code, .url, .last_visited
;
def room_columns(n):
range(n)
| (
"available_\(.)", "bathrooms_\(.)", "suite_type_\(.)",
"square_feet_start_\(.)", "square_feet_end_\(.)", "deposit_\(.)",
"price_range_start_\(.)", "price_range_end_\(.)"
)
;
def rooms(n):
. as $r
| range(n)
| $r.rooms[.]
| (
.available, .bathrooms[0], .suite_type,
.square_feet[0,1], .deposit[0], .price_range[0,1]
)
;
[ common_columns, room_columns(6) ]
, [ common, rooms(6) ]
| #csv
You can change the 6 to however many sets of room columns you need.

Related

Conditionally merging two separate JSON objects in JQ

This is how my input looks:
{
"text" : "Some text here"
}
{
"usage": {
"text_units": 1,
"text_characters": 101,
"features": 1
},
"language": "en",
"categories": [
{
"score": 0.655041,
"label": "/technology law, govt and politics/espionage and intelligence/surveillance"
},
{
"score": 0.639809,
"label": "/technology and computing/computer security/network security"
},
{
"score": 0.624533,
"label": "/business and industrial/business operations"
}
]
}
Using JQ, if the first element of array category in the second object contains /technology, I want to add a new field named relevant with 1 as value (which I managed), and copy the text field from the first object.
So, the expected output is:
{
"usage": {
"text_units": 1,
"text_characters": 101,
"features": 1
},
"language": "en",
"categories": [
{
"score": 0.655041,
"label": "/technology law, govt and politics/espionage and intelligence/surveillance"
},
{
"score": 0.639809,
"label": "/technology and computing/computer security/network security"
},
{
"score": 0.624533,
"label": "/business and industrial/business operations"
}
],
"relevant": 1,
"text": "Some text here"
}
And this is what I have done so far:
if .categories[0].label | test("/technology"; "i") then . |=( . + {"relevant": 1} + {"text": .text}) else . |= . + {"relevant": 0} end
Link to a demo on jqplay
Your input consists of two separate objects. In order to be able to access the first while processing the second, you could save the first into a variable.
. as {$text} | input | if .categories[0].label | test("/technology"; "i") then . + {relevant: 1, $text} else . + {relevant: 0} end
Online demo

Laravel querybuilder distinct function not working in search

I tried putting distinct() in my query but when i get the results in my frontend and in the api, I still get duplicate records. Does anyone know why distinct is not working in my code?
My code
$result = DB::connection('mysql2')
->table('xp_pn_ura_transactions')
->whereRaw(DB::raw("CONCAT(block, ' ', street,' ',project_name,' ', postal_code,'')LIKE '%$request->projectname%' order by STR_TO_DATE(sale_date, '%d-%M-%Y') asc"))
->limit($request->limit)
->distinct()
->get();
return \Response::json(array(
//'total_count' => $count,
'result' => $result,
));
Front end result
My response, I only get the first two objects that duplicates
{
"id": 228686,
"transtype": "RESI",
"project_name": "WATERFRONT WAVES",
"unitname": "08-06 ",
"block": "760",
"street": "Bedok Reservoir Road ",
"level": "08",
"stack": "06 ",
"no_of_units": "1",
"area": "147",
"type_of_area": "Strata",
"transacted_price": "1300500",
"nettprice": "-",
"unitprice_psm": "8847",
"unitprice_psf": "822",
"sale_date": "20-JAN-2008",
"contract_date": " ",
"property_type": "Condominium",
"tenure": "99 Yrs From 31/10/2007",
"completion_date": "Uncompleted",
"type_of_sale": "New Sale",
"purchaser_address_indicator": "Private",
"postal_district": "16",
"postal_sector": "47",
"postal_code": "479245",
"planning_region": "East Region",
"planning_area": "Bedok",
"update_time": "2019-12-09 17:14:35"
},
{
"id": 224686,
"transtype": "RESI",
"project_name": "WATERFRONT WAVES",
"unitname": "08-06 ",
"block": "760",
"street": "Bedok Reservoir Road ",
"level": "08",
"stack": "06 ",
"no_of_units": "1",
"area": "147",
"type_of_area": "Strata",
"transacted_price": "1300500",
"nettprice": "-",
"unitprice_psm": "8847",
"unitprice_psf": "822",
"sale_date": "20-JAN-2008",
"contract_date": " ",
"property_type": "Condominium",
"tenure": "99 Yrs From 31/10/2007",
"completion_date": "Uncompleted",
"type_of_sale": "New Sale",
"purchaser_address_indicator": "Private",
"postal_district": "16",
"postal_sector": "47",
"postal_code": "479245",
"planning_region": "East Region",
"planning_area": "Bedok",
"update_time": "2019-12-09 17:11:57"
}
They got different id but same records, is there a way to ignore the id and get the other fields?
You need to select the field that you need to distinct, or it will distinct all the fields that you selected:
So according to your post, the id and updated_time are not duplicated, you don't need to select it out.
Try something like this:
$result = DB::connection('mysql2')
->table('xp_pn_ura_transactions')
->whereRaw(DB::raw("CONCAT(block, ' ', street,' ',project_name,' ', postal_code,'')LIKE '%$request->projectname%' order by STR_TO_DATE(sale_date, '%d-%M-%Y') asc"))
->limit($request->limit)
# select the fields which is duplicated.(In your post, select the field without id and updated_time)
->select("transtype",
"project_name",
"unitname",
"block",
"street",
"level",
"stack",
"no_of_units",
"area",
"type_of_area",
"transacted_price",
"nettprice",
"unitprice_psm",
"unitprice_psf",
"sale_date",
"contract_date",
"property_type",
"tenure",
"completion_date",
"type_of_sale",
"purchaser_address_indicator",
"postal_district",
"postal_sector",
"postal_code",
"planning_region",
"planning_area")
->distinct()
->get();
if you need to select the fields not duplicated, you can use groupBy() instead of distinct()

Nested filtering with jq

First time user of jq and I'm wanting to filter out objects based on a value within them and I'm struggling to figure it out.
I have a big json file with lots of product data like what's below. I'm wanting to filter out based upon which website_ids they have.
Example Input:
[{
"product_id": "2",
"sku": "PROD2",
"name": "Product Name 2",
"set": "4",
"type": "simple",
"category_ids": {
"item": "15"
},
"website_ids": {
"item": [
"1",
"4"
]}
},{
"product_id": "3",
"sku": "PROD3",
"name": "Product Name 3",
"set": "4",
"type": "simple",
"category_ids": {
"item": "15"
},
"website_ids": {
"item": [
"1",
"2"
]}
}]
Desired output:
[{
"product_id": "2",
"sku": "PROD2",
"name": "Product Name 2",
"set": "4",
"type": "simple",
"category_ids": {
"item": "15"
},
"website_ids": {
"item": [
"1",
"4"
]}
}]
I've tried a few different things but I'm clearly just not getting it.
jq 'map(.website_ids.item[] | contains("4"))'
Gives me:
[
false,
true,
false,
false
]
Which seems to match the website_ids items I want, but I'm not sure how to get the full JSON object from that.
Any help would be super appreciated! Thanks.
EDIT:
I've used this and it works with my example:
map(select(.website_ids.item[] | contains("4")))
What I've realised is that my example and the file I was actually testing on have some differences.
Sometimes a product has this for the website_id items:
"website_ids": {
"item": "2"
}
Which results in the error:
Cannot iterate over string ("2")
Is there a way around this?
All you need to do is add a select call in your map function, like so:
jq 'map(select(.website_ids.item[] | contains("4")))'
After your edit, it's a bit more complicated, but it can be worked around by checking the type of .website_ids.item and then based off of that type, doing a contains check or a simple equality check:
map((select((.website_ids.item | type) == "array") | select(.website_ids.item[] | contains("4"))), (select((.website_ids.item | type) == "string") | select (.website_ids.item == "4")))
Here it is formatted a bit more readable:
map(
(select((.website_ids.item | type) == "array") | select(.website_ids.item[] | contains("4"))),
(select((.website_ids.item | type) == "string") | select (.website_ids.item == "4"))
)

How to extract labels from json?

I have this json:
"client":{
"datas": {
"number1": 14,
"number2": 2,
"number3": 1,
"city": [
"paris",
"tokyo",
"nyc",
"london"
]
},
}
I want to extract label/value peers for the "number" labels but not for the array like that:
number 1 : 14
number 2 : 2
numer 3 : 1
paris,
tokyo,
nyc,
london
Edit:
I tried this loop:
for (i in datas)
{ newDatas = clents.datas[i]; }
I get :
14
2
1
Array [
"paris",
"tokyo",
"nyc",
"london", ]

Filter by values using ./jq

Given the input of sizes:
[
{
"stock": 1,
"sales": 0,
"sizes": [
{
"countries": ["at", "be", "ch", "cy", "de", "ee", "es", "fi", "gr", "ie", "lu", "lv", "nl", "pl", "pt", "se", "si", "sk"],
"size": "EU 45,5"
},
{
"countries": ["it"],
"size": "EU 45,5"
},
{
"countries": ["fr"],
"size": "EU 45,5"
},
{
"countries": ["gb"],
"size": "EU 45,5"
}
]
}
]
I will like to get the same structure without the ones that countries hasn't "de" (Germany) and remove the field complete. Expected something like this:
[
{
"stock": 1,
"sizes": [
{
"size": "EU 45,5"
}
]
}
]
I tried this:
map(.sizes[] |= select(.countries | join(",") | contains("de"))) | map({ stock, sizes })
But the filter is not working properly, throwing jq: error (at <stdin>:48): Cannot iterate over null (null).
Tried has, in, contains, inside and nothing seems to work.
Also, how can I filter which field appears? With map({ stock, sizes }) countries still there. Can I do something like map({ stock, sizes: { size } })?
Here's a one-liner that answers your main question -- if you can't see how it works, try breaking it up into separate pieces:
map( .sizes |= map( select(.countries | index("de") ) | del(.countries) ))
Regarding the selection of fields, you can use del/1 as above, or sometimes simply using an expression such as {key1, key2} will do the trick. Consider also this function and the following example:
def query(queryobject):
with_entries( select( .key as $key | queryobject | has( $key ) ));
Example:
$ jq -c -n '{"a": 1, "b": null, "c":3} | query( {a,b,d} )'
{"a":1,"b":null}