In Cloud BigQuery, the accepted JSON format is:
One JSON object, including any nested/repeated fields, must appear on each line.
refer to: https://cloud.google.com/bigquery/data-formats#json_format
Now, given a json:
{
"1": {
"kind": "person",
"fullName": "John Doe",
"age": 22,
"gender": "Male",
"citiesLived": [
{
"place": "Seattle",
"numberOfYears": 5
},
{
"place": "Stockholm",
"numberOfYears": 6
}
]
},
"2": {
"kind": "person",
"fullName": "Jane Austen",
"age": 24,
"gender": "Female",
"citiesLived": [
{
"place": "Los Angeles",
"numberOfYears": 2
},
{
"place": "Tokyo",
"numberOfYears": 2
}
]
}
}
How to convert it into the following by jq?
{"kind": "person", "fullName": "John Doe", "age": 22, "gender": "Male", "citiesLived": [{ "place": "Seattle", "numberOfYears": 5}, {"place": "Stockholm", "numberOfYears": 6}]}
{"kind": "person", "fullName": "Jane Austen", "age": 24, "gender": "Female", "citiesLived": [{"place": "Los Angeles", "numberOfYears": 2}, {"place": "Tokyo", "numberOfYears": 2}]}
The key here is the "-c" option, which in effect tells jq to use the JSONLines output format.
In your particular case, the solution is simply:
jq -c '.[]'
Your shell might even allow you to drop the quotation marks :-)
Related
I just need to query all information of the doctor in the first index.
Here is my sample XML data
Here is my sample JSON data:
Both XML and JSON data are same. I just converted the XML to JSON format.
I have more documents like this.
I tried this query in this image
this query showing all information but I just need first doctors information.
I also tried in command prompt.Here is the query done in cmd
Here is my sample JSON code block: but I have more similar documents
{
"doctors": {
"doctor":[
{
"ssn": "257-79-xxxx",
"name": "Mavis Bxx",
"address": "xxxx Rusk Drive",
"country": "France",
"email": "",
"phone": "",
"patients": {
"patient": [
{
"gender": "Male",
"name": "Itch xxxx",
"address": "xx Cottonwood Avenue",
"revenue": "254",
"_country": "Spain",
"_id": "27"
},
{
"gender": "Male",
"name": "Damon xxxxx",
"address": "xx David Trail",
"revenue": "370",
"_country": "Germany",
"_id": "21"
}
]
},
"_id": "6"
},
{
"ssn": "179-45-xxxx",
"name": "Tobie Conxxxx",
"address": "x Comanche Center",
"country": "Spain",
"email": "tconringh#xxx.xxx",
"phone": "+86 998 262 xxxx",
"patients": {
"patient": {
"gender": "Male",
"name": "Vergil Tome",
"address": "x Melody Drive",
"revenue": "254",
"_country": "Germany",
"_id": "15"
}
},
"_id": "18"
},
{
"ssn": "777-59-xxxx",
"name": "Gertrud Macxxxx",
"address": "x Buell Drive",
"country": "USA",
"email": "gmaclaig2#xxx.com",
"phone": "+62 975 394 xxxx",
"patients": {
"patient": [
{
"gender": "Non-binary",
"name": "Dre Skxxxx",
"address": "x Becker Circle",
"revenue": "400",
"_country": "Germany",
"_id": "20"
},
{
"gender": "Female",
"name": "Arleyne Lestxxxx",
"address": "xx Farragut Court",
"revenue": "225",
"_country": "France",
"_id": "22"
}
]
},
"_id": "3"
}
You needs to change your collect data format. Your current format is single big size data. Not a quarriable format.
So
From current format
{
"doctors": {
"docker": [
{
...
"patients": {
"patient": [
{
...
},
{
...
}
]
},
},
{
...
}
]
}
}
To this format
[
{
...
"patients": [
{
...
},
{
...
}
]
},
{
...
}
]
Test JSON
[
{
"ssn": "2xx-7x-4xxx",
"name": "Mavis Bxxxed",
"address": "9xxx Rusk Drive",
"country": "France",
"email": "",
"phone": "",
"patients": [
{
"gender": "Male",
"name": "Itch Txxx",
"address": "5xx Cottonwood Avenue",
"revenue": "25x",
"_country": "Spain",
"_id": "27"
},
{
"gender": "Male",
"name": "Damon Wxxx",
"address": "09xx Dxxxx Trail",
"revenue": "370",
"_country": "Germany",
"_id": "21"
}
],
"_id": "6"
},
{
"ssn": "19-45-xxxx",
"name": "Tobie Coxxxx",
"address": "8x Comxxxx Center",
"country": "Spain",
"email": "tconxxxxx#usa.gov",
"phone": "",
"patients": [
{
"gender": "Male",
"name": "Some one",
"address": "1 Mexxxx Drive",
"revenue": "254",
"_country": "Germany",
"_id": "15"
}
],
"_id": "18"
}
]
In Compass View after add data upper data
Find doctor by name query
{ name: "Mavis Bxxxed" }
Find doctor by patient name
{ "patients.name": "Some one" }
If same search with this query
{ "patients.name": "Itch Txxx" }
OR
{ "patients.name": "Damon Wxxx" }
will same result to find the first docker (Mavis Bxxxed)
Due to it's patients start [ and end ]. it is array type a single size of data.
You can filter by project option
{
patients: {
$filter:
{
input: "$patients",
cond: { $eq: [ "$$patient.name", "Itch Txxx"] },
as: "patient"
}
}
}
OR shows only address
{ "patients.address" : 1 }
Need to parse the following Json like the following-
new.json
{
"name": "johnDoe",
"dept": 45,
"details": [
{
"salary": "76566",
"weight": "150",
"height": "160",
"sex": "male",
"country": "Usa",
"State": "NJ"
},
{
"salary": "76560",
"weight": "160",
"height": "180",
"sex": "male",
"country": "Usa",
"State": "NC"
}
]
}
{
"name": "DanLee",
"dept": 46,
"details": [
{
"salary": "76566",
"weight": "180",
"height": "160",
"sex": "male",
"country": "Usa",
"State": "NJ"
},
{
"salary": "76560",
"weight": "190",
"height": "180",
"sex": "male",
"country": "Usa",
"State": "NC"
},
{
"salary": "87888",
"weight": "170",
"height": "160",
"sex": "male",
"country": "Usa",
"State": "NY"
}
]
}
Need name, weight and State information extracted from the above Json format and get a comma separated file for analysis like this:
name,weight,State
johnDoe,150,NJ
johnDoe,160,NC
DanLee,180,NJ
DanLee,190,NC
DanLee,170,NY
I have just started to use Jq in Linux and would like to understand how such conversion can be done with a step by step explanation.
Thanks.
jq -r '.name as $name | .details[] | [$name, .weight, .State] | #csv' foo.json
gives me:
"johnDoe","150","NJ"
"johnDoe","160","NC"
"DanLee","180","NJ"
"DanLee","190","NC"
"DanLee","170","NY"
Breakdown:
.name as $name remember the current name as $name
.details[] expand the details array one by one
[$name, .weight, .State] construct an array for every element of details
#csv use the built-in CSV conversion.
-r: don't wrap output in JSON strings again.
I have a JSON object which has 100000 records. I want a select a record which has specific value to the one of the key
Eg:
[{
"name": "bindu",
"age": "24",
"qualification": "b.tech"
},
{
"name": "naveen",
"age": "23",
"qualification": "b.tech"
},
{
"name": "parvathi",
"age": "23",
"qualification": "m.tech"
},
{
"name": "bindu s",
"status": "married"
},
{
"name": "naveen k",
"status": "unmarried"
}]
now I want to combine the records which are having the name with 'bindu' and 'bindu s. We can achieve this by iterating on the JSON object but since the size is more it is taking more time. Is there any way to make this easy.
I want the output like
[{
"name": "bindu",
"age": "24",
"qualification": "b.tech",
"status": "married"
},
{
"name": "naveen",
"age": "23",
"qualification": "b.tech",
"status": "unmarried"
},
{
"name": "parvathi",
"age": "23",
"qualification": "m.tech"
"status": ""
},
This will rename and merge your objects by first name.
jq 'map(.name |= split(" ")[0]) | group_by(.name) | map(add)'
I have a JSON file with nested elements that I'm trying to manipulate into an unnested JSON file. How do I do this?
Using js, I have tried to isolate the percentages, which I was able to do. I'm not sure how to rename the percentage field. I've seen instances where it seems like value.gender or value.grade should work, but I'm also not sure how to combine it.
jq '.data[] | .id as $id | (.demographics[] | .percentage as $percentage | .gender as $gender | .grade as $grade | {"id":$id, "percentage":$percentage})' test2.json
From here, I want to be able to rename the percentage field to the gender and grades values. I then want to group by id.
Here is the original JSON file (test2.json):
{
"data": [{
"id": "abc",
"students": "elementary",
"demographics": [{
"grade": "K-2",
"percentage": "0.1",
"gender": "unspecified"
},
{
"grade": "K-2",
"gender": "male",
"percentage": "0.5"
},
{
"gender": "female",
"percentage": "0.4",
"grade": "K-2"
},
{
"grade": "3-6",
"percentage": "0.3",
"gender": "male"
},
{
"percentage": "0.2",
"gender": "unspecified",
"grade": "3-6"
},
{
"grade": "3-6",
"gender": "female",
"percentage": "0.5"
}
],
"neighborhood_name": [{
"percentage": "0.5",
"neighborhood": "atwood"
},
{
"region": "bluff",
"percentage": "0.5"
}
]
},
{
"id": "def",
"students": "midhigh",
"demographics": [{
"grade": "7-9",
"percentage": "0.2",
"gender": "unspecified"
},
{
"grade": "7-9",
"gender": "male",
"percentage": "0.2"
},
{
"gender": "female",
"percentage": "0.6",
"grade": "7-9"
},
{
"grade": "10-12",
"percentage": "0.1",
"gender": "male"
},
{
"percentage": "0.1",
"gender": "unspecified",
"grade": "10-12"
},
{
"grade": "10-12",
"gender": "female",
"percentage": "0.8"
}
],
"neighborhood_name": [{
"percentage": "0.2",
"neighborhood": "atwood"
},
{
"region": "bluff",
"percentage": "0.8"
}
]
}
]
}
Here is what I expect:
{
"id": "abc",
"students": "elementary",
"demo_K-2_unspecified": "0.1",
"demo_K-2_male": "0.5",
"demo_K-2_female": "0.4",
"demo_3-6_male": "0.3",
"demo_3-6_unspecified": "0.6",
"demo_3-6_female": "0.5",
},
{
"id": "def",
"students": "midhigh",
"demo_7-9_unspecified": "0.2",
"demo_7-9_male": "0.2",
"demo_7-9_female": "0.6",
"demo_10-12_male": "0.1",
"demo_10-12_unspecified": "0.1",
"demo_10-12_female": "0.8",
}
With the sample data, the following filter produces the desired output:
.data[]
| {id, students} as $ix
| .demographics
| map( {"demo_\(.grade)_\(.gender)": .percentage} )
| $ix + add
The main idea here is to use map to create a list of the key-value pairs, so that the composite object can then easily be created using add.
As a one-liner
jq '.data[] | {id,students} + (.demographics | map( {"demo_\(.grade)_\(.gender)": .percentage} ) | add)' test2.json
i wanted to simulate a http request by reading an Json File and its stored data. Here is how i tryed it
Data.json:
{ "jsonData": {
"data1": [{
"age": "18",
"name": "bernd",
"gender": "1"
}],
"data2": [{ "age": "18", "name": "bernd", "gender": "1"},
{ "age": "18", "name": "bernd", "gender": "1"},
{ "age": "15", "name": "bernd", "gender": "1"},
{ "age": "18", "name": "marting", "gender": "1"},
{ "age": "25", "name": "bernd", "gender": "1"},
{ "age": "29", "name": "bernd", "gender": "1"},
{ "age": "18", "name": "mike", "gender": "1"},
{ "age": "45", "name": "bernd", "gender": "1"},
{ "age": "18", "name": "bernd", "gender": "1"}
],
"data3": [{ "age": "18", "name": "mike", "gender": "1"},
{ "age": "18", "name": "monika", "gender": "2"},
{ "age": "18", "name": "martin", "gender": "1"},
{ "age": "18", "name": "monika", "gender": "2"},
{ "age": "18", "name": "monika", "gender": "2"},
{ "age": "18", "name": "monika", "gender": "2"},
{ "age": "18", "name": "bernd", "gender": "1"},
{ "age": "18", "name": "bernd", "gender": "1"},
{ "age": "18", "name": "lisa", "gender": "2"}
]}}
and a service which recieves the data
data.service:
getServers() {
return this.http.get('../assets/Data.json')
.subscribe((res: Response) => {
this.testData = res.json().data2;
} );}
The plan is to get for example the "data2" object Array into "this.testData". I tryed different versions of http.get, but just managed to get the Json object without successfully converting it into a object[].
I would really appreciate your help guys.