A have a big block of JSON l'm trying to parse, that looks basically like
{
"order": [
"hash1",
"hash2"
],
"posts": {
"hash4": {
"id": "hash4",
"message": "lorem ipsem"
},
"hash5": {
"id": "hash5",
"message": "dolor sit amet"
},
"hash6": {
"id": "hash6",
"message": "consectetur adipiscing elit"
}
}
}
The way I've been handling this so far is to just grep for messages
$ grep 'message' jq_dat.json
"message": "lorem ipsem"
"message": "dolor sit amet"
"message": "consectetur adipiscing elit"
This works for my current purposes, but l'd like to know how to get the same effect with jq. I.e.
$ jq .posts.<something>.message < jq_dat.json
"lorem ipsem"
"dolor sit amet"
"consectetur adipiscing elit"
I've tried using [] and {} in place of something, but those both spit back compile errors.
You just have one too many dot
jq .posts[].message < jq_dat.json
I have this JSON structure and need to pass it to a database. But I do not know if the structure I made is the best possible code has some repetitions that I do not think this is correct and I think it is unnecessary.
JSON:
{
"info": [
{
"atriz": "Sandra Bullock",
"atriz-id": "162",
"atriz-slug": "sandra-bullock",
"carreira": {
"id": "264",
"inicio": "08/10/91",
"final": "16/08/18",
"videos": [
{
"id": "2930500",
"titulo": "Lorem ipsum habitant commodo cubilia eget blandit",
"desc": "Suscipit augue dictum ultrices ultricies aliquam mattis nostra taciti sagittis",
"code": "Sb6y1oTwd",
"source": "http://URL",
"download": true,
"link": "http://URL",
"duration": null,
"post_status": "pending",
"category": "drama",
"upload": {
"send": true,
"id": "118840448",
"url": "http://URL",
"status": "finished"
}
},
{
"id": "2930499",
"titulo": "Lorem ipsum habitant commodo cubilia eget blandit",
"desc": "Suscipit augue dictum ultrices ultricies aliquam mattis nostra taciti sagittis",
"code": "R2G0GhTwF",
"source": "http://URL",
"download": true,
"link": "http://URL",
"duration": null,
"post_status": "pending",
"category": "acao",
"upload": {
"send": true,
"id": "118840554",
"url": "http://URL",
"status": "finished"
}
}
]
}
},
{
"atriz": "Jennifer Lawrence",
"atriz-id": "207",
"atriz-slug": "jennifer-lawrence",
"carreira": {
"id": "263",
"inicio": "02/01/88",
"final": "09/08/18",
"videos": [
{
"id": "2930443",
"titulo": "Lorem ipsum habitant commodo cubilia eget blandit",
"desc": "Suscipit augue dictum ultrices ultricies aliquam mattis nostra taciti sagittis",
"code": "DNJNYHWh",
"source": "http://URL",
"download": true,
"link": "http://URL",
"duration": null,
"post_status": "pending",
"category": "drama",
"upload": {
"send": true,
"id": "118844113",
"url": "http://URL",
"status": "finished"
}
},
{
"id": "2930442",
"titulo": "Lorem ipsum habitant commodo cubilia eget blandit",
"desc": "Suscipit augue dictum ultrices ultricies aliquam mattis nostra taciti sagittis",
"code": "OqieXJHwh",
"source": "http://URL",
"download": true,
"link": "http://URL",
"duration": null,
"post_status": "pending",
"category": "comedia",
"upload": {
"send": true,
"id": "118844112",
"url": "http://URL",
"status": "finished"
}
}
]
}
}
]
}
I do not understand much of SQL so I used an online convert and it returned me this:
SQL:
CREATE TABLE IF NOT EXISTS infos (
`info_atriz` VARCHAR(17) CHARACTER SET utf8,
`info_atriz_id` INT,
`info_atriz_slug` VARCHAR(17) CHARACTER SET utf8,
`info_carreira_id` INT,
`info_carreira_inicio` DATETIME,
`info_carreira_final` DATETIME,
`info_carreira_videos_id` INT,
`info_carreira_videos_titulo` VARCHAR(49) CHARACTER SET utf8,
`info_carreira_videos_desc` VARCHAR(78) CHARACTER SET utf8,
`info_carreira_videos_code` VARCHAR(9) CHARACTER SET utf8,
`info_carreira_videos_source` VARCHAR(10) CHARACTER SET utf8,
`info_carreira_videos_download` VARCHAR(4) CHARACTER SET utf8,
`info_carreira_videos_link` VARCHAR(10) CHARACTER SET utf8,
`info_carreira_videos_duration` INT,
`info_carreira_videos_post_status` VARCHAR(7) CHARACTER SET utf8,
`info_carreira_videos_category` VARCHAR(7) CHARACTER SET utf8,
`info_carreira_videos_upload_send` VARCHAR(4) CHARACTER SET utf8,
`info_carreira_videos_upload_id` INT,
`info_carreira_videos_upload_url` VARCHAR(10) CHARACTER SET utf8,
`info_carreira_videos_upload_status` VARCHAR(8) CHARACTER SET utf8
);
INSERT INTO infos VALUES
('Sandra Bullock',162,'sandra-bullock',264,'1991-10-08 00:00:00','2018-08-16 00:00:00',2930500,'Lorem ipsum habitant commodo cubilia eget blandit','Suscipit augue dictum ultrices ultricies aliquam mattis nostra taciti sagittis','Sb6y1oTwd','http://URL','True','http://URL',NULL,'pending','drama','True',118840448,'http://URL','finished'),
('Sandra Bullock',162,'sandra-bullock',264,'1991-10-08 00:00:00','2018-08-16 00:00:00',2930499,'Lorem ipsum habitant commodo cubilia eget blandit','Suscipit augue dictum ultrices ultricies aliquam mattis nostra taciti sagittis','R2G0GhTwF','http://URL','True','http://URL',NULL,'pending','acao','True',118840554,'http://URL','finished'),
('Jennifer Lawrence',207,'jennifer-lawrence',263,'1988-01-02 00:00:00','2018-08-09 00:00:00',2930443,'Lorem ipsum habitant commodo cubilia eget blandit','Suscipit augue dictum ultrices ultricies aliquam mattis nostra taciti sagittis','DNJNYHWh','http://URL','True','http://URL',NULL,'pending','drama','True',118844113,'http://URL','finished'),
('Jennifer Lawrence',207,'jennifer-lawrence',263,'1988-01-02 00:00:00','2018-08-09 00:00:00',2930442,'Lorem ipsum habitant commodo cubilia eget blandit','Suscipit augue dictum ultrices ultricies aliquam mattis nostra taciti sagittis','OqieXJHwh','http://URL','True','http://URL',NULL,'pending','comedia','True',118844112,'http://URL','finished');
Is this really the best possible structure for the database? This repetition in these values for example ('Sandra Bullock', 162, 'sandra-bullock', 264, '1991-10-08 00:00:00', '2018-08-16 00:00:00') does not have how to optimize this?
The answer depends on your goals you try to achieve. And also it is important to know in what way you will use this data further.
The structure is not very good. You will have several records (their count may be huge - depends on JSON) for the only JSON item. I'm not sure this will be convenient to retrieve data in such way.
It is preferred to use relational model:
Generally, each table/relation represents one "entity type" (such as customer or product). The rows represent instances of that type of entity (such as "Lee" or "chair") and the columns representing values attributed to that instance (such as address or price).
So, in your case, I would use more than 1 table.
-- videos with columns {id, titulo, desc, etc.}
-- carreira with id of related "atriz" record as a separate column and other columns {id, inicio, final}
-- carreira_videos - separate table which stores carreira-video relations. The rable may contain only two columns: carreira_id and video_id. For example, "Sandra Bullock" item in your example JSON has carreira with two videos - so, in carreira_videos table you will have two records:
carreira_id | video_id
264 | 2930500
264 | 2930499
-- atrizs with columns {atriz, atriz-id, atriz-slug} and separate column related carreira_id.
Actually, you're right about a lot of data repetition in your example. In case your JSON has more than just two items in "info" node, then there are more data duplicates. If you will use relational model (as it is preferred in SQL-based databases) and optimal schema such as described above you will avoid repetitions.
You need to study basics of relational databases and SQL before you go further. If you do not understand them then it is not real to create optimize design.
https://www.w3schools.com/sql/
https://searchdatamanagement.techtarget.com/definition/relational-database
Actually, if you want to store JSON itself then you may not need SQL-based database. You should look at NoSQL approaches and DBs https://en.wikipedia.org/wiki/NoSQL
https://www.quackit.com/json/tutorial/list_of_json_databases.cfm
All depends on your goals.
I have a problem with woocommerce rest api. I want to create a product with photos via rest api, but it works only without photos.
This snippet of code work:
{
"regular_price": "21.99",
"name": "Premium Quality",
"description": "Pellentesque habitant morbi tristique senectus et netus",
"type": "simple"
}
but this doesn't work:
{
"regular_price": "21.99",
"images": "[{\"src\":\"https://static01.nyt.com/images/2017/03/17/world/europe/oakImage-1489777706390/oakImage-1489777706390-largeHorizontal375.jpg\",\"position\":0,\"id\":0}]",
"name": "Premium Quality",
"description": "Pellentesque habitant morbi tristique senectus et netus",
"type": "simple"
}
The JSON is validate. The response from the server is:
{
"code": "woocommerce_product_invalid_image_id",
"message": "#0 is an invalid image ID.",
"data": {
"status": 400
}
}
The documentation claims:
Image ID (attachment ID). In write-mode used to attach pre-existing images.
According to that, you should send this request without the id parameter in your image object. This is because there is no pre-existing image with the id=0, I guess.
Example code taken from the docs:
{
name: 'Premium Quality',
type: 'simple',
regular_price: '21.99',
description: 'Pellentesque habitant morbi...',
short_descriptienter code hereon: 'Pellentesque ...',
categories: [
{
id: 9
},
{
id: 14
}
],
images: [
{
src: 'http://demo.woothemes.com/woocommerce/wp-content/uploads/sites/56/2013/06/T_2_front.jpg',
position: 0
},
{
src: 'http://demo.woothemes.com/woocommerce/wp-content/uploads/sites/56/2013/06/T_2_back.jpg',
position: 1
}
]
};
I am trying to get this:
[
["John Black",[
["Lorem ipsum dolor sit amet.",27],
["Ut tempus lectus ut mi.",23]
]],
["Peter Pan",[
["Quisque pulvinar odio.",22],
["Nec ut lorem quis interdum elit.",32]
]],
["Gary Halbert",[
["Placerat aliquam.",17]
]],
["Richard Gere",[
["Porttitor commodo fermentum.",28]
]]
]
Till now, this is what I got:
export A=$(cat <<'EOL'
[
["John Black",["Lorem ipsum dolor sit amet.",27]],
["Peter Pan",["Quisque pulvinar odio.",22]],
["John Black",["Ut tempus lectus ut mi.",23]],
["Gary Halbert",["Placerat aliquam.",17]],
["Peter Pan",["Nec ut lorem quis interdum elit.",32]],
["Richard Gere",["Porttitor commodo fermentum.",28]]
]
EOL
)
echo "$A" | jq 'map({(.[0]): .[1]}) | add'
Resulting this:
{
"John Black": [
"Ut tempus lectus ut mi.",
23
],
"Peter Pan": [
"Nec ut lorem quis interdum elit.",
32
],
"Gary Halbert": [
"Placerat aliquam.",
17
],
"Richard Gere": [
"Porttitor commodo fermentum.",
28
]
}
I am using jq-1.5.
Any ideas? Thanks.
This is an appropriate use case for a reducer. Most of the below is related not to joining items under shared keys, but to getting them into the desired nested-list form:
jq -n '[
inputs |
reduce .[] as $item ({}; .[$item[0]] += [$item[1]]) |
to_entries |
.[] |
[.key, .value]
]' <<<"$A"
...yields as output (edited only for compactness with regard to whitespace):
[
["John Black", [["Lorem ipsum dolor sit amet.", 27], ["Ut tempus lectus ut mi.",23]]],
["Peter Pan", [["Quisque pulvinar odio.",22], ["Nec ut lorem quis interdum elit.", 32]]],
["Gary Halbert", [["Placerat aliquam.", 17]]],
["Richard Gere", [["Porttitor commodo fermentum.", 28]]]
]
the Couchdb engine is append only, it means that each document will append into a file manager by Couchdb, and each file is related to a db.
to use properly think how to avoid updates in the same document, remember all revisions will be storage.
my suggestion for each document.
{
"name": "John Black",
"entries": [
{
"test": "Lorem ipsum dolor sit amet.",
"value": 27
},
{
"text": "Ut tempus lectus ut mi.",
"value": 23
}
],
"type": "user"
}
I created json for adding new poi with fw_media data included, based on json from example_components/fw_media.json:
{"fw_core":
{"location":{"wgs84":{"latitude":1,"longitude":1}},
"categories":["Field"],
"name":{"":"poljana 1"}},
"fw_media": {
"entities": [
{
"type": "photo",
"short_label": {
"en": "Sunset at sea"
},
"caption": {
"en": "Sunset on the Bothnian Bay, Northwest from Hailuoto summer 2013"
},
"description": {
"": "Lorem ipsum dolor sit amet, consectetur adipisci elit, sed eiusmod tempor incidunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquid ex ea commodi consequat.",
"fi": "Oli mukava retki."
},
"thumbnail": "http://www.example.org/sunset_on_sea_tbn.jpg",
"url": "http://www.example.org/sunset_on_sea.jpg",
"copyright": "Photo: Ari Okkonen"
},
{
"type": "audio",
"short_label": {
"": "Säkkijärven polkka"
},
"url": "http://www.example.org/sakkijarven_polkka.mp3"
}
],
"last_update": {
"timestamp": 1390203898
}, "source": {
"website": "http://www.cie.fi",
"name": "CIE, University of Oulu",
"id": "7c32c67d-cf00-4d11-9acc-2471141e03a3",
"license": "http://www.gnu.org/licenses/gpl.html"
}
}}
But im getting error:
JSON does not validate. Violations:
[fw_media] The property - source - is not defined and the definition
does not allow additional properties
POI data validation failed!
Can you give me example of working fw_media json?
And also is it possible to upload image with poi in fw_media?(Not just url to image but whole image)
Corrected in master branch in GitHub.
Thank you for your report.
It seems that an old example somewhere has the fw_image.source field that is not implemented. Just edit the source structure out of the data, and it should be Ok. Of course there is no sense to have one source description common for all images, etc. for the one POI.
A source field might be usable in per-item basis. So I'll introduce it to the next release.
Please, could you tell me where that erroneous Sunset at sea is, so I can go correct it. Correction is tracked in https://github.com/Chiru/FIWARE-POIDataProvider/issues/7
I feel uploading pictures a kind of Specific Enabler stuff. So it left to the community to implement. I suggest using some media repository in combination of a standard POI-DP by a specialized client software.