Merge and Reformat JSON files using jq - json

Question

I have two different JSON files returned with Python code for Twitter data. The first JSON file is in the form below:
A.json
{"tweet_id": "1212024242595926028", "username": "THPDPIO", "created_at": "2019-12-
31T14:54:32.000Z", "tweets": {"0": "Folks, it\u2019s simple!! You know what to do and
what not to do...don\u2019t drink and drive and you don\u2019t have to worry about
ANY consequences. Btw...jail is just a small part of it...think about the
possibility killing someone or killing yourself...again it\u2019s simple...
\ud83d\ude42 #stayhome"}}
{"tweet_id": "1212024242595926028", "username": "TheAliciaRanae", "created_at":
"2019-12-31T15:11:51.000Z", "tweets": {"1": "#THPDPIO Stay home and drink and pass
out and leave everyone else alone lol that\u2019s what I\u2019ll be doing lol HAPPY
NEW YEAR!"}}
{"tweet_id": "1212024242595926028", "username": "duane4343", "created_at": "2019-12-
31T15:21:37.000Z", "tweets": {"1": "#THPDPIO Happy New Year"}}
{"tweet_id": "1212024242595926028", "username": "HollyBr34731868", "created_at":
"2019-12-31T15:24:25.000Z", "tweets": {"1": "#THPDPIO Hope everyone has a safe
night."}}
{"tweet_id": "1211503874395254785", "username": "UNDPoliceDept", "created_at": "2019-
12-30T04:26:46.000Z", "tweets": {"0": "Typical North Dakotan.... #BestCopsAround
#NoTravelAdvised #StayHome"}}
{"tweet_id": "1211503874395254785", "username": "UNDPoliceDept", "created_at": "2019-
12-30T04:27:40.000Z", "tweets": {"1": "#NDHighwayPatrol"}}
{"tweet_id": "1211503874395254785", "username": "BorgenEthan", "created_at": "2019-
12-30T05:28:48.000Z", "tweets": {"1": "#UNDPoliceDept Nah i definitely look like the
first one"}}
With jq, i wrote some commands to choose the fields I want {NB: This was done in https://jqplay.org} with the command {tweet_id: .tweet_id, username: .username, reply: .tweets} | group_by(.tweet_id) but I get the error that
q: error (at :1): Cannot index string with string "tweet_id"
My desired output is to get the sample below for the first file:
{
"tweet_id": "1212024242595926028",
"username": "THPDPIO",
"reply": {
"0": "Folks, it’s simple!! You know what to do and what not to do...don’t drink and drive and you don’t have to worry about ANY consequences. Btw...jail is just a small part of it...think about the possibility killing someone or killing yourself...again it’s simple... 🙂 #stayhome",
"1": "#THPDPIO Stay home and drink and pass out and leave everyone else alone lol that’s what I’ll be doing lol HAPPY NEW YEAR!",
"1": "#THPDPIO Happy New Year",
"1": "#THPDPIO Hope everyone has a safe night."
}}
My issue: To have all replies linked to a particular tweet_id
For the 2nd file
B.json
{
"author_id": 80083199,
"tweet_id": 1212150612026151000,
"username": "CTVdavidspence",
"author_followers": 19572,
"author_tweets": 73406,
"author_description": "Retired broadcast Meteorologist. 2017 RTDNA Lifetime
Achievement Award. Best of Calgary 2018, 2019. AHS Patient and Family Advisor
(volunteer)",
"author_location": "Calgary",
"text": "The Trans Canada Highway near #Sicamous BC. #stayhome Image from
#DriveBC.",
"created_at": 1577834201000,
"retweets": 12,
"replies": 4,
"likes": 27,
"quote_count": 0}
{
"author_id": 848959032370921500,
"tweet_id": 1212024242595926000,
"username": "THPDPIO",
"author_followers": 4626,
"author_tweets": 2383,
"author_description": "Police Sgt.",
"author_location": "Terre Haute, IN",
"text": "Folks, it’s simple!! You know what to do and what not to do...don’t drink
and drive and you don’t have to worry about ANY consequences. Btw...jail is just a
small part of it...think about the possibility killing someone or killing
yourself...again it’s simple... 🙂\n#stayhome",
"created_at": 1577804072000,
"retweets": 11,
"replies": 9,
"likes": 84,
"quote_count": 1}
I will want to merge my final json file such that the tweet_id becomes the reference and the replies and username becomes the keys to merge them. So the final json file will be like the one below:
{
"author_id": 848959032370921500,
"tweet_id": 1212024242595926000,
"username": "THPDPIO",
"author_followers": 4626,
"author_tweets": 2383,
"author_description": "Police Sgt.",
"author_location": "Terre Haute, IN",
"text": "Folks, it’s simple!! You know what to do and what not to do...don’t drink
and drive and you don’t have to worry about ANY consequences. Btw...jail is just a
small part of it...think about the possibility killing someone or killing
yourself...again it’s simple... 🙂\n#stayhome",
"created_at": 1577804072000,
"retweets": 11,
"replies": 9,
"likes": 84,
"quote_count": 1,
"reply": {
"0": "Folks, it’s simple!! You know what to do and what not to do...don’t drink and
drive and you don’t have to worry about ANY consequences. Btw...jail is just a
small part of it...think about the possibility killing someone or killing
yourself...again it’s simple... 🙂 #stayhome",
"1": "#THPDPIO Stay home and drink and pass out and leave everyone else alone lol
that’s what I’ll be doing lol HAPPY NEW YEAR!",
"1": "#THPDPIO Happy New Year",
"1": "#THPDPIO Hope everyone has a safe night."
}}
I will appreciate any help in this regard. Thank you.

Answer 1

As for the second issue, you could employ JOIN on B.json and a custom index (eg. from the first issue) on A.json:
jq --slurpfile A A.json --slurpfile B B.json -n '
JOIN(
$A | reduce group_by(.tweet_id)[] as $g (
{}; .[$g[0].tweet_id].reply += ($g | map(.tweets))
);
$B[]; .tweet_id | #text; add
)
'
{
"author_id": 80083199,
"tweet_id": 1212150612026151000,
"username": "CTVdavidspence",
"author_followers": 19572,
"author_tweets": 73406,
"author_description": "Retired broadcast Meteorologist. 2017 RTDNA Lifetime Achievement Award. Best of Calgary 2018, 2019. AHS Patient and Family Advisor (volunteer)",
"author_location": "Calgary",
"text": "The Trans Canada Highway near #Sicamous BC. #stayhome Image from #DriveBC.",
"created_at": 1577834201000,
"retweets": 12,
"replies": 4,
"likes": 27,
"quote_count": 0
}
{
"author_id": 848959032370921500,
"tweet_id": 1212024242595926000,
"username": "THPDPIO",
"author_followers": 4626,
"author_tweets": 2383,
"author_description": "Police Sgt.",
"author_location": "Terre Haute, IN",
"text": "Folks, it’s simple!! You know what to do and what not to do...don’t drink and drive and you don’t have to worry about ANY consequences. Btw...jail is just a small part of it...think about the possibility killing someone or killing yourself...again it’s simple... 🙂\n#stayhome",
"created_at": 1577804072000,
"retweets": 11,
"replies": 9,
"likes": 84,
"quote_count": 1,
"reply": [
{
"0": "Folks, it’s simple!! You know what to do and what not to do...don’t drink and drive and you don’t have to worry about ANY consequences. Btw...jail is just a small part of it...think about the possibility killing someone or killing yourself...again it’s simple... 🙂 #stayhome"
},
{
"1": "#THPDPIO Stay home and drink and pass out and leave everyone else alone lol that’s what I’ll be doing lol HAPPY NEW YEAR!"
},
{
"1": "#THPDPIO Happy New Year"
},
{
"1": "#THPDPIO Hope everyone has a safe night."
}
]
}
Demo

Answer 2

This response deals with the first issue:
To have all replies linked to a particular tweet_id
As pointed out by #pmf, your sample output won't achieve the goal. Also, group_by expects an array as input. Consider therefore:
< A.json jq -n '
def accumulate_by(stream; f):
reduce stream as $x ({}; (f + [$x|f]) as $v | . + $x | f = $v );
[inputs | {tweet_id, username, reply: .tweets}] | group_by(.tweet_id)
| map( accumulate_by(.[]; .reply ))
'
Note that, by design, this ignores "collisions" in the value of .username; you may need to consider this further. Anyway, with your sample, the result would be:
[
{
"tweet_id": "1211503874395254785",
"username": "BorgenEthan",
"reply": [
{
"0": "Typical North Dakotan.... #BestCopsAround #NoTravelAdvised #StayHome"
},
{
"1": "#NDHighwayPatrol"
},
{
"1": "#UNDPoliceDept Nah i definitely look like the first one"
}
]
},
{
"tweet_id": "1212024242595926028",
"username": "HollyBr34731868",
"reply": [
{
"0": "Folks, it’s simple!! You know what to do and what not to do...don’t drink and drive and you don’t have to worry about ANY consequences. Btw...jail is just a small part of it...think about the possibility killing someone or killing yourself...again it’s simple... 🙂 #stayhome"
},
{
"1": "#THPDPIO Stay home and drink and pass out and leave everyone else alone lol that’s what I’ll be doing lol HAPPY NEW YEAR!"
},
{
"1": "#THPDPIO Happy New Year"
},
{
"1": "#THPDPIO Hope everyone has a safe night."
}
]
}
]