Insert mongodb to my json file - json

"Title": "Anti-Mage",
"Url": "antimage",
"ID": 1,
"Lore": "The monks of Turstarkuri watched the rugged valleys below their mountain monastery as wave after wave of invaders swept through the lower kingdoms. Ascetic and pragmatic, in their remote monastic eyrie they remained aloof from mundane strife, wrapped in meditation that knew no gods or elements of magic. Then came the Legion of the Dead God, crusaders with a sinister mandate to replace all local worship with their Unliving Lord's poisonous nihilosophy. From a landscape that had known nothing but blood and battle for a thousand years, they tore the souls and bones of countless fallen legions and pitched them against Turstarkuri. The monastery stood scarcely a fortnight against the assault, and the few monks who bothered to surface from their meditations believed the invaders were but demonic visions sent to distract them from meditation. They died where they sat on their silken cushions. Only one youth survived - a pilgrim who had come as an acolyte, seeking wisdom, but had yet to be admitted to the monastery. He watched in horror as the monks to whom he had served tea and nettles were first slaughtered, then raised to join the ranks of the Dead God's priesthood. With nothing but a few of Turstarkuri's prized dogmatic scrolls, he crept away to the comparative safety of other lands, swearing to obliterate not only the Dead God's magic users - but to put an end to magic altogether. ",
"SuggestedRoleLevels": {
"Carry": 2,
"Escape": 3
},
"Enabled": true,
"Side": "Radiant",
"Aliases": [
"am"
],
"AttackCapabilities": "DOTA_UNIT_CAP_MELEE_ATTACK",
"PrimaryAttribute": "DOTA_ATTRIBUTE_AGILITY",
"Initial": {
"Strength": 22,
"StrengthGain": 1.2,
"Agility": 22,
"AgilityGain": 2.8,
"Intelligence": 15,
"IntelligenceGain": 1.8,
"Health": 568,
"HealthRegen": 0.91,
"Mana": 195,
"ManaRegen": 0.61,
"Armor": 2.08,
"MagicResistance": 0.25,
"MinDamage": 49,
"MaxDamage": 53,
"AvgDamage": 51,
"IncreasedAttackSpeed": 22,
"BaseAttackTime": 1.45,
"AttackTime": 1.19,
"AttacksPerSecond": 0.84,
"AttackAnimationPoint": 0.3,
"AttackAcquisitionRange": 600,
"AttackRange": 128,
"VisionDayRange": 1800,
"VisionNightRange": 800,
"ProjectileSpeed": 0,
"MovementSpeed": 315,
"TurnRate": 0.5
},
"Abilities": [
{
"Title": "Mana Break",
"Url": "mana_break",
"HeroAbilityUrl": "antimage_mana_break",
"ID": 5003,
"Description": "Burns an opponent's mana on each attack. Mana Break deals 60% of the mana burned as damage to the target. Mana Break is a Unique Attack Modifier, and does not stack with other Unique Attack Modifiers.",
"Lore": "A modified technique of the Turstarkuri monks' peaceful ways is to turn magical energies on their owner.",
"Notes": [
"Mana Burn is blocked by spell immunity.",
"You can lifeleech the damage dealt by this skill with a Lifesteal aura."
],
"Type": "DOTA_ABILITY_TYPE_BASIC",
"Behavior": [
"DOTA_ABILITY_BEHAVIOR_PASSIVE"
],
"DamageType": "DAMAGE_TYPE_PHYSICAL",
"AbilitySpecial": [
{
"Name": "Damage Per_burn",
"Url": "damage_per_burn",
"Value": [
0.6
]
},
{
"Name": "Mana Burned Per Hit",
"Url": "mana_per_hit",
"ValueType": "FIXED",
"Value": [
28,
40,
52,
64
]
}
]
That is a slice from my json file, I wanna insert it inside a mongodb called dota2.
I was trying the command:
mongoimport --jsonArray -d dota2 -c docs --file heroes.json
the return told me:
2015-10-31T12:09:35.755-0200 connected to: localhost
2015-10-31T12:09:35.842-0200 imported 110 documents
But I won't get my data information, the command : db.dota.find()
Return empty...
Someone can help me? Yes...I am a newbie with mongodb. :/

As per your command you are inserting the data into the collection named docs present in database dota2.
So you should probably do like
use dota2;
db.docs.find();

Ok, I understand, my data are inside mongodb! But inside docs collections :) It works!

Related

Merge and Reformat JSON files using jq

I have two different JSON files returned with Python code for Twitter data. The first JSON file is in the form below:
A.json
{"tweet_id": "1212024242595926028", "username": "THPDPIO", "created_at": "2019-12-
31T14:54:32.000Z", "tweets": {"0": "Folks, it\u2019s simple!! You know what to do and
what not to do...don\u2019t drink and drive and you don\u2019t have to worry about
ANY consequences. Btw...jail is just a small part of it...think about the
possibility killing someone or killing yourself...again it\u2019s simple...
\ud83d\ude42 #stayhome"}}
{"tweet_id": "1212024242595926028", "username": "TheAliciaRanae", "created_at":
"2019-12-31T15:11:51.000Z", "tweets": {"1": "#THPDPIO Stay home and drink and pass
out and leave everyone else alone lol that\u2019s what I\u2019ll be doing lol HAPPY
NEW YEAR!"}}
{"tweet_id": "1212024242595926028", "username": "duane4343", "created_at": "2019-12-
31T15:21:37.000Z", "tweets": {"1": "#THPDPIO Happy New Year"}}
{"tweet_id": "1212024242595926028", "username": "HollyBr34731868", "created_at":
"2019-12-31T15:24:25.000Z", "tweets": {"1": "#THPDPIO Hope everyone has a safe
night."}}
{"tweet_id": "1211503874395254785", "username": "UNDPoliceDept", "created_at": "2019-
12-30T04:26:46.000Z", "tweets": {"0": "Typical North Dakotan.... #BestCopsAround
#NoTravelAdvised #StayHome"}}
{"tweet_id": "1211503874395254785", "username": "UNDPoliceDept", "created_at": "2019-
12-30T04:27:40.000Z", "tweets": {"1": "#NDHighwayPatrol"}}
{"tweet_id": "1211503874395254785", "username": "BorgenEthan", "created_at": "2019-
12-30T05:28:48.000Z", "tweets": {"1": "#UNDPoliceDept Nah i definitely look like the
first one"}}
With jq, i wrote some commands to choose the fields I want {NB: This was done in https://jqplay.org} with the command {tweet_id: .tweet_id, username: .username, reply: .tweets} | group_by(.tweet_id) but I get the error that
q: error (at :1): Cannot index string with string "tweet_id"
My desired output is to get the sample below for the first file:
{
"tweet_id": "1212024242595926028",
"username": "THPDPIO",
"reply": {
"0": "Folks, it’s simple!! You know what to do and what not to do...don’t drink and drive and you don’t have to worry about ANY consequences. Btw...jail is just a small part of it...think about the possibility killing someone or killing yourself...again it’s simple... 🙂 #stayhome",
"1": "#THPDPIO Stay home and drink and pass out and leave everyone else alone lol that’s what I’ll be doing lol HAPPY NEW YEAR!",
"1": "#THPDPIO Happy New Year",
"1": "#THPDPIO Hope everyone has a safe night."
}}
My issue: To have all replies linked to a particular tweet_id
For the 2nd file
B.json
{
"author_id": 80083199,
"tweet_id": 1212150612026151000,
"username": "CTVdavidspence",
"author_followers": 19572,
"author_tweets": 73406,
"author_description": "Retired broadcast Meteorologist. 2017 RTDNA Lifetime
Achievement Award. Best of Calgary 2018, 2019. AHS Patient and Family Advisor
(volunteer)",
"author_location": "Calgary",
"text": "The Trans Canada Highway near #Sicamous BC. #stayhome Image from
#DriveBC.",
"created_at": 1577834201000,
"retweets": 12,
"replies": 4,
"likes": 27,
"quote_count": 0}
{
"author_id": 848959032370921500,
"tweet_id": 1212024242595926000,
"username": "THPDPIO",
"author_followers": 4626,
"author_tweets": 2383,
"author_description": "Police Sgt.",
"author_location": "Terre Haute, IN",
"text": "Folks, it’s simple!! You know what to do and what not to do...don’t drink
and drive and you don’t have to worry about ANY consequences. Btw...jail is just a
small part of it...think about the possibility killing someone or killing
yourself...again it’s simple... 🙂\n#stayhome",
"created_at": 1577804072000,
"retweets": 11,
"replies": 9,
"likes": 84,
"quote_count": 1}
I will want to merge my final json file such that the tweet_id becomes the reference and the replies and username becomes the keys to merge them. So the final json file will be like the one below:
{
"author_id": 848959032370921500,
"tweet_id": 1212024242595926000,
"username": "THPDPIO",
"author_followers": 4626,
"author_tweets": 2383,
"author_description": "Police Sgt.",
"author_location": "Terre Haute, IN",
"text": "Folks, it’s simple!! You know what to do and what not to do...don’t drink
and drive and you don’t have to worry about ANY consequences. Btw...jail is just a
small part of it...think about the possibility killing someone or killing
yourself...again it’s simple... 🙂\n#stayhome",
"created_at": 1577804072000,
"retweets": 11,
"replies": 9,
"likes": 84,
"quote_count": 1,
"reply": {
"0": "Folks, it’s simple!! You know what to do and what not to do...don’t drink and
drive and you don’t have to worry about ANY consequences. Btw...jail is just a
small part of it...think about the possibility killing someone or killing
yourself...again it’s simple... 🙂 #stayhome",
"1": "#THPDPIO Stay home and drink and pass out and leave everyone else alone lol
that’s what I’ll be doing lol HAPPY NEW YEAR!",
"1": "#THPDPIO Happy New Year",
"1": "#THPDPIO Hope everyone has a safe night."
}}
I will appreciate any help in this regard. Thank you.
As for the second issue, you could employ JOIN on B.json and a custom index (eg. from the first issue) on A.json:
jq --slurpfile A A.json --slurpfile B B.json -n '
JOIN(
$A | reduce group_by(.tweet_id)[] as $g (
{}; .[$g[0].tweet_id].reply += ($g | map(.tweets))
);
$B[]; .tweet_id | #text; add
)
'
{
"author_id": 80083199,
"tweet_id": 1212150612026151000,
"username": "CTVdavidspence",
"author_followers": 19572,
"author_tweets": 73406,
"author_description": "Retired broadcast Meteorologist. 2017 RTDNA Lifetime Achievement Award. Best of Calgary 2018, 2019. AHS Patient and Family Advisor (volunteer)",
"author_location": "Calgary",
"text": "The Trans Canada Highway near #Sicamous BC. #stayhome Image from #DriveBC.",
"created_at": 1577834201000,
"retweets": 12,
"replies": 4,
"likes": 27,
"quote_count": 0
}
{
"author_id": 848959032370921500,
"tweet_id": 1212024242595926000,
"username": "THPDPIO",
"author_followers": 4626,
"author_tweets": 2383,
"author_description": "Police Sgt.",
"author_location": "Terre Haute, IN",
"text": "Folks, it’s simple!! You know what to do and what not to do...don’t drink and drive and you don’t have to worry about ANY consequences. Btw...jail is just a small part of it...think about the possibility killing someone or killing yourself...again it’s simple... 🙂\n#stayhome",
"created_at": 1577804072000,
"retweets": 11,
"replies": 9,
"likes": 84,
"quote_count": 1,
"reply": [
{
"0": "Folks, it’s simple!! You know what to do and what not to do...don’t drink and drive and you don’t have to worry about ANY consequences. Btw...jail is just a small part of it...think about the possibility killing someone or killing yourself...again it’s simple... 🙂 #stayhome"
},
{
"1": "#THPDPIO Stay home and drink and pass out and leave everyone else alone lol that’s what I’ll be doing lol HAPPY NEW YEAR!"
},
{
"1": "#THPDPIO Happy New Year"
},
{
"1": "#THPDPIO Hope everyone has a safe night."
}
]
}
Demo
This response deals with the first issue:
To have all replies linked to a particular tweet_id
As pointed out by #pmf, your sample output won't achieve the goal. Also, group_by expects an array as input. Consider therefore:
< A.json jq -n '
def accumulate_by(stream; f):
reduce stream as $x ({}; (f + [$x|f]) as $v | . + $x | f = $v );
[inputs | {tweet_id, username, reply: .tweets}] | group_by(.tweet_id)
| map( accumulate_by(.[]; .reply ))
'
Note that, by design, this ignores "collisions" in the value of .username; you may need to consider this further. Anyway, with your sample, the result would be:
[
{
"tweet_id": "1211503874395254785",
"username": "BorgenEthan",
"reply": [
{
"0": "Typical North Dakotan.... #BestCopsAround #NoTravelAdvised #StayHome"
},
{
"1": "#NDHighwayPatrol"
},
{
"1": "#UNDPoliceDept Nah i definitely look like the first one"
}
]
},
{
"tweet_id": "1212024242595926028",
"username": "HollyBr34731868",
"reply": [
{
"0": "Folks, it’s simple!! You know what to do and what not to do...don’t drink and drive and you don’t have to worry about ANY consequences. Btw...jail is just a small part of it...think about the possibility killing someone or killing yourself...again it’s simple... 🙂 #stayhome"
},
{
"1": "#THPDPIO Stay home and drink and pass out and leave everyone else alone lol that’s what I’ll be doing lol HAPPY NEW YEAR!"
},
{
"1": "#THPDPIO Happy New Year"
},
{
"1": "#THPDPIO Hope everyone has a safe night."
}
]
}
]

json.load loads a string instead of json

I have a list of dictionaries written to a data.txt file. I was expecting to be able to read the list of dictionaries in a normal way when I load, but instead, I seem to load up a string.
For example - when I print(data[0]), I was expecting the first dictionary in the list, but instead, I got "[" instead.
Below attached is my codes and txt file:
read_json.py
import json
with open('./data.txt', 'r') as json_file:
data = json.load(json_file)
print(data[0])
data.txt
"[
{
"name": "Disney's Mulan (Mandarin) PG13 *",
"cast": [
"Jet Li",
"Donnie Yen",
"Yifei Liu"
],
"genre": [
"Action",
"Adventure",
"Drama"
],
"language": "Mandarin with no subtitles",
"rating": "PG13 - Some Violence",
"runtime": "115",
"open_date": "18 Sep 2020",
"description": "\u201cMulan\u201d is the epic adventure of a fearless young woman who masquerades as a man in order to fight Northern Invaders attacking China. The eldest daughter of an honored warrior, Hua Mulan is spirited, determined and quick on her feet. When the Emperor issues a decree that one man per family must serve in the Imperial Army, she steps in to take the place of her ailing father as Hua Jun, becoming one of China\u2019s greatest warriors ever."
},
{
"name": "The New Mutants M18",
"cast": [
"Maisie Williams",
"Henry Zaga",
"Anya Taylor-Joy",
"Charlie Heaton",
"Alice Braga",
"Blu Hunt"
],
"genre": [
"Action",
"Sci-Fi"
],
"language": "English",
"rating": "M18 - Some Mature Content",
"runtime": "94",
"open_date": "27 Aug 2020",
"description": "Five young mutants, just discovering their abilities while held in a secret facility against their will, fight to escape their past sins and save themselves."
}
]"
The above list is formatted properly for easy reading but the actual file is a single line and the different lines are denoted with "\n". Thanks for any help.
remove double quote in data.txt is useful for me。
eg. modify
"[{...},{...}]"
to
[{...},{...}]
Hope it helps!

Deserializing Nested JSON API response with Django

I'm pretty new to the DRF and serializing/deserializing. I'm slowly building a dashboard for my business during the corona virus and learning to code. I am in a little deep, but after spending more than $10k on developers on upwork and not really get much result, I figured, what do I have to lose?
Our software provider has a full API for our needs https://developer.myvr.com/api/, but absolutely no dashboard to report statistics about our clients reservation data.
The end result will be a synchronization of some of the data from their API to my database which will be hosted through AWS. I chose to do it this way due to having to do some post processing of data from the API. For example, we need to calculate occupancy rates(which is not an endpoint), expenses from our accounting connection and a few other small calculations in which the data is not already in the provided API. I originally wanted to use the data from the API solely, but I'm hesitant due to the reasons above.
That's the backstory, here are the questions:
The API response is extremely complex and nested multiple times, what is the best practise to extract a replication of the structure of the data to my own Database? Would I have to create models for each field manually?
Example response:
```{
"uri": "https://api.myvr.com/v1/properties/b6b0f2fe278f612b/",
"id": "b6b0f2fe278f612b",
"key": "b6b0f2fe278f612b",
"accessDescription": null,
"accommodates": 11,
"active": false,
"addressOne": "11496 Zermatt Dr",
"addressTwo": null,
"allowTurns": true,
"amenities": "https://api.myvr.com/v1/property-amenities/?propertyId=b6b0f2fe278f612b",
"automaticallyApprove": false,
"baseNightlyRate": "395.00",
"baseRate": {
"uri": "https://api.myvr.com/v1/rates/660c299d4785c32e/",
"id": "660c299d4785c32e",
"key": "660c299d4785c32e",
"externalId": null,
"baseRate": true,
"changeoverDay": null,
"created": "2019-01-19T08:02:36Z",
"currency": "USD",
"endDate": "2020-01-18",
"minStay": 3,
"modified": "2019-01-19T08:02:36Z",
"monthly": 0,
"name": "Base Rate",
"weekNight": 39500,
"nightly": 39500,
"position": 0,
"property": {
"name": "API Demo Property",
"uri": "https://api.myvr.com/v1/properties/b6b0f2fe278f612b/",
"id": "b6b0f2fe278f612b",
"externalId": null,
"key": "b6b0f2fe278f612b",
"slug": "api-demo-property"
},
"ratePlan": {
"uri": "https://api.myvr.com/v1/rate-plans/862caa3f5267602d/",
"key": "862caa3f5267602d",
"name": "Default Rates for Property"
},
"repeat": true,
"startDate": "2020-01-18",
"weekend": 0,
"weekendNight": 0,
"weekly": 250000
},
"bathrooms": "4.0",
"bedrooms": 4,
"bookingUrl": "https://myvr.com/reservation/redirect/booking/b6b0f2fe278f612b/",
"checkInTime": "16:00:00",
"checkOutTime": "10:00:00",
"city": "Truckee",
"commissionStructure": null,
"countryCode": "US",
"created": "2016-01-19T00:01:48Z",
"currency": "USD",
"customFields": {},
"description": "Luxurious living, scenic mountain setting, entertainment galore. Located on a quiet street in Tahoe Donner, our well equipped modern home is nestled into the wilderness. A babbling creek greets visitors approaching the front step as it collects into a small pond with a cascading waterfall. <br/>\n<br/>\nInside, over 3,000 sqft of luxurious living space divides itself between two floors. On the first floor, a beautiful kitchen with granite counters, gas stove and stainless steel appliances opens to a large great room centered around a wood burning fireplace and featuring 30' soaring ceilings. A spacious loft overlooks the great room, showcasing a large poker/card table. Upstairs features a large entertainment room, complete with wet bar, shuffleboard table, and state-of-the-art television setup with surround sound. The scenic backyard is accessible from a large deck featuring a new hot tub with seating for 7.",
"externalId": null,
"feePlan": {
"uri": "https://api.myvr.com/v1/fee-plans/4d1c44383755051b/",
"key": "4d1c44383755051b",
"name": "Default Fees for Listing"
},
"headline": "Beautiful Four Bedroom Lake Front Property",
"houseRules": null,
"instantBookingsEnabled": false,
"lat": "39.3422523000",
"level": "unit",
"localAreaDescription": "Tahoe Donner is a year round activity resort. The amenities include private beach/boat launching facilities, pools, recreation center, tennis, horseback riding, golf, downhill skiing as well as cross country skiing. Truckee is a historical mining town-having a western feel but also has museums, theaters, fine dining plus 2 large supermarkets-all less than 3 miles from the house. Our home is also located within a 15 minute drive to 4 major ski resorts. Downtown Reno is a short 40 minute drive away for those seeking a night on the town or the thrill of a Nevada casino.",
"lon": "-120.2271947000",
"lowestNightlyRate": "395.00",
"manual": "",
"modified": "2019-10-18T17:18:43Z",
"name": "API Demo Property",
"owner": null,
"postalCode": "96161",
"ratePlan": {
"uri": "https://api.myvr.com/v1/rate-plans/862caa3f5267602d/",
"key": "862caa3f5267602d",
"name": "Default Rates for Property"
},
"ratePlanLocked": false,
"region": "CA",
"shortCode": "API",
"size": 3000,
"slug": "api-demo-property",
"suitableElderly": "yes",
"suitableEvents": "unknown",
"suitableGroups": "yes",
"suitableHandicap": "no",
"suitableInfants": "unknown",
"suitableKids": "yes",
"suitablePets": "no",
"suitableSmoking": "no",
"transitDescription": null,
"type": "house",
"weekendNights": [
5,
6
]
}```
I think the best way to populate the database would be to run a custom management command to run a once off script, I've done this previously with another database, however I'm still stuck as I don't really want to write these models manually. Also a concern is if a field is missing or the structure changes.
This project is definitely above my skills and extremely ambitious, but I would appreciate any feedback or advice anyone might have.
Thanks,
Darren
So I didn't really get any interest in this question, but I ended up working it out myself.
I hope someone googles it and might find it helpful.
import requests
from rest_framework.response import Response
from django.core.management.base import BaseCommand, CommandError
from reservation.models import Reservation
import time
MYVR_URL = 'https://api.myvr.com/'
MYVR_RESERVATION = 'v1/reservations/?limit=100'
headers = {
'Authorization': 'Basic SOmeAPiCodeHeRe123=',
}
class Command(BaseCommand):
help = 'Imports new properties and saves the objects in the database'
def handle(self, *args, **options):
url = MYVR_URL + MYVR_RESERVATION
print("Populating Reservations")
def looping_api(url, headers):
while url:
r = requests.request("GET", url, headers=headers)
url = r.json().get('next')
props_data = r.json().get('results')
start_time = time.time()
for prop in props_data:
try:
created = Reservation.objects.update_or_create(
myvr_key=prop.get('key'),
adults=prop.get('adults'),
children=prop.get('children'),
checkin=prop.get('checkIn'),
checkout=prop.get('checkOut'),
checkinTime=prop.get('checkInTime'),
checkoutTime=prop.get('checkOutTime'),
guestFirstName=prop.get('firstName'),
dateCreated=prop.get('created'),
dateBooked=prop.get('dateBooked'),
dateCancelled=prop.get('dateCanceled'),
contact=prop.get('contact').get('name'),
contact_key=prop.get('contact').get('key'),
guest_type=prop.get('guestType'),
property_name=prop.get('property').get('name'),
property_key=prop.get('property').get('key'),
source_code=prop.get('source').get('code'),
source_name=prop.get('source').get('name'),
total_due=prop.get('quote').get('totalDue'),
total_refundables=prop.get(
'quote').get('totalRefundableFees'),
total_nonrefundables=prop.get(
'quote').get('totalNonrefundableFees'),
reference_id=prop.get('referenceId'),
)
print(
f"Added obj {prop.get('key')}")
except AttributeError as error:
print(f"{error} attribute is null or owner booking")
url = r.json().get('next')
print(r.json().get('next'))
print(len(props_data))
end_time = time.time()
duration = (end_time - start_time)
print(duration)
looping_api(url, headers)

Wikipedia API cloud of confusion: `list` vs `generator` vs `search`

My goal is to write an API search of Wikipedia that will:
Return pages only in the category "English language films"
Of those, return pages only beginning with the letters "Avatar" (or anything, really, just using those letters for an example)
Of those, give me the url, the title, the date, a summary, and the main page image.
So far I've tried three things, none of which seem to be able to do exactly what I want.
PROBLEM: list=allpages is just generally useless
Here's a search with list=allpages:
https://en.wikipedia.org/w/api.php?action=query&format=json&prop=info%7Cpageimages%7Cextracts&list=allpages&inprop=url%7Cdisplaytitle&piprop=name%7Cthumbnail%7Coriginal&pithumbsize=100&exintro=1&explaintext=1&apprefix=Avatar&aplimit=3
Here's the result (just the first 3 pages):
{
"batchcomplete": "",
"continue": {
"apcontinue": "Avatar,_The_Last_Airbender",
"continue": "-||info|pageimages|extracts"
},
"query": {
"allpages": [
{
"pageid": 100368,
"ns": 0,
"title": "Avatar"
},
{
"pageid": 4846971,
"ns": 0,
"title": "Avatar's Abode"
},
{
"pageid": 35243953,
"ns": 0,
"title": "Avatar, Iran"
}
]
}
}
As you can see, it basically ignored all my prop requests.
PROBLEM: generator=allpages can't search by category
Here's a search using generator-allpages:
https://en.wikipedia.org/w/api.php?action=query&format=json&prop=info%7Cpageimages%7Cextracts&generator=allpages&inprop=url%7Cdisplaytitle&piprop=name%7Cthumbnail%7Coriginal&pithumbsize=100&exintro=1&explaintext=1&gapprefix=Avatar
Here's the output from that (just the first result):
{
"batchcomplete": "",
"continue": {
"gapcontinue": "Avatar's_Abode",
"continue": "gapcontinue||"
},
"query": {
"pages": {
"100368": {
"pageid": 100368,
"ns": 0,
"title": "Avatar",
"contentmodel": "wikitext",
"pagelanguage": "en",
"pagelanguagehtmlcode": "en",
"pagelanguagedir": "ltr",
"touched": "2018-05-03T11:21:07Z",
"lastrevid": 838959509,
"length": 45784,
"fullurl": "https://en.wikipedia.org/wiki/Avatar",
"editurl": "https://en.wikipedia.org/w/index.php?title=Avatar&action=edit",
"canonicalurl": "https://en.wikipedia.org/wiki/Avatar",
"displaytitle": "Avatar",
"thumbnail": {
"source": "https://upload.wikimedia.org/wikipedia/commons/thumb/a/a0/Avatars.jpg/80px-Avatars.jpg",
"width": 80,
"height": 100
},
"original": {
"source": "https://upload.wikimedia.org/wikipedia/commons/a/a0/Avatars.jpg",
"width": 357,
"height": 448
},
"pageimage": "Avatars.jpg",
"extract": "An avatar (Sanskrit: \u0905\u0935\u0924\u093e\u0930, IAST: avat\u0101ra), a concept in Hinduism that means \"descent\", refers to the material appearance or incarnation of a deity on earth. The relative verb to \"alight, to make one's appearance\" is sometimes used to refer to any guru or revered human being.\nThe word avatar does not appear in the Vedic literature, but appears in verb forms in post-Vedic literature, and as a noun particularly in the Puranic literature after the 6th century CE. Despite that, the concept of an avatar is compatible with the content of the Vedic literature like the Upanishads as it is symbolic imagery of the Saguna Brahman concept in the philosophy of Hinduism. The Rigveda describes Indra as endowed with a mysterious power of assuming any form at will. The Bhagavad Gita expounds the doctrine of Avatara but with terms other than avatar.\nTheologically, the term is most often associated with the Hindu god Vishnu, though the idea has been applied to other deities. Varying lists of avatars of Vishnu appear in Hindu scriptures, including the ten Dashavatara of the Garuda Purana and the twenty-two avatars in the Bhagavata Purana, though the latter adds that the incarnations of Vishnu are innumerable. The avatars of Vishnu are important in Vaishnavism theology. In the goddess-based Shaktism tradition of Hinduism, avatars of the Devi in different appearances such as Tripura Sundari, Durga and Kali are commonly found. While avatars of other deities such as Ganesha and Shiva are also mentioned in medieval Hindu texts, this is minor and occasional. The incarnation doctrine is one of the important differences between Vaishnavism and Shaivism traditions of Hinduism.\nIncarnation concepts similar to avatar are also found in Buddhism, Christianity and others. The scriptures of Sikhism include the names of numerous Hindu gods and goddesses, but it rejected the doctrine of savior incarnation and endorsed the view of Hindu Bhakti movement saints such as Namdev that formless eternal god is within the human heart and man is his own savior."
}
}
}
}
...and it's so close to perfect that I can't believe it... the only problem is there's no way to restrict the search to the category "English-language films".
PROBLEM: generator=categorymember won't show page images and doesn't filter by prefix, only sets start point by prefix
Here's a search using generator=categorymember:
https://en.wikipedia.org/w/api.php?action=query&format=json&prop=pageimages&generator=categorymembers&piprop=name%7Cthumbnail%7Coriginal&pithumbsize=100&gcmtitle=Category%3AEnglish-language%20films&gcmprop=&gcmtype=page&gcmlimit=10&gcmstartsortkeyprefix=Avatar
Here's the first ten results (I've left the prop=extract and prop=info parameters off--they're working fine--so you can see just the relevant detail):
{
"batchcomplete": "",
"continue": {
"gcmcontinue": "page|2953314335314b4d04354b394141011201dcbedc08|47013432",
"continue": "gcmcontinue||"
},
"query": {
"pages": {
"4273140": {
"pageid": 4273140,
"ns": 0,
"title": "Avatar (2009 film)"
},
"15945267": {
"pageid": 15945267,
"ns": 0,
"title": "Avatar (2004 film)"
},
"25813358": {
"pageid": 25813358,
"ns": 0,
"title": "Avatar 2"
},
"27442998": {
"pageid": 27442998,
"ns": 0,
"title": "Avatar 3"
},
"50071841": {
"pageid": 50071841,
"ns": 0,
"title": "Ave Maria (1918 film)"
},
"41748079": {
"pageid": 41748079,
"ns": 0,
"title": "Avenged (2013 U.S. film)"
},
"42739169": {
"pageid": 42739169,
"ns": 0,
"title": "Avenger (film)"
},
"50726142": {
"pageid": 50726142,
"ns": 0,
"title": "The Avenger (1931 film)"
},
"43707905": {
"pageid": 43707905,
"ns": 0,
"title": "The Avengers (1950 film)"
},
"22114132": {
"pageid": 22114132,
"ns": 0,
"title": "The Avengers (2012 film)"
}
}
}
}
And as you can see:
The pageimages data is nowhere to be seen
The setting gcmstartsortkeyprefix=Avatar only sets the point from which the listing starts, it doesn't actually filter out results that don't have that prefix. There's no analogue to the gapprefix parameter that's available when using generator-allpages.
PROBLEM: list=search won't show any prop values and searches by content as well as title
Here's a search using list=search, an avenue of approach I'm grateful to this page for:
https://en.wikipedia.org/w/api.php?action=query&format=jsonfm&prop=info%7Cpageimages%7Cextracts&list=search&inprop=url%7Cdisplaytitle&piprop=name%7Cthumbnail%7Coriginal&pithumbsize=100&exintro=1&explaintext=1&srsearch=Avatar+incategory:English-language_films&srlimit=3
And here's the return (just the first 3 results):
{
"batchcomplete": "",
"continue": {
"sroffset": 3,
"continue": "-||info|pageimages|extracts"
},
"query": {
"searchinfo": {
"totalhits": 224
},
"search": [
{
"ns": 0,
"title": "Avatar (2009 film)",
"pageid": 4273140,
"size": 201954,
"wordcount": 18643,
"snippet": "<span class=\"searchmatch\">Avatar</span>, marketed as James Cameron's <span class=\"searchmatch\">Avatar</span>, is a 2009 American epic science fiction film directed, written, produced, and co-edited by James Cameron, and",
"timestamp": "2018-05-01T01:52:00Z"
},
{
"ns": 0,
"title": "Avatar 2",
"pageid": 25813358,
"size": 55754,
"wordcount": 5380,
"snippet": "<span class=\"searchmatch\">Avatar</span> 2 is an upcoming American epic science fiction film directed, produced, and co-written by James Cameron, and the first of four planned sequels to",
"timestamp": "2018-05-02T10:02:34Z"
},
{
"ns": 0,
"title": "Avatar 3",
"pageid": 27442998,
"size": 17747,
"wordcount": 1374,
"snippet": "<span class=\"searchmatch\">Avatar</span> 3 is an upcoming 2021 American epic science fiction film directed, produced, co-written, and co-edited by James Cameron, scheduled to be released",
"timestamp": "2018-05-02T10:02:45Z"
}
]
}
}
Now this search does define prop=info and prop=images and prop=extracts, but they're nowhere to be seen. Plus there's also no analogue to gapprefix with this approach.
PROBLEM: generator=search won't show pageimages and searches by content as well as title
Here's the same search as above, but using generator=search:
https://en.wikipedia.org/w/api.php?action=query&format=jsonfm&prop=info%7Cpageimages%7Cextracts&generator=search&inprop=url%7Cdisplaytitle&piprop=name%7Cthumbnail%7Coriginal&pithumbsize=100&exintro=1&explaintext=1&gsrsearch=Avatar+incategory:English-language_films&gsrlimit=3
And here's the result:
{
"batchcomplete": "",
"continue": {
"gsroffset": 3,
"continue": "gsroffset||"
},
"query": {
"pages": {
"4273140": {
"pageid": 4273140,
"ns": 0,
"title": "Avatar (2009 film)",
"index": 1,
"contentmodel": "wikitext",
"pagelanguage": "en",
"pagelanguagehtmlcode": "en",
"pagelanguagedir": "ltr",
"touched": "2018-05-01T01:52:00Z",
"lastrevid": 839068297,
"length": 201954,
"fullurl": "https://en.wikipedia.org/wiki/Avatar_(2009_film)",
"editurl": "https://en.wikipedia.org/w/index.php?title=Avatar_(2009_film)&action=edit",
"canonicalurl": "https://en.wikipedia.org/wiki/Avatar_(2009_film)",
"displaytitle": "<i>Avatar</i> (2009 film)",
"extract": "Avatar, marketed as James Cameron's Avatar, is a 2009 American epic science fiction film directed, written, produced, and co-edited by James Cameron, and starring Sam Worthington, Zoe Saldana, Stephen Lang, Michelle Rodriguez, and Sigourney Weaver. The film is set in the mid-22nd century, when humans are colonizing Pandora, a lush habitable moon of a gas giant in the Alpha Centauri star system, in order to mine the mineral unobtanium, a room-temperature superconductor. The expansion of the mining colony threatens the continued existence of a local tribe of Na'vi \u2013 a humanoid species indigenous to Pandora. The film's title refers to a genetically engineered Na'vi body with the mind of a remotely located human that is used to interact with the natives of Pandora.\nDevelopment of Avatar began in 1994, when Cameron wrote an 80-page treatment for the film. Filming was supposed to take place after the completion of Cameron's 1997 film Titanic, for a planned release in 1999, but, according to Cameron, the necessary technology was not yet available to achieve his vision of the film. Work on the language of the film's extraterrestrial beings began in 2005, and Cameron began developing the screenplay and fictional universe in early 2006. Avatar was officially budgeted at $237 million. Other estimates put the cost between $280 million and $310 million for production and at $150 million for promotion. The film made extensive use of new motion capture filming techniques, and was released for traditional viewing, 3D viewing (using the RealD 3D, Dolby 3D, XpanD 3D, and IMAX 3D formats), and for \"4D\" experiences in select South Korean theaters. The stereoscopic filmmaking was touted as a breakthrough in cinematic technology.\nAvatar premiered in London on December 10, 2009, and was internationally released on December 16 and in the United States and Canada on December 18, to positive critical reviews, with critics highly praising its groundbreaking visual effects. During its theatrical run, the film broke several box office records and became the highest-grossing film of all time, as well as in the United States and Canada, surpassing Cameron's Titanic, which had held those records for twelve years. It also became the first film to gross more than $2 billion and the best-selling film of 2010 in the United States. Avatar was nominated for nine Academy Awards, including Best Picture and Best Director, and won three, for Best Art Direction, Best Cinematography and Best Visual Effects. Following the film's success, Cameron signed with 20th Century Fox to produce four sequels: Avatar 2 and Avatar 3 are currently filming, and will be released on December 18, 2020, and December 17, 2021 respectively; subsequent sequels will start shooting as soon as they wrap filming, and will be released in 2024 and 2025. Several cast members are expected to return, including Worthington, Saldana, Lang, and Weaver."
},
"25813358": {
"pageid": 25813358,
"ns": 0,
"title": "Avatar 2",
"index": 2,
"contentmodel": "wikitext",
"pagelanguage": "en",
"pagelanguagehtmlcode": "en",
"pagelanguagedir": "ltr",
"touched": "2018-05-02T10:02:34Z",
"lastrevid": 839266311,
"length": 55754,
"fullurl": "https://en.wikipedia.org/wiki/Avatar_2",
"editurl": "https://en.wikipedia.org/w/index.php?title=Avatar_2&action=edit",
"canonicalurl": "https://en.wikipedia.org/wiki/Avatar_2",
"displaytitle": "<i>Avatar 2</i>",
"extract": "Avatar 2 is an upcoming American epic science fiction film directed, produced, and co-written by James Cameron, and the first of four planned sequels to his film Avatar (2009). Cameron is producing the film with Jon Landau, with Josh Friedman originally announced as his co-writer; it was later announced that Cameron, Friedman, Rick Jaffa, Amanda Silver, and Shane Salerno took a part in the writing process of all sequels before being attributed separate scripts, making the eventual writing credits unclear. Cast members Sam Worthington, Zoe Saldana, Stephen Lang, Sigourney Weaver, Giovanni Ribisi, Joel David Moore, Dileep Rao, C. C. H. Pounder, and Matt Gerald are all expected to return.\nCameron, who had stated in 2006 that he would like to make sequels to Avatar if it were successful, announced the first two in 2010 following the widespread success of the first film, with Avatar 2 aiming for a 2014 release. However, the subsequent addition of two more sequels, and the necessity to develop new technology in order to film performance capture scenes underwater, a feature never accomplished before in motion capture history, led to significant delays to allow the crew more time to work on the writing, pre-production, and visual effects; it is currently planned for a release on December 18, 2020, exactly eleven years after the American release of the first film, with the following sequels to be released between 2021 and 2025.\nPreliminary shooting for the film started in Manhattan Beach, California on August 15, 2017, followed by principal photography simultaneously with Avatar 3 in New Zealand on September 25, 2017. The other sequels are expected to start shooting as soon as Avatar 2 and 3's filming wraps."
},
"27442998": {
"pageid": 27442998,
"ns": 0,
"title": "Avatar 3",
"index": 3,
"contentmodel": "wikitext",
"pagelanguage": "en",
"pagelanguagehtmlcode": "en",
"pagelanguagedir": "ltr",
"touched": "2018-05-02T10:02:45Z",
"lastrevid": 839266333,
"length": 17747,
"fullurl": "https://en.wikipedia.org/wiki/Avatar_3",
"editurl": "https://en.wikipedia.org/w/index.php?title=Avatar_3&action=edit",
"canonicalurl": "https://en.wikipedia.org/wiki/Avatar_3",
"displaytitle": "<i>Avatar 3</i>",
"extract": "Avatar 3 is an upcoming 2021 American epic science fiction film directed, produced, co-written, and co-edited by James Cameron, scheduled to be released on December 17, 2021. It is the second of four planned sequels to his film Avatar (2009), and will be a follow-up to Avatar 2 (2020). Cameron is producing the film with Jon Landau, with Rick Jaffa and Amanda Silver originally announced as his co-writers; it was later announced that Cameron, Jaffa, Silver, Josh Friedman and Shane Salerno took a part in the writing process of all of the sequels before being assigned to finish the separate scripts, making the eventual writing credits unclear. Cast members Sam Worthington, Zoe Saldana, Stephen Lang, Sigourney Weaver, Joel David Moore, CCH Pounder and Matt Gerald are all expected to return from the first two films.\nAvatar 3 started shooting simultaneously with Avatar 2 on August 15, 2017. Two additional sequels will start shooting as soon as the first two wrap post-production, and are expected to be released in 2024 and 2025 respectively."
}
}
}
}
...and again this one is so close to perfect, because in this version it does return the prop=info and prop=extracts results, but again it ignores prop=pageimages and I can't find any way to restrict the search to the starting letters of the titles.
Conclusion: SNAFU
...Is there any 'one search to rule them all' here? It's so tantalizing that I can almost, almost, almost get everything with one query, but in the end I can't figure out how to get 'em all. Can anyone help me through the thicket?
On a high level, list modules generate some list of pages (e.g. the pages most recently edited). Sometimes they include some additional information but mostly they are just meant to give you a list of pages. Prop modules operate on a list of pages and add some kind of extra information to each; the list can be determined by the client (via parameters like titles or pageids) or a list module (used as generator= instead of list=). You can use list= and prop= "together" but all that does is create two separate lists (one of which will be empty and not shown because there is no titles or generator parameter).
Your generator queries are fine; you'll need pilicense=any if you want non-free images in the output. Maybe you found some outdated documentation that does not mention that?
You can use intitle:... in the search term for title search (see docs).

Retrieving a specific object from JSON

This is a snippet of a JSON file that was returned by TMDB and I'm trying to access the title of every object. I've tried using the following methods like from this post How to access specific value from a nested array within an object array.
"results": [
{
"vote_count": 2358,
"id": 283366,
"video": false,
"vote_average": 6.5,
"title": "Miss Peregrine's Home for Peculiar Children",
"popularity": 20.662756,
"poster_path": "/AvekzUdI8HZnImdQulmTTmAZXrC.jpg",
"original_language": "en",
"original_title": "Miss Peregrine's Home for Peculiar Children",
"genre_ids": [
18,
14,
12
],
"backdrop_path": "/9BVHn78oQcFCRd4M3u3NT7OrhTk.jpg",
"adult": false,
"overview": "A teenager finds himself transported to an island where he must help protect a group of orphans with special powers from creatures intent on destroying them.",
"release_date": "2016-09-28"
},
{
"vote_count": 3073,
"id": 381288,
"video": false,
"vote_average": 6.8,
"title": "Split",
"popularity": 17.488396,
"poster_path": "/rXMWOZiCt6eMX22jWuTOSdQ98bY.jpg",
"original_language": "en",
"original_title": "Split",
"genre_ids": [
27,
53
],
"backdrop_path": "/4G6FNNLSIVrwSRZyFs91hQ3lZtD.jpg",
"adult": false,
"overview": "Though Kevin has evidenced 23 personalities to his trusted psychiatrist, Dr. Fletcher, there remains one still submerged who is set to materialize and dominate all the others. Compelled to abduct three teenage girls led by the willful, observant Casey, Kevin reaches a war for survival among all of those contained within him — as well as everyone around him — as the walls between his compartments shatter apart.",
"release_date": "2016-11-15"
},
var titles = results.map(function extract(item){return item.title})
The map function iterates through the array and builds the resulting array by applying the extract function on each item.
for (var i =0; i < obj.results.length; i++) {
console.log(obj.results[i].title);
}
First, we get the results key, and then, iterate over it since it is an array.
Results is nothing more than an array of objects, there's nothing nested.
You could use .forEach()
results.forEach(function(item){
console.log(item.title);
});