I retrieved a dataset from a news API in JSON format. I want to extract the news description from the JSON data.
This is my code:-
import requests
import json
url = ('http://newsapi.org/v2/top-headlines?'
'country=us&'
'apiKey=608bf565c67f4d99994c08d74db82f54')
response = requests.get(url)
di=response.json()
di = json.dumps(di)
for di['articles'] in di:
print(article['title'])
The dataset looks like this:-
{'status': 'ok',
'totalResults': 38,
'articles': [
{'source':
{'id': 'the-washington-post',
'name': 'The Washington Post'},
'author': 'Derek Hawkins, Marisa Iati',
'title': 'Coronavirus updates: Texas, Florida and Arizona officials say early reopenings fueled an explosion of cases - The Washington Post',
'description': 'Local officials in states with surging coronavirus cases issued dire warnings Sunday about the spread of infections, saying the virus was rapidly outpacing containment efforts.',
'url': 'https://www.washingtonpost.com/nation/2020/07/05/coronavirus-update-us/',
'urlToImage': 'https://www.washingtonpost.com/wp-apps/imrs.php?src=https://arc-anglerfish-washpost-prod-washpost.s3.amazonaws.com/public/K3UMAKF6OMI6VF6BNTYRN77CNQ.jpg&w=1440',
'publishedAt': '2020-07-05T18:32:44Z',
'content': 'Here are some significant developments:\r\n<ul><li>The rolling seven-day average for daily new cases in the United States reached a record high for the 27th day in a row, climbing to 48,606 on Sunday, … [+5333 chars]'}])
Please guide me with this!
There are few corrections needed in your code.. below code should work and i have removed API KEY in answer make sure that you add one before testing
import requests
import json
url = ('http://newsapi.org/v2/top-headlines?'
'country=us&'
'apiKey=<API KEY>')
di=response.json()
#You don't need to dump json that is already in json format
#di = json.dumps(di)
#your loop is not correctly defined, below is correct way to do it
for article in di['articles']:
print(article['title'])
response.json
{'status': 'ok',
'totalResults': 38,
'articles': [
{'source':
{'id': 'the-washington-post',
'name': 'The Washington Post'},
'author': 'Derek Hawkins, Marisa Iati',
'title': 'Coronavirus updates: Texas, Florida and Arizona officials say early reopenings fueled an explosion of cases - The Washington Post',
'description': 'Local officials in states with surging coronavirus cases issued dire warnings Sunday about the spread of infections, saying the virus was rapidly outpacing containment efforts.',
'url': 'https://www.washingtonpost.com/nation/2020/07/05/coronavirus-update-us/',
'urlToImage': 'https://www.washingtonpost.com/wp-apps/imrs.php?src=https://arc-anglerfish-washpost-prod-washpost.s3.amazonaws.com/public/K3UMAKF6OMI6VF6BNTYRN77CNQ.jpg&w=1440',
'publishedAt': '2020-07-05T18:32:44Z',
'content': 'Here are some significant developments:\r\n<ul><li>The rolling seven-day average for daily new cases in the United States reached a record high for the 27th day in a row, climbing to 48,606 on Sunday, … [+5333 chars]'}]}
Code:
di = response.json() # Understand that 'di' is of type 'dictionary', key-value pair
for i in di["articles"]:
print(i["description"])
"articles" is one of the keys of dictionary di, It's corresponding value is of type list. "description" , which you are looking is part of this list (value of "articles"). Further list contains the dictionary (key-value pair).You can access from key - description
I am trying to remove the lines that contain [CLS] and [SEP] in the following json file. Is there any way to do this in python? How to remove these lines with the given text?
"Tirukkollampudur Vilvavaneswarar -. Temple - Shivastalam.txt": {
"context": " may be reproduced or used in any form without permission. This Shivastalam is located 5 km south of Kodavasal and Koradacheri on the Tiruvarur Thanjavur railroad. Koovilamputhur the original name became Kollampudur. Koovilam stands for Vilvam, hence Vilvavanam. This shrine is regarded as a Muktistalam. This shrine is regarded as the 113rd in the series of Tevara Stalams in the Chola Region south of the river Kaveri. Legends The Vilva trees are said to represent splashes of the celestial nectar Amritam, and this stalam is considered on par with Banares. Sundarar is believed to have floated across the river to this temple in a boatmanless raft in a river in spate singing a Patikam . This event is celebrated in a festival in the monsoon month of Libra. The Avimukteswarar temple nearby is also associated with this legend as is the Shivastalam at Kodavasal. Shiva is said to have blessed Durvasa Muni with a vision of the Cosmic Dance here. Legend also has it that Arjuna worshipped Shiva at this shrine. The Temple There are several inscriptions here, and the Cholas have made immense contributions here.to this temple which was built during the time of Kulottunga Chola I. This temple occupies an area of over 2 acres, and its second prakaram has a 5 tiered rajagopuram. The Vinayakar in this temple is also of great Festivals Six worship services are offered each day. Kartikai Deepam, Arudra Darisanam, Sivaratri, Skanda Sashti are some of the festivals celebrated here. ",
"answers": [
[
"5 km south of kodavasal and koradacheri on the tiruvarur thanjavur railroad"
],
[
"5 km south of kodavasal and koradacheri on the tiruvarur thanjavur railroad"
],
[
" "
],
[
" "
],
[
"during the time of kulottunga chola i"
],
[
"[CLS] what are the darshan hours ? [SEP] may be reproduced or used in any form without permission . this shivastalam is located 5 km south of kodavasal and koradacheri on the tiruvarur thanjavur railroad . koovilamputhur the original name became kollampudur . koovilam stands for vilvam , hence vilvavanam . this shrine is regarded as a muktistalam . this shrine is regarded as the 113rd in the series of tevara stalams in the chola region south of the river kaveri . legends the vilva trees are said to represent splashes of the celestial nectar amritam , and this stalam is considered on par with banares . sundarar is believed to have floated across the river to this temple in a boatmanless raft in a river in spate singing a patikam . this event is celebrated in a festival in the monsoon month of libra . the avimukteswarar temple nearby is also associated with this legend as is the shivastalam at kodavasal . shiva is said to have blessed durvasa muni with a vision of the cosmic dance here . legend also has it that arjuna worshipped shiva at this shrine . the temple there are several inscriptions here , and the cholas have made immense contributions here . to this temple which was built during the time of kulottunga chola i . this temple occupies an area of over 2 acres , and its second prakaram has a 5 tiered rajagopuram . the vinayakar in this temple is also of great festivals six worship services are offered each day . kartikai deepam , arudra darisanam"
],
[
"[CLS] what is the average darshan duration ? [SEP]"
],
[
" "
],
[
" "
],
[
" "
],
[
" "
],
[
" "
],
[
" "
]
]
},
you can try the following approach. Since you got lists of sublists we can do the following.
import json
def remove_from_sublists(the_list, to_be_removed):
for each_item in list(the_list):
if isinstance(each_item, list):
remove_from_sublists(each_item, to_be_removed)
elif to_be_removed in each_item :
the_list.remove(each_item)
return the_list
dic = {}
with open('WebTempleCorpus.json') as json_file:
data = json.load(json_file)
for (i, v) in data.items():
sub_dict = v
if(v.get("answers")):
sub_dict["answers"] = remove_from_sublists(v["answers"], "CLS")
sub_dict["answers"] = remove_from_sublists(v["answers"], "SEP")
dic[i] = sub_dict
with open('result.json', 'w') as fp:
json.dump(dic, fp)
I am trying to convert JSON file into data frame in R and had this error when I ran fromJSON in 'jsonlite' package.
Error in fromJSON(content, handler, default.size, depth, allowComments, :
Invalid JSON Node
I figured that the variable order are different in each row. In specific, for example, in the first row, "categories" is the second variable whereas it is the last variable in the third row (Please take a look at the data below). This is how the original JSON file is formatted, and I have several hundred thousand rows like this so can't manually change it.
Anyone can suggest how to fix this and successfully convert the JSON into data frame in R?
[{"asin": "0188399313", "categories": [["Baby"]], "description": "Wee-Go Glass baby bottles by LifeFactory (Babylife) are designed to grow with your child. The included clear cover can also serve as an easy to hold cup. Twist on the solid cap (sold separately) and use your bottles for storing juice or snacks. Perfect for a lunchbox or traveling. The bright colored silicone sleeve (patent pending) helps to protect the bottle from breakage and provides a great gripping surface and tactile experience during feeding. The bottle and sleeve can be boiled or put in the dishwasher together. They can also go in the freezer, making breast milk storage simple.", "title": "Lifefactory 4oz BPA Free Glass Baby Bottles - 4-pack-raspberry and Lilac", "price": 69.99, "imUrl": "http://ecx.images-amazon.com/images/I/41SwthpdD9L._SX300_.jpg", "brand": "Lifefactory", "related": {"also_bought": ["B002SG7K7A", "B003CJSXW8", "B004PW4186", "B002O3JH9Q", "B002O3NLIO", "B004HGSU28"], "also_viewed": ["B003CJSXW8", "B0052QOL1Q", "B004PW4186", "B00EN0OLZ8", "B00EN0OOQY", "B0049YS46K", "B00E64CBLM", "B00F9YOOS6", "B00AH9RPVQ", "B00BCU2R7G", "B002O3NLIO", "B008NZ4X2K", "B005NIDFEW", "B00DKPJCH4", "B00CZNGWWK", "B00DAKJIQ4", "B005CT55IQ", "B0049YRJM0", "B0071IEWD0", "B00E64CA68", "B00IUB3SKK", "B00A7AA6XY", "B001F50FFE", "B002HU9EO4", "B007HP11SQ", "B009WPUMX4", "B002O3JH9Q", "B00F2FT3K6", "B00I5CR35A", "B00BCTY5EK", "B002SG7K7A", "B00F2FLU2U", "B0062ZK0GQ", "B002UOFR66", "B0055LKQQ2", "B00A0FGN8I", "B00HMYCG2W", "B00DHFLUO0", "B0040HMPA2", "B00I5CT9XE", "B008B5MMNO", "B00BQYVNGO", "B00925WM28", "B00BGKC3EY", "B005Q3LSDO", "B0038JDVCE", "B0045I6IA4"], "bought_together": ["B002SG7K7A", "B003CJSXW8"], "buy_after_viewing": ["B003CJSXW8", "B0052QOL1Q", "B004PW4186", "B002SG7K7A"]},},
{"asin": "0188399518", "categories": [["Baby"]], "description": "The Planet Wise Flannel Wipes are 10 super soft, color coordinated, high-quality wipes that fit perfectly into the Planet Wise Wipe PouchTM.", "title": "Planetwise Flannel Wipes", "price": 15.95, "imUrl": "http://ecx.images-amazon.com/images/I/41otjnA4OGL._SY300_.jpg", "brand": "Planet Wise", "related": {"also_bought": ["B00G96N3YY", "B003XSEV2O", "B000138GNY", "B005WWIE3G", "B005WWI0DA", "B005WWIMGA", "B00DS4WPNK", "B0039VCRPI", "B001QIN6ME", "B004GMGLN8", "B00CJ2OWUG", "B004BDNJW8", "B003N0JXSO", "B003X3R6TO", "B002T5Q01C", "B006J2U4T0", "B00JBJDEC2", "B00380LVLG", "B00GJVM2NW", "B003AJHDQW", "B0043EDGP0", "B0012IJBUE", "B001OI0YWG", "B005DL5970", "B00305GSKS", "B00CQ9UUK8", "B002Y27PQ4", "B00AH8J448", "B00A3JXVZY", "B00483GAJU", "B001J6O6B8", "B0019ID6G2", "B005DL7LGM", "B00APSYDXM", "B004D5KJJA", "B0021HR94K", "0757302661", "B001HX4DNE"], "also_viewed": ["B000138GNY", "B00G96N3YY", "B006J2U4T0", "B00GLBR3C0", "B0039VCRPI", "B0012IJBUE", "B001IA2XGK", "B001R2L6PS", "B003X3R6TO", "B00DS4WPNK", "B005WWIE3G", "B004BDNJW8", "B005WWIMGA", "B005WWI0DA", "B003XSEV2O", "B004A5VGPE", "B004A5TTGC", "B00JBJDEC2", "B004GMGLN8", "B009W6PM9M", "B004RGL9I0", "B0040IWMTK", "B00J3C0XC6", "B00HO1HBC8", "B00G96N18M", "B00A3V0XDA", "B0032AMM9M", "B001QIN6ME", "B00A3V0XEY", "B00CIUAB40", "B00J3C0Y1G", "B005IPE0KI", "B00L0W5B7Y", "B00FZGLKQW", "B00HBYOE3W", "B004A5TT4O", "B00B6KS578", "B002HOQOMA", "B004G792KW", "B002UD6BZS", "B002XVRKHK", "B002HOQOUW", "B002UD6C1Q", "B00APSYDXM", "B00H07AZ42", "B002LZX2BG", "B00BFQVZNO", "B00EZKNVYS", "B00DCLYYJW", "B00AHG8H1Q", "B002QZ64T8", "B003N0JXSO", "B003AJHDQW", "B00DN9NW32", "B00CHPJ0I4", "B005WXMJUY", "B00D4LFC3W"], "bought_together": ["B00G96N3YY", "B003XSEV2O"], "buy_after_viewing": ["B000138GNY", "B006J2U4T0", "B00G96N3YY", "B005WWIMGA"]},},
{"asin": "0188399399", "description": "The Planet Wise Wipe PouchTM features our patent pending no-leak design so your wipes will stay moist and not wick into your diaper bag. Features a unique snap down design to eliminate wicking from the zipper area. Most standard size wipes will fit into our Wipe PouchTM perfectly! Pouches are made of the same high quality fabrics as our Wet Bags, this pouch will last and last. Pouches are anti-microbial too! After you are done with the baby years, keep this pouch to use for just about anything you need to transport: make-up, toiletries, anything. Made in the USA!", "title": "Planetwise Wipe Pouch", "price": 10.95, "imUrl": "http://ecx.images-amazon.com/images/I/61x8h9u6mxL._SY300_.jpg", "related": {"also_bought": ["B005WWI0DA", "B005WWIMGA", "B006J2U4T0", "B000138GNY", "B003XSEV2O", "B005WWIE3G", "B00DS4WPNK", "B0039VCRPI", "B003GSLCOG", "B0018B15FE", "B002MN3JY2"], "also_viewed": ["B005WWIMGA", "B00G96N3YY", "B00DAI76TC", "B0067GKHVS", "B005WWIE3G", "B004RGL9I0", "B000138GNY", "B00JBJDEC2", "B00CMCJ2AS", "B005WWI0DA", "B002UD6C16", "B00E401FVU", "B004GMGLN8", "B0012IJBUE", "B00BFQVZNO", "B006J2U4T0", "B000VV21G4", "B002HOQOUW", "B003X3R6TO", "B000GZJIVQ", "B0052AIF00", "B005GQ6606", "B003TZHCA4", "B001QIN6ME", "B003XSEV2O", "B00GLBR3C0", "B00DW9R03G", "B00J3C0XC6", "0188399542", "B00DQL6CIE", "B002UD6C1G", "B00A3V0XDA", "B0039VCRPI", "B00EZKNVYS", "B006QRFDHQ", "B00DS4WPNK", "B00CQ9UUK8", "B0087A2PKS", "B00GGNICSW", "B0037NXP18", "B000YCNLSC", "B00FJG945W", "B00FZGLKQW", "B004A5VGPE", "B00D4LFC3W", "B00G96N18M", "B00439UNR4", "B004A5TT4O", "B005DL7LGM", "B006SD2JUW", "B006Z6HXYY", "B006QRFDZI", "B00I0P7VYK", "B00AZWDM0I", "B000ZKHVMU", "B007799PGC", "B00H58UNMU", "B00JKUU3TE"], "buy_after_viewing": ["B005WWIMGA", "B00G96N3YY", "B00DAI76TC", "B0067GKHVS"]},, "categories": [["Baby"]]},
{"asin": "0316967297", "description": "Hand crafted set includes 1 full quilt (76x86 inches) and 2 standard shams (20x26 inches). Face cloth and fill are 100 natural cotton. Prewashed for out of the bag comfort. Hand crafted with embroidery. Machine washable. Made in China", "title": "Annas Dream Full Quilt with 2 Shams", "price": 109.95, "imUrl": "http://ecx.images-amazon.com/images/I/51%2BZ1%2BNeukL._SY300_.jpg", "related": {"also_viewed": ["B009LTER3W", "B00575TI5Q", "B004NSYYJI"], "buy_after_viewing": ["B009LTER3W", "B001MX5EE6", "B00575TI5Q", "B0029009TG"]},, "categories": [["Baby"]]},
{"asin": "0615447279", "categories": [["Baby"]], "description": "Thumbuddy To Love- The Binky Fairy helps children give up pacifier sucking without the fuss and tears. The adorable book comes with a matching Binky Fairy puppet and just like the Tooth Fairy, the book is read the night before so children understand where thier pacifiers will go. They awake to find an adorable Binky Fairy puppet under thier pillow knowing that the Binky Fairy came! Each book comes with a success chart and stickers. Winner of the PTPA Awards (Parent Tested, Parent Approved). Recommended for ages 2-4.", "title": "Stop Pacifier Sucking without tears with Thumbuddy To Love s Binky Fairy Puppet and Adorable Book", "price": 16.95, "imUrl": "http://ecx.images-amazon.com/images/I/51RKKENlq%2BL._SY300_.jpg", "brand": "", "related": {"also_bought": ["0979670004", "1601310234", "B005G172KE", "1575422573", "1905417896", "B0044D0HA2", "B001GQ2CPI"], "also_viewed": ["0979670004", "1575422573", "1493535943", "0615273645", "1601310234", "0992616727", "0615273793", "0763623644", "1907152962", "0992616719", "B0016ZX7AI", "0698400488", "1608442713", "0451416058", "1581176848", "B0072CTECY", "B005G172KE", "B0016P8K3W", "0375822704", "0979201004", "0961478020", "160131048X", "1572245859"], "bought_together": ["B005G172KE"], "buy_after_viewing": ["0979670004", "1575422573", "B005G172KE", "1493535943"]},}]