I retrieved a dataset from a news API in JSON format. I want to extract the news description from the JSON data.
This is my code:-
import requests
import json
url = ('http://newsapi.org/v2/top-headlines?'
'country=us&'
'apiKey=608bf565c67f4d99994c08d74db82f54')
response = requests.get(url)
di=response.json()
di = json.dumps(di)
for di['articles'] in di:
print(article['title'])
The dataset looks like this:-
{'status': 'ok',
'totalResults': 38,
'articles': [
{'source':
{'id': 'the-washington-post',
'name': 'The Washington Post'},
'author': 'Derek Hawkins, Marisa Iati',
'title': 'Coronavirus updates: Texas, Florida and Arizona officials say early reopenings fueled an explosion of cases - The Washington Post',
'description': 'Local officials in states with surging coronavirus cases issued dire warnings Sunday about the spread of infections, saying the virus was rapidly outpacing containment efforts.',
'url': 'https://www.washingtonpost.com/nation/2020/07/05/coronavirus-update-us/',
'urlToImage': 'https://www.washingtonpost.com/wp-apps/imrs.php?src=https://arc-anglerfish-washpost-prod-washpost.s3.amazonaws.com/public/K3UMAKF6OMI6VF6BNTYRN77CNQ.jpg&w=1440',
'publishedAt': '2020-07-05T18:32:44Z',
'content': 'Here are some significant developments:\r\n<ul><li>The rolling seven-day average for daily new cases in the United States reached a record high for the 27th day in a row, climbing to 48,606 on Sunday, … [+5333 chars]'}])
Please guide me with this!
There are few corrections needed in your code.. below code should work and i have removed API KEY in answer make sure that you add one before testing
import requests
import json
url = ('http://newsapi.org/v2/top-headlines?'
'country=us&'
'apiKey=<API KEY>')
di=response.json()
#You don't need to dump json that is already in json format
#di = json.dumps(di)
#your loop is not correctly defined, below is correct way to do it
for article in di['articles']:
print(article['title'])
response.json
{'status': 'ok',
'totalResults': 38,
'articles': [
{'source':
{'id': 'the-washington-post',
'name': 'The Washington Post'},
'author': 'Derek Hawkins, Marisa Iati',
'title': 'Coronavirus updates: Texas, Florida and Arizona officials say early reopenings fueled an explosion of cases - The Washington Post',
'description': 'Local officials in states with surging coronavirus cases issued dire warnings Sunday about the spread of infections, saying the virus was rapidly outpacing containment efforts.',
'url': 'https://www.washingtonpost.com/nation/2020/07/05/coronavirus-update-us/',
'urlToImage': 'https://www.washingtonpost.com/wp-apps/imrs.php?src=https://arc-anglerfish-washpost-prod-washpost.s3.amazonaws.com/public/K3UMAKF6OMI6VF6BNTYRN77CNQ.jpg&w=1440',
'publishedAt': '2020-07-05T18:32:44Z',
'content': 'Here are some significant developments:\r\n<ul><li>The rolling seven-day average for daily new cases in the United States reached a record high for the 27th day in a row, climbing to 48,606 on Sunday, … [+5333 chars]'}]}
Code:
di = response.json() # Understand that 'di' is of type 'dictionary', key-value pair
for i in di["articles"]:
print(i["description"])
"articles" is one of the keys of dictionary di, It's corresponding value is of type list. "description" , which you are looking is part of this list (value of "articles"). Further list contains the dictionary (key-value pair).You can access from key - description
I am creating a react app and I am using Zomato api for restaurants. I was able to display the search results in the app and link to a single page with specific restaurant details. I wanted to fill up the single restaurant page with name, link to restaurant website, address, photo gallery, etc. I already have the rest of the info in the restaurant json. However. I couldn't find any info to link to the restaurant website. All I could find is several links to zomato page with info for the specific website and on this page, there's the link to the restaurant website...
Here's the specific restaurant json I got from zomato api:
R: {has_menu_status: {…}, res_id: 16935822}
all_reviews: {reviews: Array(3)}
all_reviews_count: 3
apikey: ""
average_cost_for_two: 30
book_again_url: ""
book_form_web_view_url: ""
cuisines: "Drinks Only"
currency: "$"
deeplink: "zomato://restaurant/16935822"
establishment: ["Lounge"]
events_url: "https://www.zomato.com/miami/blackbird-ordinary-downtown-miami/events#tabtop?utm_source=api_basic_user&utm_medium=api&utm_campaign=v2.1"
featured_image: "https://b.zmtcdn.com/data/reviews_photos/607/4edc23f93d257fbed50f184d8c920607_1503580424.png?fit=around%7C640%3A640&crop=640%3A640%3B%2A%2C%2A"
has_online_delivery: 0
has_table_booking: 0
highlights: (15) ["Credit Card", "Lunch", "Serves Alcohol", "Cash", "Dinner", "Live Music", "Outdoor Seating", "Live Entertainment", "Fullbar", "Valet Parking Available", "Indoor Seating", "Nightlife", "Wine", "Serves Cocktails", "Street Parking"]
id: "16935822"
include_bogo_offers: true
is_book_form_web_view: 0
is_delivering_now: 0
is_table_reservation_supported: 0
is_zomato_book_res: 0
location: {address: "729 SW 1st Avenue, Downtown Miami, Miami 33130", locality: "Downtown Miami", city: "Miami", city_id: 291, latitude: "25.7667186000", …}
menu_url: "https://www.zomato.com/miami/blackbird-ordinary-downtown-miami/menu?utm_source=api_basic_user&utm_medium=api&utm_campaign=v2.1&openSwipeBox=menu&showMinimal=1#tabtop"
mezzo_provider: "OTHER"
name: "Blackbird Ordinary"
offers: []
opentable_support: 0
phone_numbers: "(305) 671-3307"
photo_count: 34
photos: (4) [{…}, {…}, {…}, {…}]
photos_url: "https://www.zomato.com/miami/blackbird-ordinary-downtown-miami/photos?utm_source=api_basic_user&utm_medium=api&utm_campaign=v2.1#tabtop"
price_range: 3
switch_to_order_menu: 0
thumb: "https://b.zmtcdn.com/data/reviews_photos/607/4edc23f93d257fbed50f184d8c920607_1503580424.png?impolicy=newcropandfit&cropw=886&croph=886&cropoffsetx=169&cropoffsety=1&cropgravity=NorthWest&fitw=200&fith=200&fittype=ignore"
timings: "3 PM to 5 AM (Mon-Fri),5 PM to 5 AM (Sat-Sun)"
url: "https://www.zomato.com/miami/blackbird-ordinary-downtown-miami?utm_source=api_basic_user&utm_medium=api&utm_campaign=v2.1"
user_rating: {aggregate_rating: "3.3", rating_text: "Average", rating_color: "CDD614", rating_obj: {…}, votes: "14"}
__proto__: Object
I am begining to find out that I kinda don't like Zomato api, but it will take more time to switch over to something else....
I'm trying to export some API JSON data to a csv. I've tried alot of methods refered here but I'm getting the error.
My json file is as attached:
[{u'src': {}, u'product': u'WEB_MPS', u'name':
u'DOMAIN_MATCH',u'explanation':
{u'osChanges': [],
u'malwareDetected':
{u'malware':
[{
u'md5Sum': u'www.jfaewrwergwea.com',
u'name': u'Raxxxxxe.WCry.InfectionFail'}]
}},
u'vlan': 0, u'sensorIp': u'10.165.100.31', u'occurred': u'2017-08-24 01:54:54 +0530',
u'alertUrl':u'', u'applianceId': u'002590879266', u'rootInfection': 7971, u'id': 860438060,
u'action': u'notified', u'sensor': u'W-inline', u'dst': {u'mac': u'00:1b:17:00:01:10'}, u'severity': u'MINR'},
{u'src': {}, u'product': u'WEB_MPS', u'name': u'DOMAIN_MATCH',
u'explanation': {u'osChanges': [], u'malwareDetected':
{u'malware': [{u'md5Sum': u'www.iuqerea.com', u'name': u'Raxxxxre.WCry.InfectionFail'}]}},
u'vlan': 0, u'sensorIp': u'10.165.100.31', u'occurred': u'2017-08-23 19:52:56 +0530',
u'alertUrl': u'', u'applianceId': u'002590879266', u'rootInfection': 7950, u'id': 860401215,
u'action': u'notified', u'sensor': u'W0-FW-inline', u'dst': {u'mac': u'00:1b:17:00:01:10'}, u'severity': u'MINR'},
How can i get my python code to convert??
Trying to write a python script that will allow me to read a .csv file and mix up the values to a specific format/data structure in .json that I can then import into mongoDB. I'm using pedestrian data as my dataset and there are over a million entries with redundant data. I'm stuck on writing the actual script and translating that into my desired .json format.
data.csv - in table format for easier reading and raw
Id,Date_Time,Year,Month,Mdate,Day,Time,Sensor_ID,Sensor_Name,Hourly_Counts
1, 01-JUN-2009 00:00,2009,June,1,Monday,0,4,Town Hall (West),194
2, 01-JUN-2009 00:00,2009,June,1,Monday,0,17,Collins Place (South),21
3, 01-JUN-2009 00:00,2009,June,1,Monday,0,18,Collins Place (North),9
4, 01-JUN-2009 00:00,2009,June,1,Monday,0,16,Australia on Collins,39
5, 01-JUN-2009 00:00,2009,June,1,Monday,0,2,Bourke Street Mall (South),28
6, 01-JUN-2009 00:00,2009,June,1,Monday,0,1,Bourke Street Mall (North),37
7, 01-JUN-2009 00:00,2009,June,1,Monday,0,13,Flagstaff Station,1
8, 01-JUN-2009 00:00,2009,June,1,Monday,0,3,Melbourne Central,155
9, 01-JUN-2009 00:00,2009,June,1,Monday,0,15,State Library,98
10, 01-JUN-2009 00:00,2009,June,1,Monday,0,9,Southern Cross Station,7
11, 01-JUN-2009 00:00,2009,June,1,Monday,0,10,Victoria Point,8
12, 01-JUN-2009 00:00,2009,June,1,Monday,0,12,New Quay,30
Because I'll be uploading to mongoDB, the Id in my context is redundant to me so I need my script to skip that. Sensor_ID is not unique but I'm planning to make it the PK and create a list of objects differentiating the Hourly_Count.
I'm aiming to generate a JSON object like this from the data:
**data.json**
{
{
"Sensor_ID": 4,
"Sensor_Name": "Town Hall(West)",
"countList":
[
{
"Date_Time": "01-JUN-2009 00:00",
"Year":2009,
"Month": "June",
"Mdate": 1,
"Day": "Monday",
"Time": 0,
"Hourly_Counts": 194
},
{
"Date_Time": "01-JUN-2009 00:00",
"Year":2009,
"Month": "June",
"Mdate": 1,
"Day": "Monday",
"Time": 1,
"Hourly_Counts": 82
}
]
},
{
"Sensor_ID": 17,
"Sensor_Name": "Collins Place(North)",
"countList":
[
{
"Date_Time": "01-JUN-2009 00:00",
"Year":2009,
"Month": "June",
"Mdate": 1,
"Day": "Monday",
"Time": 0,
"Hourly_Counts": 21
}
]
}
}
So on so forth. I'm trying to make it so when it reads a Sensor_ID it creates an json object from the fields listed and adds it to the countList. Added in another from station_ID = 4 to the countList.
I am using python 2.7.x and I have looked at every question concerning this on stackoverflow and every other website. Very few rarely seem to want to restructure the .csv data when converting to .json so it's been a bit difficult.
What I have so far, still relatively new to python so thought this would be good to try out.
csvtojson.py
import csv
import json
def csvtojson():
filename = 'data.csv'
fieldnames = ('Id','Date_Time','Year','Month','Mdate','Day',
'Time','Sensor_ID','Sensor_Name', 'Hourly_Counts')
dataTime = ('Date_Time','Year','Month','Mdate','Day',
'Time', 'Hourly_Counts')
all_data = {}
with open(filename, 'rb') as csvfile:
reader = csv.DictReader(csvfile, fieldnames)
#skip header
next(reader)
current_sensorID = None
for row in reader:
sensor_ID = row['Sensor_ID']
sensorName = row['Sensor_Name']
data = all_data[sensor_ID] = {}
data['dataTime'] = dict((k, row[k]) for k in dataTime)
print json.dumps(all_data, indent=4, sort_keys=True)
if __name__ == "__main__":
csvtojson()
As far as I got is that countList is in is own object but it's not creating a list of objects and may mess up the import to mongoDB. It is filtering through Sensor_ID but overwriting if there are duplicates instead of adding to countList. And I can't seem to get it in the format/data structure I want - I'm not even sure if that's the right structure, ultimate goal is to import the millions of tuples into mongoDB like the way I listed. I'm trying a small set now to test it out.
Please check the following.
https://github.com/gurdyals/test-repo/tree/master/MongoDB
Use " MongoDB_py.zip " files .
I did the same to convert csv data to MongoDB dict .
Please let me know if you have any questions.
Thanks
Here is sample code for doing something similar to the above using python pandas. You could also do some aggregation in the dataframe if you wish to summarise the data to get rid of the redundant data.
import pandas as pd
import pprint as pp
import json
from collections import defaultdict
results = defaultdict(lambda: defaultdict(dict))
df = pd.read_csv('data.csv')
df.set_index(['Sensor_ID', 'Sensor_Name'],inplace=True)
df.reset_index(inplace=True)
grouped = df.groupby(['Sensor_ID', 'Sensor_Name']).apply(lambda x: x.drop(['Sensor_ID', 'Sensor_Name'], axis=1).to_json(orient='records'))
grouped.name = 'countList'
js = json.loads(pd.DataFrame(grouped).reset_index().to_json(orient='records'))
print json.dumps(js, indent = 4)
The output:
[
{
"Sensor_ID": 1,
"countList": "[{\"Id\":6,\"Date_Time\":\" 01-JUN-2009 00:00\",\"Year\":2009,\"Month\":\"June\",\"Mdate\":1,\"Day\":\"Monday\",\"Time\":0,\"Hourly_Counts\":37}]",
"Sensor_Name": "Bourke Street Mall (North)"
},
{
"Sensor_ID": 2,
"countList": "[{\"Id\":5,\"Date_Time\":\" 01-JUN-2009 00:00\",\"Year\":2009,\"Month\":\"June\",\"Mdate\":1,\"Day\":\"Monday\",\"Time\":0,\"Hourly_Counts\":28}]",
"Sensor_Name": "Bourke Street Mall (South)"
},
{
"Sensor_ID": 3,
"countList": "[{\"Id\":8,\"Date_Time\":\" 01-JUN-2009 00:00\",\"Year\":2009,\"Month\":\"June\",\"Mdate\":1,\"Day\":\"Monday\",\"Time\":0,\"Hourly_Counts\":155}]",
"Sensor_Name": "Melbourne Central"
},
{
"Sensor_ID": 4,
"countList": "[{\"Id\":1,\"Date_Time\":\" 01-JUN-2009 00:00\",\"Year\":2009,\"Month\":\"June\",\"Mdate\":1,\"Day\":\"Monday\",\"Time\":0,\"Hourly_Counts\":194}]",
"Sensor_Name": "Town Hall (West)"
},
I am not sure what's going wrong, cause it looks pretty straight forward to me. I have few things in my mind about appending the JSON response, but for that when I am trying to apply forEach loop to my Json , no data is shown in the alert
Angular code
mainApp.controller('MultiCarouselController', function($scope, $http) {
$scope.products = [];
$http.get("/get_broad_category_products/?BroadCategory=BroadCategory3")
.success(function (response) {
$scope.products = response;
angular.forEach(products,function(value,key){
alert(key+'----'+value); // can see this alerts
});
}).error(function(){
console.log('Error happened ... ');
});
});
My JSON response is some junk data of Products
[
{
"sku": "d1cd71a8-9dc5-4724-b269-473ede28a1d7",
"selectedQtyOptions": [],
"selectedSize": "",
"description": "foolish interpolates trumpet monographs ferried inboards Forster tike grammatically sunroof vaporizing Sweden demure retouching completely robbing readies unloose guiltless tatty unobservant cuffs fortieth wither rigorously paradoxically snowmobiling charts clenching planning dealing lesions bicameral pertly chaffinches grumpiness private purled insanely attainment proposal Fatima execrates pshaws chars actuators turboprop soughed kicking majors conquistadores Cynthia septuagenarians kneecaps titans attractions larvas invigorating trunking Shevat recluse Trina slenderness kinking falsified logistically hogged skyrocketing ordinal avoiding trademarked underfoot garter sacrificial pricey nosedive bachelors deiced heave dictatorial muffing prayed rewinding recopied limpidly Crichton conversion chitterlings signets Aiken Froissart turnoff snowshoe forded spiralled underwriters flourishes Sade splicer transfusions cesspools lifelike ruckus showering paean voguish Buck copings Russell watchdog magneto pored height zodiac motherland backings Venus obeys scooters nonintervention dinosaur unashamedly anathema hibernate consumerism portended worked mystically existentialist dissatisfies badgers unanimously triplicated Jenny sagacity Windex snoopier nonplusing shovelling Assam putty darn Sulawesi Italians gunnery codify develops rhinos upwards Louise welled experiences socks pinky mewed Camille claimants swirl squattest ware parenthetic bonitoes hydrangeas decolonizing omit skyjacks Gorky financiers province flywheel southeastward Bayeux updated yowl Tulsidas macintosh sprees pralines systolic uncommoner cilium tromping Asimov heinous cordoned combated camerawomen syndrome identified prizefights heavyweight vertically reflector integrity Hebrides sepulchral loner parrot smooths candidness",
"selectedQty": "1",
"title": "viragoes",
"brand": "Brand0",
"images": [
{
"image0": "/media/products/f791a316ced7b3b774bd61e138197224.jpg"
}
],
"sizeQtyPrice": [
{
"discountAttributes": "chinos theosophy misdemeanor irrigates school Pullman sombrely suspect vortex baddest",
"measureUnit": "ltr",
"discountPercent": 4,
"mrp": 3102,
"qty": 7,
"size": 66
},
{
"discountAttributes": "Molotov absurd traces pounces contracts clarions thighbone Hesse parricide constrains",
"measureUnit": "m",
"discountPercent": 16,
"mrp": 2773,
"qty": 7,
"size": 18
},
{
"discountAttributes": "detainment gunnysack vied expropriation unobtrusive collectables embracing poster hexing governess",
"measureUnit": "m",
"discountPercent": 6,
"mrp": 9920,
"qty": 6,
"size": 69
}
],
"id": 9
},
{
"sku": "838660bb-7ab9-4f2a-8be7-9602a5801756",
"selectedQtyOptions": [],
"selectedSize": "",
"description": "agreeing vizier bleariest trig appliquéing copulating commissariats Balzac lunchtimes glittery quacking Leoncavallo heehawing Tampax lizards pegged nanosecond centigrade subplots tumbrils give jawed skits nickel discontinues impinged evangelized Platonist waterlines dams symposiums intercessor cognition heavier softener dromedaries bravos immobilize consciously Clemons patch klutzier Kirkpatrick caddying designs Dulles twelfths undemocratic isolationists infected ma homering soliciting minibus pluralism fraternity catalyzed Scorpio pandemonium waxwing starter infuses rebuttals spirals rerunning interrogatories Manuel whomsoever tenderized conjoint baronesses callower parenthetic plusses extend cockier Fokker dewlap Cowper Swammerdam secs hock relaxations Judas Canadian presidency lo wildness Philippe picture beekeeper lull manuals transnational yaw chloroformed perennials distinctive Nottingham antiquaries underneath parted nervously basemen observatories scrubbed encoder egalitarians winnow caddish Hawaiians brownstones robbing exhaustible antagonist benefactresses Plasticine Peace platypi Guzman stippled shuts peacemakers butterfly Bolton grout McCain Lebanon bounce oleander Balkans endearments snowfall spoonerisms furnaces inequities billowy jutting guffaw beautifully penis newtons snuffboxes j Angelita tinkles literature depicts insouciant scribblers blinker disobediently devotees primordial sixties Kalamazoo shear contest classes cripple edging exactest cheat invocation thrived drunkenness Fuller architectures sprite Lillian constricts tucking chastisements walruses guzzlers rejoinder apprenticeships pillory spendthrift omens spoonful contortions precociously intensely motorway guts cahoot sculptor paralytics reminisce meltdown trusts lady pronghorn scurried Campbell micron flawing foals nigher",
"selectedQty": "1",
"title": "smokier",
"brand": "Brand2",
"images": [
{
"image0": "/media/products/f51a649e72694d23962ee77a97872f0e.jpg"
}
],
"sizeQtyPrice": [
{
"discountAttributes": "Beerbohm earldom Stanley seconding hypertension Sayers miserly epitome retires ditching",
"measureUnit": "m",
"discountPercent": 15,
"mrp": 5065,
"qty": 6,
"size": 83
},
{
"discountAttributes": "confine Newman bagel cornflower rears generator goaded midweeks drain cigarillo",
"measureUnit": "Kg",
"discountPercent": 12,
"mrp": 2284,
"qty": 9,
"size": 13
},
{
"discountAttributes": "eerier fizzes lessened rotisserie developer Gray industrial callused convergences ampoule",
"measureUnit": "gms",
"discountPercent": 4,
"mrp": 6816,
"qty": 8,
"size": 18
}
],
"id": 14
}
]
products isn't defined. It should be $scope.products