Scrapy - Getting duplicated items when appending items using for loop - json

Question

I am crawling the data from JSON response. Extracting data into item using for loop and all i get, is a last record rewriting all the previous records made by this loop.
Here is my code:
def parse_centers_and_ambulances(self, response):
json_response = json.loads(response.body_as_unicode())
facility = MedFacilityItem()
facility["name"] = "Med Facility #1"
centers = []
med_centers = MedCenterItem()
for center in json_response:
if center["name"].startswith("Center"):
med_centers["response_url"] = center["product_id"]
med_centers["name"] = center["name"]
med_centers["address"] = center["name_short"] + "." +
center["adr_name"] + " " +
center["adr_dom"]
med_centers["lat"] = center["latitude"]
med_centers["lon"] = center["longitude"]
med_centers["phoneInfo"] = [{"number": center["tel1"],
"description": center["tel1_descr"]},
{"number": center["tel2"],
"description": center["tel2_descr"]}]
centers.append(med_centers)
facility["facility_type"] = centers
return facility
What i am missing?

Answer 1

Since Scrapy items basically behave like dicts, I'm going to use dicts for the following examples. Consider this:
In [1]: dict_list = []
...: d = {}
...: for i in range(3):
...: d['i'] = i
...: dict_list.append(d)
...: print dict_list
...: print [id(e) for e in dict_list]
...:
[{'i': 2}, {'i': 2}, {'i': 2}]
[4557722520, 4557722520, 4557722520]
Dicts are mutable objects, and in this case you are appending the same dict instance multiple times to a list. The resulting list does not contain different items, only several references to the same dict object. The following example shows the same behaviour, appending the same dict three times to a list and then setting a value for it:
In [2]: dict_list = []
...: d = {}
...: for i in range(3):
...: dict_list.append(d)
...: d['some'] = 'value'
...: print dict_list
...:
[{'some': 'value'}, {'some': 'value'}, {'some': 'value'}]
What you need to do is create different dicts by initializing them inside the for loop, as follows:
In [3]: dict_list = []
...: for i in range(3):
...: d = {}
...: d['i'] = i
...: dict_list.append(d)
...: print dict_list
...: print [id(e) for e in dict_list]
...:
[{'i': 0}, {'i': 1}, {'i': 2}]
[4557901904, 4557724760, 4557843264]

Answer 2

You can try defining your item inside of the loop, instead outside of it.
def parse_centers_and_ambulances(self, response):
json_response = json.loads(response.body_as_unicode())
facility = MedFacilityItem()
facility["name"] = "Med Facility #1"
centers = []
# med_centers = MedCenterItem() <-- this
for center in json_response:
if center["name"].startswith("Center"):
med_centers = MedCenterItem() <-- should be here
med_centers["response_url"] = center["product_id"]
med_centers["name"] = center["name"]
med_centers["address"] = center["name_short"] + "." +
center["adr_name"] + " " +
center["adr_dom"]
med_centers["lat"] = center["latitude"]
med_centers["lon"] = center["longitude"]
med_centers["phoneInfo"] = [{"number": center["tel1"],
"description": center["tel1_descr"]},
{"number": center["tel2"],
"description": center["tel2_descr"]}]
centers.append(med_centers)
facility["facility_type"] = centers
return facility