Unable to scrape the names from json response - json

There are around 966 names in the webpage with json content but with my script I'm getting only 10 out of them. I'm very new to json that is why I'can't figure out the mistake I'm making. How can I get all the names? I'm trying with the below code:
import requests
url = 'https://www.zebra.com/bin/zebra/partnersearch?inMiles=true&start=0&numRows=10&latitude=39.5500507&longitude=-105.7820674&sortOrder=asc&sortBy=distance&country=US&searchRadius=5000'
response = requests.get(url)
data = response.json()
for item in data:
print(item['name'])
Partial json content from that page is:
[{"id":"001i000001XR9dqAAD","website":"www.resortinternet.com","type":"partner","phoneNumber":"+1.970.262.3515","name":"Resortnet, LLC","logoPresent":"No","logoExtension":"","des":"Technology provider for destination resorts","translatedName":"ResortInternet","dbaName":"ResortInternet","PR":"NA","AN":"6244306","accountType":["Reseller"],"contentType":"parent","countries":["US"],"HSA":[],"countriesAndHsa":["US"],"premierSolutionPartner":false,"premierBusinessPartner":false,"solutionPartner":true,"businessPartner":false,"advancedSpecialistBarcodePrinterSupplies":false,"advancedSpecialistCardPrinters":false,"advancedSpecialistSupplies":false,"advancedSpecialistWirelessNetworks":false,"advancedSpecialistPrintEngines":false,"advancedSpecialistRfid":false,"specialistBarcodePrinterSupplies":false,"specialistCardPrinters":false,"specialistSupplies":false,"specialistWirelessNetworks":false,"specialistPrintEngines":false,"specialistRfid":false,"advancedRepairSpecialistLabelPrinter":false,"advancedRepairSpecialistCardPrinter":false,"advancedRepairSpecialistMobilePrinter":false,"advancedRepairSpecialistPrintEngine":false,"repairSpecialistLabelPrinter":false,"repairSpecialistCardPrinter":false,"repairSpecialistMobilePrinter":false,"repairSpecialistPrintEngine":false,"registeredResellerNoSpecialization":false,"pmiWraps":[{"programName":"Solution Partner","category":"Reseller","id":"001i000001XR9dqAAD_2","type":"pmiWrap","contentType":"child"}],"partnerLocations":[{"locationType":"Headquarters","addressLine1":"117 S 6th Ave.,","addressLine2":"PO Box 2718","city":"Frisco","state":"Colorado","zipCode":"80443","country":"United States","phone":"(970) 262-3515","fax":"(970) 668-9431","latlon":"39.5754576,-106.0952117","distance":16.8,"countryCode":"US","HSA":[],"id":"001i000001XR9dqAAD_0","type":"partnerLocation","contentType":"child"},{"locationType":"Primary Location","addressLine1":"117 S 6th Ave.,","city":"Frisco","state":"Colorado","zipCode":"80443","country":"United States","phone":"+1.970.262.3515","latlon":"39.5754576,-106.0952117","distance":16.8,"countryCode":"US","HSA":[],"id":"001i000001XR9dqAAD_1","type":"partnerLocation","contentType":"child"},{"locationType":"Address","addressLine1":"RESORTINTERNET\r2718:FRISCO:80443\r117 S 6TH AVERM UNIT 2","city":"Frisco","state":"Colorado","stateCode":"CO","zipCode":"80443","country":"United States","latlon":"39.5744309,-106.0975203","distance":16.9,"countryCode":"US","HSA":[],"id":"001i000001XR9dqAAD_100","type":"partnerLocation","contentType":"child"}],"verticalHierarchyWraps":[],"primaryLocation":{"locationType":"Headquarters","addressLine1":"117 S 6th Ave.,","addressLine2":"PO Box 2718","city":"Frisco","state":"Colorado","zipCode":"80443","country":"United States","phone":"(970) 262-3515","fax":"(970) 668-9431","latlon":"39.5754576,-106.0952117","distance":16.8,"countryCode":"US","HSA":

I don't think there's a problem with your code. If you check len(data) it returns 10, which means that the list of results contains only 10 (large) JSON objects.
Is there some reason you're expecting more than 10, or are you trying to access the name property of something inside each of these larger objects?

Your JSON is an array of objects, so when you loop through the data you aren't getting the array in your item variable but instead are getting the index of the array.
You can get the array by using the item variable as the index and once you have a reference to the array you can then read property objects such as name:
Like this:
for index in data:
item = data[index]
print(item['name'])
Here it is in JavaScript:
<script>
var data = [
{
"id":"001i000001XR9dqAAD",
"website":"www.resortinternet.com",
"type":"partner",
"phoneNumber":"+1.970.262.3515",
"name":"Resortnet, LLC",
"logoPresent":"No",
"logoExtension":"",
"des":"Technology provider for destination resorts",
"translatedName":"ResortInternet",
"dbaName":"ResortInternet",
"PR":"NA",
"AN":"6244306",
"accountType":[
"Reseller"
],
"contentType":"parent",
"countries":[
"US"
],
"HSA":[
],
"countriesAndHsa":[
"US"
],
"premierSolutionPartner":false,
"premierBusinessPartner":false,
"solutionPartner":true,
"businessPartner":false,
"advancedSpecialistBarcodePrinterSupplies":false,
"advancedSpecialistCardPrinters":false,
"advancedSpecialistSupplies":false,
"advancedSpecialistWirelessNetworks":false,
"advancedSpecialistPrintEngines":false,
"advancedSpecialistRfid":false,
"specialistBarcodePrinterSupplies":false,
"specialistCardPrinters":false,
"specialistSupplies":false,
"specialistWirelessNetworks":false,
"specialistPrintEngines":false,
"specialistRfid":false,
"advancedRepairSpecialistLabelPrinter":false,
"advancedRepairSpecialistCardPrinter":false,
"advancedRepairSpecialistMobilePrinter":false,
"advancedRepairSpecialistPrintEngine":false,
"repairSpecialistLabelPrinter":false,
"repairSpecialistCardPrinter":false,
"repairSpecialistMobilePrinter":false,
"repairSpecialistPrintEngine":false,
"registeredResellerNoSpecialization":false,
"pmiWraps":[
{
"programName":"Solution Partner",
"category":"Reseller",
"id":"001i000001XR9dqAAD_2",
"type":"pmiWrap",
"contentType":"child"
}
],
"partnerLocations":[
{
"locationType":"Headquarters",
"addressLine1":"117 S 6th Ave.,",
"addressLine2":"PO Box 2718",
"city":"Frisco",
"state":"Colorado",
"zipCode":"80443",
"country":"United States",
"phone":"(970) 262-3515",
"fax":"(970) 668-9431",
"latlon":"39.5754576,-106.0952117",
"distance":16.8,
"countryCode":"US",
"HSA":[
],
"id":"001i000001XR9dqAAD_0",
"type":"partnerLocation",
"contentType":"child"
},
{
"locationType":"Primary Location",
"addressLine1":"117 S 6th Ave.,",
"city":"Frisco",
"state":"Colorado",
"zipCode":"80443",
"country":"United States",
"phone":"+1.970.262.3515",
"latlon":"39.5754576,-106.0952117",
"distance":16.8,
"countryCode":"US",
"HSA":[
],
"id":"001i000001XR9dqAAD_1",
"type":"partnerLocation",
"contentType":"child"
},
{
"locationType":"Address",
"addressLine1":"RESORTINTERNET\r2718:FRISCO:80443\r117 S 6TH AVERM UNIT 2",
"city":"Frisco",
"state":"Colorado",
"stateCode":"CO",
"zipCode":"80443",
"country":"United States",
"latlon":"39.5744309,-106.0975203",
"distance":16.9,
"countryCode":"US",
"HSA":[
],
"id":"001i000001XR9dqAAD_100",
"type":"partnerLocation",
"contentType":"child"
}
],
"verticalHierarchyWraps":[
],
"primaryLocation":{
"locationType":"Headquarters",
"addressLine1":"117 S 6th Ave.,",
"addressLine2":"PO Box 2718",
"city":"Frisco",
"state":"Colorado",
"zipCode":"80443",
"country":"United States",
"phone":"(970) 262-3515",
"fax":"(970) 668-9431",
"latlon":"39.5754576,-106.0952117",
"distance":16.8,
"countryCode":"US"
}
}
];
for (var index in data)
{
var item=data[index];
console.log(item["name"]);
console.log(item);
}
</script>

Related

How to access field in list of JSON objects in a django SerializerMethodField?

I am creating an API for a game where users get points for labelling images. If a user has labeled an image with the same label as another, the score gets updated by 5 points (and 25 if that player is their opponent). I have declared the field score inside my serializer as a SerializerMethodField.
In the get_score method, I am trying to access the tag__name field inside the list of tags_to_compare and simply extract the strings and store them in a list.
I have tried the following inside my serializer:
Update
I have updated my get_score method as follows:
serializers.py
score = serializers.SerializerMethodField('get_score')
...
def get_score(self, tagging):
score = 0
resource_id = tagging.resource.id
# A previously played gameround for this resource is coordinated for Tag verification
coordinated_gameround = Gameround.objects.all().filter(taggings__resource_id=resource_id).order_by("?").first()
# list of tag_name from coordinated gameround
coordinated_gameround_tags = coordinated_gameround.taggings.all().values_list("tag__name", flat=True)
if Tag.objects.all().filter(name=tagging) in coordinated_gameround_tags:
score += 25
return score
I tested the following line and it retrieves a list of taggings.
coordinated_gameround_tags = coordinated_gameround.taggings.all().values_list("tag__name", flat=True)
This is the JSON Object where the list of tags is:
{
"gameround": {
"id": 2014305513,
"user": {
"id": 1,
"username": "carina",
"is_superuser": true
},
"score": 50,
"tags_to_compare": [
{
"tag_id": 153,
"tag__name": "night",
"tag__language": "en",
"resource_id": 323570
},
{
"tag_id": 10437,
"tag__name": "dress",
"tag__language": "en",
"resource_id": 323570
},
{
"tag_id": 9598,
"tag__name": "sleep",
"tag__language": "en",
"resource_id": 323570
}
]
}
}
coordinated_gameround is a Gameround instance of your model. Not a Dict. So you cannot access it with coordinated_gameround[key].
If your Gameround has a field named "name" or "langage", you can get those values with
name = coordinated_gameround.name
langage = coordinated_gameround.langage
Also, there might be an error with this line
Tag.objects.all().filter(name=tagging) in coordinated_gameround_tags
Do you expect coordinated_gameround_tags to contain a list of queryset ?

Intersept a FeatureCollection in MongoDB

I have a GeoJson filled with states from Austria and I want to do a query that gives me as output which certain states intercepts my polygon.
This is my query:
db.GeoAustria.find(
{
'features.geometry':{
$geoIntersects:{
$geometry:{
type: "Polygon",
coordinates: [
[
[
16.21685028076172,
48.007381433478855
],
[
16.24225616455078,
47.98716432210271
],
[
16.256675720214844,
48.00669234420252
],
[
16.21685028076172,
48.007381433478855
]
]
]
}
}
}
}
)
But it gives me all the features, including those that don't overlap the polygon...
Where is my mistake in this query?
Basic array match misunderstanding here. The input set is a single doc with 95 polygons in an array in a single FeatureCollection object. When you do a find() on such things, any individual geo that is an intersect will cause the entire doc to be returned as a match. This is exactly the same as:
> db.foo.insert({x:["A","B","C"]})
WriteResult({ "nInserted" : 1 })
> db.foo.find({x:"A"});
{ "_id" : ObjectId("5fb1845b08c09fb8dfe8d1c1"), "x" : [ "A", "B", "C" ] }
The whole doc is returned, not just element "A".
Let's assume that you might have more than one big doc in your collection. This pipeline yields the single target geometry for Baden (I tested it on your input set):
var Xcoords = [
[
[
16.21685028076172,
48.007381433478855
],
[
16.24225616455078,
47.98716432210271
],
[
16.256675720214844,
48.00669234420252
],
[
16.21685028076172,
48.007381433478855
]
]
];
var targ = {type: "Polygon", coordinates: Xcoords};
db.geo1.aggregate([
// First, eliminate any docs where the geometry array has zero intersects. In this
// context, features.geometry means "for each element of array features get the
// geometry field from the object there", almost like saying "features.?.geometry"
{$match: {"features.geometry": {$geoIntersects: {$geometry: targ}} }}
// Next, break up any passing docs of 95 geoms into 95 docs of 1 geom...
,{$unwind: "$features"}
// .. and run THE SAME $match as before to match just the one we are looking for.
// In this context, the array is gone and "features.geometry" means get JUST the
// object named geometry:
,{$match: {"features.geometry": {$geoIntersects: {$geometry: targ}} }}
]);
Beyond this, I might recommend breaking up that FeatureCollection into something that is both indexable (FeatureCollection is NOT indexable in MongoDB) and easier to deal with. For example, this little script run against your single-doc/many-polys design will convert it in 95 docs with extra info:
db.geo2.drop();
mainDoc = db.geo1.findOne(); // the one Austria doc
mainDoc['features'].forEach(function(oneFeature) {
var qq = {
country: "Austria",
crs: mainDoc['crs'],
properties: oneFeature['properties'],
geometry: oneFeature['geometry']
};
db.geo2.insert(qq);
});
db.geo2.aggregate([
{$match: {"geometry": {$geoIntersects: {$geometry: targ}} }}
]);
// yields same single doc output (Baden)
This allows ease of matching and filtering. For more on FeatureCollection vs. GeometryCollection see https://www.moschetti.org/rants/hurricane.html.

How to check a value matches with another correpsonding value in a json response

What is the most dynamic way of checking each instance of a json response value matches another json response value within a script assertion?
What I mean is lets say I have the following response below:
{
"xxx": [{
"roomInformation": [{
"xxx": xxx
}],
"totalPrice": xxx
},
{
"roomInformation": [{
xxx: xxx
}],
"totalPrice": xxx
}
]
}
I want to check that the first room price to match with the first totalPrice and the second roomPrice to match with the second totalPrice. It has to be dynamic as I may get many different instances of this so I can't simply just look through the json with [0] and [1]. Virtually check each roomPrice matches with its corresponding totalPrice.
Thanks
So given the Json as a variable:
def jsonTxt = '''{
"hotels": [{
"roomInformation": [{
"roomPrice": 618.4
}],
"totalPrice": 618.4
},
{
"roomInformation": [{
"roomPrice": 679.79
}],
"totalPrice": 679.79
}
]
}'''
We can then use the following script:
import groovy.json.*
new JsonSlurper().parseText(jsonTxt).hotels.each { hotel ->
assert hotel.roomInformation.roomPrice.sum() == hotel.totalPrice
}
As you can see, I'm using sum to add all the roomInformation.roomPrice values together. In your example, you only have one price, so this will be fine.. And it will also cover the case where you have multiple rooms adding up to make the total
Here is the script assertion to check each roomPrice is matching or not with totalPrice.
EDIT: based on OP's full response provided here
Script Assertion:
//Check if the response is not empty
assert context.response, "Response is empty or null"
def json = new groovy.json.JsonSlurper().parseText(context.response)
def sb = new StringBuffer()
json.regions.each { region ->
region.hotels.each { hotel ->
(hotel?.totalPrice == hotel?.roomInformation[0]?.roomPrice) ?: sb.append("Room price ${hotel?.roomInformation[0]?.roomPrice} is not matching with total price ${hotel.totalPrice}")
}
}
if (sb.toString()) {
throw new Error(sb.toString())
} else { log.info 'Prices match' }

Prevent change of 'copied' object affecting original using ng-repeat in angular

I have a setup in angular which displays a json string called 'items'. Each item contains an array of field ids. By matching the field ids, it pulls information for the fields using a seperate 'fields' json string.
{
"data": [
{
"id": "1",
"title": "Item 1",
"fields": [
1,
1
]
},
{
"id": "2",
"title": "Item 2",
"fields": [
1,
3
]
},
{
"id": 3,
"title": "fsdfs"
}
]
}
You can copy or delete either the items or fields, which will modify the 'items' json.
Everything works except when I copy one item (and its fields), and then choose to delete a field for that specific item.
What happens is that it deletes the field for both the copied item AND the original.
Plunker -
http://plnkr.co/edit/hN8tQiBMBhQ1jwmPiZp3?p=preview
I've read that using 'track by' helps to index each item as unique and prevent duplicate keys, but this seems to be having no effect.
Any help appreciated, thanks
Edit -
Credit to Eric McCormick for this one, using angular.copy with array.push solved this issue.
Instead of -
$scope.copyItem = function (index) {
items.data.push({
id: $scope.items.data.length + 1,
title: items.data[index].title,
fields: items.data[index].fields
});
}
This worked -
$scope.copyItem = function (index) {
items.data.push(angular.copy(items.data[index]));
}
I recommend using angular.copy, which is a "deep copy" of the source object. This is a unique object from the source one.
It may seem slightly counter-intuitive, but a direct reference (as you're observing) interacts with the original object. If you inspect the element's scope after it's instantiated in the DOM, you can see there's a $id property assigned to the object in memory. Basically, by using angular.copy(source, destination), you ensure a copying of all the properties/values and having a unique object.
Example:
//inside the controller, a function to instantiate new copy of selected object
this.selectItem = function(item){
var copyOfItem = angular.copy(item);
//new item with same properties and values but unique object!
}
Egghead.io has a video on angular.copy.

MongoDB find() to return the sub document when a (field,value) is matched

This is a single collection which has 2 json files. I am searching for a particular field: value in an object and the entire sub document must be returned in case of a match ( That particular sub document from the collection must be returned out of the 2 sub documents in the following collection). Thanks in advance.
{
"clinical_study": {
"#rank": "379",
"#comment": [],
"required_header": {
"download_date": "ClinicalTrials.gov processed this data on March 18, 2015",
"link_text": "Link to the current ClinicalTrials.gov record.",
"url": "http://clinicaltrials.gov/show/NCT00000738"
},
"id_info": {
"org_study_id": "ACTG 162",
"secondary_id": "11137",
"nct_id": "NCT00000738"
},
"brief_title": "Randomized, Double-Blind, Placebo-Controlled Trial of Nimodipine for the Neurological Manifestations of HIV-1",
"official_title": "Randomized, Double-Blind, Placebo-Controlled Trial of Nimodipine for the Neurological Manifestations of HIV-1",
}
{
"clinical_study": {
"#rank": "381",
"#comment": [],
"required_header": {
"download_date": "ClinicalTrials.gov processed this data on March 18, 2015",
"link_text": "Link to the current ClinicalTrials.gov record.",
"url": "http://clinicaltrials.gov/show/NCT00001292"
},
"id_info": {
"org_study_id": "920106",
"secondary_id": "92-C-0106",
"nct_id": "NCT00001292"
},
"brief_title": "Study of Scaling Disorders and Other Inherited Skin Diseases",
"official_title": "Clinical and Genetic Studies of the Scaling Disorders and Other Selected Genodermatoses",
}
Your example documents are malformed - right now both clinical_study keys are part of the same object, and that object is missing a closing }. I assume you want them to be two separate documents, although you call them subdocuments. It doesn't make sense to have them be subdocuments of a document if they are both named under the same key. You cannot save the document that way, and in the mongo shell it will silently replace the first instance of the key with the second:
> var x = { "a" : 1, "a" : 2 }
> x
{ "a" : 2 }
If you just want to return the clinical_study part of the document when you match on clinical_study.#rank, use projection:
db.test.find({ "clinical_study.#rank" : "379" }, { "clinical_study" : 1, "_id" : 0 })
If instead you meant for the clinical_study documents to be elements of an array inside a larger document, then use $. Here, clinical_study is now the name of an array field which has as its elements the two values of the clinical_study key in your non-documents:
db.test.find({ "clinical_study.#rank" : "379" }, { "_id" : 0, "clinical_study.$" : 1 })