Asynchronous location lookup on GAE/webapp2 - json

Question

I do have got the following GAE datastore model (models/location.py) which I want to populate:
from google.appengine.ext import db
class Location(db.Model):
name = db.StringProperty(required=True)
country = db.StringProperty(required=False)
address = db.PostalAddressProperty(required=False)
coordinates = db.GeoPtProperty(required=False)
description = db.TextProperty(required=False)
To do so I've created a class LocationCreateHandler and a function _geocode (handlers/location.py):
from google.appengine.ext import db
from google.appengine.api import urlfetch
from webapp2_extras import json
import urllib
from handlers import BaseHandler
from models.location import Location
import logging
class LocationCreateHandler(BaseHandler):
def post(self):
name = self.request.get("name")
country = self.request.get("country")
address = self.request.get("address")
coordinates = _geocode(self, address)
description = self.request.get("description")
newLocation = Location(name=name, country=country, address=address, coordinates=coordinates, description=description)
newLocation.put()
return self.redirect("/location/create")
def get(self):
self.render_response("location/create.html")
def _geocode(self, address):
try:
logging.info("Geocode address: %s", address)
parameter = {'address': address.encode('utf-8'), 'sensor': 'false'}
payload = urllib.urlencode(parameter)
url = ('https://maps.googleapis.com/maps/api/geocode/json?%s' % payload)
logging.info("Geocode URL: %s", url)
result = urlfetch.fetch(url)
jsondata = json.decode(result.content)
location = jsondata['results'][0]['geometry']['location']
coordinates = '%s,%s' % (location['lat'], location['lng'])
logging.info("Geocode coordinates: %s", coordinates)
return coordinates
except:
return "0.0,0.0"
How would I make this asynchronous? At the moment the user would have to wait until the geocode lookup has finished.
Once I get this working I also plan to use _geocode() after updating a Location record.
I still have to figure out the _geocode part after "result =", seems to be a bug there to as I always receive 0.0,0.0.
-Luca.

Answer 1

Seems like a good use for task queues. When the POST arrives, kick off a task, passing it all the parameters for the Location entity. You can then finish the request and return to the client right away. The task can call _geocode, then create the Location entity with all the data.
Or, if you need to create the Location object in the request handler for some reason, you can do so in the POST handler, and pass the new entity's key to the task. When the task completes, it can fetch the entity and update it with the coordinates.
Also, to help determine why your urlfetch isn't working, here's a way to log the exception while still catching it:
import traceback
try:
...
except:
logging.exception(traceback.print_exc())

Answer 2

Thank you Jamie
I did now implement a GeocodeWorker (workers/googleapis.py) using task queues:
from handlers import BaseHandler
import urllib
import logging
from google.appengine.api import urlfetch
from google.appengine.ext import db
from webapp2_extras import json
from models.location import Location
class GeocodeWorker(BaseHandler):
def post(self):
address = self.request.get('address')
logging.info("Geocode address: %s", address)
parameter = {'address': address.encode('utf-8'), 'sensor': 'false'}
url = ('https://maps.googleapis.com/maps/api/geocode/json?%s' % urllib.urlencode(parameter))
logging.info("Geocode URL: %s", url)
result = urlfetch.fetch(url)
JSONData = json.decode(result.content)
location = JSONData['results'][0]['geometry']['location']
coordinates = '%s,%s' % (location['lat'], location['lng'])
logging.info("Geocode coordinates: %s", coordinates)
key = self.request.get('key')
logging.info("Geocode key: %s", key)
existingLocation = Location.get(db.Key(key))
existingLocation.coordinates = coordinates
existingLocation.put()
I also modified my location handler (handlers/location.py) to call the worker:
from google.appengine.api import taskqueue
from webapp2 import uri_for
class LocationCreateHandler(BaseHandler):
def get(self):
self.render_response('location/create.html')
def post(self):
name = self.request.get('name')
country = self.request.get('country')
address = self.request.get('address')
description = self.request.get('description')
newLocation = Location(name=name, country=country, address=address, description=description)
key = newLocation.put()
params = {'key': key, 'address': address}
taskqueue.add(url=uri_for('googleapis-geocode'), queue_name='googleapis', name=('googleapis-geocode-%s' % key), params=params)
return self.redirect('/location/create')
I did also create a queue.yaml file, defining the googleapis queue.
Also I removed the try: except: part. The google queue will automatically retry the operation and quit after a defined time.
Do you see any areas of improvements?
-Luca.