twitter geocode to google maps - google-maps

Hi
Im trying to make this tutorial to work but Im having some difficulties.
http://woorkup.com/2009/12/26/how-to-play-with-google-maps-and-twitter-api/
When I run the loadFromTwitter function I get a javascript error. It says "invalid label".
The code so far is:
loadFromTwitter = function(lat,lng,r) {
var url = 'http://search.twitter.com/search.json?geocode='
+lat+'%2C'
+lng+'%2C'+r
+'km&callback=manage_response';
var script = document.createElement('script');
script.setAttribute('src', url);
document.getElementsByTagName('head')[0].appendChild(script);
}
What am I doing wrong?

It's an error in the tutorial page, Francesca has somehow double-encoded the special characters. The %2c's should be commas and the & should be &.
loadFromTwitter = function(lat,lng,r) {
var url = 'http://search.twitter.com/search.json?geocode='
+lat+','
+lng+','+r
+'km&callback=manage_response';
var script = document.createElement('script');
script.setAttribute('src', url);
document.getElementsByTagName('head')[0].appendChild(script);
}

(I wanted to write this as a comment, but there wasn't enough room.)
Although that change gets you the info from Twitter, doing anything useful with it on a Google Map is rather trickier than suggested in that tutorial.
As mentioned in that tutorial, Twitter doesn't tell you the lat/lng values where it thinks the tweet came from, so you have to geocode the tweet.location yourself. You then discover that many tweet.location values are garbage. Presumably tweeters can write anything in that field.
For the tweets that do have valid locations, most of them are only accurate to the nearest city. You get lots of tweets in each city all of which geocode to exactly the same location at the centre of the city. There's no nice easy way to deal with that in Google Maps. It always ends up being very messy.
Also, you tend to get more tweets in the reply than the GClientGeocoder limit of calls per second, so you have to add delays between the geocode calls and add code to handle error 620.

Related

Obtain list of My Places from Google Maps

I am trying to obtain the list of places the user has saved on Google Maps. Now I know there isnt an API for this (for whatever reason), but I saw here:
"My Places" Google Maps API
That apparently there used to be a way to obtain the URL, but it does not seem to work with my list of places.
E.g.
https://www.google.com/maps/#46.889424,0.1194148,6z/data=!4m3!11m2!2s1KbZtik1IdXyNhwfXEb3P9vaZvzU!3e3
Does not seem to work if I append &output=kml or &output=json
I created this list on Google Maps, then hit share and obtained that link.
I even tried parsing the resulting HTML but it seems everything is handled by some Javascript Engine and I can't find any reference to Google Ids there --- I dont even know how they handle clicks!
Any help? There must be a way to retrieve this information programmatically!
EDIT:
I managed to get something working by visiting the shared link, then processing the html and storing the window.APP_INITIALIZATION_STATE variable. I then convert it to an javascript array and loop over it. Deep inside the array/map structure, I managed to get the google name and google place id out of that array. That seems to work a bit, but when trying with lists over 20 items long, google only gets the first 20 and is waiting for the user to 'scroll down' to get the next 20. That seems to trigger another call to get the next 20 results and looks a bit like:
https://www.google.com/search?tbm=map&fp=1&authuser=0&hl=en&gl=nl&pb=!4m8!1m3!1d54065472.4384380........
I can see the original feature id being included at the end of the url, but have no idea how to construct this url in full though to get the next 20 items.... Any ideas?
Your saved places list actually has what you call a feature ID attribute, this isn't a common practice and Google frowns upon this technique but take a look at this URL:
https://www.google.com/maps/preview/entity?authuser=0&hl=en&gl=us&pb=!1m10!1s0x0%3A0x3743ae09a161976b!3m8!1m3!1d14318.72623152007!2d-98.2296425!3d26.2070353!3m2!1i1024!2i768!4f13.1!12m3!2m2!1i392!2i106!13m57!2m2!1i203!2i100!3m2!2i4!5b1!6m6!1m2!1i86!2i86!1m2!1i408!2i200!7m42!1m3!1e1!2b0!3e3!1m3!1e2!2b1!3e2!1m3!1e2!2b0!3e3!1m3!1e3!2b0!3e3!1m3!1e8!2b0!3e3!1m3!1e3!2b1!3e2!1m3!1e9!2b1!3e2!1m3!1e10!2b0!3e3!1m3!1e10!2b1!3e2!1m3!1e10!2b0!3e4!2b1!4b1!9b0!14m3!1snyc5W-WeHY3r5gLwkoRI!7e81!15i10112!15m19!2b1!5m4!2b1!3b1!5b1!6b1!10m1!8e3!14m1!3b1!17b1!24b1!25b1!26b1!30m1!2b1!36b1!52b1!53b1!21m28!1m6!1m2!1i0!2i0!2m2!1i458!2i768!1m6!1m2!1i974!2i0!2m2!1i1024!2i768!1m6!1m2!1i0!2i0!2m2!1i1024!2i20!1m6!1m2!1i0!2i748!2m2!1i1024!2i768!22m1!1e81!29m0!30m1!3b1
Highlighted is the feature ID from the link you posted:
https://www.google.com/maps/#46.889424,0.1194148,6z/data=!4m3!11m2!2s1KbZtik1IdXyNhwfXEb3P9vaZvzU!3e3
Along with other maps parameters; when you hit that link you're actually manually triggering the same callback that Google's own scripts in maps use to parse the data to feed back to the maps UI; if you look at array item 2, or {c:..} you'll find a stringified array with the contents of your list, now depending on the program language you're using all it takes is a little tweaking (find/replace, loop through, lint and trim, etc.) to this array and you can pull your results; the cool thing is if you add or remove a place the next time you hit that end point it's updated in real-time.
Some people may call it a "hack"; but it gets the job done. :)
Hope I pointed you to a direction in the event you haven't found a solution; give this a shot.
Note the URL has to be pasted in its entirety, SO truncated the hyperlink; copy and paste the whole thing in one shot and a text file from Google with the arrays will be produced; in my case I curl the URLs I need and parse the returned strings as needed to pull data from Google where their API has limitations. Just a tip. :)
Also check Joel's Answer who did some research and refined some of the following information.
Pagination
You can use this tool to decrypt the pb-parameter. PB stands for protocol buffer (protobuf) and Google uses its own kind of it for maps. You can find different decoders for this by googling it.
In my case, the pagination was done via one parameter (8iX0). It seems, that it always comes with another similar parameter (7i20) but I don't know that it does. I can't yet confirm that this is always the case, but from my experience you're basically looking for two integers that are 20/40/60 etc. apart.
Here's what this looks like for me:
page 2 (7i20, 8i20)
page 3 (7i20, 8i40)
page 4 (7i20, 8i60)
From this information, I tried 7i20 8i00 for page 1, that seemed to work. For lists with >100 items, it just continues like that (8i120, 8i140 etc.)
Here's a code snippet in python (quick & dirty). Make sure to add (long) delays if your list has many pages as you will get rate-limited by captchas eventually if you don't. Notice the 8i%s0 in the url, make sure to put the %s back when you paste your pb-block.
url = "https://www.google.com:443/search?tbm=map&pb=!7i20!8i%s0!..."
headers = {"Referer": "https://www.google.com/"}
def fetch_stops_from_maps():
new_results = -1
page = 0
results = []
while new_results != 0:
new_results = 0
x = requests.get(url % page, headers=headers)
txt = html.unescape(x.text)
txt = txt.split("\n")[1]
results = re.findall(r"\[null,null,[0-9]{1,2}\.[0-9]{4,15},[0-9]{1,2}\.[0-9]{4,15}]", txt)
print(len(results))
for cord in results:
# curr = the description you can manually type in when saving
curr = txt.split(cord)[1].split("\"]]")[0]
curr = curr[curr.rindex(",\"") + 2:]
cords = str(cord).split(",")
lat = cords[2]
lon = cords[3][:-1]
results.append(s)
new_results += 1
page += 2
Actually getting the correct url
Getting the correct url currently seems to be the hardest part when doing this and I have not fully figured this out aswell. However, for my use-case this is not really important, so I extracted the correct pb-block once and called it a day.
As explained in the other answers, the id of the list is visible in the basic url (here, the 2sXX...) when you navigate to the list in your browser. It seems to usually be 24-32 (?) characters long.
.../maps/<coords>/data=!4m3!11m2!2sXXXX...XXXX!3e3
If you have this id, you can put it into an existing protobuf-block and it may work (I only tested this with 3 different lists, which were all created by the same account, so this theory is far from proven).
Now, how do you get the block? I would just share the one I have, but because I only understand parts of what it does, I fear that it may contain some personal info. Instead, I will share my process of getting it. For this I use Burpsuite. It's a program mainly used for web-security testing and has a free community edition, however for our use-case it is the perfect tool, because with it you can easily tinker with requests, change small parts in the request, send it again and immediately see if your changes changed the response. However for extracting the pb-block, one should also be able to use any program that can intercept browser traffic.
Heres the basic rundown with burp:
From GMaps, share a list that has >20 items (this is important) and copy the public link
In Burp, go to the tab "Proxy", make sure "Intercept" is off and click "Open browser" to open the integrated chromium browser
There, paste the link and wait until maps loaded completely
In Burp, turn "Intercept" on, then in google maps, scroll down in the list, until it starts loading new results (always blocks of 20)
Burp now intercepted all requests the browser made since you turned intercepting on. Click "Forward" and go through all requests, until you see a request in the format
GET /search?tbm=map&authuser=0&hl=de&gl=de&pb=!7i20....
This is what you're looking for.
Optionally, you can now right click into the request-text and click "send to repeater", then switch to the repeater-tab. Here you can edit the request and then send it again, being able to see the response immediately. For example, removing the authuser, hl, gl, q, ech, psi url parameters, the request still works flawlessly. If you remove the tch=1 parameter, the response you get will be in a more human readable format.
In the request-text you should now be able to just search for the list-id you got from the link previously and replace it with the id of another list (search bar is at the bottom in burp). As I said, this worked for me, but it may be possible that the pb-block contains some additional metadata that makes lists from different google-accounts or different types of lists incompatible with specific pb-blocks. Just a theory though. Let me know how it goes!
Further automating
I have theorised that one could automate getting the pb-block using requests-html because it can load html-sites fully but it doesn't get updated anymore. Another option (probably the better one) is Selenium Wire, as you should be able to load the page and intercept the requests, like we did in burp. Seems like a whole lot of work tho :D
This was the only API was able to find was this:
https://www.google.com/bookmarks/?output=xml
Used in a browser you would have to first log in through Google's OAuth. It would then return your saved places. Not sure at the moment how you would embedded the authentication to do this programmatically, but this might send you in the right direction.
I was able to extract the data I needed from my google maps list. Below are some comments that expand on some of the other comments here, along with a script that extracts all of the relevant data points from the network response.
Obtaining the underlying URL
You can easily find this URL by just opening the devtools on your browser, going to the network tab, and refreshing the webpage or scrolling down on the list until it loads new results (the list must be larger than 20 results). You should be able to find the network request that starts with https://www.google.com/search?tbm=map&pb... and go from there.
Increase the results size
I was able to increase the number of results returned from the request by changing the value of the 7i20 parameter. From what I can tell, the 71XX parameter is the size of the page, and the 8iXX parameter is the starting point. I haven't tested how large you can make the page limit, but I tested 100 and it seemed to work fine. This should make dealing with larger lists much easier.
Parsing out the data
Instead of using regex to parse out the relevant data from the response, I found that the response is basically just a massive JSON object and I was able to identify the indexes for specific types of data, such as the name of the place, location, notes, etc. See the script below.
If you look at the buildResults function in the script below, you can see the exact indexes used to extract specific pieces of information. This of course may change over time if the network response changes format at all, so use these as a starting point in the case where the specific values aren't at those indexes anymore. Hopefully they would be close to those locations
Script to parse the data (javascript / node.js)
// Insert the raw text content from the network response from the
// https://www.google.com/search?tbm=map&pb... url below.
const rawInput = null
function prepare(input) {
// There are 5 random characters before the JSON object we need to remove
// Also I found that the newlines were messing up the JSON parsing,
// so I removed those and it worked.
const preparedForParsing = input.substring(5).replace(/\n/g, '')
const json = JSON.parse(preparedForParsing)
const results = json[0][1].map(array => array[14])
return results
}
function prepareLookup(data) {
// this function takes a list of indexes as arguments
// constructs them into a line of code and then
// execs the retrieval in a try/catch to handle data not being present
return function lookup(...indexes) {
const indexesWithBrackets = indexes.reduce((acc, cur) => `${acc}[${cur}]`, '')
const cmd = `data${indexesWithBrackets}`
try {
const result = eval(cmd)
return result
} catch(e) {
return null
}
}
}
function buildResults(preparedData) {
const results = []
for (const place of preparedData) {
const lookup = prepareLookup(place)
// Use the indexes below to extract certain pieces of data
// or as a starting point of exploring the data response.
const result = {
address: {
street_address: lookup(183, 1, 2),
city: lookup(183, 1, 3),
zip: lookup(183, 1, 4),
state: lookup(183, 1, 5),
country_code: lookup(183, 1, 6),
},
name: lookup(11),
tags: lookup(13),
notes: lookup(25,15,0,2),
placeId: lookup(78),
phone: lookup(178,0,0),
coordinates: {
long: lookup(208,0,2),
lat: lookup(208,0,3)
}
}
results.push(result)
}
return results
}
const preparedData = prepare(rawInput)
const listResults = buildResults(preparedData)
console.log(listResults)

Google Maps API results vs place results return different names

I am using google maps to provide directions to multiple locations within a website. Users are in Japan, but are non-Japanese, so results should be in English.
In certain examples, even when the name is in the query parameter, a link like this location, returns an alternate Japanese place name (主教座聖堂牧師館), instead of "St. Andrew's Tokyo."
This link is dynamically generated, so I can change the parameters if need be, but I can't figure out how to force results that look more like this, without hardcoding the entire link. Here is what builds the URL:
//handle directions links; send to Apple Maps (iOS), or Google Maps (everything else)
var iOS = !!navigator.platform && /iPad|iPhone|iPod/.test(navigator.platform);
$body.on('click', 'a.tsml-directions', function(e){
e.preventDefault();
var directions = (iOS ? 'maps://?' : 'https://maps.google.com/?') + $.param({
daddr: $(this).attr('data-latitude') + ',' + $(this).attr('data-longitude'),
saddr: 'Current Location',
q: $(this).attr('data-location')
});
window.open(directions);
});
I've had a look at your sample URL https://www.google.com/maps?daddr=35.6603676,139.7444553&saddr=Yotsuya,%20Shinjuku,%20Tokyo%20160-0004&q=St.%20Andrew%27s%20Tokyo.
I understand that your intention is getting a directions on Google Maps. In the aforementioned URL you specify parameters for origin saddr and destination daddr, the q parameter shouldn't affect directions in this case. So, the destination address is just coordinate 35.6603676,139.7444553. When I reverse geocode this coordinate I get the 'Japan, 〒105-0011 Tōkyō-to, Minato-ku, Shibakōen, 3 Chome−6−18 主教座聖堂牧師館' address as shown in Geocoder tool:
https://google-developers.appspot.com/maps/documentation/utils/geocoder/#q%3D35.660368%252C139.744455
The 主教座聖堂牧師館 corresponds to premise address component and I suspect it is not translated to English in Google database, because web service call with language set to English returns this component in original language as well
https://maps.googleapis.com/maps/api/geocode/json?latlng=35.6603676%2C139.7444553&language=en&key=YOUR_API_KEY
If your destination should be St. Andrew's, use it as a destination parameter.
And the most important part: Google has Google Maps URLs as an official, recommended and documented method to construct URLs. I would suggest using this API in order to create your directions URLs. These URLs are cross-platform, so there is no need to create different URLs for iOS, Android or web browser.
Your example will convert into
https://www.google.com/maps/dir/?api=1&origin=Yotsuya,%20Shinjuku,%20Tokyo%20160-0004&destination=St.%20Andrew%27s%20Tokyo&travelmode=driving
The result is shown in the screenshot
Your code might be something like
$body.on('click', 'a.tsml-directions', function(e){
e.preventDefault();
var directions = 'https://www.google.com/maps/dir/?' + $.param({
api: "1",
destination: $(this).attr('data-location'),
travelmode: "driving"
});
window.open(directions);
});
Note I don't specify origin parameter as it is optional and in this case it should be interpreted as my current location.
I hope this helps!

Need help parsing through a page & getting all comments found into an Array using a RegEx pattern in Google Apps Script?

My problem lies in 2 parts however I'm hoping solving 1 will fix the other. I've been trying to parse through a page and get all the comments found within a forum thread.
The comments are found using a RegEx pattern and the idea is that whatever lies in the comment will be read into an array until there aren't any more comments left. Each comment div follows this format
<div id="post_message_480683" style="margin-right:2px;"> something </div>
I'm trying to locate up to "post_message_[some number]" since each number seems to be generated randomly and then get whatever is between that particular div. My 1st problem is my RegEx just doesn't seem to be working I've tried a few but none yielded any results (except for when I insert the post message no. in manually), Here's the code so far:
function GetPosts() {
var posts = new Array(60);
var url = "http://forums.blackmesasource.com/showthread.php?p=480683";
var geturl = UrlFetchApp.fetch(url).getContentText().toString();
var post_match = geturl.match(/<div id="post_message_(.+)" style="margin-right:2px;">(\w.+)<\/div>/m);
Logger.log(post_match);
}
Edit: I initially tried getting this info via GAS's Xml.Parse() class but after grabbing the URL I just didn't know what to do since suffixing
.getElement().getElement('div') (I also tried .getElements('div') and other variations with 'body' & 'html')
would cause an error. Here is the last code attempt I tried before trying the RegEx route:
function TestArea() {
var url = "http://forums.blackmesasource.com/showthread.php?p=480683";
var geturl = UrlFetchApp.fetch(url).getContentText().toString();
//after this point things stop making sense
var parseurl = Xml.parse(geturl, true);
Logger.log(geturl);
//None of this makes sense because I don't know HOW!
//The idea: Store each cleaned up Message Div in an Array called posts
//(usually it's no more than 50 per page)
//use a for loop to write each message into a row in GoogleSpreasheet
for (var i = 0; i <= parseurl - 1; i++) {
var display = parseurl[i];
Logger.log(parseurl); }
}
Thanks for reading!
In general like the comment points out - be aware of parsing HTML with RegEx.
In my past personal experience, I've used Yahoo's YQL platform to run the HTML through and using XPath on their service. Seems to work decently well for simple reliable markup. You can then turn that into a JSON or XML REST service that you can grab via UrlFetch and work on that simplified response. No endorsement here, but this might be easier than to bring down the full raw HTML into Google Apps Script. See below for the YQL Console. I also don't know what their quotas are - you should review that.
Of course, the best course is to convince the site owner to provide an RSS feed or an API.

Changing the state of apps script webapp

I'm making standalone web app in Google Apps Script. I have somekind of task flow in the app. First search something from spreadsheet, then do some selections what to do with the collected data and after that fill spreadsheet row with some userinputs. So I need to run between few states.
I'm fairly sure I don't know enough about web tech so this might be very stupid question but here it goes.
I know that the e.parameters comes from the url like this www.google.com&value=helloworld
, e.parameters.value returns helloworld.
So now the problem: How do I set parameter e values in doGet(e) and call this same page or even different page/script in same script project? In otherwords how do I call self&value=helloworld ?
Is there some other way to do this? In GWT I just store everything to database and keep session in cookies. But cookies are not allowed so how to keep the state of the webapp in google apps script?
EDIT: This is how I pass the parameters for doGet(e).
function doGet(e) {
var app = UiApp.createApplication();
var newValue = 0;
if(e.parameter == undefined || e.parameter.value == undefined){
newValue = 1;
}else{
newValue = 1+parseInt(e.parameter.value);
}
var link = 'https://script.google.com/a/macros/domain.com/s/<insertidhere>/dev';
var anchor = app.createAnchor('Next',link+'?&value='+newValue);
anchor.setTarget('_self');
app.add(anchor);
var label = app.createLabel(newValue);
app.add(label);
return app;
}
The link format has changed a bit so instead of concatenate &value=... you need to add ? first like this ?&value1=helloworld&value2=....
Failing to use ? led me to think that there is bug and I need to use old format for the link and using that old format forbit any other than the script owner using it.
Hopefully this helps somebody to solve similar problems.
You've almost answered yourself. Append the URL paramenters to the end of your web app's URL and you can access them in your script.
Let's say you have a URL like
http://script.google.com/.../exec
When you hit the URL
http://script.google.com/.../exec?value=helloworld
,
inside doGet, you can read these URL parameters as
e.parameter.value;
IMO - instead of using ? to separate multiple parameters try keeping all the values in a single anchor with some other character separator of your choice such as # since after the first ? its possibly ignoring the other one and everything afterwards. Would be great to hear back on what you find.

Trigger Google Apps Script by email

I'm looking for examples of a pattern where a demon script running within a GoogleAppsForBusiness domain can parse incoming email messages. Some messages will will contain a call to yet a different GAScript that could, for example, change the ACL setting of a specific document.
I'm assuming someone else has already implemented this pattern but not sure how I go about finding examples.
thx
You can find script examples in the Apps Script user guide and tutorials. You may also search for related discussions on the forum. But I don't think there's one that fits you exactly, all code is out there for sure, but not on a single script.
It's possible that someone wrote such script and never published it. Since it's somewhat straightforward to do and everyone's usage is different. For instance, how do you plan on marking your emails (the ones you've already read, executed, etc)? It may be nice to use a gmail filter to help you out, putting the "command" emails in a label right away, and the script just remove the label (and possibly set another one). Point is, see how it can differ a lot.
Also, I think it's easier if you can keep all functions in the same script project. Possibly just on different files. As calling different scripts is way more complicated.
Anyway, he's how I'd start it:
//set a time-driven trigger to run this function on the desired frequency
function monitorEmails() {
var label = GmailApp.getUserLabelByName('command');
var doneLabel = GmailApp.getUserLabelByName('executed');
var cmds = label.getThreads();
var max = Math.min(cmds.length,5);
for( var i = 0; i < max; ++i ) {
var email = cmds[i].getMessages()[0];
var functionName = email.getBody();
//you may need to do extra parsing here, depending on your usage
var ret = undefined;
try {
ret = this[functionName]();
} catch(err) {
ret = err;
}
//replying the function return value to the email
//this may make sense or not
if( ret !== undefined )
email.reply(ret);
cmds[i].removeLabel(label).addLabel(doneLabel);
}
}
ps: I have not tested this code
You can create a google app that will be triggered by an incoming email message sent to a special address for the app. The message is converted to an HTTP POST which your app receives.
More details here:
https://developers.google.com/appengine/docs/python/mail/receivingmail
I havn't tried this myself yet but will be doing so in the next few days.
There are two ways. First you can use Google pub/sub and handle incomming notifications in your AppScrit endpoint. The second is to use the googleapis npm package inside your AppScript code an example here. Hope it helps.
These are the steps:
made a project on https://console.cloud.google.com/cloudpubsub/topicList?project=testmabs thing?
made a pubsub topic
made a subscription to the webhook url
added that url to the sites i own, i guess? I think I had to do DNS things to confirm i own it, and the error was super vague to figure out that was what i had to do, when trying to add the subscription
added permission to the topic for "gmail-api-push#system.gserviceaccount.com" as publisher (I also added ....apps.googleusercontent.com and youtrackapiuser.caps#gmail.com but i dont think I needed them)
created oauth client info and downloaded it in the credentials section of the google console. (oauthtrash.json)