Support needed for web scraping to Tableau - html

Theres a site which is daily updating. I need to get it directly to tableau and it need to refresh automatically. I have tried google sheets but it gives me an error saying please try again. No luck at all. Is there any way to upload the table in the below link to tableau which is refreshing daily?
Its only showing 25 rows. I need to get all the rows there.
https://www.cse.lk/pages/trade-summary/trade-summary.component.html
Can you anyone please help ?

Try to find the link for the api of the particular site. Do a python web scraping using requests and you can do the task easily

Related

Program/tool for reading/logging http requests and responses?

I'm trying to log autocomplete responses from a gmail "people" server, and I'm trying to find a program or tool to do this, like the network section in chrome dev tools. I tried using a web crawler script, but authentication would be a mess. I also tried Wireshark, but it was all garble to me and probably the wrong way to go. Is this even possible? Can somebody help me, I'm totally out of my comfort zone here. please see attached image
It would be best to find an API that google makes public for the data that you need and use that to query the data. For people, google does have an API. Have a look here: https://developers.google.com/people
Tools commonly used to make HTTP queries are https://www.postman.com/ and https://curl.haxx.se/

Google drive preview "/image" API doesn't work anymore?

Few days ago I was able to make request to:
https://drive.google.com/a/{{domain}}/file/d/{{docDriveId}}/image?pagenumber=1&w=400
Via this API I was able to get image preview on multi-page documents.
Seems it's there, but times-out after some time. It doesn't work anymore. Does anyone know what happened? It seems there's no documentation on this API.
On the other hand, I am aware of the thumbnailLink URL when getting document object via API, but this will only get 1st page preview.
Any alternative solutions to this?
Thanks a lot,
M
That URL is not a part of the official Drive API and is not guaranteed to remain stable. The Drive API doesn't include the ability to fetch per-page image previews of documents, but you may want to investigate using the embedLink as an alternative: https://developers.google.com/drive/v2/reference/files

sites.google - anyway to harvest the data

We have a shiny new Google Site that tracks the position of cyclists over a 1200km event.
The issue is that the content in the list needs to go into a mySQL database and I wondered if anyone had managed to loop through or transfer the content into a csv or similar from the list with test rider. I can see that working through json might be able to do this but I am concerned that this approach could fail.
I've worked a little in Google scripts for spreadsheets but not entirely sure where to start or even where to look. The site is at...
https://sites.google.com/site/audaxsm2013progress/sydney-canberra-day-1
Start with sitesApp in apps script and come back if your siteApp code is not working..
https://developers.google.com/apps-script/reference/sites/sites-app

Google Picker shows just a blank page

I'm trying to develop javascript application which would allow user to select files from their Google drive,and then pass IDs of selected files to PHP backend and download them to my server.
For selecting files I'm using Google Picker, and I could use it normaly for a few days but suddenly It stopped to work. When i click and try to open Picker dialog, loadPicker() function pass without any error reported by firebug but i get only blank iframe. I have tested it on several computers with several various browsers, all with the same result
First I was thinking that I've put wrong API key, but every time I try to load picker that attempt is properly loged inGoogle API Console statistic pages, altrought every time i get just a blank screen. I have also tryed to make a new application an just copy/paste Google Picker hello world app source code from Google documentation, but I'm still getting just a blank page.
Any suggestions why this is happening? Is it possible that Google blocked Picker service for my account?
So as disscussed at google groups thread it was a bug on Google side, it should'nt happen anymore.
I've had that problem too, while the picker was working correctly few days ago. I found the reason : in the meantime i had to create a Service Account key for other purposes (access to Google Apps Script apis). Using the clientId of the Web Application Key was causing a picker window with title and buttons but "No files found" inside, while my folder parent Id didn't changed.... I replaced the clientId by the one of the Service Account key, and it solved the issue. Good intuition.
I recently had the same problem. I can't beleive I spent almost two hours trying to figure the issue. In case you've tried everything expect closing the developer tools, please do that.
The file picker won't show unless the developer tools is closed.

ScriptDb and Google Visualization API

I have a web app that data is submitted from a Ui embedded in a Google Site and saved in a spreadsheet. I am thinking about switching to scriptDB instead, but I am unsure about querying and displaying data dynamically. The last time I checked it was not possible to refresh a sites page from within a script, so as a work around I wrote a .html page with javascript that accesses the spreadsheet via the key and uses the Visualization DataTables to produce a quick and nicely formatted query. I would like to retain this functionality if possible. Could anyone make some possible suggestions?
Regards,
Shawn
This is doable today via UiApp (see the Charts api, and note that you can add a Chart to a UiApp).
To auto refresh:
You can use CheckBox.setValue(value, fireEvents) - but you may hit quota issues (there is an example somewhere around). If you hit quota issues, then I suspect HTML service is your best option. It also gives you better control.
Here is an open item requesting expanded Charts service integration with Gviz.
Although, there is limited integration at present, it needs to be expanded to group variables in tables to facilitate basic count and sum operations.
I suggest you “star” the issue to vote for it and to be notified of new developments.