I am trying to retrieve data from a website for which the parameters that you need to define does not show up in the url ie. http://www.vmm.be/webrap/ibmcognos/cgi-bin/cognosisapi.dll?b_action=cognosViewer&ui.action=run&ui.object=%2fcontent%2ffolder[%40name%3d%27Water%27]%2ffolder[%40name%3d%27Afvalwater%27]%2freport[%40name%3d%27Individuele%20analyseresultaten%20per%20RWZI%27]&ui.name=Individuele%20analyseresultaten%20per%20RWZI&run.outputFormat=HTML&run.prompt=false&ui.backURL=%2fwebrap%2fibmcognos%2fcgi-bin%2fcognosisapi.dll%3fb_action%3dxts.run%26m%3dportal%2fcc.xts%26m_folder%3di5DDA04E5A00C4B6AB6DF44BB4FAD7CEC&p_RwziNr=51&run.prompt=false
how can I extract the data for different years and parameters in a programmatic way?
I am using matlab's urlread but since the data I want to import does not show up in the html-code (I have checked this with the Web Developer Toolbar in FireFox), nothing is being read in. I have no experience with websites, only matlab and c-programming, so I have no idea how the data can be shown in the browser if it is not showing up in the html-source code so could someone point me to the right direction on how to get this job done? Is it at all possible? I hope so because I will have to repeat this for around 500 measuring stations, each 10 years so I am not planning on copying the required data manually as I did before when I just needed one station.
it turns out it is not possible to do what I want in Matlab. I however did manage to get the required data programmatically by using a combination of Selenium with C# and chrome driver. It's slow, but it's working and I can do other stuff in the meanwhile so I can recommend it to everyone who is downloading data from servers in a tedious way.
Related
I'm embedding GMStudio game in browser using . I need to send some data to the game from site's frontend in JSON and to receive some data from the game in frontend to make consequent actions.
So, my idea before was to save data in cookies/localStorage and to get it in the game somehow, using HTTP functionality or DLL's. Also, I'd like to emit messages from the game using window.parent.postMessage and receive them in frontend correctly.
Alas, I did not find a way to implement this. I hope there's some consistent approach to this problem about which I do not know.
The backup plan is to use Game Maker http_post_string and web sockets to get user's data before game's start and to make frontend do something after game's ending. It's clumsy and insecure, however.
The standard approach is to make a JavaScript extension.
That is done by creating a blank extension, adding a blank JS file to it, defining the functions via the context menu on it, and then adding the implementations into the JS file. Then you'll be able to call them from GML side as per usual.
This way you can access LocalStorage\Cookies, transmit\receive data from JS backends, and overall mess with the runtime as you please (with various degrees of understanding required to access internal data).
My idea is to have an arduino board that will communicate with the browser.
I want the arduino board to react (eg. blink led) when user is connected to a certain website.
User inputs on the board ( eg. press button) will affect the browser (eg. close tab, switch tab).
I started learning and creating simple examples using chrome extensions tutorial.
However, since I am not myself a skilled programmer, I'd like to know if it is possible to achieve those things aforementioned.
How I imagine it right now it will be:
Chrome extension writes into a json. Arduino reads data from json -> blink led.
Arduino writes values in json. Chrome extension can automatically see changes in the file and react accordingly -> close tab( so, without having a user re-installing each time the extension).
Are this scenarios possible? Which would be the easiest way to achieve this?
I would recommend to take a look at firebase. That's a really easy real-time database with a lot of good tutorials (even some by Google).
Then you should use a library like this one on your Arduino.
(there are some examples to look at)
And in your chrome extension you can fetch the values from the database really easy again.
We are building an offline version of our online store.
This is for reps to take with them on a tablet when they are out. The reason it needs to be offline is because there will be no connection in a lot of the places and we aren't using tablets with 3/4g connectivity.We use Windows 8.1/RT based tablets.
Since php relies on a server and the tablets cannot be installed with xampp or the like, I have rebuilt the site using html and css.
So far this works and has some flexibility in it. The website is turned into an application using googles create application button in its tools (though technically it's not really an application). It has all the product info, pictures and videos the reps need. Another advantage to this is that the application and files are stored on a one drive cloud account shared to all the tablets. This way, I can update the app/website from my machine and have it up to date on all the reps machines. This current set up works for now but we are looking to add some more functionality.
What we want is a button on each product that will let a rep add an amount of that particular product to a quote form. Because each product sits on a different page, it can't be in the one page form. So as the rep is pressing the buttons on each product, they are getting stored somewhere. Then at the end, the rep can turn all those values into a word doc/pdf/excel by hitting a final submit button.
I have looked at web storage in html 5 but still not quite sure if I can get what I need using it.
Going through the explination here:
http://diveintohtml5.info/storage.html
looks like sites can store info but not sure how to turn this into a form or document at the end. This document is what the reps will email back to the head office.
Has anyone got any pointers on what I could do. Since the site/app has been created in html already, I would like to just build into the existing framework. Are there any other pieces of software that I could use? I do remember using spreadsheet converter to turn an excel file into a web form that exported a pdf but the form needs to be on a single page.
All help appreciated.
Thanks
I currently have a swing Application with a WebView panel(via some javafx in swing). The WebPanel interfaces with the application itself by using 'invalid' protocols, keywords and hooks, and basically is not distinguishable from the rest of the UI.
All of that information is relevant, because on intial load, the webpage has to load, which can take a bit of time depending on server and client side variables.
What I need to do is have a way to save an offline copy of the webpage, and display it immediately on load. Since this is more critical and will not follow the rules of the actual underlying cache, I think it would be better if I manually did this instead.
Any information to help out here would be appreciated, like the best way to go about saving an offline copy of a website. I am hoping their is an easy way to achieve this without scanning the html for other references, recursing into all of these references, and reassigning the address to a local one, which seem a bit much for what I am trying to achieve.
The reason for the web based ui is because this portion of the UI is regularly updated, and gives the greatest freedom without pushing unnecessary program updates.
Background:
I check my grades weekly online for school. I then use print-screen and ms paint to save images of the website with my grades. (I know this is really time consuming and inefficient, but my computer has connectivity problems with my printer and it doesn't recognize any PDF makers)
Possible Solution:
Write some program or something that automatically:
opens a browser
goes to the website
logs in
opens each individual subject
creates an image (idc what process it uses to do so)
Thanks E'rybody
Don't use a browser. send Http requests in perl or something like that.
You can simply parse the HTML instead of grabbing screenshots.
Alternatively ask the school to provide your grades via email or on bloddy paper?
Personally I'd go with curl from the command line and then parse the html you get back:
curl http://website.com/grades
However, if you need to do it through a web browser (they might have some funky javascript) then there's a framework called selenium which might be usable: http://seleniumhq.org/
It's generally used for testing but can be used for your scenario. It's got clients for lots of different languages too.