webRequest API not capturing all page requests from application - google-chrome

I am trying to download JSON data from a web application. The URL/API is static and I can use it to call the webpage that returns the data. There is a session variable parameter that needs to be added to the URL/API call to connect to the server and download the JSON data which is created when you launch the application, but times out if the application is not actively used. My current process is to open the developer tools, launch the web application and when the specific JSON request is made I copy the parameter value then add it to a script that mimics the page request and downloads the JSON data.
I am trying to avoid manually copying and pasting this session variable parameter. I want to be able to automatically capture the web request, parse out the value that I need, set a cookie on my machine and then pick up the cookie by a php script to initiate the JSON data download with the valid session value.
I have looked into creating an extension in chrome using the chrome.webRequest.onResponseStarted with the following code:
chrome.webRequest.onCompleted.addListener(function(details) {
console.log(details);
chrome.cookies.set(
{ url: "http://localhost/MySite/", name: "MyCookie", value: "Tested" }
);
}, {urls:["<all_urls>"]} );
This code works for the main web requests but it doesn’t pick up all the JSON data requests that are made by the application. The application is swf format which is most likely the problem, but I can see the requests in the Network Panel tab of the Developer Tools and they are captured using chrome://net-internals which that leads me to believe that I should be able to capture them somehow.
I have looked into chrome.devtools.network but I cannot seem to figure out how that is supposed to work. Any advice or direction would be greatly appreciated.

Related

HTML junk returned when JSON is expected

The following code used to work but not anymore and I'm seeing junk HTML with success code of 200 returned.
response = urlopen('https://www.tipranks.com/api/stocks/stockAnalysisOverview/?tickers='+symbol)
data = json.load(response)
If you open the page in chrome you will see the JSON file format. But when opened in python I'm now getting:
f1xx.v1xx=v1xx;f1xx[374148]=window;f1xx[647467]=e8NN(f1xx[374148]);f1xx[125983]=n3EE(f1xx[374148]);f1xx[210876]=(function(){var
P6=2;for(;P6 !== 1;){switch(P6){case 2:return {w3:(function(v3){var
v6=2;for(;v6 !== 10;){switch(v6){case 2:var O3=function(W3){var
u6=2;for(;u6 !== 13;){switch(u6){case 2:var o3=[];u6=1;break;case
14:return E3;break;case 8:U3=o3.H8NN(function(){var Z6=2;for(;Z6 !==
1;){switch(Z6){case 2:return 0.5 - B8NN.P8NN();break;}}.....
What should I be doing to adapt to the new backend change so that I can parse the JSON again.
It is a bot protection, to prevent people from doing what you are doing. This API endpoint is supposed to be used only by the website itself, not by some Python script!
If you delete your site data and then freshly access the page in the browser, you'll see it first loads the HTML page that you see which loads some JavaScript, which then executes a POST to another URL with some data. Somewhere in the process a number of cookies get set and finally the code refreshes the page which then loads the JSON data. At this point visiting the URL directly returns the data because the correct cookies are already set.
If you look at those requests, you'll see the server returns a header server: rhino-core-shield. If you google that, you can see that it's part of the Reblaze DDoS Protection Platform.
You may have luck with a headless browser like ghost.py or pyppetteer but I'm not sure how effective it will be, you'll have to try. The proper way to do this would be to find an official (probably paid) API for getting the information you need instead of relying on non-public endpoints.

Trying to retrieve JSON data from website api

Bottom line is that I'm trying to download my order history data from Lowes.com. I'm assuming there's an API endpoint that can be used but I haven't figured out the method or maybe what parameters need passed to get a successful request.
I looked at using python and scrapy but I don't believe the format of the webpages are going to be easy to parse the data.
I have found references to APIs in some of the javascript code for the website and the mobile app. Some of the relevant urls I've found:
From website -
ORDER_HISTORY_USER: '/wcs/resources/store/%0/member/%1/orderhistory/v1_0' (from function named Conduit on www.lowescdn.com)
From mobile app:
"url": "https://lwssvcs.lowes.com/IntegrationServices/resources/mylowes/user/order/list/v1_0"
"url": "https://lwssvcs.lowes.com/IntegrationServices/resources/mylowes/user/order/instore/v1_0"
So far I've only attempted to use these links via a browser after logging into my account. Assuming I'm on the right path, what method should I use to successfully get the data?

Caching API response from simple ruby script

I'm working on a simple ruby script with cli that will allow me to browse certain statistics inside the terminal.
I'm using API from the following website: https://worldcup.sfg.io/matches
require 'httparty'
url = "https://worldcup.sfg.io/matches"
response = HTTParty.get(url)
I have to goals in mind. First is to somehow save the JSON response (I'm not using a database) so I can avoid unnecessary requests. Second is to check if the new data is available, and if it is, to override the previously saved response.
What's the best way to go about this?
... with cli ...
So caching in memory is likely not available to you. In this case you can save the response to a file on disk.
Second is to check if the new data is available, and if it is, to override the previously saved response.
The thing is, how can you check if new data is available without doing a request for the data? Not possible (given the information you provided). So you can simply keep fetching data every 5 minutes or so and updating your local file.

Flash - How come I do not receive a security warning to allow cross-site requests?

I am using the URLLoader class to send data to a script that sends an email with that data; the URLLoader is also used to receive the return data on the status of the request. This email script is located on another domain (webscript.io, in this case; they host scripts written in LUA that can perform various functions when called). I don't want to have to get them to host a crossdomain.xml file, but when I visit the page to use the Flash app, I don't even get a security warning. Is that not the default action when there is cross-domain scripting in a Flash application without a crossdomain.xml file?
For your reference, here is the application: http://www.canadadocks.ca/build-dock-app/
The cross domain issues are specifically for data that your SWF is retrieving from other domains. So if you're sending data (via a HTTP POST/GET), then this is not a problem.
Also, unless you're using the debug player, you won't see the security exception that occurs.
There are also various things that will not trigger the exception: like downloading and displaying an image from another domain. However, if you then try to access the bitmap data of that image, you'll get the security exception (unless the crossdomain.xml permits it).
So it can depend on what you're retrieving and what you're doing w/the data you fetch.
If you're still curious, you should further explain what type of data you're retrieving and what you do with it. As usual, it's better to explain that with code than with words ;)

Retrieve URL JSON data in MS Access

There is a web service that allows me to go to a URL, with my API-key, and request a page of data. The data is returned as JSON. The JSON is well-formed, I ran it through JSONLint and confirmed its OK.
What I would like to do is retrieve the JSON data from within MS Access (2003 or 2007), if possible, and build a table from that data (first time thru), then append/update the table on subsequent calls to that URL. I would settle for "pre-step" where I retrieve this information via another means. Since I have an API key in the URL, I do not want to do this server-side. I would like to keep it all within Access, run it on my PC at home (its for personal use anyway).
If I have to use another step before the database load then Javascript? But I dont know that very well. I dont even really know what JSON is other than what I have read in Wikipedia. The URL looks similar to:
http://www.SomeWebService.com/MyAPIKey?p=1&s=50
where: p = page number
s = records per page
Access DB is a JavaScript Lib for MS Access, quick page search says they play nicely with JSON, and you can input/output with. boo-ya.
http://www.accessdb.org/
EDIT:
dead url; wayback machine ftw:
http://web.archive.org/web/20131007143335/http://www.accessdb.org/
also sourceforge
http://sourceforge.net/projects/accessdb/