I am trying to intercept a website - https://www.kroger.com/pl/chicken/05002. In the chrome network tab, I see the request as below, with the details of the products nicely listed as JSON
I copied the cURL as bash and imported it as raw text in Postman. It ran forever without any response. Then I used the intercept feature and still it is running forever.
When both the requests are exactly same, why is it running in Chrome and not in Postman? What am i missing? Any help is appreciated, thanks in advance.
This is probably happening because they don't want you to do what you are trying to do. Note the "filter.verified" param in the URL.
You may want to try reaching out to them for an external API token - especially if you are creating an app or extension to compare competitive prices with the intention of distributing said app or extension - regardless of if it is for financial compensation or not.
Ethically questionable workaround (which would defintely need to be improved upon - this is simply an example of how you could solve your problem...):
GET https://www.kroger.com/search?query=chicken&searchType=default_search&fulfillment=all
const html = cheerio(responseBody);
var results = [];
html.find('div[class="AutoGrid-cell min-w-0"] > div').each(function (i, e)
{
results.push({
"Item": e.children[e.children.length-3].children[0].children[0].children[0]["data"],
"Price": e.children[e.children.length-4].children[0].attribs["value"]
})
});
console.log(results);
If you are unable to obtain an API token from them, this would probably be a legal way to accomplish what you want.
Related
Please point me on the right path. I use angular for the client and I get my data with:
private getOrders(query: {}) {
return from(this._feathers.service('orders').find({ query }));
}
this works great and I get an observable in return.
But I don't know how to get the messages on the client side.
For instance, the app template in channels.ts mentions something like this:
app.service('messages').publish(() => {
return [
app.channel(`userIds/${data.createdBy}`),
app.channel(`emails/${data.recipientEmail}`)
];
});
Well, how can I get the data from the client for emails/${data.recipientEmail}?
What is the syntax?
Things are actually pretty straight forward!
My confusion was that I had experience with socket.io and I created a channel per each service I had back then.
In feathers, the channels are a means of sending the data. So every service can post to any channel and it all depends to which channel you subscribe to!
On the client side I had the confusion of how I can access that data thinking that I need a 'custom' channel name and method. This actually does not matter as the clients are subscribed to different channels and in the end data does come from a service and method in that service!
I hope this makes sense and clears up confusion for people that are like me. :)
This article clears things up: https://blog.feathersjs.com/feathersjs-channel-subscriptions-647c771ca6c8
I've been researching this and cannot find or understand some of the solutions so i'm hoping to get some help here. I'm using Asp.net and building an application that needs to use a bible api. I like the two listed in the question. Every time I call esvapi it comes back successful, but I cannot view the data. I get an error in the console.
XMLHttpRequest cannot load http://www.esvapi.org/v2/rest`/passageQuery?key=8834092f0c58fcda&passage=James2. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost:59324' is therefore not allowed access.`
I've seen other with this error and I have questions.
If I'm understanding this correct I get this because the server is preventing me from seeing the data for security purposes. Maybe even the browser( this is not just a chrome issue) problem. So if I need to add a info to the response header from Angularjs to stop this how is that done. Anyone with experience?
Would I need to contact anyone to be able to prevent the server from responding this way...I doubt this, but thought I would ask. I already have valid api key.
the bible.org website api key is confusing to apply to my code. on esvapi i just add a header with key: "keypass" and I only have the CORS issue. But with bible.org I can't figure out how to implement the api key and password. see below... Do I say token:key: username. If i put the api in the browser I get a popup to add username and password. the username is my key and the password is ignored. I tried putting in username as key, but that didn't cut it. Regardless I need to fix the CORS issue and add info to response headers to see response data.
$scope.search = function() {
return $http.get("http://www.esvapi.org/v2/rest/passageQuery?&passage=" + $scope.bo + $scope.chap, {
headers: {
"key?token?orusername?": "",
///thought i saw someone do this...don't know if this is right
"Access-Control-Expose-Headers": "Content-Disposition",
}
}).success(function (data, status, headers, config) {
$scope.book = data.Book;
$scope.chapter = data.Chapter;
$scope.output = data.Output;
}).error(function (data, status, headers, config) {
$scope.message = "Oops... something went wrong";
});
Any input would be helpful. Thanks!
I actually have a bible api working...just a version that I don't like and there is not another version on that webites api.
Change the get $http.get call to $http.jsonp and hope it works. You're using cross-site scripting. Sometimes you can get away with a JSONP call in these cases and sometimes you can't.
I have a UI where the same image URL could be requested by several UIImageViews at varying times. Obviously if a request from one of them has finished then returning the cached version works as expected. However, especially with slower networks, I'd like to be able to piggy-back requests for an image URL onto any currently running/waiting HTTP request for the same URL.
On an HTTP server this called request combining and I'd love to do the same in the client - to combine the different requests for the same URL into a single request and then callback separately to each of the callers). The requests for that URL dont happen to start at the same time.
What's the best way to accomplish this?
I think re-writing UIImageView+AFNetworking might be the easiest way:
check the af_sharedImageRequestOperationQueue to see if it has an operation with the same request
if I do already have an operation in the queue or running then add myself to some list of callbacks/blocks to be called on success/failure
if I don't have the operation, then create it as normal
in the setCompletionBlockWithSuccess to call each of the blocks in turn.
Any simpler alternatives?
I encountered a similar problem and decided that your way was the most straightforward. One added bit of complexity is that these downloads require special credentials and so must go through their own operation queue. Here's the code from my UIImageView category to check whether a particular URL is inflight:
NSUInteger foundOperation = [[ConnectionManager sharedConnectionManager].operationQueue.operations indexOfObjectPassingTest:^BOOL(AFHTTPRequestOperation *obj, NSUInteger idx, BOOL *stop) {
BOOL URLAlreadyInFlight = [obj.request.URL.absoluteString isEqualToString:URL.absoluteString];
if (URLAlreadyInFlight) {
NSBlockOperation *updateUIOperation = [NSBlockOperation blockOperationWithBlock:^{
[[NSOperationQueue mainQueue] addOperationWithBlock:^{
self.image = [[ImageCache sharedImageCache] cachedImageForURL:URL];
}];
}];
//Makes updating the UI dependent on the completion of the matching operation.
[updateUIOperation addDependency:obj];
}
return URLAlreadyInFlight;
}];
Were you able to come up with a better solution?
EDIT: Well, it looks like my method of updating the UI just can't work, as the operation's completion blocks are run asynchronously, so the operation finishes before the blocks are run. However, I was able to modify the image cache to be able to add callbacks for when certain URLs are cached, which seems to work correctly. So this method will properly detect when certain URLs are in flight and be able to take action with that knowledge.
Good Evening!
I've been looking into the possibility of using GAS(Google Apps Script) to host a small bit of javascript that lets me use the new Google finance apps api. The intention being that I'll be using the stock information for a project which involves the use of stock data. I know that there are a few ways to get stock information from Google, but the data that the finanace app returns is more in-line with other sources we are using. (One constraint on this project is that we have multiple sources).
I've written the javascript and I can call a httpc:request to the URL for the script given to me from Google. In the browser the JS returns the json object as I want it, however when the call is made from Erlang I'm getting it in a list of ascii. From checking the values it appears to be a document starting like:
Below is the javascript and the url to see the json:
https://script.google.com/macros/s/AKfycbzEvuuQl4jkrbPCz7hf9Zv4nvIOzqAkBxL1ixslLBxmSEhksQM/exec
function doGet() {
var stock = FinanceApp.getStockInfo('LON:TSCO');
return ContentService.createTextOutput(JSON.stringify(stock))
.setMimeType(ContentService.MimeType.JSON);
}
For the erlang, it's a simple request but I've not been doing erlang long, so perhaps I've messed something up here (The URL being the one mentioned above). I've got crypto / ssl / inets when I'm testing this on the command line.
{ok, {Version, Headers, Body}} = httpc:request(get, URL, []}, [], []).
I think it's also worth mentioning that when i curl it from Cygwin, I get a massive load of HTML also, I've included it below, but if you see it you'll thank me for not posting it in here! http://pastebin.com/UtJHXjRm
I've been updating the script as I go with the new versions but I'm at a bit of a loss as to why it's not returning correctly.
If anyone can give me any pointers I'd be very grateful! I get the feeling that it's not intended to be used this way, perhaps only within other Google products and such.
Cheers!
It would be necessary to review how are you deploying the Web App, specifically the Who has access to the app, to access without authentication should be configured as shown in the image:
See Deploying Your Script as a Web App from the documentation.
In my test, by running:
curl -L https://script.google.com/macros/s/************/exec
Get the following result:
{
"priceopen":358,
"change":2.199981689453125,
"high52":388.04998779296875,
"tradetime":"2013-10-11T15:35:18.000Z",
"currency":"GBX",
"timezone":"Europe/London",
"low52":307,
"quote":357.8999938964844,
"name":"Tesco PLC",
"exchange":"LON",
"marketcap":28929273763,
"symbol":"TSCO",
"volumedelay":0,
"shares":8083060703,
"pe":23.4719295501709,
"eps":0.15248000621795654,
"price":357.8999938964844,
"has_stock_data":true,
"volumeavg":14196534,
"volume":8885809,
"changepct":0.6184935569763184,
"high":359.5,
"datadelay":0,
"low":355.8999938964844,
"closeyest":355.70001220703125
}
Possibly your GET is not following the REDIRECT that happens when you use contentService. Look at the html returned there is a redirect in there.
I want to create some kind of AJAX script or call that continuously will check a MySQL database if any new messages has arrived. When there is a new message in the database, the AJAX script should invoke a kind of alert box or message box.
I’m not quite a AJAX expert (yet anyway) and have Googled around to find a solution but I’m having a hard time to figure out where to begin. I imagine that it is kind of the same method that an AJAX chat is using to see if any new chat-message has been send.
I’ve also tried to search for AJAX (httpxmlrequest) call in a continuously and infinity loop but still haven’t got a solution yet.
I hope there is someone, which can help me with such a AJAX script or maybe nudge me in the right direction.
Thanks
Sincerely
Mestika
Step 1 - You need a server-side page that you can call that checks to see if something new has arrived.
Step 2 - You could adapt the sequential AJAX request script from here (it uses jQuery to simplify the AJAX requests):
http://www.stevefenton.co.uk/Content/Blog/Date/201004/Blog/AJAX-and-Race-Conditions/
Currently, this script is for queuing a list of sequential AJAX requests, but you could use it to continually check by changing it like this...
var InfiniteAjaxRequest = function (uri) {
$.ajax({
url: uri,
success: function(data) {
// do something with "data"
if (data.length > 0) {
alert(data);// Do something sensible with it!
}
InfiniteAjaxRequest (uri);
},
error: function(xhr, ajaxOptions, thrownError) {
alert(thrownError);
}
});
};
InfiniteAjaxRequest ("CheckForUpdate.php");
What are the benefits of using this script?
Well, rather than checking every "x" seconds, it will only check once the previous request has been received, so it chains the requests. You could add in a delay to throttle this constant request, which I would highly recommend you do - otherwise you will be hitting your site with way too much traffic. You would add that delay in AFTER you've done something with the response, but BEFORE you call back into "InfiniteAjaxRequest".
Here's your nudge:
Get one of the available JavaScript frameworks (jQuery seems to be the most common, but there are others)
flip though the documentation on the AJAX methods it provides, choose a method for your task that seems appropriate
build a request to your site that fetches the info and reacts on the response (shows a message box or updates some part of your page), wrap that in a function
make sure request errors do not go unnoticed by implementing an error handler
check out setInterval() to call that function you've just made repeatedly
final step: make sure that the interval will be stopped in case of an error condition (or provide a on/off button for the user, even) so the server is not hammered needlessly
There is a technique called Comet where-by your client-side script would instantiate a HTTP request which remains open for a long time. The server can then push data into the response as they happen. It's a technique to deliver a push notification.
The Wikipedia link has more information on real-world implementations.
Instead of polling the server with AJAX calls you could also use push technology (COMET).
This way you can push the results to the client(s) as soon as the server is done with it's work.
There are many frameworks available like:
JQuery plugin
Cometd
Atmosphere (if your on java)