I have been using Mandrill webhooks from a long time and till now I haven't encountered this error.
But now I see this error, I am not sure what has caused this ?
Please let me know why this might be happening and what might be the possible solution for the same.
Is it related to my server handling capacity because I have checked for that as well and Mandrill doesnt have too many concurrent request that it is sending to my Apache server, so according to me that is not an issue and also mysql also doesn't seem to be causing the bottleneck, but then I I have not used any benchmarking tool to determine the same.
Please let me know the solution if you guys have encountered something like this.
It seems that the URL is not responding to the request. There could be a few reasons:
If the URL points to an internal server, a firewall could be blocking it or a port number (if given).
Once set up the webhook will send via a POST HTTP verb, however for testing it sends a HEAD request. Quite often web servers (e.g. IIS) will limit what verbs they respond to and will only respond to GET and POST requests.
If that's working your URL should respond with just headers only to acknowledge the request. (HEAD doesn't allow any page content to be sent) so it should only do something like this for a HEAD request:
<?php header( 'Content-Type:' ); // returning 200 ?>
More details on their site
http://help.mandrill.com/entries/22024856-Why-can-t-my-webhook-or-inbound-route-URL-be-verified-
You may wish to try this tool to see what HTTP header result is being returned (if any) or if another error is being returned, just remember that if the URL is internal, it could be blocked to the outside world.
https://chrome.google.com/webstore/detail/postman-rest-client/fdmmgilgnpjigdojojpjoooidkmcomcm?hl=en
Related
I'm trying to get data from this site: [1] https://www.eurobet.it/it/scommesse/#!/calcio/?temporalFilter=TEMPORAL_FILTER_OGGI_DOMANI
I found this link where I can get the data in JSON format: [2] https://www.eurobet.it/detail-service/sport-schedule/services/discipline/calcio?prematch=1&live=0&temporalFilter=TEMPORAL_FILTER_OGGI_DOMANI
But there is a problem:
The JSON link Doesn't work every time in fact sometimes I get a 404 error.
I noticed that if I open the first link [1] before opening the second [2] it works perfectly.
This error is also more frequent when I try to scrape other data on the same site: [3] https://www.eurobet.it/detail-service/sport-schedule/services/discipline/calcio/piu-giocate/u-o-goal?prematch=1&live=0&temporalFilter=TEMPORAL_FILTER_OGGI_DOMANI
In this link [3] I try to get all "u-o-goal" odds but this link works only if (before starting my program to scrape data) in the main link [1] I press the "U/O GOAL" button -> https://i.stack.imgur.com/Nei5u.png
In my code, I'm using Java and htmlunit to scrape the data.
My question is: how this webpage works, why couldn't I open directly the links [2]/[3], I know that there is a sort of request and approval system behind but I can't see where.
You cannot directly open these URLs since the website (and many like it) will use cookies and bot-prevention techniques/session tracking so they can gather data about usage of their website. eg. they set a "Referer".
I'm not going to code a solution for you but I can at least help you understand what you need to do to get to where you want...
I've attempted to summarise how I'd typically unpick a request like this to recreate it, but in its essence, you need to understand the sequence of HTTP requests being made (this is how the web works - HTTP requests).
First you typically start with no session cookies and you access the site directly (no referer).
Once you access a website, typically the server responds with a session cookie for you to communicate back to the server a unique session ID so it has some sort of record of your browser having already been in contact.
Your browser may make more requests (asynchronously) and in doing so typically sends the cookies and the referring URL (usually the base Url will work... just don't use something that starts with something other than "https://www.eurobet.it"
anything else you're going to need to figure it out. Lots of headers are optional. Lots of query params have defaults.
https://stackoverflow.com/a/64671815/7619034 - here's an answer I've given before that answers this type of question which comes up often enough.
so to explain a bit further, for your specific scenario...
When you access https://www.eurobet.it/it/scommesse/#!/calcio/?temporalFilter=TEMPORAL_FILTER_OGGI_DOMANI, the server responds with HTTP headers:
...
set-cookie: __cfduid=dd38d***********41125; ...
...
The rest doesn't look that relevant:
Going straight to the other request: https://www.eurobet.it/detail-service/sport-schedule/services/discipline/calcio?prematch=1&live=0&temporalFilter=TEMPORAL_FILTER_OGGI_DOMANI
This HTTP request takes (as input):
cookie: __cfduid=dd38d***********41125; mbox=session#6661556c.....b6e8cc1fa6f03#1608242987; at_check=true; s_ecid=MCMID%***********2021453010; AMCVS_45F10C3A53DAEC9F0A490D4D%40AdobeOrg=1; AMCV_45F10C3A53DAEC9F0A490D4D%40AdobeOrg=1075005958%7CMCIDTS%7C18614%7CMCMID%7C91883906030825914429183258312021453010%7CMCAID%7CNONE%7CMCOPTOUT-1608248327s%7CNONE%7CvVersion%7C4.4.1; s_cc=true
...
referer: https://www.eurobet.it/it/scommesse/
...
x-eb-accept-language: it_IT
x-eb-marketid: 5
x-eb-platformid: 1
Cookies are set in an initial request (typically) using Set-Cookie header and then are passed back to the server in subsequent requests using the cookie header.
I'm not certain how many of these values are relevant but you'd need to figure out where each came from in the chain of HTTP requests between the initial one and this one and you'd need to replicate them (see url above of my previous answer - warning this can be time consuming).
The other headers can be set statically most likely since they probably aren't due to change.
If you have access to curl on the command line, you can attempt to reconstruct some of these requests by hand. Some will be time sensitive since cookies do expire after an amount of time (see set-cookie header details for exactly when). Once you've reconstructed a working request, you can then start coding it in your application.
If you can work all this out you should be able to re-construct the chain of HTTP GET requests to get the JSON data you want. Good luck!
I am writing a http handler for a server and I am looking directly at the http requests when they come in from different clients. I can easily deal with normal http requests. The problem occurs when I get a GET or POST request. I do not know how to access the data from the GET or the POST therefore I cannot continue. Could someone please point me in the direction of some where which deals with the issue on how to access the data. Thanks in advance.
Answer:
to do this:
In a GET request the data comes in the URL itself therefore just parse the URL from the HTTP request and look for the question mark and the arguments.
For a POST request there are 2 different ways however the main one means that the arguments are put in the body of the request like this:
q=hello&v=world
The length is specified in the request as well so if you need it is under Content-Length:
Why if I submit an URL like:
localhost:38245/TeamWork/Group/1/LONG-COMMENT-POSTED-BY-USER ,
I get an error that says "Bad Request - invalid URL HTTP Error 400. The request URL is invalid." ?
The default maximum length of a URL is 4096 in IIS. You may well be breaching that with the comment posted by the user. Generally, the best way to transfer that kind of data is by POST rather than in a URL. You can either trim the comment to a more suitable size of increase the maxUrl value in the requestLimits section of your IIS config: http://msdn.microsoft.com/en-us/library/ms689462%28v=vs.90%29.aspx
Firstly, HTTP 400 error is intended for cases when the client has made an error. As you've noticed yourself it's a bad request error. So that means possibly due to bad syntax the request can not be carried out. As Mike Brind mentions, you're most probably exceeding the maximum length of the URL (the default value anyway)
Secondly, why are you trying to "submit" a long post by URL? Use the POST method instead. A way to do this is by using Jquery's Post method to do this. See this. Or simply by calling the form's submit method through OnClick method or however the user is submitting the form.
POST is perfect for transmitting large amounts of data. So submitting data from the client side, the easiest way by far is to use POST to archieve this.
As you have yourself stated in the Question code part,
localhost:38245/TeamWork/Group/1/LONG-COMMENT-POSTED-BY-USER
the Long comment by user, must be inside the limits that are in the web.config file of your website.
Secondly, no Server would allow a very Long Url thinking that the user might be trying to post a malware. Also, using short URLs is good for Data transfer as they consume less data.
From the limits on Requests one would easily come to understand that the URL must be less than 4096bytes, only 4KB. So do you think, all that URL would be 4KB?
As Mike has suggested to use POST request to send the long data, I would also suggest you to use HttpPost requests to send this type of long data. This way, Browser would encrypt the data and sending it as an attachment to the request.
I'm trying to find the correct HTTP status code for a page where the content is temporary unavailable however there is no redirect, instead a message is displayed on the page informing the user the content is temporarily unavailable.
307 Temporary Redirect isn't applicable as there is no redirect.
404 Not Found might possibly be applicable, however I'm not sure if this is the correct response to give as the content is found, just not available.
410 Gone isn't applicable as the content will be available again some time in the future.
None of the other codes seemed even remotely applicable. Does anyone know the correct code to use and can explain why?
It sounds like the 4XX series of responses are appropriate here. From the RFC:
The 4xx class of status code is intended for cases in which the
client seems to have erred. Except when responding to a HEAD request,
the server SHOULD include an entity containing an explanation of the
error situation, and whether it is a temporary or permanent
condition.
With this in mind, I think 403 forbidden is the most appropriate:
10.4.4 403 Forbidden
The server understood the request, but is refusing to fulfill it.
Authorization will not help and the request SHOULD NOT be repeated.
If the request method was not HEAD and the server wishes to make
public why the request has not been fulfilled, it SHOULD describe the
reason for the refusal in the entity. If the server does not wish to
make this information available to the client, the status code 404
(Not Found) can be used instead.
I suggest this for three reasons:
It's not an exotic code, so it will work fine in the browser. This, to me, is the most important reason - you will be able to serve a page that explains why the content isn't available, and you can be fairly certain it will be displayed correctly.
It's appropriate for the server to say "I understand your request, but I won't serve you that content at this time", and that's exactly what the first two lines of the description say.
It doesn't explicity say "forget you ever knew about this content" to any robots (or for that matter, people).
For completeness, here's why I ruled out the other response code categories:
2XX Success: This class of status code indicates that the client's request was
successfully received, understood, and accepted.
But, we're not accepting the request in this case. I don't think 2XX is right.
3XX Redirection: This class of status code indicates that further action needs to be
taken by the user agent in order to fulfill the request.
I suppose that you could argue "further action" to mean "please wait until the content is available before trying again", but reading the other 3XX codes, "further action" usually means "immediate redirect", which as you've already pointed out, isn't appropriate.
5XX Server error: Response status codes beginning with the digit "5" indicate cases in
which the server is aware that it has erred or is incapable of
performing the request.
Nothing has gone wrong on the server, you just don't want to serve the content right now.
HTTP STATUS CODE 204
i.e. NO CONTENT
Read More about it here:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
section 10.2.5
i.e.
The server successfully processed the request, but is not returning any content.
New here. I've searched quite a bit for a working solution to my problem, but even though I have found posts with promising titles, none of the solutions have worked.
I am deploying an MVC2 web app to a client's server.
I did my development on Win2k8 Server, but they are running Win2k3 sever.
The app's only purpose is to receive some record ID information as HTTP parameters, check in the database for the status of the given record or records, and then return the status information as a simple string such as "Completed" or "Incomplete" in JSON format.
This getJSON code works fine in the development environment.
Inexplicably to me, on the client's server, the getJSON request receives a null response from the application.
There is no cross-domain action AFAIK... the result is the same from the client's server or from my machine via VPN.
In the MVC model's Json code, a common solution for people is to add the "JsonRequestBehavior.AllowGet" attribute to the Json result being returned. I did this long before trying to deploy it, and as I said, it has worked fine in the dev environment.
Using Firebug, I have watched the same request URL get sent to both my local server and the client server - the response headers from both servers are the same, but the response content from my server is shown as:
{"Result":"No Data"}
Which is what I want.
There is literally no content shown in the response from the client's server..? But the request gets an HTTP 200 code and is recorded as a success in the reponse's status attribute.
The response header content type in both situations is "application/json"
But wait, there is more!
If I manually enter the request to each server in the Firefox nav bar, and hit enter, in both cases it responds with:
{"Result":"No Data"}
Which is what I want. So why can I get the result I want from the MVC app on the client's server only when I hand-enter the request URL in Firefox, but not from the Javascript code?
I have tried forcing different output content types ... using the jQuery ajaxSetup method...
$.ajaxSetup({
async: false,
dataType: 'text'
});
or
$.ajaxSetup({
async: false,
dataType: 'html'
});
and again wtih 'script', and 'json'. I also tried the conversion options such as 'text json' or 'html json' or 'json text' and so forth.
Some of the posts I'm reading, and my gut feeling, though, suggest the problem is not the jQuery code making the request that is at fault... I don't see how the same jQuery request point to a different server running the same app would suddenly cause that server to send back a 'null' value.
By null, I want to be clear... I mean nothing is sent. There is no {} or {null} or any sign of JSON... just blank whiteness of non-existence :P
Even if nobody knows the answer, I would love some input perhaps suggesting where I should focus my sleuthing ... client or server?
If the problem is the server, it seems hard to really know that the MVC stuff is running 100% on the IIS6 server, but in general it seems to work. I have a different MVC app running on the client server which responds to the virtual paths, and generally runs the same as on dev machine.
I have found one thing ... the request headers are somewhat different? The Request Headers sent to the IIS7 setup include an "X-Requested-With: XMLHttpRequest", "referrer" , and "cookie" field/value.
I could guess that the lack of the "X-requested-with: XMLHttpRequest" in the IIS6 request headers is a clue, but I do not see then how the same javascript code pointing at a different server can generate different request headers itself. So how else are those being generated?
The javascript is embedded in an ASP.NET page, btw.
Oooh.. frustration!
Thanks for any input.
Odd Progress ... apparently there is some sort of issue with IIS6 handling the query. Although I have not payed any attention to JSONP, a post elsewhere suggested that sometimes use the "&callback=?" parameter at the end of a .getJSON request URL would force it into GET mode and that worked frequently for problems getting data from the server. So I did that... and it did work, sort of. The proper {"Result":"No Data"} was returned in response to the request... which seems good. However, the way that the JSONP callback works, it generates its own script to do the calling and fetching and interpreting of the incoming JSON. In this case, it interprets the JSON to need a label which it does not have, thus an error is thrown "invalid label" ... there must be some way to hack things to just deliver the JSON, but the whole required use of JSONP callbacks suggests that the server configuration is wrong, right? Or why does it work without JSONP for IIS7 and not IIS6?
Despite my not liking the callback JSONP solution, it appears to work ok. The error is still returned about an invalid label, but that does not appear to stop the remaining javascript from running... and so the application is working with IIS6 now. I have not tested the fix of using the callbacks and JSONP against IIS7 yet, but I expect it will work well enough.
Here is a link to the discussion that lead me to my current solution. I do still hope to find a more elegant solution, however.
NeoWin.net
Are you certain that your App Extension Mappings are set up correct?
Follow this article for running MVC2 on IIS6 and ensure all the different configurations have been done, that's probably the first step before going further and investigating specifics.
I'm really inclined to believe it's related to HTTP Verbs.