How do I load HTML source code? I tried a few functions with no luck. This one works for me on many sites, but not all of them:
function LoadWebPageToString(MyUrl: String): String; //load HTML content from a webpage to a string
begin
Result := 'Error';
with TIdHTTP.Create do
begin
try
Result := Get(MyUrl);
finally
Free;
end;
end;
end;
When it fails, I get this error:
HTTP/1.1 403 forbidden
The target page is just a normal page. It loads normally via HTTP, doesn't require (think nor supports) HTTPS. Maybe it is about cookies or something? I don't know.
One of the reasons of the HTTP/1.1 403 forbidden is which the server doesn't recognizes the user agent of the client, so try setting the useragent property like so .
Request.UserAgent:='Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0';
Related
I am sending JSON to an endpoint and I have a problem. The execution works, but the form gets stuck until HTTP.Post() ends, after it has finished the screen releases for use. I'm sure I'm doing something wrong.
Here is the button action that sends the JSON:
procedure TForm1.Button1Click(Sender: TObject);
var
HTTP: TIdHTTP;
vJsonAEnviar: TStringStream;
Json:String;
begin
Json := '{ '+
' "user":"Lucy"'+
' "execute":"ok"'+
' } ';
HTTP := TIdHTTP.Create;
HTTP.Request.ContentType := 'application/json';
HTTP.Request.UserAgent := 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:12.0) Gecko/20100101 Firefox/12.0';
vJsonAEnviar := TStringStream.Create(UTF8Encode(Json));
HTTP.Post('http://localhost:8080/exportaManual', vJsonAEnviar);
FreeAndNil(HTTP);
FreeAndNil(vJsonAEnviar);
end;
On the other side, it takes time to finish and the screen stays blockrd a long time.
You are not doing anything wrong (well, except for a complete lack of any error handling). This is simply how Indy is designed to operate (see Introduction to Indy). Indy uses blocking socket operations. When you perform an operation, the calling thread is blocked until the operation is complete. This is normal.
If you don't want your UI frozen while the POST is in progress, you can either:
drop a TIdAntiFreeze component onto your Form. It will pump UI messages in the background while Indy is blocking the main UI thread.
move the POST code to its own worker thread, using TThread, TTask, TIdThreadComponent, etc, and have it notify the main UI thread when finished.
1) start local web server
C:\Users\Public\Documents\Rick>http-server . -p 8832 --cors
Starting up http-server, serving . on: http://0.0.0.0:8832<br/>
Hit CTRL-C to stop the server<br/><br/>
**partial log** from (node.js) http-server . -p 8832 --cors<br/><br/>
[Mon, 15 Jun 2015 18:14:57 GMT] "GET /2015_03_19_Try6a3D_dae/2015_03_19_Try6a3D/scrn_ground.png" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.124 Safari/537.36"<br/><br/>
2) start html file that loads 2015_03_19_Try6a3D_dae/2015_03_19_Try6a3D.dae
from collada.html (javascript console)<br/><br/>
Uncaught SecurityError: Failed to execute 'texImage2D' on 'WebGLRenderingContext': The cross-origin image at http://localhost:8832/2015_03_19_Try6a3D_dae/2015_03_19_Try6a3D/scrn_ground.png may not be loaded.<br/><br/>
I tried to post the javascript that loads the dae, here, but could not get it to format correctly.
3) There is a brief flash of something before the texture loading errors happen. This dae has been loaded in Sketchup where all the textures appear. Of course, I am confused because cross-origin loading had to be working to load 2015_03_19_Try6a3D.dae in the first place. I will gladly send anyone collada.html, 2015_03_19_Try6a3D.dae, and all related files for them to play with.
I had the same problem. ColladaLoader.js currently does not address CORS out-of-the-box. In order to render your textures, it implements either the Loader class or the ImageLoader class (depending upon the situation). Both need to have the CORS origin assigned to either '' or 'anonymous' if you want to avoid cross-origin errors in all cases for Collada references.
Go to this line in ColladaLoader.js:
texture = loader.load( url );
Add this line right above it:
loader.crossOrigin = '';
Then go to this line in the same script:
loader = new THREE.ImageLoader();
And add this line right below it:
loader.setCrossOrigin( '' );
And voila! My cross-origin errors went away after I made this change.
My company is hosting an ecom shop based on Infinity Shop System. Our logs say that there are HTTP calls to this path which lead to 404 errors since the file does not exist:
http://{domain}/{somePath}/skin/default/images/tb-collectingalarm-red-low_inject.png
However, this reference is not made by us as I cannot find this path in any line of our source code.
The logs also state that only (some?) Firefox users do this call:
User Agent Mozilla/5.0 (Windows NT 6.3; rv:35.0) Gecko/20100101
Firefox/35.0
So, since this does cause quite some 404 errors, does anyone know what could cause these requests?
We already followed the referrer URL which lead to one of our sites but within its html markup we could not find any reference.
I want to get html code from windows phone market pages. So far I have not run into any problems but today following error is displayed every time I retrieve data.
[...] Your request appears to be from an automated process.
If this is incorrect, notify us by clicking here to be redirected [...].
I tried to use proxy in case to many requests are called from one IP but this does not bring any progression. Do you happen to know why this problem takes place, any ideas about possible way outs? Any help would be very much appreciated. The main goal is to somehow get information about windows phone app from market.
It seems that they detect the user agent and block the request if it is not valid / known for a device.
I managed to make it work with curl with eg.
curl -A 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.9) Gecko/20071025 Firefox/2.0.0.9' http://www.windowsphone.com/en-us/store/app/pinpoint-by-foundbite/ff9fdf41-aabd-4cac-9086-8710bd327da9
For asp.net, if you use HttpRequest to get the html content, try the following:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.9) Gecko/20071025 Firefox/2.0.0.9";
For PHP you can set your user agent as well via curl_setopt.
I was not able to find out, whether there is an IP-based block after several requests.
On my machines I find that SignalR client functions are not called in Chrome.
My SignalR test app works fine with with IE9, Firefox and even Safari on my iphone. Looking at Fiddler, these browsers all seem to negotiate transport=longPolling. But Chrome negotiates a connection with transport=serverSentEvents, and I'm assuming this is why client functions are not called in Chrome.
More detail:
I'm using full IIS (not IIS express) on Windows 7. I'm using SignalR version 1.0.0-rc2. I've disabled my AVG firewall and the Windows firewall is not running. Chrome is version 24.0.1312.56, and was up-to-date at the time of writing. The app is invoked on localhost.
On Chrome, the signalR connection seems to take place OK - the $.connection.hub.start().done callback function is invoked. But after that, the client function is never called, even while the other browsers work just fine.
In the client-side code, I've turned on logging with
$.connection.hub.logging = true;
I can see log messages in Chrome's javascript console that correspond to a successful connection.
For reference, those log messages are
[20:22:16 GMT+0800 (W. Australia Standard Time)] SignalR: Negotiating with '/SignalRChat-RC/signalr/negotiate'. jquery.signalR-1.0.0-rc2.js:54
[20:22:16 GMT+0800 (W. Australia Standard Time)] SignalR: Attempting to connect to SSE endpoint 'http://localhost/SignalRChat-RC/signalr/connect?transport=serverSentEvents&…7-22c5dbf27e0d&connectionData=%5B%7B%22name%22%3A%22chathub%22%7D%5D&tid=3' jquery.signalR-1.0.0-rc2.js:54
[20:22:16 GMT+0800 (W. Australia Standard Time)] SignalR: EventSource connected jquery.signalR-1.0.0-rc2.js:54
[20:22:16 GMT+0800 (W. Australia Standard Time)] SignalR: Now monitoring keep alive with a warning timeout of 40000 and a connection lost timeout of 60000 jquery.signalR-1.0.0-rc2.js:54
But there are no messages logged in Chrome's javascript console when a client-side method is invoked.
Interestingly, the send method works OK on Chrome. The other clients display a message sent from Chrome even through Chrome itself can't see it.
The application is pretty much the chat application from the signalR tutorial at http://www.asp.net/signalr/overview/getting-started/tutorial-getting-started-with-signalr
If I explicitly specify longPolling in the start method, i.e.
$.connection.hub.start({ transport: 'longPolling' })
then Chrome works OK. But my expectation was that I should be able to allow the browsers to negotiate their connection, and things would Just Work.
For reference, the relevant part of my client-side code looks like this:
$(function () {
// Turn on logging to the javascript console
$.connection.hub.logging = true;
// set up an error-handling function
$.connection.hub.error(function (err) {
alert("Error signalR:" + JSON.stringify(err));
});
// Declare a proxy to reference the hub.
var chat = $.connection.chatHub;
// Create a function that the hub can call to broadcast messages.
// This function is never called when running in Chrome with the default signalR connection
chat.client.broadcastMessage = function (name, message) {
// Html encode display name and message.
var encodedName = $('<div />').text(name).html();
var encodedMsg = $('<div />').text(message).html();
// Add the message to the page.
$('#discussion').append('<li><strong>' + encodedName
+ '</strong>: ' + encodedMsg + '</li>');
};
// Get the user name and store it to prepend to messages.
$('#displayname').val(prompt('Enter your name:', ''));
// Set initial focus to message input box.
$('#message').focus();
// Start the connection.
// Use $.connection.hub.start({ transport: 'longPolling' }) for reliability
// Use $.connection.hub.start() to demonstrate that Chrome doesn't receive messages
$.connection.hub.start().done(function () {
// Enable the "Send" button
$('#sendmessage').removeAttr('disabled');
$('#sendmessage').click(function () {
// Call the Send method on the hub.
chat.server.send($('#displayname').val(), $('#message').val());
// Clear text box and reset focus for next comment.
$('#message').val('').focus();
});
});
});
Can anybody see what I'm doing wrong?
I tried the sample in Win7 Google Chrome 24 and it works fine.
You can troubleshoot installing Fiddler and setting breakpoints in the javascript
POST /signalr/send?transport=serverSentEvents&connectionId=6ff0bffa-c31e-4d85-9aff-24f4528555ee HTTP/1.1
Host: localhost:43637
Connection: keep-alive
Content-Length: 113
Accept: application/json, text/javascript, */*; q=0.01
Origin:
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.57 Safari/537.17
Content-Type: application/x-www-form-urlencoded
Referer: /index.html
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
data=%7B%22H%22%3A%22chathub%22%2C%22M%22%3A%22Send%22%2C%22A%22%3A%5B%22gus%22%2C%22hello%22%5D%2C%22I%22%3A0%7D
May be it is related with buffering the responses.
https://github.com/SignalR/SignalR/issues/1944
Try setting EnableJSONP = False on the server hub. This fixed a similar issue I was having.