Does this URL exist? RCurl says no - html

The URL of interest is:
http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&u=/netahtml/PTO/search-adv.htm&r=10&f=G&l=50&d=PTXT&OS=AN/(nortel)&RS=AN/nortel&Query=AN/(nortel)&Srch1=nortel.ASNM.&NextList1=Next 50 Hits
The chosen function to test its existence is:
> url.exists("http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&u=/netahtml/PTO/search-adv.htm&r=10&f=G&l=50&d=PTXT&OS=AN/(nortel)&RS=AN/nortel&Query=AN/(nortel)&Srch1=nortel.ASNM.&NextList1=Next 50 Hits")
[1] FALSE
Why dis no work? The URL clearly exists and resolves in chrome and using htmlTreeParse on the URL works just fine.

My guess would be that url.exists is using a HTTP HEAD-request, which the server seems unable to handle:
$ telnet patft.uspto.gov 80
Trying 151.207.240.26...
Connected to patft.uspto.gov.
Escape character is '^]'.
HEAD /netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&u=/netahtml/PTO/search-adv.htm&r=10&f=G&l=50&d=PTXT&OS=AN/(nortel)&RS=AN/nortel&Query=AN/(nortel)&Srch1=nortel.ASNM.&NextList1=Next+50+Hits HTTP/1.1
Host: patft.uspto.gov
Connection: close
Connection closed by foreign host.
So server broken, not RCurl.

Related

How to do a website preview using a subdomain?

I'm trying to make a website preview with subdomain
e.g.
I've https://www.sub.example.com and CNAME to https://www.sub2.example2.com
When I do a PING command sub2.example2.com. response me, but I a navigator don't open the sub2.example2.com.
Both Domain are using a different Wildcard and I do not want to use a .htaccess
what options I have?
Don't get confused/mix things, PING behaves differently than HTTP that's why when you ping you may always get a response since you are doing the requests to the same web server or load balancer.
Regarding the HTTP request, what could be missing is a server block/virtual host to handle your request for your defined HOST: sub2.example2.com.
Once you have your vhosts's defined you could test using curl with something like this:
curl -I -H 'Host: sub.example.com' your-web-server.tld
Check the returned headers (option -I) that could give you a hint.

Mailgun blocking my IP?

Trying to set up mailgun through nullmailer for email delivery from a box (let's call it a.example.com).
After some experimenting I found out that this works well for my /etc/nullmailer/remotes:
smtp.mailgun.org smtp --port=587 --user=postmaster#mg.example.com --pass=PASSWORD
But this only works on b.example.com, not on a.example.com that I wanted to set this up on. Looking further into this it looks as if mailgun is blocking my server.
On my laptop and b.example.com this works:
$ telnet smtp.mailgun.org 587
Trying 52.38.21.43...
Connected to smtp.mailgun.org.
Escape character is '^]'.
220 ak47 ESMTP ready
But on a.example.com it stops:
$ telnet smtp.mailgun.org 587
Trying 52.38.21.43...
Now I wonder if Mailgun is blocking a.example.com? And if so, if this is only temporary. And if not, what is going on here?
Found out that Scaleway.com (where a.example.com is) decided to block outgoing SMTP traffic by default.

Using outbound Node-RED email to local SMTP server

I have a local SMTP email server I use for testing purposes running on my machine. It listens for SMTP on port 25. I am able to send and receive emails to it using a regular email client.
When I build a Node-RED flow that contains an e-mail output node and configure its properties with:
to: <email address>
server: localhost
port: 25
and submit a flow, I get the error:
25 Feb 16:43:24 - [error] [e-mail:<email address>] Error: 101057795:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:openssl\ssl\s23_clnt.c:794:
I am at a loss on how to proceed. Looking at the messages, it almost appears that there is some form of SSL negotiation/test at play here. Switching on trace on my SMTP server, I find the following logs each time I try and run a flow:
"TCPIP" 10708 "2016-02-25 16:43:08.294" "TCP - 127.0.0.1 connected to 127.0.0.1:25."
"DEBUG" 10708 "2016-02-25 16:43:08.298" "Creating session 22"
"SMTPD" 10708 22 "2016-02-25 16:43:08.298" "127.0.0.1" "SENT: 220 WIN7-X64 ESMTP"
"DEBUG" 9772 "2016-02-25 16:43:08.299" "Ending session 22"
It appears that the Node-RED node is sending a connection request, getting back the SMTP 220 response and then failing immediately after that.
I came across the same problem and have a nasty hack that will enable mail to go via my local exchange server's plain SMTP, with no auth.
Edit the .../61-email.js file and change it thusly:
var smtpTransport = nodemailer.createTransport({
host: node.outserver,
port: node.outport,
secure: false,
ignoreTLS: true //,
// auth: {
// user: node.userid,
// pass: node.password
// }
});
I see Dave has replied to the github issue but just to close the loop on this question.
At this time (Feb 2016) the node assumes SSL is always available and enabled, at some point we need to go back to the email node and find a simple way to expose a lot more of the nodemailer options to allow connections to a wider range of email providers both public and private.

POST some JSON through TELNET

I am told to upload an image to a server by sending a JSON as a request.
JSON is sth like below:
{"action":"setMap","data":{"mapName":"myMapName","mapURL":"http://tinypic.com/myimg"}}
I do not know how to use TELNET to POST a JSON.
i guess i should write something like below
terminal>telnet my.ip.num.ber port
POST /setMap HTTP/1.1
but dont know how to continue.
Should i write
DATA : {"action":"setMap","data":{"mapName":"myMapName","mapURL":"http://tinypic.com/myimg"}}
How can i get the JSON sent?
I can't understand why you want to use Telnet. Telnet can be useful to quickly test chatty protocols, and even if HTTP is chatty to some degree, it's very cumbersome to upload an image (plus, from the given service name, setMap, I guess the service doesn't really let you upload an image, but just insert a record in the database pointing to an image accessible on another service).
What you are asking is something like:
$ telnet example.com 80
> POST /setMap HTTP/1.1
> Host: www.example.com
> Content-Type: application/json; charset=utf-8
> Content-Length: 1234
>
> {"mapName":"myMapName","mapURL":"http://tinypic.com/myimg"}
>
Note that it's just an example. You have to replace connection parameters (host, port), content-type, content-length and the actual JSON data - and this we can't know because depends on the actual service implementation.

'Sec-WebSocket-Accept' header is missing in Chrome 17

Edit:
I tried this phpwebsocket: http://www.wilky.it/Shared/phpwebsocket.zip and it works in Firefox, but my question still remains: how do I get websockets to work with a php server in Chrome 17?
I'm following the tutorial here: http://net.tutsplus.com/tutorials/javascript-ajax/start-using-html5-websockets-today/
It appears as though the client connects, and then immediately disconnects. I noticed this error in the console:
Error during WebSocket handshake: 'Sec-WebSocket-Accept' header is missing
I'm trying it in Chrome 17.0.963.56 on my WAMP localhost with the php_sockets extension enabled.
I saw mentioned somewhere that Chrome had changed what it supported, but it didn't go in depth on how to fix it. I was hoping someone could step me through it. (I'm brand new to websockets).
Server:
{PATH}>php startDaemon.php
2012-02-20 07:02:51 System: Socket Resource id #7 created.
2012-02-20 07:02:51 System: Socket bound to localhost:8000.
2012-02-20 07:02:51 System: Start listening on Socket.
2012-02-20 07:03:01 WebSocket: Resource id #8 CONNECTED!
2012-02-20 07:03:01 WebSocket: Requesting handshake…
2012-02-20 07:03:01 WebSocket: Handshaking…
2012-02-20 07:03:01 WebSocket: Done handshaking…
2012-02-20 07:03:01 WebSocket: Resource id #8 disconnected!
Client:
Socket Status: 0
Socket Status: 3 (Closed)
I have the same problem (and I do not seem to be able to post a comment here, so I post a reply).
Actually, I just downloaded and tested phpwebsocket.
On safari 5.1.4, it works just fine.
On Chrome 17, I got the same error in the script log console:
Error during WebSocket handshake: 'Sec-WebSocket-Accept' header is missing
So, in websocket.class.php, I added to the header returned by server:
$accept = base64_encode(SHA1($key."258EAFA5-E914-47DA-95CA-C5AB0DC85B11"));
And I get the error:
Error during WebSocket handshake: Sec-WebSocket-Accept mismatch
Now, the header received by the server is:
GET /websocket/server.php HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: localhost:12345
Origin: http://localhost:8888
Sec-WebSocket-Key: OqMJI0t/cOl6d6JNE+Op0g==
Sec-WebSocket-Version: 13
And the header sent back by the server is:
HTTP/1.1 101 WebSocket Protocol Handshake
Upgrade: WebSocket
Connection: Upgrade
Sec-WebSocket-Origin: http://localhost:8888
Sec-WebSocket-Location: ws://localhost:12345/websocket/server.php
Sec-WebSocket-Accept: ZjY5ODliNTViYzJlOTNkMjk4OTg3Y2U2NjQ3MTBlZjZiNzliYzk4Yg==
The Sec-WebSocket-Accept seems good, but still there is a mismatch error. Do you see a mistake somewhere? Maybe the protocol has changed to calculate the Sec-WebSocket-Accept, but I don't find it... Thanks for your help!
Edit: Here seems to be the solution (for me, at least): adding the parameter true to the SHA1 function, as found in files given in this issue thread. So, the Sec-WebSocket-Accept must be found like this:
$accept = base64_encode(SHA1($key."258EAFA5-E914-47DA-95CA-C5AB0DC85B11", true));
And, Sec-WebSocket-Key1 and Sec-WebSocket-Key2 does not seem to be present anymore in the client request, instead, $key must be extracted from the header: "Sec-WebSocket-Key".
New issue: It seems too that even if the web socket connection now works on the handshake, it disconnects when the first message is sent.
I noticed that in the console of Chrome 19:
A server must not mask any frames that it sends to the client.
Maybe this is the problem. It disconnects as soon as a message is sent. It works fine in Firefox.
I fixed this websocket problem and it works in chrome now.
First I used:
Then I used the encode function from:
https://github.com/lemmingzshadow/php-websocket
I fixed the replaced the encode function with the one in the connection.php file in lemmingzshadow’s github and it started working. The function is called: hybi10Encode in the \server\lib\WebSocket\connection.php file.
change this parameter in the function encode: $masked = true to $masked = false
An EASY way to fix is add Sec-WebSocket-Accept information when do_handshake, code as below:
list($resource,$host,$origin,$key) = $this->getheaders($buffer);
$accept = base64_encode(SHA1($key."258EAFA5-E914-47DA-95CA-C5AB0DC85B11", true));
$upgrade = "HTTP/1.1 101 Web Socket Protocol Handshake\r\n" .
"Upgrade: WebSocket\r\n" .
"Connection: Upgrade\r\n" .
"WebSocket-Origin: {$origin}\r\n" .
"WebSocket-Location: ws://{$host}{$resource}\r\n".
"Sec-WebSocket-Accept: " . $accept . "\r\n\r\n";
$this->handshakes[$socket_index] = true;
socket_write($socket,$upgrade,strlen($upgrade));
where,
$accept = base64_encode(SHA1($key."258EAFA5-E914-47DA-95CA-C5AB0DC85B11", true));
$key is Sec-WebSocket-Key got from $buffer, you can print_r($buffer) to have a look.
Hope this can solve your problem..