Using REST in TCL to POST in JSON format? - json

Essentially what I'm attempting to do is post to a REST API, but no matter what I do I end up with HTTP 400. Here is my extremely quick and extremely dirty code:
package require rest
package require json
::http::register https 443 ::tls::socket
set credentials {username admin password LabPass1}
set url1 [format "%s/%s" "https://127.0.0.1:8834" session]
set unformattedToken [dict get [::json::json2dict [::rest::post $url1 $credentials]] token]
set cookie [format "token=%s" $unformattedToken]
set header [list X-Cookie $cookie Content-type application/json]
set config [list method post format json headers $header]
set url [format "%s/%s" "https://127.0.0.1:8834" scans]
set uuid 7485-2345-566
set name "Testing TCL Network Scan"
set desc "Basic Network Scan using API"
set pid 872
set target 127.0.0.1
set data {{"uuid":"$uuid","settings": {"name":"$name","description":"$desc", "policy_id":"$pid","text_targets":"$target", "launch":"ONETIME","enabled":false,"launch_now":true}}}
set jsonData [json::json2dict $data]
set response [::rest::simple $url $jsonData $config]
I've tried using the above code and I've also tried removing the json::json2dict call and just sending the data. I believe, and I could be wrong, that my issue is the data is going as line-based text data:
POST /scans HTTP/1.1
Host: 127.0.0.1:8834
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 10.0) http/2.8.9 Tcl/8.6.4
Connection: close
X-Cookie: token=301b8dcdf855a29b5b902cf8d93c49750935c925a965445e
Content-type: application/json
Accept: */*
Accept-Encoding: gzip,deflate,compress
Content-Length: 270
uuid=7485-2345-566&settings=name%20%7BTesting%20TCL%20Network%20Scan%7D%20description%20%7BBasic%20Network%20Scan%20using%20API%7D%20policy_id%20872%20text_targets%20127.0.0.1%20launch%20ONETIME%20enabled%20false%20launch_now%20true
I've reviewed the JSON documentation, and the REST documentation but I'm having a hard time finding an example of posting using JSON format. Here is what this looks like in a curl command:
curl https://127.0.0.1:8834/scans -k -X POST -H 'Content-Type: application/json' -H 'X-Cookie: token= <token>' -d '{"uuid":"7485-2345-566","settings":{"name":"Testing TCL Network Scan","description":"Basic Network Scan using API", "policy_id":"872","text_targets":"127.0.0.1", "launch":"ONETIME","enabled":false,"launch_now":true}'

One problem you have is that the values in the query aren't evaluated. "uuid":"$uuid" becomes "uuid":"$uuid", for instance. This is because of the braces around the value that data is set to.
The best solution would seem to be to not create a json object and then convert it to a dict, but instead create the dict directly, like this:
set data [list uuid $uuid settings [list name $name description $desc policy_id $pid text_targets $target launch ONETIME enabled false launch_now true]]
or like this, for shorter lines:
dict set data uuid $uuid
dict set data settings name $name
dict set data settings description $desc
dict set data settings policy_id $pid
dict set data settings text_targets $target
dict set data settings launch ONETIME
dict set data settings enabled false
dict set data settings launch_now true
or by some other method.
Documentation: dict, list, set

Related

service discovery in tcl

I'm writing a little Tcl/Tk script, that uses a (custom) web-application (written in Python) to retrieve information from some central storage.
Everything works nicely, but only as long as the address of the webserver is known beforehand.
So I thought about adding some kind of service discovery, where my script would discover all running instances of my web-application on the local network, and automatically use them.
My first idea was to use Zeroconf/Bonjour/Avahi, and have my web-application publish a _my-web-service._tcp service with the actual query path (that the tcl client script should use to access the data) encoded in the TXT field:
avahi-publish-service MyService _my-web-service._tcp 8000 path=/data
Unfortunately, I haven't found anything that brings zeroconf-like service-discovery into the Tcl-world.
In particular, I was looking at the DNS entry on the Tcl Wiki but that only gets me as far as mDNS (and i currently have no clue how to proceed from there to zeroconf stack).
I'm not especially bound to Zeroconf/Bonjour/Avahi, but would like to run my script on Linux/Windows/macOS, and keep my build requirements minimal (that is: i would prefer it, if i don't have to compile my own wrapper code to interface with the service-discovery for each platform). Telling the users to install Bonjour or whatnot from 3rd-party sources would be tolerable though.
In particular, I was looking at the DNS entry on the Tcl Wiki but that
only gets me as far as mDNS (and i currently have no clue how to
proceed from there to zeroconf stack).
You were looking at the right corner, but the code snippet at the Tcl'ers Wiki appears outdated. I was curious and gave it some love.
This was tested using:
% package req Tcl
8.6.12
% package req dns
1.4.1
% package req udp
1.0.11
... and by announcing an exemplary service on macOS via:
dns-sd -R "Index" _http._tcp . 80 path=/index22.html
I managed to discover the above service using the patched dns package, by retrieving the DNS-SD (RFC 6763) specific records, mainly the target and port from the SRV record(s), and extras (e.g., a path) from the corresponding TXT record(s):
set instanceName "Index._http._tcp.local"
set tok [::dns::resolve $instanceName -protocol mdns -type SRV]
if {[::dns::wait $tok] eq "ok"} {
set res [dict create {*}[lindex [::dns::result $tok] 0]]; # Pick first answer record, only!
array set SRV [dict get $res rdata]
::dns::cleanup $tok
set tok [::dns::resolve $instanceName -protocol mdns -type TXT]
if {[::dns::wait $tok] eq "ok"} {
array set TXT {}
foreach txt [::dns::result $tok] {
lassign [split [dict get $txt rdata] "="] k v
set TXT($k) $v
}
::dns::cleanup $tok
}
set tok [::dns::resolve $SRV(target) -protocol mdns -type A]
if {[::dns::wait $tok] eq "ok"} {
set res [dict create {*}[lindex [::dns::result $tok] 0]]; # Pick first answer record, only!
puts "Service IP: [dict get $res rdata]"
puts "Service port: $SRV(port)"
puts "Service options: [array get TXT]"
}
::dns::cleanup $tok
}
This will print:
Service IP: 192.168.0.14
Service port: 80
Service options: path /index222.html
Patching tcllib's dns
The snippet from Tcl'ers Wiki needs to be modified, yielding:
proc ::dns::UdpTransmit {token} {
# FRINK: nocheck
variable $token
upvar 0 $token state
# setup the timeout
if {$state(-timeout) > 0} {
set state(after) [after $state(-timeout) \
[list [namespace origin reset] \
$token timeout\
"operation timed out"]]
}
if {[llength [package provide ceptcl]] > 0} {
# using ceptcl
set state(sock) [cep -type datagram $state(-nameserver) $state(-port)]
chan configure $state(sock) -blocking 0
} else {
# using tcludp
set state(sock) [udp_open]
if { $state(-protocol) eq "mdns" } {
set state(-nameserver) "224.0.0.251"
set state(-port) 5353
chan configure $state(sock) -mcastadd $state(-nameserver);
}
}
chan configure $state(sock) -remote [list $state(-nameserver) $state(-port)] \
-translation binary \
-buffering none;
set state(status) connect
chan event $state(sock) readable [list [namespace current]::UdpEvent $token]
puts -nonewline $state(sock) $state(request)
return $token
}
Background:
This is mainly to reflect changes to udp API (udp_conf is replaced by chan configure /socket/ -remote)
... but also: the original snippet does not join the multicast IP to listen for the mDNS responses (-mcastadd).
Other than that, dns is capable of decoding the DNS resource records (SRV, TXT) etc. just fine.
(that is: i would prefer it, if i don't have to compile my own wrapper
code to interface with the service-discovery for each platform)
This way, you do not have to interface with any third-party library or exec to some executable (dns-sd), but you will have to bundle your Tcl/Tk script with the platform-specific TclUDP extension, as a starpack or starkit, maybe?

Reuse TCP socket for HTTP requests

I'm trying to get the Tcl http package to reuse the initial TCP connection to a server to follow a redirect response. From the manual page I get the impression that specifying the option -keepalive 1 should accomplish that. However, in my tests I observe that two separate TCP connections are used. This is my code:
package require http
# Redirect test page
set url http://jigsaw.w3.org/HTTP/300/301.html
# Make the HTTP library report what it is doing
proc http::Log {args} {
puts [join $args]
}
set tok [http::geturl $url -keepalive 1]
if {[http::ncode $tok] in {301 302}} {
# Determine the new URL
set meta [http::meta $tok]
set key [lsearch -exact -nocase -inline [dict keys $meta] location]
set loc [dict get $meta $key]
http::cleanup $tok
puts "Redirecting to: $loc"
set tok [http::geturl $loc -keepalive 1]
}
puts [http::code $tok]
http::cleanup $tok
This produces the following output, clearly showing that a socket is opened and closed for each request:
^A1 URL http://jigsaw.w3.org/HTTP/300/301.html - token ::http::1
Using sock560e29012cc0 for jigsaw.w3.org:80 - token ::http::1 keepalive
^B1 begin sending request - token ::http::1
^C1 end sending request - token ::http::1
^D1 begin receiving response - token ::http::1
^E1 end of response headers - token ::http::1
^F1 end of response body (unchunked) - token ::http::1
Closing connection jigsaw.w3.org:80 (sock sock560e29012cc0)
Redirecting to: http://jigsaw.w3.org/HTTP/300/Overview.html
^A2 URL http://jigsaw.w3.org/HTTP/300/Overview.html - token ::http::2
Using sock560e28f6e820 for jigsaw.w3.org:80 - token ::http::2 keepalive
^B2 begin sending request - token ::http::2
^C2 end sending request - token ::http::2
^D2 begin receiving response - token ::http::2
^E2 end of response headers - token ::http::2
^F2 end of response body (unchunked) - token ::http::2
Closing connection jigsaw.w3.org:80 (sock sock560e28f6e820)
HTTP/1.1 200 OK
What else do I need to do to send multiple request using the same TCP connection?

headers parameters in http::getUrl

I am passing headers parameters in http::getUrl as
-headers {Authorization {Bearer $token} Content-Type application/json}
but it is not substituting the token value.
Right now I am creating a dict a step above, and then passing it directly
-headers $data
It their any way to pass the value directly, where variable substitution occurs
This is because string in {} are not evaluated, so you have 2 way here.
The first is using the list command:
-headers [list \
Authorization [list Bearer $token] \
Content-Type application/json ]
The second is by the subst command:
-headers [subst -nocommands \
{Authorization {Bearer $token} Content-Type application/json}]
The first method is the proper to be used as it is the equivalent to the OP example, just a different expression of a list.
The purpose of the second way, is in case you have the parameter input as a variable, and could not change the format in your code.
Notice that the case with subst works on a string and not a list as the previous method.
It should work the same as a string could be converted to a list easy:
$ tclsh
% set x "A B C D"
A B C D
% lindex $x 0
A
%
As you see the variable x is a string, but you could access as it is a list without problems.

Formatting HTTP Response Headers in tcl

I just started messing with tcl tonight and I've hit my first hurdle. Here is my code:
package require http
package require tls
::http::register https 443 ::tls::socket
set url https://127.0.0.1:8834/session
set body [http::formatQuery username admin password myPassword]
set login [http::geturl $url -query $body]
set authToken [http::data $login]
http::cleanup $token
This is how authToken looks after the code runs:
% puts $authToken
{"token":"d52e61030d93824128cea67e2b99dde6f3fd61b25e9a0440"}
I only need d52e61030d93824128cea67e2b99dde6f3fd61b25e9a0440 stored. I cannot figure out how to do this.
I tried using the rest package, but I couldn't get that to work either:
package require rest
package require tls
::http::register https 443 ::tls::socket
set nessus(login) {
url https://127.0.0.1:8834/session
method POST
req_args {username: password:}
}
rest::create_interface nessus
set token [nessus::login -username admin -password myPassword]
Here is my output from running this code:
% puts $token
token ba9c5b4256ef701bf7d8ae151c01261cb8a3267f1b8c2787
This may be a bit easier to format than the http code, I'm assuming.
In Python I was able to retrieve just that token key by pulling 'token' out of the json but I'm not sure how to accomplish that in tcl.
The response you are getting is in the form of a json object, with one key and one value. You can use the json package to turn it into a Tcl dict structure:
package require json
::json::json2dict [http::data $login]
# -> token d52e61030d93824128cea67e2b99dde6f3fd61b25e9a0440
The rest package autodetects xml and json, and apparently converts the object received into a dict before delivering it.
You can get the value by normal dictionary access:
dict get $token token
# -> ba9c5b4256ef701bf7d8ae151c01261cb8a3267f1b8c2787
# or
dict get [::json::json2dict [http::data $login]] token
# -> d52e61030d93824128cea67e2b99dde6f3fd61b25e9a0440
The lindex command could also be used, but it's probably better practice in the long run to use dictionary access.
Documentation: dict, http, json package, lindex, package
You can try this code:
package require rest
package require tls
::http::register https 443 ::tls::socket
set nessus(login) {
url https://127.0.0.1:8834/session
method POST
req_args {username: password:}
}
rest::create_interface nessus
set token [lindex [nessus::login -username admin -password myPassword] 1]

Simple JSON request with cURL to Mochiweb

I have a very simple 'hello world' Mochiweb server (I just started my introduction into it), which takes a JSON request and sends it back:
'POST' ->
case Path of
"dummy" ->
Data = Req:parse_post(),
Json = proplists:get_value("json", Data),
Struct = mochijson2:decode(Json),
Action_value = struct:get_value(<<"action">>, Struct),
Action = list_to_atom(binary_to_list(A)),
Result = [got_json_request, Action],
DataOut = mochijson2:encode(Result),
Req:ok({"application/json",[],[Result]});
The thing is that when I try to make a request to it with cURL it fails:
curl -i -H "Content-Type: application/json" -H "Accept: application/json" -X POST -d '{"action":"dummy"}' http://localhost:8080/dummy
The Mochiweb log is quite difficult to read, but as I understand the error happens at this line:
Json = proplists:get_value("json", Data)
I have put a couple of io:formats in the code and found out that both Data and Json variables are [] after I make a request with cURL.
On the other hand, when I do a very simple request with cURL:
curl -d '{"action":"dummy"}' http://localhost:8080/dummy
both Data and Json are [{"{\"action\":\"dummy\"}",[]}], but in that case the line Struct = mochijson2:decode(Json) fails.
For some strange reason Mochiweb does not see the JSON data in the POST request in case the header has the "application/json" value.
So, the question is: How do I make a correct POST request with JSON data to a Mochiweb server?
EDIT: Json variable has the undefined value.
Try something along the lines of
Data = Req:recv_body(),
Json = mochijson2:decode(Data),
...
You should at least ensure method post and the content type ahead of this.
This is not about POST nor get.
It's about how you post your data to send to your server
When you send a json data to server, you need to make it as key=value
curl -d "key=value" "http://your.domain.com/path"
Therefore, if you want to post json as '{"action":"dummy"}', for GET request
curl -d "json='{\"action\":\"dummy\"}'" http://localhost:8080/dummy
For POST request as a file,
curl -F "json=#filename.json" http://localhost:8080/dummy
of course, when you send as a file, you need to read the posted file from the server side.