I am making a sideband connection in F5 Big-IP and getting the response headers as a utf-8 string
I want to extract the value of the cookies from a string like:
"Set-Cookie {__aaaa=724131337970; HttpOnly; path=/; Expires=Sun, 27-Nov-22 06:43:44 GMT ; Max-Age=15724800; SameSite=Lax} Set-Cookie {__bbbb=1653806624; HttpOnly; path=/; Expires=Sun, 27-Nov-22 06:43:44 GMT ; Max-Age=15724800; SameSite=Lax} Set-Cookie {__ccc=1653806624; HttpOnly; path=/; Expires=Sun, 27-Nov-22 06:43:44 GMT ; Max-Age=15724800; SameSite=Lax}"
I need to comeup with a proc to extract the value of the cookies from "_aaa=..." to the end of its value and the proc shoud get all the cookie values no matter if there is two cookies or three cookies or even more
in the end I need to have a list of those cookies like:
{ {__aaaa=724131337970; HttpOnly; path=/; Expires=Sun, 27-Nov-22 06:43:44 GMT ; Max-Age=15724800; SameSite=Lax} {__bbbb=1653806624; HttpOnly; path=/; Expires=Sun, 27-Nov-22 06:43:44 GMT ; Max-Age=15724800; SameSite=Lax} {__ccc=1653806624; HttpOnly; path=/; Expires=Sun, 27-Nov-22 06:43:44 GMT ; Max-Age=15724800; SameSite=Lax} }
What is the best way to do it? using string range, findstr or using regexp?
That input descriptor looks a lot like a list where every second item is the content of a Set-Cookie header. The spec for handling cookies is complicated and what you see in practice are not exactly the things that the RFC describes. (I wrote Tcl 8.7's cookie support; I looked at this in some depth.)
The header is a semicolon-separated list where the first item is the cookie name and value, and subsequent parts are options (all of those are key/value with = separating the two). The key option you might care about here is the Expires header, which is absent in session cookies, present and in the future for persistent cookies, and present and in the past to delete a cookie. (That last part is not documented anywhere!) However, handling that gets really complicated — there isn't a consistent workable standard for what the format of the date is, and then you've got to do additional work to handle storage and so on — that I'll omit it here.
# Much much easier to do some of this work in a procedure
proc parseCookieHeader {header} {
set pieces [split $header ";"]
if {![regexp {^([^=]+)=(.*)$} [lindex $pieces 0] -> key value]} {
error "malformed Set-Cookie header"
}
# Cookie option parsing, at least at a basic level
set options {}
foreach option [lrange $pieces 1 end] {
if {[regexp {^([^=]+)=(.*)$} $option -> k v]} {
dict set options [string tolower [string trim $k]] [string trim $v]
}
}
return [list [string trim $key] [string trim $value] $options]
}
foreach {a b} $input { # $input is the string you started with
# Drop the options for simplicity
lassign [parseCookieHeader $b] key value
# Store them all in an array; the order shouldn't matter
set cookies($key) $value
}
parray cookies
# cookies(__aaaa) = 724131337970
# cookies(__bbbb) = 1653806624
# cookies(__ccc) = 1653806624
If you have multiple cookies with the same name, the last one wins. This is how it works with real sites too, and is another part of why handling cookies is really messy. (The code also assumes it doesn't have to do any complex policy enforcement such as limiting what paths or hosts they apply to.)
Related
How to extract the username value from this query string (HTTP url-encoded): username=james&password=pwd in Tcl?
I can get it through Java's request.getParameter("username"); but how to get using Tcl?
The first stage is to split the query string up, and form a dictionary of it (which isn't strictly correct, but I'm guessing you don't care about the case where someone puts multiple username fields in the query string!). However, you also need to decode the encoding of the contents, and that's pretty awful:
proc QueryStringToDict {qs} {
set mapping {}
foreach item [split $qs "&"] {
if {[regexp {^([^=]+)=(.*)$} $item -> key value]} {
dict set mapping [DecodeURL $key] [DecodeURL $value]
}
}
return $mapping
}
proc DecodeURL {string} {
# This *is* tricky! The URL encoding of fields is way nastier than you thought!
set mapped [string map {+ { } \[ "\\\[" \] "\\\]" $ "\\$" \\ "\\\\"} $string]
encoding convertfrom utf-8 \
[subst [regsub -all {%([[:xdigit:]]{2})} $string {[format %c 0x\1]}]]
}
set qs "username=james&password=pwd"
set info [QueryStringToDict $qs]
puts "user name is [dict get $info username]"
In 8.7 (currently in alpha) it'll be much simpler to do that inner encoding; there won't need to be that subst call in there for example. But you haven't got that version of Tcl; nobody has (except for people who insist on being right on the bleeding edge and get themselves into trouble over it).
Assuming this is a CGI environment, where the environment will contain
REQUEST_METHOD=GET
QUERY_STRING='username=james&password=pwd'
or
REQUEST_METHOD=POST
CONTENT_LENGTH=27
# and stdin contains "username=james&password=pwd"
then use tcllib's ncgi module
$ cat > cgi.tcl
#!/usr/bin/env tclsh
package require ncgi
::ncgi::parse
array set params [::ncgi::nvlist]
parray params
$ printf "username=james&password=pwd" | env REQUEST_METHOD=POST CONTENT_LENGTH=27 ./cgi.tcl
params(password) = pwd
params(username) = james
$ env REQUEST_METHOD=GET QUERY_STRING='username=james&password=pwd' ./cgi.tcl
params(password) = pwd
params(username) = james
An alternative to Donal's suggestion, sharing the spirit, but building on battery pieces: tcllib rest package:
(1) To process the query (as part of a valid URL)
% package req rest
1.3.1
% set query [rest::parameters ?username=jo%3Dhn]; # http:// is default scheme, ? is minimum URL boilerplate
username jo%3Dhn
(2) Run a URL decoder (e.g., the one by Donal or the one from Rosetta code):
% proc urlDecode {str} {
set specialMap {"[" "%5B" "]" "%5D"}
set seqRE {%([0-9a-fA-F]{2})}
set replacement {[format "%c" [scan "\1" "%2x"]]}
set modStr [regsub -all $seqRE [string map $specialMap $str] $replacement]
return [encoding convertfrom utf-8 [subst -nobackslash -novariable $modStr]]
}
then:
% set info [lmap v $query {urlDecode $v}]
username jo=hn
% dict get $info username
jo=hn
Essentially what I'm attempting to do is post to a REST API, but no matter what I do I end up with HTTP 400. Here is my extremely quick and extremely dirty code:
package require rest
package require json
::http::register https 443 ::tls::socket
set credentials {username admin password LabPass1}
set url1 [format "%s/%s" "https://127.0.0.1:8834" session]
set unformattedToken [dict get [::json::json2dict [::rest::post $url1 $credentials]] token]
set cookie [format "token=%s" $unformattedToken]
set header [list X-Cookie $cookie Content-type application/json]
set config [list method post format json headers $header]
set url [format "%s/%s" "https://127.0.0.1:8834" scans]
set uuid 7485-2345-566
set name "Testing TCL Network Scan"
set desc "Basic Network Scan using API"
set pid 872
set target 127.0.0.1
set data {{"uuid":"$uuid","settings": {"name":"$name","description":"$desc", "policy_id":"$pid","text_targets":"$target", "launch":"ONETIME","enabled":false,"launch_now":true}}}
set jsonData [json::json2dict $data]
set response [::rest::simple $url $jsonData $config]
I've tried using the above code and I've also tried removing the json::json2dict call and just sending the data. I believe, and I could be wrong, that my issue is the data is going as line-based text data:
POST /scans HTTP/1.1
Host: 127.0.0.1:8834
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 10.0) http/2.8.9 Tcl/8.6.4
Connection: close
X-Cookie: token=301b8dcdf855a29b5b902cf8d93c49750935c925a965445e
Content-type: application/json
Accept: */*
Accept-Encoding: gzip,deflate,compress
Content-Length: 270
uuid=7485-2345-566&settings=name%20%7BTesting%20TCL%20Network%20Scan%7D%20description%20%7BBasic%20Network%20Scan%20using%20API%7D%20policy_id%20872%20text_targets%20127.0.0.1%20launch%20ONETIME%20enabled%20false%20launch_now%20true
I've reviewed the JSON documentation, and the REST documentation but I'm having a hard time finding an example of posting using JSON format. Here is what this looks like in a curl command:
curl https://127.0.0.1:8834/scans -k -X POST -H 'Content-Type: application/json' -H 'X-Cookie: token= <token>' -d '{"uuid":"7485-2345-566","settings":{"name":"Testing TCL Network Scan","description":"Basic Network Scan using API", "policy_id":"872","text_targets":"127.0.0.1", "launch":"ONETIME","enabled":false,"launch_now":true}'
One problem you have is that the values in the query aren't evaluated. "uuid":"$uuid" becomes "uuid":"$uuid", for instance. This is because of the braces around the value that data is set to.
The best solution would seem to be to not create a json object and then convert it to a dict, but instead create the dict directly, like this:
set data [list uuid $uuid settings [list name $name description $desc policy_id $pid text_targets $target launch ONETIME enabled false launch_now true]]
or like this, for shorter lines:
dict set data uuid $uuid
dict set data settings name $name
dict set data settings description $desc
dict set data settings policy_id $pid
dict set data settings text_targets $target
dict set data settings launch ONETIME
dict set data settings enabled false
dict set data settings launch_now true
or by some other method.
Documentation: dict, list, set
Using tcl http package I am connecting to url with the credentials required. After that I want to send a query, but geturl proc doesn't have an option to use existing token id.
set auth "Basic [base64::encode test:test123]"
set headerl [list Authorization $auth]
set tok [http::geturl http://192.168.2.77:9001 -headers $headerl -timeout 10000 -type text/html]
http::wait $tok
if {![string compare [http::status $tok] "ok"]} {
puts [http::data $tok]
} else {
puts stderr [http::error $tok]
}
Here I want to send a query for the subsequent page, but i couldn't find an option to do that
If I do geturl again then it is throwing error saying that authorization failed.
set tok [http::geturl http://192.168.4.77:9001/index.html?action=stopall -timeout 10000 -type text/html]
http::wait $tok
if {![string compare [http::status $tok] "ok"]} {
puts [http::data $tok]
} else {
puts stderr [http::error $tok]
}
<head>
<title>Error response</title>
</head>
<body>
<h1>Error response</h1>
<p>Error code 401.
<p>Message: Unauthorized.
</body>
Thanks in advance
You need a cookie jar for holding the response cookies. Here's an example that adds cookies to a jar and sends them back. Be aware that it doesn't check cookie attributes which you need to do to avoid exposing session keys.
#!/usr/bin/tclsh
package require http
proc wget {url {jarname COOKIE_JAR}} {
set o_headers {User-Agent {Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0)}}
upvar $jarname cookiejar
set cookies [list]
# WARNING - Blindly sending cookies in jar. You chould check attributes.
# domain, path, expires, httponly, secure, max-age
# Adding cookiename=value without attributes. Attributes aren't for sending.
foreach {value} $cookiejar {lappend cookies [lindex [split $value ";"] 0]}
# Add cookies to the header.
lappend o_headers "Cookie" [join $cookies "; "]
set tok [::http::geturl $url -timeout 10000 -method GET -query "" -headers $o_headers]
# add/replace cookies in the return headers.
foreach {key value} [::http::meta $tok] {
if {[string tolower $key] != "set-cookie"} {continue}
set cookie_key [lindex [regexp -inline {\s*([^=]+)} $value] 1]
if {[set index [lsearch -glob $cookiejar "$cookie_key=*"]] != -1} {
# Replace if cookie already exists
lset cookiejar $index $value
continue
}
lappend cookiejar $value
}
return [::http::data $tok]
}
set google_cookies [list]
set page http://www.google.com
puts "Cookies before request.\n$google_cookies"
wget $page google_cookies
puts "Cookies after request.\n$google_cookies"
Output:
./wget
Cookies before request.
Cookies after request.
{PREF=ID=xxx:FF=0:TM=xxx:LM=xx:S=xxx-xxx; expires=Sun, 18-Dec-2016 02:26:37 GMT; path=/; domain=.google.com} {NID=12=x_xx_xxx-xxxx_xxxxx; expires=Sat, 20-Jun-2015 02:26:37 GMT; path=/; domain=.google.com; HttpOnly}
I couldn't think of better keywords to Google this issue, so I apologize if this is a duplicate.
Here is my logout.pl script that basically erases cookie:
#!/usr/bin/perl -w
use strict;
use warnings;
use CGI;
my $q = new CGI;
print $q->header('text/html');
my $cookie = $q->cookie(
-name => 'CGISESSID',
-value => '',
-expires => '-1d'
);
print $q->header(-cookie=>$cookie);
print $q->redirect('welcome.pl');
exit;
When I run this script in a browser, it prints the following:
Set-Cookie: CGISESSID=; path=/; expires=Mon, 17-Feb-2014 09:05:42 GMT Date: Tue, 18 Feb 2014 09:05:42 GMT Content-Type: text/html; charset=ISO-8859-1 Status: 302 Found Location: welcome.pl
What I want, however, is for the browser to delete the cookie and redirect to welcome.pl.
When you print $q->header, that prints all the headers, including the blank line which signals the end of headers, making anything after it content. You need to only print $q->header once, no more.
There is actually one more problem you might not figure out on your own. The “clear” cookie you’re trying to send to expire the session must be sent with the redirect. The -w switch is not usually what you want, just the use warnings you have too. Also, redirect URLs RFC:MUST be absolute. "welcome.pl" will in most likelihood work but it’s not a good practice and I had relative URIs bite very badly in a modperl app once. So, amended–
#!/usr/bin/env perl
use strict;
use warnings;
use CGI;
use URI;
my $q = CGI->new;
my $cookie = $q->cookie(
-name => 'CGISESSID',
-value => '',
-expires => '-1d'
);
my $welcome = URI->new_abs("welcome.pl", $q->url);
print $q->redirect( -uri => $welcome,
-cookie => $cookie,
-status => 302 );
exit;
You should use $q->header only once in your script and that should be before using anything printable on page
I am a newbie in expect / TCL and trying to parse an HTML page that has output some thing like below:
<li><p>Timestamp: Wed, 14 Nov 2012 16:37:50 -0800
<li><p>Your IP address: 202.76.243.10</p></li>
<li><p class="XXX_no_wrap_overflow_hidden">Requested URL: /</p></li>
<li><p>Error reference number: 1003</p></li>
<li><p>Server ID: FL_23F7</p></li>
<li><p>Process ID: PID_1352939870.809-1-428432242</p></li>
<li><p>User-Agent: </p></li>
My script is below. I am able to get the web page which I am not able to parse the line "Your IP address:" which is giving me errors:
#!/usr/bin/expect -f
set timeout -1
spawn telnet www.whatismyip.com 80
send "GET /\r\n"
expect
set output $expect_out(buffer)
foreach line [split $output \n] {
regexp {.*<li><p>Your IP Address Is:.*?(\d+\.\d+\.\d+\.\d+)} $line ip
if {[string length ${ip}]} {
puts $ip
}
}
The error is:
Connection closed by foreign host.
can't read "ip": no such variable
while executing
"string length ${ip}"
("foreach" body line 3)
invoked from within
"foreach line [split $output \n] {
regexp {.*<li><p>Your IP Address Is:.*?(\d+\.\d+\.\d+\.\d+)} $line ip
if {[string length ${ip}]} {
..."
(file "./t4" line 7)
Any pointers where I am doing wrong?
The regular expression did not match, so the variable was not assigned. You should check the result of regexp to see if the match succeeded; when not using the -all option to regexp, you can treat it like a boolean. Try this:
foreach line [split $output \n] {
if {[regexp {<li><p>Your IP Address Is:.*?(\d+\.\d+\.\d+\.\d+)(?!\d)} $line -> ip]} {
puts $ip
}
}
The -> is really a (weird!) variable name which will hold the whole matched string; we're not interested in it (just the parenthetical part) so we use the non-alphabetic to mnemonically say “this is going to there” (the submatch to the ip variable).
Your line contains "address" (lowercase) but you're trying to match "Address" (uppercase). Add the
-nocase option to the regexp command. Also, Tcl regular expressions cannot have mixed greediness -- the first quantifier determines if the whole expression is greedy or non-greedy (I can't find where this is documented right now).
regexp -nocase {IP Address.*(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})} $line -> ip
If your ultimate goal is to get your host's external IP, then go with an API solution, such as one from exip.org:
#!/usr/bin/env tclsh
set api http://api-nyc01.exip.org/?call=ip
if {[catch {exec curl --silent $api} output]} {
puts "Failed to acquire external IP"
} else {
puts "My external IP is $output"
}
Please visit their API site for more information, especially if you live outside the USA. This solution requires curl, which you might need to install.