JSON string fails to convert to correct Lua table - json

I'm performing a request to a foreign API using luasec, lua-socket and converting the data, a JSON string, to a lua table with cjson. I've read the docs of said modules and unfortunately none of it helped me with my problem. Can't link more than 2 websites with current account, sorry.
Summary: I get the response and the appropiate string using the posted request function, when turning said string into a lua table via cjson.decode the output table isn't the desired one, it's a copy of my response header, which is not intentional.
The following code is how I do my request:
local function request (req_t)
local res_t = {}
resp = https.request {
url = const.API_URL .. req_t.url,
method = req_t.method,
headers = req_t.headers,
sink = ltn12.sink.table(res_t)
}
return table.concat(res_t), resp.headers, resp.code
end
Using the following call
local res, headers = request({ ... })
I receive the proper response as a string but my goal is to do data manipulation with it, so turning said response(string) to a lua table with
local resJson = cjson.decode(res)
Does not produce the correct output. It does produce a table which is exactly the same as my response header. Here is the following output from my terminal alongside the code
When out of function type is: string
Desired response in string:
{"total_photos":221926,"photo_downloads":"186029632.0"}
When out of function type is: string
Desired response in string:
{"total_photos":221926,"photo_downloads":"186029632.0"}
After decode, type is: table
server Cowboy
strict-transport-security max-age=31536000
access-control-allow-headers *
x-ratelimit-limit 50
x-ratelimit-remaining 46
x-cache-hits 0, 0
accept-ranges bytes
access-control-request-method *
x-request-id ee5a74fd-2b10-4f46-9c25-5cfc53aeac6c
access-control-expose-headers Link,X-Total,X-Per-Page,X-RateLimit-Limit,X-RateLimit-Remaining
content-type application/json
connection close
content-length 55
fastly-debug-digest f62d52c08b1ef74db89a66a0069f0a35c49e52230567905240dacf08c9ea1813
vary Origin
cache-control no-cache, no-store, must-revalidate
x-timer S1496524765.369880,VS0,VE111
x-cache MISS, MISS
x-served-by cache-iad2123-IAD, cache-mad9429-MAD
via 1.1 vegur, 1.1 varnish, 1.1 varnish
date Sat, 03 Jun 2017 21:19:25 GMT
age 0
access-control-allow-origin *
x-runtime 0.011667
Printing header
server Cowboy
strict-transport-security max-age=31536000
access-control-allow-headers *
x-ratelimit-limit 50
x-ratelimit-remaining 46
x-cache-hits 0, 0
accept-ranges bytes
access-control-request-method *
x-request-id ee5a74fd-2b10-4f46-9c25-5cfc53aeac6c
access-control-expose-headers Link,X-Total,X-Per-Page,X-RateLimit-Limit,X-RateLimit-Remaining
content-type application/json
connection close
content-length 55
fastly-debug-digest f62d52c08b1ef74db89a66a0069f0a35c49e52230567905240dacf08c9ea1813
vary Origin
cache-control no-cache, no-store, must-revalidate
x-timer S1496524765.369880,VS0,VE111
x-cache MISS, MISS
x-served-by cache-iad2123-IAD, cache-mad9429-MAD
via 1.1 vegur, 1.1 varnish, 1.1 varnish
date Sat, 03 Jun 2017 21:19:25 GMT
age 0
access-control-allow-origin *
x-runtime 0.011667
Function that produces said log
local res, headers = request({ ... })
print('When out of function type is: ' ..type(res) .. '\n')
print('Desired response in string:')
print(res .. '\n')
resJson = cjson.decode(res)
print('\nAfter decode, type is: ' .. type(resJson) .. '\n')
pTable(resJson)
print('\nPrinting header\n')
pTable(headers)
pTable is just a function to output a table to stdout.
Thanks in advance

Posted function and routines are correct. The problem was located in my print table function, which I somehow had hardcoded my headers.

Related

Karate properties.json throws ReferenceError when Reading Properties in *.feature file

Having updated Karate from 0.6.2 to 0.9.5 recently I've had a number of ReferenceError's w.r.t the properties.json I've used throughout my test cases.
I've the following setup:
test-properties.json
{
"headers": {
"x-client-ip": "192.168.3.1",
"x-forwarded-for": "192.168.3.1"
}
}
test-auth.feature
Background:
* def props = read('properties/test-properties.json')
I then use props further down in my first scenario:
And header User-Agent = props.headers.Accept-Language
And header X-Forwarded-For = props.headers.x-forwarded-for
However, when running this I get the following issue:
com.intuit.karate.exception.KarateException: test-auth.feature:14 - javascript evaluation failed: props.headers.Accept-Language, ReferenceError: "Language" is not defined in <eval> at line number 1
I've tried adding the properties file into the same package as the test-auth.feature to no avail. The issue seems to be with reading the json file. I'm aware Karate 0.6.2 could evaluate the file type and parse it internally in its native format. Is this still the case? If not, what is the solution to reading from properties.json in Karate 0.9.5.
Nothing should have changed when it comes to reading JSON files. Karate evaluates the RHS as JS, so I think this is the solution:
And header User-Agent = props.headers['Accept-Language']
And header X-Forwarded-For = props.headers['x-forwarded-for']
EDIT: this works for me:
* def props = { headers: { 'Accept-Language': 'foo', 'x-forwarded-for': 'bar' } }
* url 'http://httpbin.org/headers'
* header User-Agent = props.headers['Accept-Language']
* header X-Forwarded-For = props.headers['x-forwarded-for']
* method get
Resulting in:
1 > GET http://httpbin.org/headers
1 > Accept-Encoding: gzip,deflate
1 > Connection: Keep-Alive
1 > Host: httpbin.org
1 > User-Agent: foo
1 > X-Forwarded-For: bar
So if you are still stuck, please follow this process: https://github.com/intuit/karate/wiki/How-to-Submit-an-Issue

FCM Device group BadJsonFormat - properly formed json

I was flooded with a primitive json body for fcm:
Body = mochijson2:encode([ {<<"operation">>, <<"create">>},{<<"notification_key_name">>, <<"console group">>},{<<"registration_ids">>, [<<"02aa6XXXX3c9b6d">>,<<"APA91bGtaXXXXXXXXXXXXoi4UH8vIdZk1X67A_9izpSFSHV3BXxdIwG">>]}]).
And send POST-request to create group according to documentation:
httpc:request(post, {Url, [{"Authorization", KeyApi}, {"project_id", ProjectId}], "application/json", Body},[{timeout, 5000}], []).
But I got error BadJsonFormat:
{ok,{{"HTTP/1.1",400,"Bad Request"},
[{"cache-control","private, max-age=0"},
{"date","Fri, 10 Mar 2017 16:19:37 GMT"},
{"accept-ranges","none"},
{"server","GSE"},
{"vary","Accept-Encoding"},
{"content-length","25"},
{"content-type","application/json; charset=UTF-8"},
{"expires","Fri, 10 Mar 2017 16:19:37 GMT"},
{"x-content-type-options","nosniff"},
{"x-frame-options","SAMEORIGIN"},
{"x-xss-protection","1; mode=block"},
{"alt-svc","quic=\":443\"; ma=2592000; v=\"36,35,34\""}],
"{\"error\":\"BadJsonFormat\"}"}}
But mochijson2:decode(Body) works fine, and it looks like properly formed json, but I get the error BadJsonFormat anyway.
What was wrong? How can I fix this?
The function mochijson2:encode doesn't return a string or a binary, but an iolist:
1> Body = mochijson2:encode([ {<<"operation">>, <<"create">>},{<<"notification_key_name">>, <<"console group">>},{<<"registration_ids">>, [<<"02aa6XXXX3c9b6d">>,<<"APA91bGtaXXXXXXXXXXXXoi4UH8vIdZk1X67A_9izpSFSHV3BXxdIwG">>]}]).
[123,
[34,<<"operation">>,34],
58,
[34,<<"create">>,34],
44,
[34,<<"notification_key_name">>,34],
58,
[34,<<"console group">>,34],
44,
[34,<<"registration_ids">>,34],
58,
[91,
[34,<<"02aa6XXXX3c9b6d">>,34],
44,
[34,<<"APA91bGtaXXXXXXXXXXXXoi4UH8vIdZk1X67A_9izpSF"...>>,
34],
93],
125]
There is nothing wrong with that, by itself. Using iolists instead of strings or binaries means that you don't have to create an expensive flat data structure, that you would just write to a file or a socket, after which you'd throw it away. Function like file:write_file and gen_tcp:send handle iolists just as well as strings or binaries.
However, httpc:request doesn't!
Let's test that by starting a server on port 1111 with netcat in a shell:
$ nc -l 1111
And then make a request from the Erlang shell:
3> httpc:request(post, {"http://127.0.0.1:1111", [], "application/json", Body},[{timeout, 5000}], []).
The netcat server shows this output:
POST / HTTP/1.1
content-type: application/json
content-length: 13
te:
host: 127.0.0.1:1111
connection: keep-alive
{"operation":"create",....
Note that the content-length is 13 instead of 159! httpc:request is able to send the iolist, but it uses the function length instead of iolist_size to generate the content-length header, and as a result the server only considers the first 13 bytes of the JSON object, which is not valid JSON by itself.
The solution is to pass iolist_to_binary(Body) to httpc:request instead of just Body.

Json parse error using POST in django rest api

I am trying to implement a simple GET/POST api via Django REST framework
views.py
class cuser(APIView):
def post(self, request):
stream = BytesIO(request.DATA)
json = JSONParser().parse(stream)
return Response()
urls.py
from django.conf.urls import patterns, url
from app import views
urlpatterns = patterns('',
url(r'^challenges/',views.getall.as_view() ),
url(r'^cuser/' , views.cuser.as_view() ),
)
I am trying to POST some json to /api/cuser/ (api is namespace in my project's urls.py ) ,
the JSON
{
"username" : "abhishek",
"email" : "john#doe.com",
"password" : "secretpass"
}
I tried from both Browseable API page and httpie ( A python made tool similar to curl)
httpie command
http --json POST http://localhost:58601/api/cuser/ username=abhishek email=john#doe.com password=secretpass
but I am getting JSON parse error :
JSON parse error - Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
Whole Debug message using --verbose --debug
POST /api/cuser/ HTTP/1.1
Content-Length: 75
Accept-Encoding: gzip, deflate
Host: localhost:55392
Accept: application/json
User-Agent: HTTPie/0.8.0
Connection: keep-alive
Content-Type: application/json; charset=utf-8
{"username": "abhishek", "email": "john#doe.com", "password": "aaezaakmi1"}
HTTP/1.0 400 BAD REQUEST
Date: Sat, 24 Jan 2015 09:40:03 GMT
Server: WSGIServer/0.1 Python/2.7.9
Vary: Accept, Cookie
Content-Type: application/json
Allow: POST, OPTIONS
{"detail":"JSON parse error - Expecting property name enclosed in double quotes: line 1 column 2 (char 1)"}
The problem that you are running into is that your request is already being parsed, and you are trying to parse it a second time.
From "How the parser is determined"
The set of valid parsers for a view is always defined as a list of classes. When request.data is accessed, REST framework will examine the Content-Type header on the incoming request, and determine which parser to use to parse the request content.
In your code you are accessing request.DATA, which is the 2.4.x equaivalent of request.data. So your request is being parsed as soon as you call that, and request.DATA is actually returning the dictionary that you were expecting to parse.
json = request.DATA
is really all you need to parse the incoming JSON data. You were really passing a Python dictionary into json.loads, which does not appear to be able to parse it, and that is why you were getting your error.
I arrived at this post via Google for
"detail": "JSON parse error - Expecting property name enclosed in double-quotes":
Turns out you CANNOT have a trailing comma in JSON.
So if you are getting this error you may need to change a post like this:
{
"username" : "abhishek",
"email" : "john#doe.com",
"password" : "secretpass",
}
to this:
{
"username" : "abhishek",
"email" : "john#doe.com",
"password" : "secretpass"
}
Note the removed comma after the last property in the JSON object.
Basically, whenever you are trying to make a post request with requests lib, This library also contains json argument which is ignored in the case when data argument is set to files or data. So basically when json argument is set with json data. Headers are set asContent-Type: application/json. Json argument basically encodes data sends into a json format. So that at DRF particularly is able to parse json data. Else in case of only data argument it is been treated as form-encoded
requests.post(url, json={"key1":"value1"})
you can find more here request.post complicated post methods

Socrata API for Delete in .csv?

I am wondering if there is a way to send a DELETE request using .csv format instead of .json format?
API Reference: http://dev.socrata.com/publishers/direct-row-manipulation.html
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Date: Thu, 27 Mar 2014 00:48:42 GMT
[
{
"typ": "delete",
"id": "row-evac~sxbs~gm8t"
}
]
I tried something along the lines of:
typ, id
delete, row-evac~sxbs~gm8t
to no avail.
You can delete using a CSV payload by adding a ":deleted" column and passing a value of true. You can also omit all the other columns but the ID. Example:
Earthquake ID,:deleted
15215753, true
Make sure you use a Content-Type of "text/csv" as well.

HTML5 video and partial range HTTP requests

I'm trying to modify a custom web server app to work with HTML5 video.
It serves a HTML5 page with a basic <video> tag and then it needs to handle the requests for actual content.
The only way I could get it to work so far is to load the entire video file into the memory and then send it back in a single response. It's not a practical option. I want to serve it piece by piece: send back, say, 100 kb, and wait for the browser to request more.
I see a request with the following headers:
http_version = 1.1
request_method = GET
Host = ###.###.###.###:##
User-Agent = Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20100101 Firefox/16.0
Accept = video/webm,video/ogg,video/*;q=0.9,application/ogg;q=0.7,audio/*;q=0.6,*/*;q=0.5
Accept-Language = en-US,en;q=0.5
Connection = keep-alive
Range = bytes=0-
I tried to send back a partial content response:
HTTP/1.1 206 Partial content
Content-Type: video/mp4
Content-Range: bytes 0-99999 / 232725251
Content-Length: 100000
I get a few more GET requests, as follows
Cache-Control = no-cache
Connection = Keep-Alive
Pragma = getIfoFileURI.dlna.org
Accept = */*
User-Agent = NSPlayer/12.00.7601.17514 WMFSDK/12.00.7601.17514
GetContentFeatures.DLNA.ORG = 1
Host = ###.###.###.###:##
(with no indication that the browser wants any specific part of the file.) No matter what I send back to the browser, the video does not play.
As stated above, the same video will play correctly if I try to send the entire 230 MB file at once in the same HTTP packet.
Is there any way to get this all working nicely through partial content requests? I'm using Firefox for testing purposes, but it needs to work with all browsers eventually.
I know this is an old question, but if it helps you can try the following "Model" that we use in our code base.
class Model_DownloadableFile {
private $full_path;
function __construct($full_path) {
$this->full_path = $full_path;
}
public function get_full_path() {
return $this->full_path;
}
// Function borrowed from (been cleaned up and modified slightly): http://stackoverflow.com/questions/157318/resumable-downloads-when-using-php-to-send-the-file/4451376#4451376
// Allows for resuming paused downloads etc
public function download_file_in_browser() {
// Avoid sending unexpected errors to the client - we should be serving a file,
// we don't want to corrupt the data we send
#error_reporting(0);
// Make sure the files exists, otherwise we are wasting our time
if (!file_exists($this->full_path)) {
header('HTTP/1.1 404 Not Found');
exit;
}
// Get the 'Range' header if one was sent
if (isset($_SERVER['HTTP_RANGE'])) {
$range = $_SERVER['HTTP_RANGE']; // IIS/Some Apache versions
} else if ($apache = apache_request_headers()) { // Try Apache again
$headers = array();
foreach ($apache as $header => $val) {
$headers[strtolower($header)] = $val;
}
if (isset($headers['range'])) {
$range = $headers['range'];
} else {
$range = false; // We can't get the header/there isn't one set
}
} else {
$range = false; // We can't get the header/there isn't one set
}
// Get the data range requested (if any)
$filesize = filesize($this->full_path);
$length = $filesize;
if ($range) {
$partial = true;
list($param, $range) = explode('=', $range);
if (strtolower(trim($param)) != 'bytes') { // Bad request - range unit is not 'bytes'
header("HTTP/1.1 400 Invalid Request");
exit;
}
$range = explode(',', $range);
$range = explode('-', $range[0]); // We only deal with the first requested range
if (count($range) != 2) { // Bad request - 'bytes' parameter is not valid
header("HTTP/1.1 400 Invalid Request");
exit;
}
if ($range[0] === '') { // First number missing, return last $range[1] bytes
$end = $filesize - 1;
$start = $end - intval($range[0]);
} else if ($range[1] === '') { // Second number missing, return from byte $range[0] to end
$start = intval($range[0]);
$end = $filesize - 1;
} else { // Both numbers present, return specific range
$start = intval($range[0]);
$end = intval($range[1]);
if ($end >= $filesize || (!$start && (!$end || $end == ($filesize - 1)))) {
$partial = false;
} // Invalid range/whole file specified, return whole file
}
$length = $end - $start + 1;
} else {
$partial = false; // No range requested
}
// Determine the content type
$finfo = finfo_open(FILEINFO_MIME_TYPE);
$contenttype = finfo_file($finfo, $this->full_path);
finfo_close($finfo);
// Send standard headers
header("Content-Type: $contenttype");
header("Content-Length: $length");
header('Content-Disposition: attachment; filename="' . basename($this->full_path) . '"');
header('Accept-Ranges: bytes');
// if requested, send extra headers and part of file...
if ($partial) {
header('HTTP/1.1 206 Partial Content');
header("Content-Range: bytes $start-$end/$filesize");
if (!$fp = fopen($this->full_path, 'r')) { // Error out if we can't read the file
header("HTTP/1.1 500 Internal Server Error");
exit;
}
if ($start) {
fseek($fp, $start);
}
while ($length) { // Read in blocks of 8KB so we don't chew up memory on the server
$read = ($length > 8192) ? 8192 : $length;
$length -= $read;
print(fread($fp, $read));
}
fclose($fp);
} else {
readfile($this->full_path); // ...otherwise just send the whole file
}
// Exit here to avoid accidentally sending extra content on the end of the file
exit;
}
}
You then use it like this:
(new Model_DownloadableFile('FULL/PATH/TO/FILE'))->download_file_in_browser();
It will deal with sending part of the file or the full file etc and works well for us in this and lots of other situations. Hope it helps.
I want partial range requests, because I'll be doing realtime transcoding, I can't have the file completely transcoded and available upon request.
For response which you don't know the full body content yet (you can't guess the Content-Length, live encoding), use chunk encoding:
HTTP/1.1 200 OK
Content-Type: video/mp4
Transfer-Encoding: chunked
Trailer: Expires
1E; 1st chunk
...binary....data...chunk1..my
24; 2nd chunk
video..binary....data....chunk2..con
22; 3rd chunk
tent...binary....data....chunk3..a
2A; 4th chunk
nd...binary......data......chunk4...etc...
0
Expires: Wed, 21 Oct 2015 07:28:00 GMT
Each chunk is send when it's available: when few frames are encoded or when the output buffer is full, 100kB are generated, etc.
22; 3rd chunk
tent...binary....data....chunk3..a
Where 22 give the chunk byte length in hexa (0x22 = 34 bytes),
; 3rd chunk is extra chunk infos (optional) and tent...binary....data....chunk3..a is the content of the chunk.
Then, when the encoding is finished and all chunks are sent, end by:
0
Expires: Wed, 21 Oct 2015 07:28:00 GMT
Where 0 means there not more chunks, followed by zero or more trailer (allowed header fields) defined in the header (Trailer: Expires and Expires: Wed, 21 Oct 2015 07:28:00 GMT are not required) to provide checksums or digital signatures, etc.
Here is the equivalent of the server's response if the file was already generated (no live encoding):
HTTP/1.1 200 OK
Content-Type: video/mp4
Content-Length: 142
Expires: Wed, 21 Oct 2015 07:28:00 GMT
...binary....data...chunk1..myvideo..binary....data....chunk2..content...binary....data....chunk3..and...binary......data......chunk4...etc...
For more information: Chunked transfer encoding — Wikipedia, Trailer - HTTP | MDN