POST issue with json service using cpprest-sdk, interfacing to jeasyui - json

Question

I'm developing a web service to serve json objects to a jeasyui async tree. My HTML has the following:
<ul id="tt" method="POST" class="easyui-tree" url="http://w.x.y.z:1024/testrest">
</ul>
Assume w.x.y.z is my server's IP address. According to the jeasyui documentation for their PHP json service, I need to return an array of dictionary objects that have keys id, text, and state. Okay, so far so good. I am attempting to develop a json service in c++ using the cpprest-sdk from Microsoft. I compiled and installed this library on RHEL 7.2 and am able to write some basic services using it. The problem lies (I think) with the encoding of the json that gets sent back to the client.
Here's a fully-functional example json server written with cpprest-sdk that handles POST requests and replies with a singly-populated array of dictionary objects that conform to the protocol expected by jeasyui:
#include <cpprest/http_listener.h>
#include <cpprest/json.h>
#pragma comment(lib, "cpprestlib" )
using namespace web;
using namespace web::http;
using namespace web::http::experimental::listener;
#include <iostream>
#include <map>
#include <set>
#include <string>
using namespace std;
#define TRACE(msg) wcout << msg
void handle_request(http_request request, function<void(const json::value &, json::value &, bool)> action)
{
json::value answer;
TRACE("\nHandle_request\n");
// Spit out the HTTP header to the console...
const auto HeaderString = request.to_string();
wcout << HeaderString.c_str() << endl;
request
.extract_json()
.then([&answer, &action](pplx::task<json::value> task) {
try
{
const auto & jvalue = task.get();
if (!jvalue.is_null())
{
action(jvalue, answer, false);
}
else
{
action(jvalue, answer, true);
}
}
catch (http_exception const & e)
{
wcout << "HTTP exception in handle_request: " << e.what() << endl;
}
})
.wait();
request.reply(status_codes::OK, answer);
}
void handle_post(http_request request)
{
TRACE("\nHandle POST\n");
handle_request(
request,
[](const json::value & jvalue, json::value & answer, bool bNull)
{
const utility::string_t sID("id");
const utility::string_t sText("text");
const utility::string_t sState("state");
if( bNull )
{
wcout << "jvalue must be null, setting some default values..." << endl;
json::value group;
group[sID] = json::value::string("1");
group[sText] = json::value::string("Hello");
group[sState] = json::value::string("closed");
answer[0] = group;
}
else
{
// To be written once the null case is sorted
}
}
);
}
int main()
{
uri_builder uri("http://w.x.y.z:1024/testrest");
http_listener listener(uri.to_uri());
listener.support(methods::POST, handle_post);
try
{
listener
.open()
.then([&listener]()
{
TRACE(L"\nStarting to listen\n");
})
.wait();
while (true);
}
catch (exception const & e)
{
wcout << e.what() << endl;
}
return 0;
}
This compiles cleanly and I can start the service on the linux server with the following:
./testrest &
Starting to listen
To aid in debugging, I've been using curl to serve as a POST client directly on the same linux server. I've been using the following command to send a POST request with 0 content-length:
curl -i -X POST -H 'Content-Type: application/json' http://w.x.y.z:1024/testrest
The output from curl is the following:
HTTP/1.1 200 OK
Content-Length: 44
Content-Type: application/json
[{"id":"1","state":"closed","text":"Hello"}]
and the console messages from my service are as such:
Handle POST
Handle_request
POST /testrest HTTP/1.1
Accept: */*
Content-Type: application/json
Host: w.x.y.z:1024
User-Agent: curl/7.29.0
jvalue must be null, setting some default values...
The first two lines correspond to the TRACE calls in the code. The middle section is generated by this section of code:
// Spit out the HTTP header to the console...
const auto HeaderString = request.to_string();
wcout << HeaderString.c_str() << endl;
Based on the curl output, which is an array of dictionary objects exactly one entry long, I would expect that this service should work just fine with the jeasyui javascript on the client. However, it does not. My async tree never populates and I don't see anything at all.
I suspect there's something wrong with the encoding, and so I wrote another service using web2py to test to see if it would work there. The following code exists in my default.py controller:
#service.json
def testweb2py():
aRet=[]
if request.post_vars.id is None:
mydict={'id':'1','text':'Hello','state':'closed'}
aRet.append(mydict)
return aRet
after modifying my client easyui-tree HTML to point to the web2py URL, it populates perfectly and I can see the node. I hit the web2py service.json code with curl just to see how the output might differ:
HTTP/1.1 200 OK
Date: Mon, 23 Jan 2017 18:17:17 GMT
Server: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.1e-fips mod_wsgi/3.4 Python/2.7.5
X-Powered-By: web2py
Expires: Mon, 23 Jan 2017 18:17:18 GMT
Pragma: no-cache
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Content-Length: 99
Content-Type: application/json; charset=utf-8
[{"text": "Hello", "state": "closed", "id": "1"}]
Aside from the content header being quite different, there's one line that I suspect might have something to do with it:
Content-Type: application/json; charset=utf-8
In the call to the cpprest service, the header output from curl does not include charset=utf-8 in it. If I dump the curl output to a file using the -o switch, I don't see any clear difference between the encoding. The only thing that I can see different in the format of the json is some extra whitespace and the ordering:
[{"text": "Hello", "state": "closed", "id": "1"}] // web2py version
[{"id":"1","state":"closed","text":"Hello"}] // cpprest version
I'm unable to gain any control over the order in which the json dictionary is sent, but I doubt that has anything to do with it anyways. The extra whitespace prefixing the value entry seems irrelevant as well.
I've poured over the cpprest documentation over at microsoft.github.io/cpprestsdk/index.html,
and I cannot find anything that relates to setting the output encoding. There are a number of overrides to http_request::reply that include options for setting content-type, and I've gone down the road of calling them with hard-coded strings for both the json body and the content-type of json/application; charset=utf-8, all to no avail. I don't see how those overrides can be used with json::value objects at any rate, so I don't think that's the optimal path or a viable use of this cpprest library.
The jeasyui javascript code appears to be intentionally obfuscated, and I have little faith in being able to figure out what it is doing with the reply from the POST call. Maybe someone familiar with jeasyui can point to a viable means for debugging the async POST?
Please help!

Answer 1

So I figured out what was happening. Opened the developer tools console in Chrome and discovered the following error message:
XMLHttpRequest cannot load http://w.x.y.z:1024/testrest. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://w.x.y.z' is therefore not allowed access.
So it had nothing to do with the format or encoding of my json data, but rather the fact that the json service was identified as being a different resource from the web server that generated the client HTML, which it is, and so Chrome was blocking it. To solve this problem, I had to add some header fields in the response I sent back to the client, as well as add a support method to handle OPTIONS queries from any client that might need them.
In my main() function, I added:
listener.support(methods::OPTIONS, handle_options);
Then I wrote the corresponding function:
void handle_options(http_request request)
{
http_response response(status_codes::OK);
response.headers().add(U("Allow"), U("POST, OPTIONS"));
// Modify "Access-Control-Allow-Origin" header below to suit your security needs. * indicates allow all clients
response.headers().add(U("Access-Control-Allow-Origin"), U("*"));
response.headers().add(U("Access-Control-Allow-Methods"), U("POST, OPTIONS"));
response.headers().add(U("Access-Control-Allow-Headers"), U("Content-Type"));
request.reply(response);
}
Finally, I had to add the same headers to the request.reply in my handle_request:
http_response response(status_codes::OK);
// Without these headers, the client browser will likely refuse the data and eat it
response.headers().add(U("Access-Control-Allow-Origin"), U("*"));
response.headers().add(U("Access-Control-Allow-Methods"), U("POST, OPTIONS"));
response.headers().add(U("Access-Control-Allow-Headers"), U("Content-Type"));
response.set_body(answer);
request.reply(response);
There were additional problems, as well... the most prominent being the fact that the jeasyui class easyui_tree does not POST data with a Content-Type of application/json. Instead, it posts a Content-Type of application/x-www-form-urlencoded, so I had to add a function to parse the url encoding using libcurl. This also meant replacing request.extract_json() with request.extract_string() and related modifications to the corresponding lambda functions used by cpprest.
Here's the final example code, maybe it's useful to others working in these areas. It's a fully-functional example of writing a json service with cpprest (on linux, no less) that responds to asynchronous POST requests from easyui_tree. Dependencies: boost, cpprest, and libcurl-devel.
#include <boost/algorithm/string/replace.hpp>
#include <cpprest/http_listener.h>
#include <cpprest/json.h>
#include <curl/curl.h>
#pragma comment(lib, "cpprestlib" )
using namespace web;
using namespace web::http;
using namespace web::http::experimental::listener;
#include <iostream>
#include <map>
#include <vector>
#include <set>
#include <string>
using namespace std;
#define TRACE(msg) wcout << msg
void build_json( const utility::string_t &source, json::value &jvalue )
{
// Use libcurl to unescape the POST body for us
vector<string> splitvec;
// We don't own the string created by curl_easy_unescape, so add a custom deleter
string text = shared_ptr<char>( curl_easy_unescape( 0, source.c_str(), 0, 0 ), curl_free).get();
// This works for this specific example of jeasyui, the class 'easyui-tree', which only passes id=... in the POST.
// Need custom handler to deal with more complicated data formats
boost::split( splitvec, text, boost::is_any_of("="));
if( splitvec.size() == 2 )
{
jvalue[splitvec.at(0)] = json::value::string(splitvec.at(1));
}
}
void handle_request(http_request request, function<void(const json::value &, json::value &, bool)> action)
{
json::value answer;
auto objHeader = request.headers();
auto sContentType = objHeader["Content-Type"];
// Two cases:
// 1) The very first call from easyui_tree, when the HTML is first loaded, will make a zero-length POST with no 'Content-Type' in the header
// 2) Subsequent calls from easyui_tree (e.g. when user opens a node) will have a Content-Type of 'application/x-www-form-urlencoded'
// Nowhere does easyui_tree send json data in the POST, although it expects json in the reply
if( sContentType.size() == 0 ||
!strncasecmp( sContentType.c_str(), "application/x-www-form-urlencoded", strlen("application/x-www-form-urlencoded") ) )
{
request
.extract_string()
.then([&answer, &action](pplx::task<utility::string_t> task) {
try
{
const auto & svalue = task.get();
json::value jvalue;
if ( svalue.size() == 0 )
{
action(jvalue, answer, true);
}
else
{
build_json( svalue, jvalue );
action(jvalue, answer, false);
}
}
catch (http_exception const & e)
{
wcout << "HTTP exception in handle_request: " << e.what() << endl;
}
})
.wait();
}
else
{
// This Content-Type doesn't appear with easyui_tree, but perhaps it's still useful for future cases...
if( !strncasecmp( sContentType.c_str(), "application/json", strlen("application/json") ) )
{
request
.extract_json()
.then([&answer, &action](pplx::task<json::value> task) {
try
{
const auto & jvalue = task.get();
if (!jvalue.is_null())
{
action(jvalue, answer, false);
}
else
{
action(jvalue, answer, true);
}
}
catch (http_exception const & e)
{
wcout << "HTTP exception in handle_request: " << e.what() << endl;
}
})
.wait();
}
}
http_response response(status_codes::OK);
// Without these headers, the client browser will likely refuse the data and eat it
response.headers().add(U("Access-Control-Allow-Origin"), U("*"));
response.headers().add(U("Access-Control-Allow-Methods"), U("POST, OPTIONS"));
response.headers().add(U("Access-Control-Allow-Headers"), U("Content-Type"));
response.set_body(answer);
request.reply(response);
}
void handle_options(http_request request)
{
http_response response(status_codes::OK);
response.headers().add(U("Allow"), U("POST, OPTIONS"));
// Modify "Access-Control-Allow-Origin" header below to suit your security needs. * indicates allow all clients
response.headers().add(U("Access-Control-Allow-Origin"), U("*"));
response.headers().add(U("Access-Control-Allow-Methods"), U("POST, OPTIONS"));
response.headers().add(U("Access-Control-Allow-Headers"), U("Content-Type"));
request.reply(response);
}
void handle_post(http_request request)
{
handle_request(
request,
[](const json::value & jvalue, json::value & answer, bool bInitialize)
{
if( bInitialize )
{
// First time the tree is being loaded, first id will be 16, which will yield us 16 child nodes when it POSTs back
json::value jreply;
jreply[U("id")] = json::value::string("16");
jreply[U("text")] = json::value::string("Parent");
jreply[U("state")] = json::value::string("closed");
answer[0] = jreply;
}
else
{
// User has opened a node
if( jvalue.type() == json::value::value_type::Object )
{
if( jvalue.has_field( "id" ) )
{
auto & key = jvalue.at( "id" );
if( key.is_string() )
{
auto value = key.as_string();
int id = atoi(value.c_str());
stringstream ss;
ss << (id / 2); // Each successive layer has half as many child nodes as the one prior
for( int i = 0; i < id; i++ )
{
json::value jreply;
jreply[U("id")] = json::value::string(ss.str());
jreply[U("text")] = json::value::string("Child");
jreply[U("state")] = json::value::string("closed");
answer[i] = jreply;
}
}
}
}
}
}
);
}
int main()
{
uri_builder uri("http://yourserver.com:1024/testrest");
http_listener listener(uri.to_uri());
listener.support(methods::POST, handle_post);
listener.support(methods::OPTIONS, handle_options);
try
{
listener
.open()
.then([&listener]()
{
TRACE(L"\nStarting to listen\n");
})
.wait();
while (true);
}
catch (exception const & e)
{
wcout << e.what() << endl;
}
return 0;
}
And then of course the corresponding HTML, assuming all the jeasyui scripts are referenced in the header:
<ul id="tt" method="POST" class="easyui-tree" url="http://yourserver.com:1024/testrest">
</ul>