I am trying to retrieve POST data from html form using program written in C.
At the moment I am using:
char *formdata = getenv("QUERY_STRING");
if(formdata == NULL) /* no data retrieved */
This seems to be working fine with form "GET" method but not with "POST" method. How do I retrieve POST data?
POST data is appended to the request header, after a double newline. In a CGI-BIN environment, you read it from STDIN.
Be warned that the server IS NOT REQUIRED to send you an EOF character (or some termination indicator) at the end of the POST data. Never read more than CONTENT_LENGTH bytes.
If I remember right, read stdin for POST data.
Edit for untested snippet
len_ = getenv("CONTENT_LENGTH");
len = strtol(len_, NULL, 10);
postdata = malloc(len + 1);
if (!postdata) { /* handle error or */ exit(EXIT_FAILURE); }
fgets(postdata, len + 1, stdin);
/* work with postdata */
free(postdata);
Why reinvent that wheel? Just use a library: http://libcgi.sourceforge.net/
Related
I am automating Adobe InDesign to create documents using JSON data gathered from a web API with a SQL Server backend. I am using the Sockets object to make an HTTP 1.0 call to our server. Sometimes the response received is missing about 1700 characters from various points within the JSON string, yet when I call the same API endpoint using curl or Postman I get a complete and valid response.
The response should be about 150k characters long, and I'm using conn.read(99999999) to read it. In addition, the appearance of the end of the string looks correct, so I don't believe it's any kind of truncation problem.
The problem only seems to occur when I request a UTF-8 encoding. If I request ASCII I get a complete and valid response, but missing various Unicode characters. If I request BINARY I get a complete and valid response but the JavaScript/ExtendScript seems to be handling any multi-byte Unicode characters received as individual bytes, rather than as the Unicode characters we want to display.
Here is an illustration of the behavior I'm seeing, using bogus data...
"Expected" response...
[{"Id":1, "name":"Random Name", "Text":"A bunch of text", "AnotherId": 1}]
"Actual" response...
[{"Id":1, "name":"Random Name", "Text":"A bunc": 1}]
The problem first manifested itself as a JSON2 parsing error, for obvious reasons, but the root of it seems to be the fact that parts of the data are going missing in-transit.
So far we've only seen this problem when making the call using the InDesign Sockets object, and not every response exhibits this behavior.
Any help or insights you could offer would be appreciated.
Here is the function I'm using to call for data...
function httpRequest(url, encoding) {
try {
var response = "";
var hostName = getHostFromUrl(url);
var pathAndQuery = getPathAndQueryFromUrl(url);
var httpGet = "GET ";
httpGet += pathAndQuery;
httpGet += " HTTP/1.0\r\nHost: ";
httpGet += hostName;
httpGet += "\r\n";
var conn = new Socket;
conn.timeout = 30;
//conn.encoding = encoding || "UTF-8";
//conn.charset = "UTF-16";
if (conn.open(hostName + ":80", encoding || "UTF-8")) {
// send a HTTP GET request
conn.writeln(httpGet);
// and read the server's response
response = conn.read(99999999);
conn.close();
}
return parseHttpResponse(response);
}
catch (e) {
$.writeln(e);
$.global.alert("There was a problem making an HTTP Request: " + e);
return null;
}
}
It turns out my handling of the HTTP response was too simplistic and needed extra logic to handle Unicode characters properly.
The solution, in my case, was to use the GetURL method made available by Kris Coppieter here.
I'm trying to fetch one value from the data source website Quandlto be used within a MetaTrader4 script. The data source site provides a method to export data via API formats including .csv, .json or .xml. I have chosen the .csv format, which the data source website then provides an API call for me to use in the following format:
https://www.quandl.com/api/v3/datasets/ADB/LAB_UNEMP_JPN.csv?rows=1&api_key=my_api_key
By using the rows=1parameter in the above API call, I can choose to just export one value (which is the latest value).
Q1. Can I fetch the value straight from Quandl or do I have to save the dataset as a .csv file?
Because Quandl provides the API call (as seen above), would I be correct in assuming I can just fetch the value from their website and won't have to save the dataset to my computer as a .csvfile, which I would then have to fetch the latest value from? I would much prefer to fetch the value straight from Quandl without saving any files.
Q2. How can I fetch the value to be used within my MT4 script?
I have unsuccessfully tried a method using FileOpen() to access the data on the site, and have then tried to print it so that I can compare the value to others. Is FileOpen() only for .csv files only saved to my computer? I'd like to be able to print the value within my script once retrieved so that I can use it. Here is what I have so far:
int start() {
while (!IsStopped()) {
Sleep(2000);
int handle;
int value;
handle=FileOpen("https://www.quandl.com/api/v3/datasets/ADB/LAB_UNEMP_JPN.csv?rows=1&api_key=my_api_key", FILE_CSV, ';');
if(handle>0)
{
value=FileReadNumber(handle);
Print(handle);
FileClose(handle);
}
}
If anyone could aid me in my pursuit to fetch this value and print it within my script, it would be a huge help.
A1: No, you need not use a proxy-file for this API
If one tries the API call, using a published Quandl syntax of: <pragma>://<URL.ip>/<relative.URL>[?<par.i>=<val.i>[&<par.j>=<val.j>[&...]]]
the server side will push you the content of:
Date,Value
2013-12-31,4.0
So, your code may use Quandl API with like this:
void OnStart()
{
string cookie = NULL,
headers;
char post[],
result[];
int res;
/* TODO: *
* Must allow MT4 to access the server URL, *
* you should add URL "https://www.quandl.com/api/v3/datasets/ADB/LAB_UNEMP_JPN.csv" *
* in the list of allowed URLs *
* ( MT4 -> Tools -> Options -> [Tab]: "Expert Advisors" ): */
string aDataSOURCE_URL = "https://www.quandl.com/api/v3/datasets/ADB/LAB_UNEMP_JPN.csv";
string aDataSOURCE_API = "rows = 1&api_key=<My_API_Key>";
//-- Create the body of the POST request for API specifications and API-authorization
ArrayResize( post,
StringToCharArray( aDataSOURCE_API, // string text |--> [in] String to copy.
post, // uchar &array[] <--| [out] Array of uchar type.
0, // int start = 0 |--> [in] Position from which copying starts. Default - 0.
WHOLE_ARRAY, // int count = -1 |--> [in] Number of array elements to copy. Defines length of a resulting string. Default value is -1, which means copying up to the array end, or till terminating '\0'. Terminating zero will also be copied to the recipient array, in this case the size of a dynamic array can be increased if necessary to the size of the string. If the size of the dynamic array exceeds the length of the string, the size of the array will not be reduced.
CP_UTF8 // uint cp = CP_ACP |--> [in] The value of the code page. For the most-used code pages provide appropriate constants.
)
- 1
);
//-- Reset the last error code
ResetLastError();
//-- Loading a html page from Quandl
int timeout = 5000; //-- Timeout below 1000 (1 sec.) is not enough for slow Internet connection
res = WebRequest( "POST", // const string method |--> [in] HTTP method.
aDataSOURCE_URL, // const string URL |--> [in] URL.
cookie, // const string cookie |--> [in] Cookie value.
NULL, // const string referrer |--> [in] Value of the Referer header of the HTTP request.
timeout, // int timeout |--> [in] Timeout in milliseconds.
post, // const char &data |--> [in] Data array of the HTTP message body
ArraySize( post ), // int data_size |--> [in] Size of the data[] array.
result, // char &result <--| [out] An array containing server response data.
headers // string &result_headers <--| [out] Server response headers.
);
//-- Check errors
if ( res == -1 )
{ Print( "WebRequest Error. Error code = ", GetLastError() ); //-- Perhaps the URL is not listed, display a message about the necessity to add the address
MessageBox( "Add the address '" + aDataSOURCE_URL + "' in the list of allowed URLs on tab 'Expert Advisors'", "Error", MB_ICONINFORMATION );
}
else //-- Load was successful
{
PrintFormat( "The data has been successfully loaded, size = %d bytes.", ArraySize( result ) );
//-- parse the content ---------------------------------------
/*
"Date,Value
2013-12-31,4.0"
*/
//-- consume the content -------------------------------------
//...
}
}
There are 4 principal items to take care of:
0: an MT4 permission to use a given URL
1: an API URL setup - <pragma>://<URL.ip>/<relative.URL>
2: an API const char &data[] assy. [?<par.i>=<val.i>[&<par.j>=<val.j>[&...]]]
3: an API int data_size length calculation
Addendum: This is more a list of reasons, why avoiding use of the New-MQL4.56789 WebRequest() function variants:
Whereas MQL4 documentation promises a simple use of WebRequest() function variants, (cit.:) "1. Sending simple requests of type "key=value" using the header Content-Type: application/x-www-form-urlencoded.", the reality is far from a promised simple use-case:
0: DONE: an MT4 administrative step ( weakness: cannot enforce MT4 to communicate { http | https } protocol(s) over other than their default port(s) ~ { :80 | :443 }
1: URL consists of two ( three, if using a :port specifier, which does not work in MT4 (ref. 0: right above ) ) parts. <URL.ip_address> is the first one and can be expressed in a canonical IPv4 form ( 10.38.221.136 ) or in a DNS-translateable form ( MT4_APAC_PRIMARY.broker.com ). The second part, the <relative.URL>, specifies the HttpServer itself, where to locate a file ( it is a HttpServer--relative file location ). Published WebRequest permit to use the both parts joined together, ref. aDataSOURCE_URL.
3: WebServer API, if constructed so, may permit to add some additional parameters, that can be specified and presented to the WebServer. The presentation depends whether the { HTTP GET | HTTP POST } protocol-option is selected in on a caller side.
4: each call to MT4 WebRequest() also requires the caller to specify a length of a data content parameter ( ref. the use of ArraySize( post ), // int data_size )
I am attempting to use cURL with the Pusher API (pusher.com). However I keep getting the response "Invalid JSON provided (could not parse)". Any help would be appreciated, here is my trigger function:
function trigger(name, data, channel)
string_to_sign = "POST\n/apps/"..pusher_app_id.."/events\n"..params
signature = hmac.digest("sha256", string_to_sign, pusher_secret)
md5 = md5.sumhexa('{"name":"foo","channel":"test-channel","data":"{\"some\":\"data\"}"}');
c = curl.new()
c:setopt(curl.OPT_URL, pusher_server..'apps/'..pusher_app_id..'/events'..'?'..params..'&auth_signature='..signature..'&body_md5='..md5)
c:setopt(curl.OPT_POST, true)
c:setopt(curl.OPT_HTTPHEADER, "Content-Type: application/json")
c:setopt(curl.OPT_POSTFIELDS, '{"name":"'..name..'","channel":"'..channel..'","data":"{\"some\":\"data\"}"}')
c:perform()
c:close()
end
If I print the JSON I am putting in OPT_POSTFIELDS and paste it into a json validator, it is indeed completely valid. According to the docs this is the proper usage for /events and my authentication is also working fine.
I had gone back through my function applying the suggestions moteus commented and was able to resolve my problem by fixing the md5 and applying it to the signature. I am also using the luajson module to take care of encoding. This seemed to fix the issue.
function trigger(name, data, channel)
data_table = {
["name"] = name,
["channel"] = channel,
["data"] = data
}
json_data = json.encode(data_table)
md5 = md5.sumhexa(json_data)
string_to_sign = "POST\n/apps/"..pusher_app_id.."/events\n"..params.."&body_md5="..md5
signature = hmac.digest("sha256", string_to_sign, pusher_secret)
c:setopt(curl.OPT_URL, pusher_server..'apps/'..pusher_app_id..'/events'..'?'..params..'&auth_signature='..signature..'&body_md5='..md5)
c:setopt(curl.OPT_POST, true)
c:setopt(curl.OPT_HTTPHEADER, "Content-Type: application/json")
c:setopt(curl.OPT_POSTFIELDS, json_data)
c:perform()
c:close()
end
I would like to know what can I do to upload attachments in CouchDB using the update function.
here you will find an example of my update function to add documents:
function(doc, req){
if (!doc) {
if (!req.form._id) {
req.form._id = req.uuid;
}
req.form['|edited_by'] = req.userCtx.name
req.form['|edited_on'] = new Date();
return [req.form, JSON.stringify(req.form)];
}
else {
return [null, "Use POST to add a document."]
}
}
example for remove documents:
function(doc, req){
if (doc) {
for (var i in req.form) {
doc[i] = req.form[i];
}
doc['|edited_by'] = req.userCtx.name
doc['|edited_on'] = new Date();
doc._deleted = true;
return [doc, JSON.stringify(doc)];
}
else {
return [null, "Document does not exist."]
}
}
thanks for your help,
It is possible to add attachments to a document using an update function by modifying the document's _attachments property. Here's an example of an update function which will add an attachment to an existing document:
function (doc, req) {
// skipping the create document case for simplicity
if (!doc) {
return [null, "update only"];
}
// ensure that the required form parameters are present
if (!req.form || !req.form.name || !req.form.data) {
return [null, "missing required post fields"];
}
// if there isn't an _attachments property on the doc already, create one
if (!doc._attachments) {
doc._attachments = {};
}
// create the attachment using the form data POSTed by the client
doc._attachments[req.form.name] = {
content_type: req.form.content_type || 'application/octet-stream',
data: req.form.data
};
return [doc, "saved attachment"];
}
For each attachment, you need a name, a content type, and body data encoded as base64. The example function above requires that the client sends an HTTP POST in application/x-www-form-urlencoded format with at least two parameters: name and data (a content_type parameter will be used if provided):
name=logo.png&content_type=image/png&data=iVBORw0KGgoA...
To test the update function:
Find a small image and base64 encode it:
$ base64 logo.png | sed 's/+/%2b/g' > post.txt
The sed script encodes + characters so they don't get converted to spaces.
Edit post.txt and add name=logo.png&content_type=image/png&data= to the top of the document.
Create a new document in CouchDB using Futon.
Use curl to call the update function with the post.txt file as the body, substituting in the ID of the document you just created.
curl -X POST -d #post.txt http://127.0.0.1:5984/mydb/_design/myddoc/_update/upload/193ecff8618678f96d83770cea002910
This was tested on CouchDB 1.6.1 running on OSX.
Update: #janl was kind enough to provide some details on why this answer can lead to performance and scaling issues. Uploading attachments via an upload handler has two main problems:
The upload handlers are written in JavaScript, so the CouchDB server may have to fork() a couchjs process to handle the upload. Even if a couchjs process is already running, the server has to stream the entire HTTP request to the external process over stdin. For large attachments, the transfer of the request can take significant time and system resources. For each concurrent request to an update function like this, CouchDB will have to fork a new couchjs process. Since the process runtime will be rather long because of what is explained next, you can easily run out of RAM, CPU or the ability to handle more concurrent requests.
After the _attachments property is populated by the upload handler and streamed back to the CouchDB server (!), the server must parse the response JSON, decode the base64-encoded attachment body, and write the binary body to disk. The standard method of adding an attachment to a document -- PUT /db/docid/attachmentname -- streams the binary request body directly to disk and does not require the two processing steps.
The function above will work, but there are non-trivial issues to consider before using it in a highly-scalable system.
I'm trying to post a json data via Arduino.When ı'm trying to this code.ı will send a json data with QueryString.If ı try this code the server answer me with Wrong QueryString format.Which mean is ı'm connected to server and server got my data.
if (client.connect(server, 80)) {
Serial.println("connected");
// Make a HTTP request:
client.println("POST /URL?query=jsondata HTTP/1.1");
client.println("Host: **.**.**.**");
client.println("Connection: close\r\nContent-Type: application/x-www-form-urlencoded");
client.print("Content-Length: ");
client.println(PostData.length());
client.println();
client.println(PostData);
}
But My Main Plan is send my json data with querystring.If ı Try this code ;
client.println("POST /URL?query={request:{Header:{Username:kullaniciAdi,Password:123456},Item:{Serial:ABC123QWE,Data:100, DateOn:23/11/1986 15:45:24}}} HTTP/1.1");
I get a HTTP Error 400. The request is badly formed.
Anyone Has a any idea?
Yes, your URI contains spaces and may contain other characters to confuse the format of the post request. You need to encode these characters.
As far as I can tell, the Arduino standard libraries do not include any form of urlEncode method, which is common in other languages and libraries, so you will either have to create your own or look for one.
Your resulting code would be something like:
String request = "/URL?query={request:{Header:{Username:kullaniciAdi,Password:123456},Item:{Serial:ABC123QWE,Data:100, DateOn:23/11/1986 15:45:24}}}";
String encRequest = uriEncode(request); // need to write your own method for this...
String post = "POST " + encRequest + " HTTP/1.1");
client.println( post);
Some discussion on creating a uriEncode function is on the Arduino Forum and there also appears to be a working method on hardwarefun.com