I am building a project with ESP8266, SPIFFS and AsyncWebServerRequest.
The idea is to have a config.json file in the FS, that the ESP reads on start.
The config is big(around 2K) because it stores a lot of variables.
In addition I have index.html page, that I serve to the client via AsyncWebServerRequest
server.on("/", HTTP_GET, [](AsyncWebServerRequest *request){
request->send(SPIFFS, "/index.html", "text/html");
});
When the page is loaded it makes second request to the ESP for the configuration and uses JS to propagate the configuration to the fields.
When you change the config from the web-page and click "Save Config" it encodes the form data in to JSON and make another request to the ESP(POST Request "Content-Type", "application/json" with the config. When the config is small it looks OK, but when the config is bigger it starts adding unidentified chars. I am not sure if I collect the http body properly:
server.on(
"/config/upload",
HTTP_POST,
[](AsyncWebServerRequest * request){},
NULL,
[](AsyncWebServerRequest * request, uint8_t *data, size_t len, size_t index, size_t total) {
for (size_t i = 0; i < len; i++) {
conf_f+=char(data[i]);
}
newConfig = 1;
request->send(200);
});
I think the issue is with the async, because it interrupts the concatenation process of the String. When I Serial.print the chars one by one it adds two of three new lines, that do not even have a char code.
Here is an example of the end of the config after failed upload: "enabled":"1","modbus_refresh_rate":"5","slaves_count":1���
This of course disrupt the ArduinoJSON and the system is not booting with the correct config.
My question is: Is there a better way to manipulate the HTTP body, because the one I am using looks wrong.
Related
I am trying to send post request with esp8266 programmed on arduino IDE, but there's lack of examples how to do it. I would like to send request to json server with raw input so it would look like this:
http://ip:port/something
BODY
{
"valuename":value
}
Would be grateful if anyone could show me such an example.
Greetings
For handling HTTP requests, you can use a RestClient library rather than writing all the low level requests. It saves a lot of time and is less error-prone.
For example, for a GET request, all you have to do is:
String response = "";
int statusCode = client.post("/", "foo=bar", &response);
One good such library with SSL support is written by github user DaKaz.
You can use it for your GET request. The returned response will be without the HTTP header. The function will return the response from the server without the headers.
Now you can use the ArduinoJson Library by bblanchin for decoding the JSON object.
Details can be seen here.
Or you can do plain string manipuation to get the values though it is not a recommended route to take and is prone to errors.
Here is an example to send JSON via HTTP library :
#include <ESP8266WiFi.h>
#include <ArduinoJson.h>
#include <ArduinoHttpClient.h>
#define JSON_BUF_SIZE 256
WiFiClient wifi;
HttpClient poster = HttpClient(wifi, IP, PORT);
void HTTPPost(){
String contentType = "application/json";
StaticJsonBuffer<JSON_BUF_SIZE> jsonBuffer;
JsonObject& jsonData = jsonBuffer.createObject();
jsonData["valuename"] = "value";
String postData = "";
jsonData.printTo(postData);
poster.post("/", contentType, postData);
printf("Trace : ResponseCode : %d\n", poster.responseStatusCode());
printf("Trace : Incoming Body : %s\n", poster.responseBody().c_str());
}
I'm using POST:parameters:constructingBodyWithBlock:. I believe the pieces I'm attempting to send to be correct, but I'm getting a HTTP 400 from the server. So something is wrong with the overall packet or how a piece of it is being encoded. I have a few theories about that, but before I do anything else I'd like to see what the data I'm sending actually looks like.
What is the best way to see what AFNetworking is actually sending for debugging purposes?
I've tried tracing quite deep, but when I reach AFMultipartBodyStream I get lost. I'm not sure how to capture its data without severely hacking at AFNetworking.
For anyone curious later, I've found this works nicely in the failure or success block:
NSMutableData *data = [NSMutableData data];
NSInputStream *stream = [operation.request.HTTPBodyStream copy];
[stream open];
BOOL done = NO;
while (!done) {
NSMutableData *buffer = [NSMutableData dataWithLength:1024];
buffer.length = [stream read:buffer.mutableBytes maxLength:buffer.length];
if (buffer.length) {
[data appendData:buffer];
} else {
done = YES;
}
}
[stream close];
The resulting data object can be written to disk, but can only be converted to a string directly if the parameters were entirely string (and not a binary file).
I would like to know what can I do to upload attachments in CouchDB using the update function.
here you will find an example of my update function to add documents:
function(doc, req){
if (!doc) {
if (!req.form._id) {
req.form._id = req.uuid;
}
req.form['|edited_by'] = req.userCtx.name
req.form['|edited_on'] = new Date();
return [req.form, JSON.stringify(req.form)];
}
else {
return [null, "Use POST to add a document."]
}
}
example for remove documents:
function(doc, req){
if (doc) {
for (var i in req.form) {
doc[i] = req.form[i];
}
doc['|edited_by'] = req.userCtx.name
doc['|edited_on'] = new Date();
doc._deleted = true;
return [doc, JSON.stringify(doc)];
}
else {
return [null, "Document does not exist."]
}
}
thanks for your help,
It is possible to add attachments to a document using an update function by modifying the document's _attachments property. Here's an example of an update function which will add an attachment to an existing document:
function (doc, req) {
// skipping the create document case for simplicity
if (!doc) {
return [null, "update only"];
}
// ensure that the required form parameters are present
if (!req.form || !req.form.name || !req.form.data) {
return [null, "missing required post fields"];
}
// if there isn't an _attachments property on the doc already, create one
if (!doc._attachments) {
doc._attachments = {};
}
// create the attachment using the form data POSTed by the client
doc._attachments[req.form.name] = {
content_type: req.form.content_type || 'application/octet-stream',
data: req.form.data
};
return [doc, "saved attachment"];
}
For each attachment, you need a name, a content type, and body data encoded as base64. The example function above requires that the client sends an HTTP POST in application/x-www-form-urlencoded format with at least two parameters: name and data (a content_type parameter will be used if provided):
name=logo.png&content_type=image/png&data=iVBORw0KGgoA...
To test the update function:
Find a small image and base64 encode it:
$ base64 logo.png | sed 's/+/%2b/g' > post.txt
The sed script encodes + characters so they don't get converted to spaces.
Edit post.txt and add name=logo.png&content_type=image/png&data= to the top of the document.
Create a new document in CouchDB using Futon.
Use curl to call the update function with the post.txt file as the body, substituting in the ID of the document you just created.
curl -X POST -d #post.txt http://127.0.0.1:5984/mydb/_design/myddoc/_update/upload/193ecff8618678f96d83770cea002910
This was tested on CouchDB 1.6.1 running on OSX.
Update: #janl was kind enough to provide some details on why this answer can lead to performance and scaling issues. Uploading attachments via an upload handler has two main problems:
The upload handlers are written in JavaScript, so the CouchDB server may have to fork() a couchjs process to handle the upload. Even if a couchjs process is already running, the server has to stream the entire HTTP request to the external process over stdin. For large attachments, the transfer of the request can take significant time and system resources. For each concurrent request to an update function like this, CouchDB will have to fork a new couchjs process. Since the process runtime will be rather long because of what is explained next, you can easily run out of RAM, CPU or the ability to handle more concurrent requests.
After the _attachments property is populated by the upload handler and streamed back to the CouchDB server (!), the server must parse the response JSON, decode the base64-encoded attachment body, and write the binary body to disk. The standard method of adding an attachment to a document -- PUT /db/docid/attachmentname -- streams the binary request body directly to disk and does not require the two processing steps.
The function above will work, but there are non-trivial issues to consider before using it in a highly-scalable system.
I have to pass a json array consisting of json objects from the servlet to the jsp page. This data transfer slows down page responsiveness. Is there any way to optimize performance while passing large json from servlet to jsp.
code looks like:
request.setAttribute("jsonStringForDataTable", jsonArrayForDataTable);
response.setContentType("text/plain");
response.setContentLength(jsonArrayForDataTable.toString().getBytes().length);
ServletOutputStream out=response.getOutputStream();
out.print(jsonArrayForDataTable.toString().replace('_',' '));
out.close();
...
Any help is highly appreciated
You're using ajax, right? It's hard to tell since you're setting that request attribute, as if about to forward to a jsp, but then you're writing directly to the response. If this is ajax, you can compress the response with gzip if the client accepts it. You'll have to try it to know if it will speed things up in your situation. You'll be reducing the size of the response by a ratio that depends on the original content, but you'll be increasing the amount of processor work on the server (and on the client, which has to decompress the response).
ServletOutputStream out = response.getOutputStream();
response.setContentType("application/json");
String strVal = jsonArrayForDataTable.toString().replace('_',''));
if (request.getHeader("Accept-Encoding") != null && request.getHeader("Accept-Encoding").contains("gzip"))
{
ByteArrayOutputStream baos = new ByteArrayOutputStream(strVal.length());
GZipOutputStream gzip = new GZIPOutputStream(baos);
gzip.write(strVal.getBytes());
gzip.close();
response.setHeader("Content-Encoding", "gzip");
out.write(baos.toByteArray());
baos.close();
}
else
{
out.print(strVal);
}
On the client side, you have to be prepared for a gzipped response by setting the Accept-Encoding XMLHttpRequest header to gzip. The browser will decompress it into the json string.
Recently I started taking this guide to get myself started on downloading files from the internet. I read it and came up with the following code to download the HTTP body of a website. The only problem is, it's not working. The code stops when calling the recv() call. It does not crash, it just keeps on running. Is this my fault? Am I using the wrong approch? I intent to use the code to not just download the contents of .html-files, but also to download other files (zip, png, jpg, dmg ...). I hope there's somebody that can help me. This is my code:
#include <stdio.h>
#include <sys/socket.h> /* SOCKET */
#include <netdb.h> /* struct addrinfo */
#include <stdlib.h> /* exit() */
#include <string.h> /* memset() */
#include <errno.h> /* errno */
#include <unistd.h> /* close() */
#include <arpa/inet.h> /* IP Conversion */
#include <stdarg.h> /* va_list */
#define SERVERNAME "developerief2.site11.com"
#define PROTOCOL "80"
#define MAXDATASIZE 1024*1024
void errorOut(int status, const char *format, ...);
void *get_in_addr(struct sockaddr *sa);
int main (int argc, const char * argv[]) {
int status;
// GET ADDRESS INFO
struct addrinfo *infos;
struct addrinfo hints;
// fill hints
memset(&hints, 0, sizeof(hints));
hints.ai_socktype = SOCK_STREAM;
hints.ai_flags = AI_PASSIVE;
hints.ai_family = AF_UNSPEC;
// get address info
status = getaddrinfo(SERVERNAME,
PROTOCOL,
&hints,
&infos);
if(status != 0)
errorOut(-1, "Couldn't get addres information: %s\n", gai_strerror(status));
// MAKE SOCKET
int sockfd;
// loop, use first valid
struct addrinfo *p;
for(p = infos; p != NULL; p = p->ai_next) {
// CREATE SOCKET
sockfd = socket(p->ai_family,
p->ai_socktype,
p->ai_protocol);
if(sockfd == -1)
continue;
// TRY TO CONNECT
status = connect(sockfd,
p->ai_addr,
p->ai_addrlen);
if(status == -1) {
close(sockfd);
continue;
}
break;
}
if(p == NULL) {
fprintf(stderr, "Failed to connect\n");
return 1;
}
// LET USER KNOW
char printableIP[INET6_ADDRSTRLEN];
inet_ntop(p->ai_family,
get_in_addr((struct sockaddr *)p->ai_addr),
printableIP,
sizeof(printableIP));
printf("Connection to %s\n", printableIP);
// GET RID OF INFOS
freeaddrinfo(infos);
// RECEIVE DATA
ssize_t receivedBytes;
char buf[MAXDATASIZE];
printf("Start receiving\n");
receivedBytes = recv(sockfd,
buf,
MAXDATASIZE-1,
0);
printf("Received %d bytes\n", (int)receivedBytes);
if(receivedBytes == -1)
errorOut(1, "Error while receiving\n");
// null terminate
buf[receivedBytes] = '\0';
// PRINT
printf("Received Data:\n\n%s\n", buf);
// CLOSE
close(sockfd);
return 0;
}
void *get_in_addr(struct sockaddr *sa) {
// IP4
if(sa->sa_family == AF_INET)
return &(((struct sockaddr_in *) sa)->sin_addr);
return &(((struct sockaddr_in6 *) sa)->sin6_addr);
}
void errorOut(int status, const char *format, ...) {
va_list args;
va_start(args, format);
vfprintf(stderr, format, args);
va_end(args);
exit(status);
}
If you want to grab files using HTTP, then libcURL is probably your best bet in C. However, if you are using this as a way to learn network programming, then you are going to have to learn a bit more about HTTP before you can retrieve a file.
What you are seeing in your current program is that you need to send an explicit request for the file before you can retrieve it. I would start by reading through RFC2616. Don't try to understand it all - it is a lot to read for this example. Read the first section to get an understanding of how HTTP works, then read sections 4, 5, and 6 to understand the basic message format.
Here is an example of what an HTTP request for the stackoverflow Questions page looks like:
GET http://stackoverflow.com/questions HTTP/1.1\r\n
Host: stackoverflow.com:80\r\n
Connection: close\r\n
Accept-Encoding: identity, *;q=0\r\n
\r\n
I believe that is a minimal request. I added the CRLFs explicitly to show that a blank line is used to terminate the request header block as described in RFC2616. If you leave out the Accept-Encoding header, then the result document will probably be transfered as a gzip-compressed stream since HTTP allows for this explicitly unless you tell the server that you do not want it.
The server response also contains HTTP headers for the meta-data describing the response. Here is an example of a response from the previous request:
HTTP/1.1 200 OK\r\n
Server: nginx\r\n
Date: Sun, 01 Aug 2010 13:54:56 GMT\r\n
Content-Type: text/html; charset=utf-8\r\n
Connection: close\r\n
Cache-Control: private\r\n
Content-Length: 49731\r\n
\r\n
\r\n
\r\n
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" ... 49,667 bytes follow
This simple example should give you an idea what you are getting into implementing if you want to grab files using HTTP. This is the best case, most simple example. This isn't something that I would undertake lightly, but it is probably the best way to learn and appreciate HTTP.
If you are looking for a simple way to learn network programming, this is a decent way to start. I would recommend picking up a copy of TCP/IP Illustrated, Volume 1 and UNIX Network Programming, Volume 1. These are probably the best way to really learn how to write network-based applications. I would probably start by writing an FTP client since FTP is a much simpler protocol to start with.
If you are trying to learn the details associated with HTTP, then:
Buy HTTP: the Definitive Guide and read it
Read RFC2616 until you understand it
Try examples using telnet server 80 and typing in requests by hand
Download the cURL client and use the --verbose and --include command line options so that you can see what is happening
Read Fielding's dissertation until HTTP really makes sense.
Just don't plan on writing your own HTTP client for enterprise use. You do not want to do that, trust me as one who has been maintaining such a mistake for a little while now...
The problem is, you have to implement the HTTP protocol. Downloading a file is not just a matter of connecting to the server, you have to send HTTP requests (along with proper HTTP header) before you get a response. After this, you would still need to parse the returned data to strip out more HTTP headers.
If you're just trying to download files using C, I suggest the cURL library, which does the HTTP work for you.
You have to send an HTTP request before expecting a response.
You code currently just waits for a response which never comes.
Also, don't write comments in all caps.