Sending a POST request (Apache httpclient, here Kotlin source code):
val httpPost = HttpPost("http://localhost:8000")
val builder = MultipartEntityBuilder.create()
builder.addBinaryBody("file", File("testFile.zip"),
ContentType.APPLICATION_OCTET_STREAM, "file.ext")
val multipart = builder.build()
httpPost.entity = multipart
val r = httpClient.execute(httpPost)
r.close()
I receive the request in my post handler as a via spark-java Request-object. How do I retrieve the original file (plus the file name as a bonus) from the post request? The request.bodyAsBytes() method seems to add some bytes because the body is larger than the original file.
Thanks, Jörg
Near the bottom of Spark's Documentation page there is a section "Examples and FAQ". The first example is "How do I upload something?".
From there, it links further to an example on GitHub.
In short:
post("/yourUploadPath", (request, response) -> {
request.attribute("org.eclipse.jetty.multipartConfig", new MultipartConfigElement("/temp"));
try (InputStream is = request.raw().getPart("file").getInputStream()) {
// Use the input stream to create a file
}
return "File uploaded";
});
To access the original file name:
request.raw().getPart("file").getSubmittedFileName()
To handle multiple files or parts, I usually have code similar to the following (assuming only files are included in the multi-part encoded upload):
for (Part part : req.raw().getParts()) {
try (InputStream stream = part.getInputStream()) {
String filename = part.getSubmittedFileName();
// save the input stream to the filesystem, and the filename to a database
}
}
I have a client makes file upload request via dojo.io.iframe to pass a binary data to a web application running on websphere app server and the servlet class inside of the web app makes a rest web service call to an external system to pass the binary data. So there are three layers involved.
The problem I am having is trying to display returned json data which contains exact exception happened in the last layer after making rest web service call back to the client.
The client html page looks like this
<form id="myForm" method="post" enctype="multipart/form-data">
<div id="output"></div>
<input id="filename" type="file" name="filename" />
<input id="submitButton" type="submit" value="Submit"></input>
</form>
The client javascript looks like this I am having trouble in printing out RuntimeException caught in a servlet to a client.
require(["dojo/io/iframe","dojo/dom","dojo/on","dojo/dom-construct","dojo/domReaddy!"],function(iframe,dom,on,domConst) {
on(dom.byId("submitButton"),"click",function() {
iframe.send({
form: "myForm",
handleAs: "json",
url: "/rootContext/myServlet"
}).then(function(data) {
domConst.place("<p>" + data + "</p>","output");
}, function(err) {
domConst.place("<p>" + err + "</p>","output"); // hope to print out exception caught in third layer
});
});
});
Servlet looks like this.
protected void doPost(HttpServletRequest request, HttpServletResponse response) {
....
ServletOutputStream sos = null;
....
try {
domSomething(payLoad);
} catch (RuntimeException e) [
sos = response.getOutputStream();
sos.print(e.getMessage());
}
}
private void doSomething(String payLoad) {
URL url = null;
HttpURLConnection conn = null;
OutputStream os = null;
resource = new URL("someWhere");
conn = (HttpURLConnection) resource.openConnection();
conn.setDoOutput(true);
conn.setRequestMethod("POST");
os = conn.getOutputStream();
os.write(payLoad.getBytes());
os.close();
if (conn.getResponseCode() != HttpURLConnection.HTTP_CREATED) {
String retMsg = null;
StringBuilder sb = new StringBuilder();
BufferedReader br = new BufferedReader(new InputStreamReader(conn.getErrorStream()));
while ((retMsg = br.readLine()) != null) {
sb.append(retMsg);
}
br.close();
throw new RuntimeException(sb.toString()); // throws JSON format data
}
The servlet class makes rest web service call to an external system and if uploaded file data transmission fails it returns exception in json to the servlet class and I was trying to get that error data back to a client. This servlet class works fine for file uploading and if a corrupted file is uploaded then exception is generated. I didn't include logging mechanism in here, but it is implemented so that there is a log file in the app server that registers data transmitted included returned exception.
I am new to Dojo and having trouble using iframe api to display json data passed by a servlet class. I'd appreciate your advice on this matter. Thank you!
first of all I'd appreciate others who have contributed ideas to my question. The problem I was having fixed. I had to do following steps.
In the servlet class, wrap json data by html/body/textarea tags
From client side javascript, set handleAs: "json"
function(data) will be invoked and data would be json object so call whatever property name via . then mapping value will be printed out
Came to right solution via many trials/errors. One of the mistake I made was I used wrong slash for closing textarea tag by mistake in the servlet class.
I'am using Mule Studio 3.4.0 Community Edition.
I have a big problem about how to parse a large CSV file incoming with File Endpoint. The scenario is that I have 3 CSV files and I would putting the files'content into a database.
But when I try to load a huge file (about 144MB) I get the "OutOfMemory" Exception. I thought as solution to divide/split my the large CSV into smaller size CSVs (I don't know if this solution is the best) o try to find a way to process CSV without throwing an exception.
<file:connector name="File" autoDelete="true" streaming="true" validateConnections="true" doc:name="File"/>
<flow name="CsvToFile" doc:name="CsvToFile">
<file:inbound-endpoint path="src/main/resources/inbox" moveToDirectory="src/main/resources/processed" responseTimeout="10000" doc:name="CSV" connector-ref="File">
<file:filename-wildcard-filter pattern="*.csv" caseSensitive="true"/>
</file:inbound-endpoint>
<component class="it.aizoon.grpBuyer.AddMessageProperty" doc:name="Add Message Property"/>
<choice doc:name="Choice">
<when expression="INVOCATION:nome_file=azienda" evaluator="header">
<jdbc-ee:csv-to-maps-transformer delimiter="," mappingFile="src/main/resources/companies-csv-format.xml" ignoreFirstRecord="true" doc:name="CSV2Azienda"/>
<jdbc-ee:outbound-endpoint exchange-pattern="one-way" queryKey="InsertAziende" queryTimeout="-1" connector-ref="jdbcConnector" doc:name="Database Azienda">
<jdbc-ee:query key="InsertAziende" value="INSERT INTO aw006_azienda VALUES (#[map-payload:AW006_ID], #[map-payload:AW006_ID_CLIENTE], #[map-payload:AW006_RAGIONE_SOCIALE])"/>
</jdbc-ee:outbound-endpoint>
</when>
<when expression="INVOCATION:nome_file=servizi" evaluator="header">
<jdbc-ee:csv-to-maps-transformer delimiter="," mappingFile="src/main/resources/services-csv-format.xml" ignoreFirstRecord="true" doc:name="CSV2Servizi"/>
<jdbc-ee:outbound-endpoint exchange-pattern="one-way" queryKey="InsertServizi" queryTimeout="-1" connector-ref="jdbcConnector" doc:name="Database Servizi">
<jdbc-ee:query key="InsertServizi" value="INSERT INTO ctrl_aemd_unb_servizi VALUES (#[map-payload:CTRL_ID_TIPO_OPERAZIONE], #[map-payload:CTRL_DESCRIZIONE], #[map-payload:CTRL_COD_SERVIZIO])"/>
</jdbc-ee:outbound-endpoint>
</when>
<when expression="INVOCATION:nome_file=richiesta" evaluator="header">
<jdbc-ee:csv-to-maps-transformer delimiter="," mappingFile="src/main/resources/requests-csv-format.xml" ignoreFirstRecord="true" doc:name="CSV2Richiesta"/>
<jdbc-ee:outbound-endpoint exchange-pattern="one-way" queryKey="InsertRichieste" queryTimeout="-1" connector-ref="jdbcConnector" doc:name="Database Richiesta">
<jdbc-ee:query key="InsertRichieste" value="INSERT INTO ctrl_aemd_unb_richiesta VALUES (#[map-payload:CTRL_ID_CONTROLLER], #[map-payload:CTRL_NUM_RICH_VENDITORE], #[map-payload:CTRL_VENDITORE], #[map-payload:CTRL_CANALE_VENDITORE], #[map-payload:CTRL_CODICE_SERVIZIO], #[map-payload:CTRL_STATO_AVANZ_SERVIZIO], #[map-payload:CTRL_DATA_INSERIMENTO])"/>
</jdbc-ee:outbound-endpoint>
</when>
</choice>
</flow>
Please, I do not know how to fix this problem.
Thanks in advance for any kind of help
As SteveS said, the csv-to-maps-transformer might try to load the entire file to memory before process it. What you can try to do is split the csv file in smaller parts and send those parts to VM to be processed individually.
First, create a component to achieve this first step:
public class CSVReader implements Callable{
#Override
public Object onCall(MuleEventContext eventContext) throws Exception {
InputStream fileStream = (InputStream) eventContext.getMessage().getPayload();
DataInputStream ds = new DataInputStream(fileStream);
BufferedReader br = new BufferedReader(new InputStreamReader(ds));
MuleClient muleClient = eventContext.getMuleContext().getClient();
String line;
while ((line = br.readLine()) != null) {
muleClient.dispatch("vm://in", line, null);
}
fileStream.close();
return null;
}
}
Then, split your main flow in two
<file:connector name="File"
workDirectory="yourWorkDirPath" autoDelete="false" streaming="true"/>
<flow name="CsvToFile" doc:name="Split and dispatch">
<file:inbound-endpoint path="inboxPath"
moveToDirectory="processedPath" pollingFrequency="60000"
doc:name="CSV" connector-ref="File">
<file:filename-wildcard-filter pattern="*.csv"
caseSensitive="true" />
</file:inbound-endpoint>
<component class="it.aizoon.grpBuyer.AddMessageProperty" doc:name="Add Message Property" />
<component class="com.dgonza.CSVReader" doc:name="Split the file and dispatch every line to VM" />
</flow>
<flow name="storeInDatabase" doc:name="receive lines and store in database">
<vm:inbound-endpoint exchange-pattern="one-way"
path="in" doc:name="VM" />
<Choice>
.
.
Your JDBC Stuff
.
.
<Choice />
</flow>
Maintain your current file-connector configuration to enable streaming. With this solution the csv data can be processed without the need to load the entire file to memory first.
HTH
I believe that the csv-to-maps-transformer is going to force the whole file into memory. Since you are dealing with one large file, personally, I would tend to just write a Java class to handle it. The File endpoint will pass a filestream to your custom transformer. You can then make a JDBC connection and pick off the information a row at a time without having to load the whole file. I have used OpenCSV to parse the CSV for me. So your java class would contain something like the following:
protected Object doTransform(Object src, String enc) throws TransformerException {
try {
//Make a JDBC connection here
//Now read and parse the CSV
FileReader csvFileData = (FileReader) src;
BufferedReader br = new BufferedReader(csvFileData);
CSVReader reader = new CSVReader(br);
//Read the CSV file and add the row to the appropriate List(s)
String[] nextLine;
while ((nextLine = reader.readNext()) != null) {
//Push your data into the database through your JDBC connection
}
//Close connection.
}catch (Exception e){
}
In an application I am developing RESTful API and we want the client to send data as JSON. Part of this application requires the client to upload a file (usually an image) as well as information about the image.
I'm having a hard time tracking down how this happens in a single request. Is it possible to Base64 the file data into a JSON string? Am I going to need to perform 2 posts to the server? Should I not be using JSON for this?
As a side note, we're using Grails on the backend and these services are accessed by native mobile clients (iPhone, Android, etc), if any of that makes a difference.
I asked a similar question here:
How do I upload a file with metadata using a REST web service?
You basically have three choices:
Base64 encode the file, at the expense of increasing the data size by around 33%, and add processing overhead in both the server and the client for encoding/decoding.
Send the file first in a multipart/form-data POST, and return an ID to the client. The client then sends the metadata with the ID, and the server re-associates the file and the metadata.
Send the metadata first, and return an ID to the client. The client then sends the file with the ID, and the server re-associates the file and the metadata.
You can send the file and data over in one request using the multipart/form-data content type:
In many applications, it is possible for a user to be presented with
a form. The user will fill out the form, including information that
is typed, generated by user input, or included from files that the
user has selected. When the form is filled out, the data from the
form is sent from the user to the receiving application.
The definition of MultiPart/Form-Data is derived from one of those
applications...
From http://www.faqs.org/rfcs/rfc2388.html:
"multipart/form-data" contains a series of parts. Each part is
expected to contain a content-disposition header [RFC 2183] where the
disposition type is "form-data", and where the disposition contains
an (additional) parameter of "name", where the value of that
parameter is the original field name in the form. For example, a part
might contain a header:
Content-Disposition: form-data; name="user"
with the value corresponding to the entry of the "user" field.
You can include file information or field information within each section between boundaries. I've successfully implemented a RESTful service that required the user to submit both data and a form, and multipart/form-data worked perfectly. The service was built using Java/Spring, and the client was using C#, so unfortunately I don't have any Grails examples to give you concerning how to set up the service. You don't need to use JSON in this case since each "form-data" section provides you a place to specify the name of the parameter and its value.
The good thing about using multipart/form-data is that you're using HTTP-defined headers, so you're sticking with the REST philosophy of using existing HTTP tools to create your service.
I know that this thread is quite old, however, I am missing here one option. If you have metadata (in any format) that you want to send along with the data to upload, you can make a single multipart/related request.
The Multipart/Related media type is intended for compound objects consisting of several inter-related body parts.
You can check RFC 2387 specification for more in-depth details.
Basically each part of such a request can have content with different type and all parts are somehow related (e.g. an image and it metadata). The parts are identified by a boundary string, and the final boundary string is followed by two hyphens.
Example:
POST /upload HTTP/1.1
Host: www.hostname.com
Content-Type: multipart/related; boundary=xyz
Content-Length: [actual-content-length]
--xyz
Content-Type: application/json; charset=UTF-8
{
"name": "Sample image",
"desc": "...",
...
}
--xyz
Content-Type: image/jpeg
[image data]
[image data]
[image data]
...
--foo_bar_baz--
Here is my approach API (i use example) - as you can see, you I don't use any file_id (uploaded file identifier to the server) in API:
Create photo object on server:
POST: /projects/{project_id}/photos
body: { name: "some_schema.jpg", comment: "blah"}
response: photo_id
Upload file (note that file is in singular form because it is only one per photo):
POST: /projects/{project_id}/photos/{photo_id}/file
body: file to upload
response: -
And then for instance:
Read photos list
GET: /projects/{project_id}/photos
response: [ photo, photo, photo, ... ] (array of objects)
Read some photo details
GET: /projects/{project_id}/photos/{photo_id}
response: { id: 666, name: 'some_schema.jpg', comment:'blah'} (photo object)
Read photo file
GET: /projects/{project_id}/photos/{photo_id}/file
response: file content
So the conclusion is that, first you create an object (photo) by POST, and then you send second request with the file (again POST). To not have problems with CACHE in this approach we assume that we can only delete old photos and add new - no update binary photo files (because new binary file is in fact... NEW photo). However if you need to be able to update binary files and cache them, then in point 4 return also fileId and change 5 to GET: /projects/{project_id}/photos/{photo_id}/files/{fileId}.
I know this question is old, but in the last days I had searched whole web to solution this same question. I have grails REST webservices and iPhone Client that send pictures, title and description.
I don't know if my approach is the best, but is so easy and simple.
I take a picture using the UIImagePickerController and send to server the NSData using the header tags of request to send the picture's data.
NSMutableURLRequest *request = [[NSMutableURLRequest alloc] initWithURL:[NSURL URLWithString:#"myServerAddress"]];
[request setHTTPMethod:#"POST"];
[request setHTTPBody:UIImageJPEGRepresentation(picture, 0.5)];
[request setValue:#"image/jpeg" forHTTPHeaderField:#"Content-Type"];
[request setValue:#"myPhotoTitle" forHTTPHeaderField:#"Photo-Title"];
[request setValue:#"myPhotoDescription" forHTTPHeaderField:#"Photo-Description"];
NSURLResponse *response;
NSError *error;
[NSURLConnection sendSynchronousRequest:request returningResponse:&response error:&error];
At the server side, I receive the photo using the code:
InputStream is = request.inputStream
def receivedPhotoFile = (IOUtils.toByteArray(is))
def photo = new Photo()
photo.photoFile = receivedPhotoFile //photoFile is a transient attribute
photo.title = request.getHeader("Photo-Title")
photo.description = request.getHeader("Photo-Description")
photo.imageURL = "temp"
if (photo.save()) {
File saveLocation = grailsAttributes.getApplicationContext().getResource(File.separator + "images").getFile()
saveLocation.mkdirs()
File tempFile = File.createTempFile("photo", ".jpg", saveLocation)
photo.imageURL = saveLocation.getName() + "/" + tempFile.getName()
tempFile.append(photo.photoFile);
} else {
println("Error")
}
I don't know if I have problems in future, but now is working fine in production environment.
FormData Objects: Upload Files Using Ajax
XMLHttpRequest Level 2 adds support for the new FormData interface.
FormData objects provide a way to easily construct a set of key/value pairs representing form fields and their values, which can then be easily sent using the XMLHttpRequest send() method.
function AjaxFileUpload() {
var file = document.getElementById("files");
//var file = fileInput;
var fd = new FormData();
fd.append("imageFileData", file);
var xhr = new XMLHttpRequest();
xhr.open("POST", '/ws/fileUpload.do');
xhr.onreadystatechange = function () {
if (xhr.readyState == 4) {
alert('success');
}
else if (uploadResult == 'success')
alert('error');
};
xhr.send(fd);
}
https://developer.mozilla.org/en-US/docs/Web/API/FormData
Since the only missing example is the ANDROID example, I'll add it.
This technique uses a custom AsyncTask that should be declared inside your Activity class.
private class UploadFile extends AsyncTask<Void, Integer, String> {
#Override
protected void onPreExecute() {
// set a status bar or show a dialog to the user here
super.onPreExecute();
}
#Override
protected void onProgressUpdate(Integer... progress) {
// progress[0] is the current status (e.g. 10%)
// here you can update the user interface with the current status
}
#Override
protected String doInBackground(Void... params) {
return uploadFile();
}
private String uploadFile() {
String responseString = null;
HttpClient httpClient = new DefaultHttpClient();
HttpPost httpPost = new HttpPost("http://example.com/upload-file");
try {
AndroidMultiPartEntity ampEntity = new AndroidMultiPartEntity(
new ProgressListener() {
#Override
public void transferred(long num) {
// this trigger the progressUpdate event
publishProgress((int) ((num / (float) totalSize) * 100));
}
});
File myFile = new File("/my/image/path/example.jpg");
ampEntity.addPart("fileFieldName", new FileBody(myFile));
totalSize = ampEntity.getContentLength();
httpPost.setEntity(ampEntity);
// Making server call
HttpResponse httpResponse = httpClient.execute(httpPost);
HttpEntity httpEntity = httpResponse.getEntity();
int statusCode = httpResponse.getStatusLine().getStatusCode();
if (statusCode == 200) {
responseString = EntityUtils.toString(httpEntity);
} else {
responseString = "Error, http status: "
+ statusCode;
}
} catch (Exception e) {
responseString = e.getMessage();
}
return responseString;
}
#Override
protected void onPostExecute(String result) {
// if you want update the user interface with upload result
super.onPostExecute(result);
}
}
So, when you want to upload your file just call:
new UploadFile().execute();
I wanted send some strings to backend server. I didnt use json with multipart, I have used request params.
#RequestMapping(value = "/upload", method = RequestMethod.POST)
public void uploadFile(HttpServletRequest request,
HttpServletResponse response, #RequestParam("uuid") String uuid,
#RequestParam("type") DocType type,
#RequestParam("file") MultipartFile uploadfile)
Url would look like
http://localhost:8080/file/upload?uuid=46f073d0&type=PASSPORT
I am passing two params (uuid and type) along with file upload.
Hope this will help who don't have the complex json data to send.
You could try using https://square.github.io/okhttp/ library.
You can set the request body to multipart and then add the file and json objects separately like so:
MultipartBody requestBody = new MultipartBody.Builder()
.setType(MultipartBody.FORM)
.addFormDataPart("uploadFile", uploadFile.getName(), okhttp3.RequestBody.create(uploadFile, MediaType.parse("image/png")))
.addFormDataPart("file metadata", json)
.build();
Request request = new Request.Builder()
.url("https://uploadurl.com/uploadFile")
.post(requestBody)
.build();
try (Response response = client.newCall(request).execute()) {
if (!response.isSuccessful()) throw new IOException("Unexpected code " + response);
logger.info(response.body().string());
#RequestMapping(value = "/uploadImageJson", method = RequestMethod.POST)
public #ResponseBody Object jsongStrImage(#RequestParam(value="image") MultipartFile image, #RequestParam String jsonStr) {
-- use com.fasterxml.jackson.databind.ObjectMapper convert Json String to Object
}
Please ensure that you have following import. Ofcourse other standard imports
import org.springframework.core.io.FileSystemResource
void uploadzipFiles(String token) {
RestBuilder rest = new RestBuilder(connectTimeout:10000, readTimeout:20000)
def zipFile = new File("testdata.zip")
def Id = "001G00000"
MultiValueMap<String, String> form = new LinkedMultiValueMap<String, String>()
form.add("id", id)
form.add('file',new FileSystemResource(zipFile))
def urld ='''http://URL''';
def resp = rest.post(urld) {
header('X-Auth-Token', clientSecret)
contentType "multipart/form-data"
body(form)
}
println "resp::"+resp
println "resp::"+resp.text
println "resp::"+resp.headers
println "resp::"+resp.body
println "resp::"+resp.status
}
I have an ant build script that needs to pull files down from a web server. I can use the "get" task to pull these files down one by one. However, I'd like to be able to get a list of these files first and then iterate over the list with "get" to download the files. The webserver will report the list of files in json format, but I'm not sure how to parse json with ant.
Are there any ant plugins that allow for json parsing?
I used Dave's suggestion above and it worked out pretty well. Here's what I came up with:
(Note, I ripped this out of my actual build file and tried to remove anything specific and just leave the example parts, so forgive me if it's missing anything or whatever, but it should give you an idea of how this works).
<?xml version="1.0"?>
<project name="jsonExample" default="all">
<target name="all" depends="example" />
<target name="example">
<!-- This uses Rhino - an Open Source implementation of JavaScript written in Java -
to parse JSON. -->
<script language="javascript"> <![CDATA[
importClass(java.io.File);
importClass(java.io.FileReader);
importClass(java.io.BufferedReader);
importClass(java.io.FileWriter);
importClass(java.io.BufferedWriter);
var file = new File("/path/to/myJSON.js");
fr = new FileReader(file);
br = new BufferedReader(fr);
// Read the file we just retrieved from the webservice that contains JSON.
var json = br.readLine();
// Evaluate the serialized JSON
var struct = eval("(" + json + ")");
// Get the data from
var value = struct.data.VALUE;
echo = example.createTask("echo");
echo.setMessage("Value = " + value);
echo.perform();
]]>
</script>
</target>
You can use a <script> task to run JavaScript to decode your JSON.
Here is the macro I use to load json-properties.
<macrodef name="json-properties">
<attribute name="jsonFile"/>
<sequential>
<local name="_jsonFile"/>
<property name="_jsonFile" value="#{jsonFile}"/>
<script language="javascript">//<![CDATA[
var json = new Packages.java.lang.String(
Packages.java.nio.file.Files.readAllBytes(
Packages.java.nio.file.Paths.get(project.getProperty("_jsonFile"))), "UTF-8");
var properties = JSON.parse(json);
for(key in properties) {
project.setProperty(key, properties[key]);
}
//]]></script>
</sequential>
</macrodef>