Related
As part of simulating what the front-end of an application will do while I work on the backend, I have been running various curl commands. Was easy to get just a file to be sent as Content-Type:application/octet-stream or just json with Content-Type:application/json
I sent json inline with this:
curl -X POST -H "Content-Type: application/json" -d '{"username":"xyz","password":"xyz"}' http://127.0.0.1:8088
Pulled the json out of a file and sent it like this:
curl -X POST -H "Content-Type: application/json" -H 'Accept: application/json' --data-binary #test.json http://127.0.0.1:8088
(any ideas what the 'Accept' does? does not work without it.., solution from here)
Sent just a file by itself like this:
curl --request POST -H "Content-Type:application/octet-stream" --data-binary "#photo.jpg" http://127.0.0.1:8088
Multipart with json inline and a picture goes very nicely like this:
curl --verbose --request POST --header "Content-Type:multipart/form-data" --form key1=value1 --form upload=#photo.jpg http://127.0.0.1:8088
(solution from here)
The trouble starts when I try to pull both the key-value pairs from a file and the photo, or to paste json into the curl command which also uploads a file. Can curl be convinced to send "Content-Type:multipart/form-data" with key value pairs coming from a file and a file attachment coming from a file?
john.json
{
"surname" : "Doe",
"name" : "John",
"city" : "Manchester",
"address" : "5 Main Street",
"hobbies" : ["painting","lawnbowls"]
}
and
john.jpg
I have tried some things but it just gives me error messages:
Tried inline, pasting in json:
$ curl --verbose --request POST --header "Content-Type:multipart/form-data" --data '{"surname" : "Doe","name" : "John","city" : "Manchester","address" : "5 Main Street", "hobbies" : ["painting","lawnbowls"]}' --form upload=#john.jpg http://127.0.0.1:8088
Warning: You can only select one HTTP request method! You asked for both POST
Warning: (-d, --data) and multipart formpost (-F, --form).
Then I tried to get them both from a file, but it didn't like that either:
$ curl --verbose --request POST --header "Content-Type:multipart/form-data" --form upload#john.json --form upload=#john.jpg http://127.0.0.1:8088
Warning: Illegally formatted input field!
curl: option --form: is badly used here
curl: try 'curl --help' or 'curl --manual' for more information
Any ideas how to make this work?
I think you're on the right track, but taking a look at the curl manual page might get you further.
Some key take aways from the --form option documentation:
The difference between # and < is then that # makes a file get attached in the post as a file upload, while the < makes a text field and just get the contents for that text field from a file.
So for the JSON use <, but for the picture use #.
I think also you should specify the content type of each section to help the web server know how to parse each section. Although it probably has expectations for each field.
You can also tell curl what Content-Type to use by using 'type=', in a manner similar to:
curl -F "web=#index.html;type=text/html" example.com
You'll have to look up the MIME types for JSON and jpegs.
And then the final piece to keep in mind: This option can be used multiple times.
Most of this is just an echo of what #Tanaike is saying above, but with more explanation from the documentation. I would recommend reading it in further detail.
I think the largest complain curl has with your command is that each part of the form has the key upload. That's sort of ambiguous. Is there one key for JSON and one key for the picture?
It's also really important to know what the webserver expects from each field in the form. There's a wide difference between a file upload and a text field.
With the help from #Breedly and #Tanaike got the following command to work, someone may find it useful one day:
curl --verbose --request POST --header "Content-Type:multipart/form-data" --form "upload1=<john.json" --form "upload2=#john.jpg" http://127.0.0.1:8088
It is happy to have "Content-Type:multipart/form-data" just once to cover both of them in this case. It really however wants the "<" not a "#" for the json.
The following also works:
curl --verbose --request POST --form "upload1=<john.json;type=application/json" --form "upload2=#john.jpg;type=multipart/form-data" http://127.0.0.1:8088
I recently upgraded to Elasticsearch version 6.1.1 and now I can't bulk index documents from a JSON file. When I do it inline, it works fine. Here are the contents of the document:
{"index" : {}}
{"name": "Carlson Barnes", "age": 34}
{"index":{}}
{"name": "Sheppard Stein","age": 39}
{"index":{}}
{"name": "Nixon Singleton","age": 36}
{"index":{}}
{"name": "Sharron Sosa","age": 33}
{"index":{}}
{"name": "Kendra Cabrera","age": 24}
{"index":{}}
{"name": "Young Robinson","age": 20}
When I run this command,
curl -XPUT 'localhost:9200/subscribers/ppl/_bulk?pretty' -H 'Content-Type: application/json' -d #customers_full.json
I get this error:
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "The bulk request must be terminated by a newline [\n]"
}
],
"type" : "illegal_argument_exception",
"reason" : "The bulk request must be terminated by a newline [\n]"
},
"status" : 400
It works fine if I send the data inline and in Elasticsearch 5.x. I tried adding newlines as well as the newline character to the end of the file. Doesn't seem to work.
Add empty line at the end of the JSON file and save the file and then try to run the below command
curl -XPOST localhost:9200/subscribers/ppl/_bulk?pretty --data-binary #customers_full.json -H 'Content-Type: application/json'
As the document says:
use the --data-binary flag instead of plain -d
-d doesn’t preserve newlines and doesn't format the json.
I faced this problem because of JSON formatting.
The error is pretty clear:
The bulk request must be terminated by a newline [\n]
So you simply need to add a newline at the end of your customers_full.json file and you'll be ok.
I ran into the same issue and spent hours adding and removing newlines before somebody pointed out I mis-typed the file name... So note that curl will throw the same error if the file is not actually present, making this super-confusing.
I had a similar issue when working with Elasticsearch 7.3.
Here's how I solved it.
Locate the .json file, say products.json file.
Double click to open the .json file in your text editor.
Scroll to the end of the .json file and then press the ENTER key on your keyboard.
Close the .json file. This will create a new line at the end of .json file.
Go back to your terminal and run the command below.
N/B: For the command below, the .json file name is products.json which I am importing to http://localhost:9200/ecommerce/product
curl -H "Content-type: application/json" -XPOST "http://localhost:9200/ecommerce/product/_bulk?pretty" --data-binary "#products.json"
That's all.
I hope this helps
For anyone using postman to make requests to ElasticSearch
Just press enter to create an empty new line!
And voila, problem solved
This worked for me:
curl -H "Content-Type: application/x-ndjson" -XPOST "localhost:9200/bank/_bulk?pretty&refresh" --data-binary "#C:\Program Files\Elastic\Elasticsearch\7.2.0\accounts.json"
I had the same problem running on Windows 10, using ElasticSearch 7.5.1.
I tried all the answers; none of them worked. I was certain I had a newline at the end of my file.
To get it to work, I had to ensure the file I was uploading was using UNIX end-of-line characters (0A only, no 0D), and also the encoding had to be UTF-8.
Using Notepad++, you can edit the metadata of the file.
Finally some good news:
Press Enter end of the line inside the JSON file and run the command again.
curl -H "Content-Type: application/x-ndjson" -XPOST 'localhost:9200/customers/personal/_bulk?pretty&refresh' --data-binary #"generated.json"
I just forgot to add an # symbol before file name like this
--data-binary "#products.json"
you just need to open json file and then go to the end of the file ( Ctrl+end) and then please Enter to break a new line.
I was struggling with this for a hot minute. Mine was caused by a space in my curl request between the --data and the -binary and gave the same error - must end with new line [\\n]}.
So double-check that in the curl req it's --data-binary not --data - binary
For me, the issue was only due to the wrong file name.
I have used customer_full.json in command whereas the file was named customer_full in my file system (without the extension).
So in my case,this command worked for me:
curl -H "Content-Type: application/x-ndjson" -XPOST 'http://localhost:9200/customers/personal/_bulk?pretty&refresh' --data-binary #"customer_full"
I faced a similar issue on windows using elastic 7.9.1 When I used below CURL command.
curl -s -H "Content-Type: application/json" -XPOST localhost:9200/accounts/docs/_bulk?filter_path=items.*.error --data-binary "#textoutES.json" >> erroredAtES.json"
I tried to manually add Newline at the end of the file but did not work.
I have created my JSON by extracting data from MySQL database like below to make sure my records should end with LINE FEED and CARRIAGE RETURN.
Then, it is working for me:
SELECT CONCAT('{"index":{"_id":"',id,'"}}\r\n',request_data,'\r\n') reqestData FROM cards
More importantly you End-of-File should have a carriage-return and Line-Feed (CRLF)if you are using windows. Also if any line in JSON contains a CR but no LF then you will get parsing exception Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper#2d5ed2ca
Windows CRLF and EOF
You need to use --data-binary instead of -d in your curl request. Please see: Bulk API
This worked in my local set-up.
curl -H "Content-type:application/json" -XPOST "http://localhost:9200/customer/personal/_bulk?pretty" --data-binary #"generated.json"
How do you do that if you are not using a data-file? I am having the issue but not sending data from a file.
const data1 = {
"amount" : "100",
"#timestamp" : `${UTC_timestamp}`,
"transaction_attributes" : {
"channel" : "channel-foobarbaz",
"session_id" : "session-1234",
"information" : "iinformation-foobarbaznformation-foobarbaz"
},
"currency" : {
"currency_description" : "my currency description",
},
"external_timestamp" : "2021-12-03T11:22:55.206229500Z" };
// execute a post
let res = http.post(url,JSON.stringify(data1),params);
A few things to check:
The file ends with new line (\n).
The new line is using Unix eol (LF) and not mac or windows eol.
When specifying the file name in the curl command, make sure "#" was added before the file name.
I have been trying to call the CloudFlare API v4, using an example provided in their own documentation.
This is the code of the example
curl -X PUT "https://api.cloudflare.com/client/v4/zones/023e105f4ecef8ad9ca31a8372d0c353/dns_records/372e67954025e0ba6aaa6d586b9e0b59" \ -H "X-Auth-Email: user#example.com" \ -H "X-Auth-Key: c2547eb745079dac9320b638f5e225cf483cc5cfdda41" \ -H "Content-Type: application/json" \ --data '{"id":"372e67954025e0ba6aaa6d586b9e0b59","type":"A","name":"example.com","content":"1.2.3.4","proxiable":true,"proxied":false,"ttl":120,"locked":false,"zone_id":"023e105f4ecef8ad9ca31a8372d0c353","zone_name":"example.com","created_on":"2014-01-01T05:20:00.12345Z","modified_on":"2014-01-01T05:20:00.12345Z","data":{}}'
Which can also be found at
Update DNS Records
Using Windows cmd.exe to run this command, I need to make it single line first, so I removed the "" and reformatted it (twice) making sure I altered no part in the process.
This is the same code in one line:
curl -X PUT "https://api.cloudflare.com/client/v4/zones/023e105f4ecef8ad9ca31a8372d0c353/dns_records/372e67954025e0ba6aaa6d586b9e0b59" -H "X-Auth-Email: user#example.com" -H "X-Auth-Key: c2547eb745079dac9320b638f5e225cf483cc5cfdda41" -H "Content-Type: application/json" --data '{"id":"372e67954025e0ba6aaa6d586b9e0b59","type":"A","name":"example.com","content":"1.2.3.4","proxiable":true,"proxied":false,"ttl":120,"locked":false,"zone_id":"023e105f4ecef8ad9ca31a8372d0c353","zone_name":"example.com","created_on":"2014-01-01T05:20:00.12345Z","modified_on":"2014-01-01T05:20:00.12345Z","data":{}}'
When I run this single-liner in cmd, it works but I get a malformed JSON in request body, however, a visual check, formatting on Notepad++ and a run through the JSON validator are all positive, this JSON (copied from the CloudFlare documentation) is not malformed.
Error Message
{"success":false,"errors":[{"code":6007,"message":"Malformed JSON in request body"}],"messages":[],"result":null}
Googling this error message or the error code gives me nothing and this same command works on a PC running Linux.
Can someone tell me if this is a known bug, if the JSON really is malformed or if something else comes to mind?
I found the answer in the blog post: "Expecting to find valid JSON in request body..." curl for Windows.
For example, for Purge everything --data value will be:
# On Linux
--data '{"purge_everything":true}'
# On Windows
--data "{\"purge_everything\":true}"
For Windows:
Replace the single quotes with double quotes: ' --> "
Escape the double quotes with a backslash: " --> \"
cmd.exe doesn't support single quotes, to run those commands straight from the docs you can use Bash.
Bash can be enabled in Windows 10 : https://www.laptopmag.com/uk/articles/use-bash-shell-windows-10
or Git Bash comes with Git for windows: https://gitforwindows.org/
I've been posting a file with this curl command:
curl -i -F file=#./File.xlsm -F name=file -X POST http://example.com/new_file/
Now I want to send some information about the file (as JSON) along with the file.
curl -i -H "Content-Type: application/json" -d '{"metadata": {"comment": "Submitting a new data set.", "current": false }, "sheet": 1, "row": 7 }' -F file=#./File.xlsm -F name=file http://example.com/new_file/
Curl is very grumpy about being used in this completely incorrect way, and in this case it says "You can only select one HTTP request!" OK, fair enough, so how do I get the file attachment and those POST variables into a single curl HTTP request?
I've had success developing similar endpoints that accept multiple files along with their metadata in JSON format.
curl -i -X POST -H "Content-Type: multipart/mixed" -F "blob=#/Users/username/Documents/bio.jpg" -F "metadata={\"edipi\":123456789,\"firstName\":\"John\",\"lastName\":\"Smith\",\"email\":\"john.smith#gmail.com\"};type=application/json" http://localhost:8080/api/v1/user/
Notice the addition of ;type=application/json at the end of the metadata request part. When uploading multiple files of different types, you can define the mime type at the end of the -F value.
I have confirmed that this works for Spring MVC 4.3.7 using #RequestPart. The key in that instance is to not provide the consumes value on the #RequestMapping annotation.
You could just add another form field:
curl -X POST http://someurl/someresource -F upload=#/path/to/some/file -F data="{\"test\":\"test\"}"
Note: due to the content type, this does not really equate to sending json to the web service.
This worked for me:
curl -v -H "Content-Type:multipart/form-data" -F "meta-data=#C:\Users\saurabh.sharma\Desktop\test.json;type=application/json" -F "file-data=#C:\Users\saurabh.sharma\Pictures\Saved Pictures\windows_70-wallpaper.jpg" http://localhost:7002/test/upload
test.json has the json data I want to send.
From #nbrooks comment, adding an additional header HTTP header works fine as shown below by using several -H or --header flags to your curl command:
curl -H "comment: Submitting a new data set." -H "current: false" -H "sheet: 1" -H "row: 7" -F file=#./File.xlsm -F name=file http://example.com/new_file/
comment and current can be combined into "metadata" in the
request.headers processing part on the flask web server.
I am new to Elasticsearch and have been entering data manually up until this point. For example I've done something like this:
$ curl -XPUT 'http://localhost:9200/twitter/tweet/1' -d '{
"user" : "kimchy",
"post_date" : "2009-11-15T14:12:12",
"message" : "trying out Elastic Search"
}'
I now have a .json file and I want to index this into Elasticsearch. I've tried something like this too, but no success:
curl -XPOST 'http://jfblouvmlxecs01:9200/test/test/1' -d lane.json
How do I import a .json file? Are there steps I need to take first to ensure the mapping is correct?
The right command if you want to use a file with curl is this:
curl -XPOST 'http://jfblouvmlxecs01:9200/test/_doc/1' -d #lane.json
Elasticsearch is schemaless, therefore you don't necessarily need a mapping. If you send the json as it is and you use the default mapping, every field will be indexed and analyzed using the standard analyzer.
If you want to interact with Elasticsearch through the command line, you may want to have a look at the elasticshell which should be a little bit handier than curl.
2019-07-10: Should be noted that custom mapping types is deprecated and should not be used. I updated the type in the url above to make it easier to see which was the index and which was the type as having both named "test" was confusing.
Per the current docs, https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html:
If you’re providing text file input to curl, you must use the
--data-binary flag instead of plain -d. The latter doesn’t preserve newlines.
Example:
$ curl -s -XPOST localhost:9200/_bulk --data-binary #requests
We made a little tool for this type of thing https://github.com/taskrabbit/elasticsearch-dump
One thing I've not seen anyone mention: the JSON file must have one line specifying the index the next line belongs to, for every line of the "pure" JSON file.
I.E.
{"index":{"_index":"shakespeare","_type":"act","_id":0}}
{"line_id":1,"play_name":"Henry IV","speech_number":"","line_number":"","speaker":"","text_entry":"ACT I"}
Without that, nothing works, and it won't tell you why
I'm the author of elasticsearch_loader
I wrote ESL for this exact problem.
You can download it with pip:
pip install elasticsearch-loader
And then you will be able to load json files into elasticsearch by issuing:
elasticsearch_loader --index incidents --type incident json file1.json file2.json
I just made sure that I am in the same directory as the json file and then simply ran this
curl -s -H "Content-Type: application/json" -XPOST localhost:9200/product/default/_bulk?pretty --data-binary #product.json
So if you too make sure you are at the same directory and run it this way.
Note: product/default/ in the command is something specific to my environment. you can omit it or replace it with whatever is relevant to you.
Adding to KenH's answer
$ curl -s -XPOST localhost:9200/_bulk --data-binary #requests
You can replace #requests with #complete_path_to_json_file
Note: #is important before the file path
just get postman from https://www.getpostman.com/docs/environments give it the file location with /test/test/1/_bulk?pretty command.
You are using
$ curl -s -XPOST localhost:9200/_bulk --data-binary #requests
If 'requests' is a json file then you have to change this to
$ curl -s -XPOST localhost:9200/_bulk --data-binary #requests.json
Now before this, if your json file is not indexed, you have to insert an index line before each line inside the json file. You can do this with JQ. Refer below link:
http://kevinmarsh.com/2014/10/23/using-jq-to-import-json-into-elasticsearch.html
Go to elasticsearch tutorials (example the shakespeare tutorial) and download the json file sample used and have a look at it. In front of each json object (each individual line) there is an index line. This is what you are looking for after using the jq command. This format is mandatory to use the bulk API, plain json files wont work.
As of Elasticsearch 7.7, you have to specify the content type also:
curl -s -H "Content-Type: application/json" -XPOST localhost:9200/_bulk --data-binary #<absolute path to JSON file>
I wrote some code to expose the Elasticsearch API via a Filesystem API.
It is good idea for clear export/import of data for example.
I created prototype elasticdriver. It is based on FUSE
If you are using the elastic search 7.7 or above version then follow below command.
curl -H "Content-Type: application/json" -XPOST "localhost:9200/bank/_bulk? pretty&refresh" --data-binary #"/Users/waseem.khan/waseem/elastic/account.json"
On above file path is /Users/waseem.khan/waseem/elastic/account.json.
If you are using elastic search 6.x version then you can use the below command.
curl -X POST localhost:9200/bank/_bulk?pretty&refresh --data-binary #"/Users/waseem.khan/waseem/elastic/account.json" -H 'Content-Type: application/json'
Note: Make sure in your .json file at the end you will add the one empty
line otherwise you will be getting below exception.
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "The bulk request must be terminated by a newline [\n]"
}
],
"type" : "illegal_argument_exception",
"reason" : "The bulk request must be terminated by a newline [\n]"
},
`enter code here`"status" : 400
if you are using VirtualBox and UBUNTU in it or you are simply using UBUNTU then it can be useful
wget https://github.com/andrewvc/ee-datasets/archive/master.zip
sudo apt-get install unzip (only if unzip module is not installed)
unzip master.zip
cd ee-datasets
java -jar elastic-loader.jar http://localhost:9200 datasets/movie_db.eloader
If you want to import a json file into Elasticsearch and create an index, use this Python script.
import json
from elasticsearch import Elasticsearch
es = Elasticsearch([{'host': 'localhost', 'port': 9200}])
i = 0
with open('el_dharan.json') as raw_data:
json_docs = json.load(raw_data)
for json_doc in json_docs:
i = i + 1
es.index(index='ind_dharan', doc_type='doc_dharan', id=i, body=json.dumps(json_doc))