I am new to elasticsearch, running elasticsearch from chrome:extension- Postman.
I want to enter bulk data into it, from JSON using Bulk API.
I have seen the command :
curl -s -XPOST 'http://jfblouvmlxecs01:9200/_bulk' --data-binary #bulk.json
While using Postman, I do not know where to store the file #bulk.json,
Currently I have stored it at C:\elasticsearch-1.5.2\bin\bulk.JSON
The command I am using is http://localhost:9200/_bulk --data-binary #bulk.JSON
This is throwing following error:
"error": "InvalidIndexNameException[[_bulk --data-binary #bulk.JSON] Invalid index name [_bulk --data-binary #bulk.JSON], must not contain the following characters [\, /, *, ?, \", <, >, |, , ,]]",
"status": 400 }
Can someone please suggest, where to store the JSON file.
Or am I doing something wrong here.
You can store your file anywhere and then pass the path like this:
curl -s -XPOST 'http://jfblouvmlxecs01:9200/_bulk' --data-binary #"/blah/blah/bulk.json"
If you want to do it with postman there is an answer here that explains how
But if you want to use this for large data, I wouldn't recommend using postman or sense. Use curl directly or logstash
See also this: https://github.com/taskrabbit/elasticsearch-dump
Related
Using curl, I have a GET function that is working great to retrieve the data (from Trello's API), but it comes out as a JSON that is all on one line. The software I'm using to read and interpret the retrieved data can only read lines of 32,000 characters.
Does anyone know a way to get it to include carriage returns in the way it does when you open it in Notepad++ and choose JSON Viewer (or any other manner).
The curl code is:
{LocationOfCurlOnMyServer} -H "Content-Type: text; charset=utf-8"
-o "{OutputFileName}"
--request GET --url "{TrelloURL}"
You can use jq JSON parser along with curl :
curl -L "http://citibikenyc.com/stations/json" | jq '.' > output.json
I have used
curl -XPOST "http://localhost:9200/<my_index_name>" -d #<absolute_path_to_my_json_file>
Then when I tried to get the data using
curl -XGET "http://localhost:9200/<my_index_name>"
its giving me data only for first line of my json file. (including other stuff also - settings, mappings,alias etc).
But why is it not able to load the entire json file ?
BTW, I am using ES 2.4.0. If I have to use bulk, what is the syntax ?
Try using these
curl -XPUT "http://localhost:9200/<my_index_name>" -d #<absolute_path_to_my_json_file>
GET index does not search actually.
You have also to run something like GET index/_search.
Currently I am able to PUT a single json file to a document in Cloudant using this : curl -X PUT 'https://username.cloudant.com/dummydb/doc3' -H "Content-Type: application/json" -d #numbers.json.I have many JSON files to be uploaded as different documents in the same DB.How can it be done?
So you definitely want to use Cloudant's _bulk_docs API endpoint in this scenario. It's more efficient (and cost-effective) if you're doing a bunch of writes. You basically POST an array that contains all your JSON docs. Here's the documentation on it: https://docs.cloudant.com/document.html#bulk-operations
Going one step further, so long as you've structured your JSON file properly, you can just upload the file to _bulk_docs. In cURL, that would look something like this: curl -X POST -d #file.json <domain>/db/_bulk_docs ... (plus the content type and all that other verbose stuff).
One step up from that would be using the ccurl (CouchDB/Cloudant cURL) tool that wraps your cURL statements to Cloudant and makes them less verbose. See https://developer.ibm.com/clouddataservices/2015/10/19/command-line-tools-for-cloudant-and-couchdb/ from https://stackoverflow.com/users/4264864/glynn-bird for more.
Happy Couching!
You can create a for loop and create documents from each JSON file.
For example, in the command below I have 4 JSON files in my directory and I create 4 documents in my people database:
for file in *.json
> do
> curl -d #$file https://username:password#myinstance.cloudant.com/people/ -H "Content-Type:application/json"
> done
{"ok":true,"id":"763a28122dad6c96572e585d56c28ebd","rev":"1-08814eea6977b2e5f2afb9960d50862d"}
{"ok":true,"id":"763a28122dad6c96572e585d56c292da","rev":"1-5965ef49d3a7650c5d0013981c90c129"}
{"ok":true,"id":"763a28122dad6c96572e585d56c2b49c","rev":"1-fcb732999a4d99ab9dc5462593068bed"}
{"ok":true,"id":"e944282beaedf14418fb111b0ac1f537","rev":"1-b20bcc6cddcc8007ef1cfb8867c2de81"}
I'm working on a cli for the Pushbullet HTTP API using Bash scripting. Sending pushes (notes and links), as well as creating, deleting, and modifying contacts & devices are all straight forward using curl and Bash. However, sending SMS and files are a bit more complex, as both require sending more complex JSON-formatted requests to the server (multiple JSON-formatted requests, in the case of pushing files).
I've tried sending many variations on the following (both with and without escape characters), but the server keeps replying about JSON-formatting errors. The following code is based off of the example given in the Pushbullet HTTP API documentation.
curl -su $auth_id: -X POST https://api.pushbullet.com/v2/ephemerals --header "Content-Type: application/json"
--data-binary '{ "\"type"\": "\"push"\", "\"push"\": { \
"\"type"\": "\"messaging_extension_reply"\", \
"\"package_name"\": "\"com.pushbullet.android"\", \
"\"source_user_iden"\": "\"$source_idens"\", \
"\"target_device_iden"\": "\"$target_idens"\", \
"\"conversation_iden"\": "\"$sms_device"\", \
"\"message"\": "\"Hello"\" \
} }'
Using bash -x, I can see that this is (supposedly) what is being sent to the server:
--data-binary '{"type": "push", "push": {
"type": "messaging_extension_reply",
"package_name": "com.pushbullet.android",
"source_user_iden": "<source_idens>",
"target_device_iden": "<device_idens>",
"conversation_iden": "<sms_phone_number>",
"message": "Hello" } }'
In all cases, the server returns:
{"error":{"type":"invalid_request","message":"Failed to parse JSON body.","cat":"(=^‥^=)"}}
What is the appropriate formatting of a JSON request using curl to send an SMS via the Pushbullet API? Am I overlooking something obvious? I'm trying to accomplish this using only curl and Bash, I see no reason why it's not possible (maybe not the fastest or most elegant way, but certainly possible).
I found the solution to my issue so I thought I'd share it. It was actually very simple:
Because the curl command includes a JSON-formatted response with single quotes, variable expansion was not occurring. This is a limitation (or perhaps a feature) of Bash. So, even though the server responded with { } indicating no errors in the request, the requests were actually being sent without the proper values for parameters, such asuser_iden,source_user_iden, etc.
Solution:
Enclose all variable expansions inside the JSON-formatted request in a double-quote and single-quote, like so:
"'"$user_idens"'"
First I'd like to apologize for how bad the API is, especially file upload and sending SMS. I was thinking of adding multipart or base64 file uploads to /v2/pushes. I think the first one might help you with curl, not sure about the base64 one. multipart is a huge pain though, so I'd prefer to make it better than the current setup if possible, rather than about equally as bad. Suggestions are welcome.
I tried your command line and it seemed to work, so I'm not sure what is going wrong. Here's the command line I did. Perhaps your quote escaping or newlines are causing the JSON error?
curl -u <access_token> -X POST https://api.pushbullet.com/v2/ephemerals --header "Content-Type: application/json" --data-binary '{"type": "push", "push": {"type": "messaging_extension_reply","package_name": "com.pushbullet.android","source_user_iden": "iden","target_device_iden": "device_idens", "conversation_iden": "sms_phone_number","message": "Hello" } }'
I'm getting an invalid JSON error when trying to add new JSON data to a couchapp.
I have JSON data that I converted from a CSV, and have been trying to add it with _bulk_docs to a couchapp. I solved the most obvious JSON errors with a JSON validator: http://jsonlint.com/
But one of the files is still giving an error, even thought the validator says the JSON is valid.
curl -d #songs_raw2.json -H "Content-Type: application/json" -X POST http://username:password#127.0.0.1:5984/songs/_bulk_docs
{"error":"bad_request","reason":"invalid_json"}
The file is here:
https://github.com/anatighe/sacred-harp/blob/master/songs_raw2.json
Looks like your json is not valid. This doesn't work for me:
$ curl https://raw.githubusercontent.com/anatighe/sacred-harp/master/songs_raw2.json | python -m json.tool | view -
Might have to do with special characters / encoding?
It looks like all of the single quote characters are messing it up. In every place there should be a ', there appears to be a � for me. I'm on a mac, I don't know if that matters.
When I replaced the weird characters with regular single quotes the curl request worked fine.