elasticsearch bulk insert JSON file - json

I have the following JSON file
I have used awk to get rid of empty spaces, trailing, next line
awk -v ORS= -v OFS= '{$1=$1}1' data.json
I have added a create request at the top of my data.json followed by \n and the rest of my data.
{"create": {"_index":"socteam", "_type":"products"}}
When I issue bulk submit request, I get the following error
CURL -XPUT http://localhost:9200/_bulk
{
"took": 1,
"errors": true,
"items": [
{
"create": {
"_index": "socteam",
"_type": "products",
"_id": "AVQuGPff-1Y7OIPIJaLX",
"status": 400,
"error": {
"type": "mapper_parsing_exception",
"reason": "failed to parse",
"caused_by": {
"type": "not_x_content_exception",
"reason": "Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes"
}
}
}
}
]
Any idea on what this error mean? I haven't created any mapping, I'm using vanilla elasticsearch

Accordingly to this doc, you have to specify index and type in URL:
curl -XPUT 'localhost:9200/socteam/products/_bulk?pretty' --data-binary "#data.json"
It works for PUT and POST methods.
And your data.json file should have structure like:
{"index":{"_id":"1"}}
{"name": "John Doe" }
{"index":{"_id":"2"}}
{"name": "Jane Doe" }
Maybe there present another method to import data, but i know just this... Hope it'll help...

Related

How to import a big JSON-file to a Docker-swarm cluster with ELK stack?

Basically I want to import JSON-data into (Logstash->ElasticSearch->) Kibana, but I'm completely new and stuck at the different methods, which I do not fully understand and get errors or no output.
What I've got is a file test.json containing Wikipedia-data in this format:
{
"results": [
{
"curr": "Ohio_\"Heartbeat_Bill\"",
"n": 43,
"prev": "other-external",
"type": "external"
},
{
"curr": "Ohio_\"Heartbeat_Bill\"",
"n": 1569,
"prev": "other-search",
"type": "external"
},
{
"curr": "Ohio_\"Heartbeat_Bill\"",
"n": 11,
"prev": "other-internal",
"type": "external"
},
...
And so on. The file is 1.3Mb big, because I've deleted some of the largest examples.
I tried the curl command:
cat test.json | jq -c '.[] | {"index": {}}, .' | curl -XPOST localhost:9200/_bulk --data-binary #-
and
curl -s -XPOST localhost:9200/_bulk --data-binary #test.json
and
write "{ "index" : { } }" at the beginning of the document
I also tried:
curl -XPUT http://localhost:9200/wiki -d '
{
"mappings" : {
"_default_" : {
"properties" : {
"curr" : {"type": "string"},
"n" : {"type": "integer"},
"prev" : {"type": "string"},
"type" : {"type": "string"}
}
}
}
}
';
But I always get this error:
{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}
Or when I use:
curl localhost:9200/wiki -H "Content-type:application/json" -X POST -d #test.json
I get:
{"error":"Incorrect HTTP method for uri [/wiki] and method [POST], allowed: [GET, HEAD, DELETE, PUT]","status":405}
And when I replace "wiki" with "_bulk", like all the examples seem to have in common, then I get:
{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication token for REST request [/_bulk]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication token for REST request [/_bulk]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401
I also have copy-pasted-and-adjusted-as-far-as-I-understood the conf-file in Kibana-Logstash-Pipeline like this:
input
{
file
{
codec => multiline
{
pattern=> '^\{'
negate=> true
what=> previous
}
path => ["/home/user/docker-elastic/examples/pretty.json"]
start_position => "beginning"
sincedb_path => "/dev/null"
exclude => "*.gz"
}
}
filter
{
mutate
{
replace => [ "message", "%{message}}" ]
gsub => [ 'message','\n','']
}
if [message] =~ /^{.*}$/
{
json { source => message }
}
}
output
{
elasticsearch {
protocol => "http"
codec => json
host => "localhost"
index => "wiki_json"
embedded => true
}
stdout { codec => rubydebug }
}
But when I click "create and deploy" nothing happens.
So I have tried some examples, but like I said - I don't fully understand them and have therefore trouble getting my data to Kibana. I've written Logstash and ElasticSearch, because I would love to pass the data by using those, too.
Can somebody please explain to me, how I can pass this data directly, without manually altering the file? Many answers said that the data cannot be passed in the structure I have but must be "one line, one input"-only. But I cannot alter the whole file with nearly 40000 data by hand and I would like not have to write a python-script for it..
Maybe there is a tool or something? Or maybe I'm just too stupid to understand the syntax and am doing something wrong?
Any help is appreciated!
Thank you in advance!
Like #Ian Kemp answered in the comment section, the Problem was that I used POST and not PUT. After that I got an error saying that authentification failed, so I googled for it and got the final answer:
curl elastic:changeme#localhost:9200/wiki -H "Content-type: application/json" -X PUT -d #test.json
with the index line in the file.
This is the structure of how I finally got the data to be in Elasticsearch :)
THANK YOU very much Ian Kemp!

How to iterate objects from JSON array file in batch script

I have a windows batch script to perform POST request using curl and reads the data from JSON file, it works fine with only a single object in the file and it looks like this.
curl -u username#password -H "Content-Type: application/json" -d #file.json http://apiurl.com
and the json file is this:
{
"name": "Empty name",
"properties": {
"active": "True",
"subcity_zone": "East Hararge",
"woreda": "Meta"
}
}
But now I want to send the request each object in the array by iterating each item. So, How do I iterate each JSON object from the file?
Here is what the new JSON array file looks like:
[{
"name": "test facility I",
"properties": {
"active": "True",
"city": "",
"subcity_zone": "East Hararge",
"woreda": "Meta"
}
},
{
"name": "test facility II",
"properties": {
"active": "True",
"subcity_zone": "East Hararge",
"woreda": "Girawa"
}
}]
Using jq:
jq -c '.[]' file | while read js; do
curl -u username#password -H "Content-Type: application/json" -d #<(echo "$js") http://apiurl.com
done
The jq command extracts the each object in one line that is read by read command into the $js variable.
The <(echo "$js") creates a temporary file that is passed to curl.

How to send JSON file as part of request body in CURL POST command

I am using CURL command line to send HTTP POST to a web service. I want to include a file's contents as a PART of the body of the POST command. Is this possible? I know I can send a file as the entire body as answered here. But I only want a part of the body to be the content of the file.
For example
curl -d '{ "name": "rahul", "speed": "fast", "data": { "number": 1, "letter": "abd", "letter2": "efg"} }' 'http://...'
Here I only want data as the file's content. Not the entire body. How can I do this?
Set a variable to contain the file contents:
data=$(cat /path/to/file)
then substitute it into the JSON:
curl -d '{ "name": "rahul", "speed": "fast", "data": "'$data'" }' 'http://...'
You accepted #Barmar's answer, but for anyone reading this, #Barmar switched the double- and single-quotes, which will cause the command to not work as intended.
The following works:
data="$(cat filename)" && \
curl -d '{ "name": "rahul", "speed": "fast", "data": "'$data'" }' 'http://...'
Notice that the $data variable is surrounded by single-quotes first, then double-quotes.

Github API v3 JSON passing

how can I avoid the parsing errors for
curl -H 'Accept: application/vnd.github.VERSION.raw' -XPUT -g 'https://api.github.com/repos/USER/l1/contents/PATH/FILENAME.json?ref=gh-pages&access_token=57eef6413b12cb439b837b8fc4751b3291650de1' -d '{
"message": "update from api",
"committer": {
"name": "USER",
"email": "USERe#MAIL.com"
},
"content": "[{"a": "aaa","b": "bbb"}]",
"sha": "c321fe9f6418053ecb87eb3cd2518a4xdfc83ebf"
}'
Answer:
{
"message": "**Problems parsing JSON**",
"documentation_url": "https://developer.github.com/v3/repos/contents/"
}
Instead of
"[{"a": "aaa","b": "bbb"}]"
I've tried
"[{\"a\": \"aaa",\"b\": \"bbb\"}]"
but then I get the following error:
{
"message": "**content is not valid Base64**",
"documentation_url": "https://developer.github.com/v3/repos/contents/"
}
Best,
If you open the URL to the documentation given in the error response, you'll see that the content needs to be Base64 encoded.

Parse error on Google GeoLocate API sample JSON

Has Google changed their GeoLocation api and not updated the documentation?
I have been following their example code verbatim off of the following page
https://developers.google.com/maps/documentation/geolocation/intro
I pasted the sample request into a file on my system called ex.json. I double checked that my Google Maps Geolocation API is set to on and executed the following curl command
curl -d ex.json -H "Content-Type: application/json" -i "https://www.googleapis.com/geolocation/v1/geolocate?key=[My key, yes I pasted my actual key in]"
I received the following response
{
"error": {
"errors": [
{
"domain": "global",
"reason": "parseError",
"message": "Parse Error"
}
],
"code": 400,
"message": "Parse Error"
}
}
Which according to the documentation means that there is something wrong with the example json they provided. Just for completeness the sample json looks like
{
"homeMobileCountryCode": 310,
"homeMobileNetworkCode": 260,
"radioType": "gsm",
"carrier": "T-Mobile",
"cellTowers": [
{
"cellId": 39627456,
"locationAreaCode": 40495,
"mobileCountryCode": 310,
"mobileNetworkCode": 260,
"age": 0,
"signalStrength": -95
}
],
"wifiAccessPoints": [
{
"macAddress": "01:23:45:67:89:AB",
"signalStrength": 8,
"age": 0,
"signalToNoiseRatio": -65,
"channel": 8
},
{
"macAddress": "01:23:45:67:89:AC",
"signalStrength": 4,
"age": 0
}
]
}
JsonLint verified that it is proper Json, and the documentation says that all fields are optional. What am I missing, was some required field added after the documentation was written?
Curl needs "-X POST" as extra parameter. Works like this:
curl 'https://www.googleapis.com/geolocation/v1/geolocate?key=YOURKEY' -X POST -H "Content-Type: application/json" -d #yourjsonfile.json
Found the solution, it was a silly mistake :-
my file name was "sampledata.json" i changed this to "#sampledata.json"
I tried with three different curl commands. I got response.