Send JSON/XML/TXT/CSV files to ElasticSearch - json

I have been trying to send some data in various formats to ElasticSearch running on my computer.
I am running Ubuntu 17 (the latest release) and am unable to do so.
Here are the commands I'm using on Terminal:
curl -X POST 'http://localhost:9200/json/test/2' -d #json.json
I am in the correct directory where the files are. Here is the error I'm getting:
{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}
I've searched online to no avail.
What am I doing wrong?

You can do something like this :
curl -XPUT 'localhost:9200/twitter/tweet/1?pretty' -H 'Content-Type: application/json' -d'
{
"user" : "kimchy",
"post_date" : "2009-11-15T14:12:12",
"message" : "trying out Elasticsearch"
}
'
The reason your request wasn't passing because you didnt specify the Content-Type as JSON. Plus, you should use PUT and not POST :) I copied this request from this documentation : https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html. Cheers.

The error you're seeing is due to the version of the Elastic Search being different. Before -XPUT, add -H 'Content-Type: application/json'
Here is an example:
curl -H 'Content-Type: application/json' -XPUT 127.0.0.1:9200/

Related

User not authenticated

Im trying to download a file as described in forge documentation.
So im getting an access token with scope=data:read for the first 2 steps :
curl
-v 'https://developer.api.autodesk.com/authentication/v1/authenticate'
-X 'POST'
-H 'Content-Type: application/x-www-form-urlencoded'
-d 'client_id=xxx&client_secret=xxx&grant_type=client_credentials&scope=data:read'
It works up to the GET hubs/:hub_id/projects endpoint.
Then, the GET projects/:project_id/folders/:folder_id/contents endpoint requires data:write, so im adding it:
curl
-v 'https://developer.api.autodesk.com/authentication/v1/authenticate'
-X 'POST'
-H 'Content-Type: application/x-www-form-urlencoded'
-d 'client_id=xxx&client_secret=xxx&grant_type=client_credentials&scope=data:read%20data:write'
It returns an access token without any issue, but when i use for this endpoint, i get a "User not authenticated" error.
I honestly dont know where its coming from, i tried without the url-encoded space as well, no difference.
Thanks in advance for any pointers in the right direction.

How create Issues with API/CURL to Bitbucket

I'm trying to create issues in Bitbucket with the Windows CURL command, but it doesn't work, I'm getting this error message:
{"type": "error", "error": {"message": "No import job started"}}
My command CURL:
curl -u username:password -X GET https://api.bitbucket.org/2.0/repositories/name/name2/issues/import -H "Content-Type: application/json" --data #data.jsonson
I'm trying to send the following result in JSON:
{
"title": "title"
}
But it doesn't work.
Does anyone know how I can create issues?
I suppose, you should use a different URL path for posting issues, please see here for post method. It should be like this: https://api.bitbucket.org/2.0/repositories/username/reponame/issues
For posting just an issue with a title you could use this:
curl --ssl-no-revoke -u username#password -X POST -H "Content-Type: application/json" -d "{\"title\" : \"test2\"}" https://api.bitbucket.org/2.0/repositories/username/reponame/issues
To post a content of the issue use content tag:
curl --ssl-no-revoke -u username#password -X POST -H "Content-Type: application/json" -d "{\"title\" : \"test2\",\"content\": {\"raw\": \"just test text\", \"markup\": \"plaintext\"}}" https://api.bitbucket.org/2.0/repositories/username/reponame/issues
For example for me this worked (it's my private repo):
curl --ssl-no-revoke -u myaccountname#mypasswordname -X POST -H "Content-Type: application/json" -d "{\"title\" : \"test2\",\"content\": {\"raw\": \"just test texts\", \"markup\": \"plaintext\"}}" https://api.bitbucket.org/2.0/repositories/dvmochalov/testrepo/issues
--ssl-no-revoke - works only for windows curl and it's just in case if you are running Windows with antivirus software or working with proxy.

How to GET data of a key in json using curl?

Sorry Pretty noob to json.
Basically I have a simple server where I can upload data in there.
E.g:
curl -vX PUT "http://IP:port/ABC" -H "Content-Type: application/json" -d #"Once Upon a time."
After when I do:
curl -vX GET "http://IP:port/ABC" -H "Content-Type: application/json"
I get:
{"reverse_shell":
{"aliases":{},"mappings":{},"settings":
{"index":{"creation_date":"1561863982371","number_of_shards":"5","number_of_replicas":"1","uuid":"IAWE83rYQqmtKW-9svkBVg","version":{"created":"6040299"},"provided_name":"ABC"}
}
}
}
As you can see there is no where mentioning Once Upon a time, so is there I am missing? or how do I get that data from json using curl?
I am in kali linux env.
It looks like you are trying to post a string as a file.
When you specify a "#" with -d this tells curl to send the data from a file called "Once Upon a time."
If you are trying to put a file then you should do:
my_text_file.txt
Once Upon a time.
curl -vX PUT "http://IP:port/ABC" -H "Content-Type: application/json" -d #./my_text_file.txt https://server/api/path

use curl to POST multipart/form-data, file and lots of key-value pairs

As part of simulating what the front-end of an application will do while I work on the backend, I have been running various curl commands. Was easy to get just a file to be sent as Content-Type:application/octet-stream or just json with Content-Type:application/json
I sent json inline with this:
curl -X POST -H "Content-Type: application/json" -d '{"username":"xyz","password":"xyz"}' http://127.0.0.1:8088
Pulled the json out of a file and sent it like this:
curl -X POST -H "Content-Type: application/json" -H 'Accept: application/json' --data-binary #test.json http://127.0.0.1:8088
(any ideas what the 'Accept' does? does not work without it.., solution from here)
Sent just a file by itself like this:
curl --request POST -H "Content-Type:application/octet-stream" --data-binary "#photo.jpg" http://127.0.0.1:8088
Multipart with json inline and a picture goes very nicely like this:
curl --verbose --request POST --header "Content-Type:multipart/form-data" --form key1=value1 --form upload=#photo.jpg http://127.0.0.1:8088
(solution from here)
The trouble starts when I try to pull both the key-value pairs from a file and the photo, or to paste json into the curl command which also uploads a file. Can curl be convinced to send "Content-Type:multipart/form-data" with key value pairs coming from a file and a file attachment coming from a file?
john.json
{
"surname" : "Doe",
"name" : "John",
"city" : "Manchester",
"address" : "5 Main Street",
"hobbies" : ["painting","lawnbowls"]
}
and
john.jpg
I have tried some things but it just gives me error messages:
Tried inline, pasting in json:
$ curl --verbose --request POST --header "Content-Type:multipart/form-data" --data '{"surname" : "Doe","name" : "John","city" : "Manchester","address" : "5 Main Street", "hobbies" : ["painting","lawnbowls"]}' --form upload=#john.jpg http://127.0.0.1:8088
Warning: You can only select one HTTP request method! You asked for both POST
Warning: (-d, --data) and multipart formpost (-F, --form).
Then I tried to get them both from a file, but it didn't like that either:
$ curl --verbose --request POST --header "Content-Type:multipart/form-data" --form upload#john.json --form upload=#john.jpg http://127.0.0.1:8088
Warning: Illegally formatted input field!
curl: option --form: is badly used here
curl: try 'curl --help' or 'curl --manual' for more information
Any ideas how to make this work?
I think you're on the right track, but taking a look at the curl manual page might get you further.
Some key take aways from the --form option documentation:
The difference between # and < is then that # makes a file get attached in the post as a file upload, while the < makes a text field and just get the contents for that text field from a file.
So for the JSON use <, but for the picture use #.
I think also you should specify the content type of each section to help the web server know how to parse each section. Although it probably has expectations for each field.
You can also tell curl what Content-Type to use by using 'type=', in a manner similar to:
curl -F "web=#index.html;type=text/html" example.com
You'll have to look up the MIME types for JSON and jpegs.
And then the final piece to keep in mind: This option can be used multiple times.
Most of this is just an echo of what #Tanaike is saying above, but with more explanation from the documentation. I would recommend reading it in further detail.
I think the largest complain curl has with your command is that each part of the form has the key upload. That's sort of ambiguous. Is there one key for JSON and one key for the picture?
It's also really important to know what the webserver expects from each field in the form. There's a wide difference between a file upload and a text field.
With the help from #Breedly and #Tanaike got the following command to work, someone may find it useful one day:
curl --verbose --request POST --header "Content-Type:multipart/form-data" --form "upload1=<john.json" --form "upload2=#john.jpg" http://127.0.0.1:8088
It is happy to have "Content-Type:multipart/form-data" just once to cover both of them in this case. It really however wants the "<" not a "#" for the json.
The following also works:
curl --verbose --request POST --form "upload1=<john.json;type=application/json" --form "upload2=#john.jpg;type=multipart/form-data" http://127.0.0.1:8088

Why am I getting a 409 Conflict deleting a document through Couchbase Sync Gateway?

Can anyone explain to me why I'm getting this response trying to delete a document?
curl -X DELETE --header 'Accept: application/json' 'http://localhost:4985/mydb/uprofile:testing'
When I run this, I get:
{
"error": "conflict",
"reason": "Document exists"
}
How come?
A conflict usually occurs when two writers are offline and save a different revision of the same document.
As quick fix first get document from syn-gateway and
curl -X GET --header 'Accept: application/json' 'https://localhost:4985/uprofile%3Atesting/mydb?attachments=false&revs=false&show_exp=false'
Then get _rev value from document.. Then try delete as following
curl -X DELETE --header 'Accept: application/json' 'https://localhost:4985/uprofile%3Atesting/mydb?rev={_rev value}'
I am sure this will work,
but for permanent solution where to resolve conflicts dynamically check documentation https://developer.couchbase.com/documentation/mobile/current/guides/sync-gateway/resolving-conflicts/index.html