Use cURL to add a JSON web page's data in Solr - json

I see from the UpdateJSON page how to use a command prompt to index a standalone file stored locally. Using this example I was able to successfully make a .json file accessible through Solr:
curl 'http://localhost:8983/solr/update/json?commit=true' --data-binary #books.json -H 'Content-type:application/json'
What I'm not able to find is the proper syntax to do the same for a webpage containing JSON data. I've tried with the #:
curl 'http://localhost:8983/solr/update/json?commit=true' --data-binary #[URL] -H 'Content-type:application/json'
and without:
curl 'http://localhost:8983/solr/update/json?commit=true' --data-binary [URL] -H 'Content-type:application/json'
Both ways lead to errors. How do I configure a command to prompt Solr to index the contents at [URL]?

According to documentation, (https://wiki.apache.org/solr/ContentStream) you should first ensure remote streaming is enabled (solrconfig, search for enableRemoteStreaming)
Then the command should be of the kind:
curl 'http://localhost:8983/solr/update/json?commit=true&stream.url=YOURURL' -H 'Content-type:application/json'

Related

Generate json file using curl xml

I'm trying to generate json file using curl and also assign specific path where in the json file will store once generated, but I tried some commands but no json output.
May I know what I need to add or change with my command?
curl -v -H "Accept: application/json" --user "admin:Test1234" https://test.com/adventure/
I test a simple curl for a json public api and send de response to a file and the result is a JSON output.
curl -H "Accept: application/json" https://catfact.ninja/fact >> cat.json
You can try it using https://reqbin.com/req/javascript/c-vdhoummp/curl-get-json-example
Or simply using postman and see the code snippet option to check cUrl code snippet.
https://imgur.com/a/LXqN8YH
I'm able to generate JSON file. I add -k on my command since my URL is HTTPS.

How to GET data of a key in json using curl?

Sorry Pretty noob to json.
Basically I have a simple server where I can upload data in there.
E.g:
curl -vX PUT "http://IP:port/ABC" -H "Content-Type: application/json" -d #"Once Upon a time."
After when I do:
curl -vX GET "http://IP:port/ABC" -H "Content-Type: application/json"
I get:
{"reverse_shell":
{"aliases":{},"mappings":{},"settings":
{"index":{"creation_date":"1561863982371","number_of_shards":"5","number_of_replicas":"1","uuid":"IAWE83rYQqmtKW-9svkBVg","version":{"created":"6040299"},"provided_name":"ABC"}
}
}
}
As you can see there is no where mentioning Once Upon a time, so is there I am missing? or how do I get that data from json using curl?
I am in kali linux env.
It looks like you are trying to post a string as a file.
When you specify a "#" with -d this tells curl to send the data from a file called "Once Upon a time."
If you are trying to put a file then you should do:
my_text_file.txt
Once Upon a time.
curl -vX PUT "http://IP:port/ABC" -H "Content-Type: application/json" -d #./my_text_file.txt https://server/api/path

curl xput windows Couldn't read data from file

I am using cURL in windows using command propmpt. When I am executing this command:
curl -XPUT "localhost:8983/solr/techproducts/schema/feature-store" --data-binary "#/path/myFeatures.json" -H "Content-type:application/json"
I am getting following error:
Warning: Couldn't read data from file "/path/myFeatures.json", this makes an
Warning: empty POST.
I have updated the file permissions and file is also present at the specific path.
What can be the possible solutions?
If you really have a file named myFeatures.json into a folder named path in the current folder where you're trying to submit curl, just remove the leading slash from the path.
curl -XPUT "localhost:8983/solr/techproducts/schema/feature-store" --data-binary "#path/myFeatures.json" -H "Content-type:application/json"
On the other hand, try to specify the absolute path to your myFeatures.json.
curl -XPUT "localhost:8983/solr/techproducts/schema/feature-store" --data-binary "#C:\your\complete\path\myFeatures.json" -H "Content-type:application/json"
I had the same problem, and in my case, it turned out that this was caused by using ~ in my path.
The simplest solution is to change the extension of a file from ".json" to ".txt".

Watson API - Retrieve and Rank - Error uploading JSON

I'm following the tutorial at Retrieve and Rank - Get Started, and I'm at the following step:
Issue the following command to upload the cranfield_data.json data to the example_collection collection. Replace {username}, {password}, {solr_cluster_id}, and {/path_to_file} with your information:
$ curl -X POST -H "Content-Type: application/json" -u "{username}":"{password}" "https://gateway.watsonplatform.net/retrieve-and-rank/api/v1/solr_clusters/{solr_cluster_id}/solr/example_collection/update" --data-binary #{/path_to_file}/cranfield_data.json
I'm lobbing the request with the correct username and password, and correct cluster_id and path to the json, but I get the following error:
$ curl -X POST -H "Content-Type: application/json" -u "username":"password" "https://gateway.watsonplatform.net/retrieve-and-rank/api/v1/solr_clusters/cluster_id/solr/example_collection/update" --data-binary #forum_data/parsed_answers.json
Error: WRRCSH006: Error forwarding request [/solr/example_collection/update] for Solr cluster [sc5b47c5e3_bab3_4aff_a818_f0d786d6dece].
Turns out there were characters in the JSON causing it be malformed.
Just verify the JSON "parsed_answers.json", to check for all the punctuations i.e. ";","," to be correctly placed and as per the defined schema, and try to re-upload

Github returns "Problems parsing JSON" when trying to create repo using curl

I'm attempting to create a repo on github using curl. No matter how I attempt to format my JSON it always returns "Problems parsing JSON". Is there something I'm missing? Here is my statement:
curl --data '{"name":"newRepoName"}' -XPOST -u username:password http://giturl
Strange since it works for me. Are you using the correct GitHub API endpoint (/user/repos)?
curl --data '{"name":"testrepo"}' -X POST -u username https://api.github.com/user/repos
Try including your token or password like this
curl --data '{"name":"testrepo"}' \
-X POST -u username:$TOKEN \
https://api.github.com/user/repos