Using curl, I have a GET function that is working great to retrieve the data (from Trello's API), but it comes out as a JSON that is all on one line. The software I'm using to read and interpret the retrieved data can only read lines of 32,000 characters.
Does anyone know a way to get it to include carriage returns in the way it does when you open it in Notepad++ and choose JSON Viewer (or any other manner).
The curl code is:
{LocationOfCurlOnMyServer} -H "Content-Type: text; charset=utf-8"
-o "{OutputFileName}"
--request GET --url "{TrelloURL}"
You can use jq JSON parser along with curl :
curl -L "http://citibikenyc.com/stations/json" | jq '.' > output.json
Related
Currently I am able to PUT a single json file to a document in Cloudant using this : curl -X PUT 'https://username.cloudant.com/dummydb/doc3' -H "Content-Type: application/json" -d #numbers.json.I have many JSON files to be uploaded as different documents in the same DB.How can it be done?
So you definitely want to use Cloudant's _bulk_docs API endpoint in this scenario. It's more efficient (and cost-effective) if you're doing a bunch of writes. You basically POST an array that contains all your JSON docs. Here's the documentation on it: https://docs.cloudant.com/document.html#bulk-operations
Going one step further, so long as you've structured your JSON file properly, you can just upload the file to _bulk_docs. In cURL, that would look something like this: curl -X POST -d #file.json <domain>/db/_bulk_docs ... (plus the content type and all that other verbose stuff).
One step up from that would be using the ccurl (CouchDB/Cloudant cURL) tool that wraps your cURL statements to Cloudant and makes them less verbose. See https://developer.ibm.com/clouddataservices/2015/10/19/command-line-tools-for-cloudant-and-couchdb/ from https://stackoverflow.com/users/4264864/glynn-bird for more.
Happy Couching!
You can create a for loop and create documents from each JSON file.
For example, in the command below I have 4 JSON files in my directory and I create 4 documents in my people database:
for file in *.json
> do
> curl -d #$file https://username:password#myinstance.cloudant.com/people/ -H "Content-Type:application/json"
> done
{"ok":true,"id":"763a28122dad6c96572e585d56c28ebd","rev":"1-08814eea6977b2e5f2afb9960d50862d"}
{"ok":true,"id":"763a28122dad6c96572e585d56c292da","rev":"1-5965ef49d3a7650c5d0013981c90c129"}
{"ok":true,"id":"763a28122dad6c96572e585d56c2b49c","rev":"1-fcb732999a4d99ab9dc5462593068bed"}
{"ok":true,"id":"e944282beaedf14418fb111b0ac1f537","rev":"1-b20bcc6cddcc8007ef1cfb8867c2de81"}
I'm working on a cli for the Pushbullet HTTP API using Bash scripting. Sending pushes (notes and links), as well as creating, deleting, and modifying contacts & devices are all straight forward using curl and Bash. However, sending SMS and files are a bit more complex, as both require sending more complex JSON-formatted requests to the server (multiple JSON-formatted requests, in the case of pushing files).
I've tried sending many variations on the following (both with and without escape characters), but the server keeps replying about JSON-formatting errors. The following code is based off of the example given in the Pushbullet HTTP API documentation.
curl -su $auth_id: -X POST https://api.pushbullet.com/v2/ephemerals --header "Content-Type: application/json"
--data-binary '{ "\"type"\": "\"push"\", "\"push"\": { \
"\"type"\": "\"messaging_extension_reply"\", \
"\"package_name"\": "\"com.pushbullet.android"\", \
"\"source_user_iden"\": "\"$source_idens"\", \
"\"target_device_iden"\": "\"$target_idens"\", \
"\"conversation_iden"\": "\"$sms_device"\", \
"\"message"\": "\"Hello"\" \
} }'
Using bash -x, I can see that this is (supposedly) what is being sent to the server:
--data-binary '{"type": "push", "push": {
"type": "messaging_extension_reply",
"package_name": "com.pushbullet.android",
"source_user_iden": "<source_idens>",
"target_device_iden": "<device_idens>",
"conversation_iden": "<sms_phone_number>",
"message": "Hello" } }'
In all cases, the server returns:
{"error":{"type":"invalid_request","message":"Failed to parse JSON body.","cat":"(=^‥^=)"}}
What is the appropriate formatting of a JSON request using curl to send an SMS via the Pushbullet API? Am I overlooking something obvious? I'm trying to accomplish this using only curl and Bash, I see no reason why it's not possible (maybe not the fastest or most elegant way, but certainly possible).
I found the solution to my issue so I thought I'd share it. It was actually very simple:
Because the curl command includes a JSON-formatted response with single quotes, variable expansion was not occurring. This is a limitation (or perhaps a feature) of Bash. So, even though the server responded with { } indicating no errors in the request, the requests were actually being sent without the proper values for parameters, such asuser_iden,source_user_iden, etc.
Solution:
Enclose all variable expansions inside the JSON-formatted request in a double-quote and single-quote, like so:
"'"$user_idens"'"
First I'd like to apologize for how bad the API is, especially file upload and sending SMS. I was thinking of adding multipart or base64 file uploads to /v2/pushes. I think the first one might help you with curl, not sure about the base64 one. multipart is a huge pain though, so I'd prefer to make it better than the current setup if possible, rather than about equally as bad. Suggestions are welcome.
I tried your command line and it seemed to work, so I'm not sure what is going wrong. Here's the command line I did. Perhaps your quote escaping or newlines are causing the JSON error?
curl -u <access_token> -X POST https://api.pushbullet.com/v2/ephemerals --header "Content-Type: application/json" --data-binary '{"type": "push", "push": {"type": "messaging_extension_reply","package_name": "com.pushbullet.android","source_user_iden": "iden","target_device_iden": "device_idens", "conversation_iden": "sms_phone_number","message": "Hello" } }'
I am new to elasticsearch, running elasticsearch from chrome:extension- Postman.
I want to enter bulk data into it, from JSON using Bulk API.
I have seen the command :
curl -s -XPOST 'http://jfblouvmlxecs01:9200/_bulk' --data-binary #bulk.json
While using Postman, I do not know where to store the file #bulk.json,
Currently I have stored it at C:\elasticsearch-1.5.2\bin\bulk.JSON
The command I am using is http://localhost:9200/_bulk --data-binary #bulk.JSON
This is throwing following error:
"error": "InvalidIndexNameException[[_bulk --data-binary #bulk.JSON] Invalid index name [_bulk --data-binary #bulk.JSON], must not contain the following characters [\, /, *, ?, \", <, >, |, , ,]]",
"status": 400 }
Can someone please suggest, where to store the JSON file.
Or am I doing something wrong here.
You can store your file anywhere and then pass the path like this:
curl -s -XPOST 'http://jfblouvmlxecs01:9200/_bulk' --data-binary #"/blah/blah/bulk.json"
If you want to do it with postman there is an answer here that explains how
But if you want to use this for large data, I wouldn't recommend using postman or sense. Use curl directly or logstash
See also this: https://github.com/taskrabbit/elasticsearch-dump
I'm getting an invalid JSON error when trying to add new JSON data to a couchapp.
I have JSON data that I converted from a CSV, and have been trying to add it with _bulk_docs to a couchapp. I solved the most obvious JSON errors with a JSON validator: http://jsonlint.com/
But one of the files is still giving an error, even thought the validator says the JSON is valid.
curl -d #songs_raw2.json -H "Content-Type: application/json" -X POST http://username:password#127.0.0.1:5984/songs/_bulk_docs
{"error":"bad_request","reason":"invalid_json"}
The file is here:
https://github.com/anatighe/sacred-harp/blob/master/songs_raw2.json
Looks like your json is not valid. This doesn't work for me:
$ curl https://raw.githubusercontent.com/anatighe/sacred-harp/master/songs_raw2.json | python -m json.tool | view -
Might have to do with special characters / encoding?
It looks like all of the single quote characters are messing it up. In every place there should be a ', there appears to be a � for me. I'm on a mac, I don't know if that matters.
When I replaced the weird characters with regular single quotes the curl request worked fine.
I'm wondering if there is a way to send headers from YQL (or the YQL console) like there is in cURL.
I would like to return JSON by specifying the header Accept: application/json.
I am able to return JSON in with cURL and the command line like this:
curl -H 'Accept: application/json' http://www.opendatabc.ca/data?=births
but I can't figure out how to set the header when sending YQL.
You can do this with YQL Open Data Tables.
Here is a simple demonstration.
You can find the gist for the example Open Data Table XML file here: https://gist.github.com/2042904 (Check out the documentation here.)
You will notice in my example XML that I'm using y.xmlToJson on the response object received from the get() request. This is because YQL converts JSON taken from web services into E4X. More about that in a question of mine.
You should use format parameter when querying YQL over API, instead of headers. Either by format=json or format=xml.
JSON example:
curl -G --data-urlencode 'q=SELECT * FROM html WHERE url = "example.com"' http://query.yahooapis.com/v1/public/yql?format=json
XML example:
curl -G --data-urlencode 'q=SELECT * FROM html WHERE url = "example.com"' http://query.yahooapis.com/v1/public/yql?format=xml
use "jq", if have not installed it yet, run this command first
sudo apt-get install jq
then you can curl your url like this
curl -H 'Accept: application/json' http://www.opendatabc.ca/data?=births | jq '.'