Influx, how do I upload a JSON with Rest Api?
When i read data from Influx using say Rest+query, it comes in JSON format. Now to upload they are saying json is deprecated and we have to do it in binary format, really?
Read using this gives me data in JSON format
curl -G 'http://localhost:8086/query' --data-urlencode “db=my_db" --data-urlencode "q=select * from \”server1.rte.set\" limit 1">test.txt
Write has to be this binary format
curl -i -XPOST 'http://localhost:8086/write?db=my_db' --data-binary 'cpu_load_short,host=server01,region=us-west value=0.64 1434055562000000000'
Why would anybody do this? Keep both json or keep both binary.
The current version of InfluxDB doesn't support the JSON write path or a binary protocol. The predominant reason it was deprecated was that decoding JSON was the largest performance bottleneck in the system.
For a more information see the github issue comments 107043910 and 106968181.
Related
I want to know is there a possibility to post the data in json format to a dashboard from linux server.
My JSON data is in the below format.
{
"data":"{\"actiontodo\":\"Action to do for test nr:
1\",\"critical\":\"LOW\",\"fixstatus\":\"NOTCONCERN\",\"host\":\"MTR_SOME_HOST\",
\"message\":\"$message\",\"mgsApplication\":\"MTR\",\"sMxtype\":\"PROD\",
\"scriptname\":\"$scriptname\"}",
"msg":"NotificationReceiveDTO without dict to send at: 2020-06-07T11:14:09.794 Created at: 2020-
06-07T11:14:09.797",
"msgType":"DATA"
}
From the man page for wget it is possible to post data to website, using either the --post-data=string or --post-file=file argument.
I have a Kinesis stream in AWS and can send data to it (JSON) using kinesis command and can get it back from a stream with:
SHARD_ITERATOR=$(aws kinesis get-shard-iterator --shard-id shardId-000000000000 --shard-iterator-type TRIM_HORIZON --stream-name mystream --query 'ShardIterator' --profile myprofile)
aws kinesis get-records --shard-iterator $SHARD_ITERATOR --profile myprofile
Output of this looks like something like:
HsKCQkidmlkZW9Tb3VyY2UiOiBbCgkJCXsKCQkJCSJicmFuZGluZyI6IHt9LAoJCQkJInByb21vUG9vbCI6IFtdLAoJCQkJImlkIjogbnVsbAoJCQl9CgkJXSwKCQkiaW1hZ2VTb3VyY2UiOiB7fSwKCQkibWV0YWRhdGFBcHByb3ZlZCI6IHRydWUsCgkJImR1ZURhdGUiOiAxNTgzMzEyNTA0ODAzLAoJCSJwcm9maWxlIjogewoJCQkiY29tcG9uZW50Q291bnQiOiAwLAoJCQkibmFtZSI6ICJTUUVfQVRfUFJPRklMRSIsCgkJCSJpZCI6ICJTUUVfQVRfUFJPRklMRV9JRCIsCgkJCSJwYWNrYWdlQ291bnQiOiAwLAoJCQkicGFja2FnZXMiOiBbCgkJCQl7CgkJCQkJIm5hbWUiOiAiUEVBQ09DSy1MVEEiLAoJCQkJCSJpZCI6ICJmZDk5NTRmZC03NDYwLTRjZjItOTU5Ni05YzBhMjcxNTViODgiCgkJCQl9CgkJCV0KCQl9LAoJCSJ3b3JrT3JkZXJJZCI6ICJTUUVfQVRfSk9CX1NVQk1JU1
How do I get actual JSON message in raw format (to look as JSON) - same way as it was in original when I sent it?
Thanks
As per the docs, you need to use a Base64 decoding tool or use KCL library to get the data in the format it was sent:
The first thing you'll likely notice about your record in this part of the tutorial is that the data appears to be garbage –; it's not the clear text testdata we sent. This is due to the way put-record uses Base64 encoding to allow you to send binary data. However, the Kinesis Data Streams support in the AWS CLI does not provide Base64 decoding because Base64 decoding to raw binary content printed to stdout can lead to undesired behavior and potential security issues on certain platforms and terminals. If you use a Base64 decoder (for example, https://www.base64decode.org/) to manually decode dGVzdGRhdGE= you will see that it is, in fact, testdata. This is sufficient for the sake of this tutorial because, in practice, the AWS CLI is rarely used to consume data, but more often to monitor the state of the stream and obtain information, as shown previously (describe-stream and list-streams). Future tutorials will show you how to build production-quality consumer applications using the Kinesis Client Library (KCL), where Base64 is taken care of for you. For more information about the KCL, see Developing KCL 1.x Consumers.
on unix, you can use the base64 --decode command to decode the base64 encoded kinesis record data.
for example, to decode the data of the first record:
# define the name of the stream you want to read
KINESIS_STREAM_NAME='__your_stream_name_goes_here__';
# define the shard iterator to use
SHARD_ITERATOR=$(aws kinesis get-shard-iterator --shard-id shardId-000000000000 --shard-iterator-type TRIM_HORIZON --stream-name $KINESIS_STREAM_NAME --query 'ShardIterator');
# read the records, use `jq` to grab the data of the first record, and base64 decode it
aws kinesis get-records --shard-iterator $SHARD_ITERATOR | jq -r '.Records[0].Data' | base64 --decode
I am using an API to get data from NetFlow Analyzer. I get a JSON file formatted like this;
{"startTime":"2017-12-29 11:58","resultVector":[{"port":"*","app":"Unknown_App","dscpCode":"0","traffic":"4.77 MB","dscp":"Default","src":"20.xx.xx.2","dst":"10.xx.xx.1","dstport":"*","prot":"Unknown"}],"Type":"DestinationIN","devDetails":{"deviceID":"5000006","Total":"4.77 MB"},"TimeZone":"America/Chicago","endTime":"2018-01-05 11:58"}
I have been trying to use json2csv (https://github.com/jehiah/json2csv) found at github, and did have success using for a different API & JSON output format. When I run
json2csv -k port,app,dscpCode,traffic,dscp,src,dst,dstport,prot -i filein.json -o fileout2.csv
I get a csv file with nothing but ",,,,,". What I am trying to get are the traffic, source IP, and destination IP.
Running;
json2csv -k startTime,resultVector -i filein.json -o fileout2.csv
Gives me this output which while close, it not csv really
2017-12-29 11:58,[map[dscpCode:0 src:20.xx.xx.2 dst:10.xx.xx.1 prot:Unknown port:* app:Unknown_App dstport:* traffic:4.77 MB dscp:Default]]
Checked few online sites that report this is a valid RFC 4627 JSON. Anyone else familiar with json2csv, or if nothing else another cli tool for Linux that I can use in a script to convert?
This is a good job for jq processor:
jq -r '.resultVector[] | [.traffic, .src, .dst] | #csv' filein.json > fileout2.csv
The final fileout2.csv contents:
"4.77 MB","20.xx.xx.2","10.xx.xx.1"
Typically I prefer cli tools too
If you want to quickly format some json into csv you might also checkout this:
https://json-csv.com/
Provides file upload as well as copy pasting for quick results.
I'm using Jenkins and curl to post a file in a form that is analyzed and returns an ID that is to be used to download the contents.
My problem is how to parse the json response in order to use as my next curl get request.
This is the post command:
curl --form file=#"%WORKSPACE%\results.zip" https://host.com
This returns a json response like: {"request_id":"XXXXXX","message:null","error":false}
I want to pass the pair "request_id=XXXX" in my next curl request like:
curl https://host.com/downloadreport?request_id=XXXXX
Is there a way to do this? Saving the json response to a file and parsing it somehow? or maybe chaining both requests and manipulating the json response?
Thanks in advance
Found a partial solution:
for /f "tokens=1,2,3,4,5,6 delims=:," %%a in ("%requestId%") do set request=%%a&set id=%%b&set msg=%%c&set contents=%%d&set error=%%e&set code=%%f
It isn't very robust as it requires to know the response setup and sometimes response changes order and can't really know what is in each variable.
I have a large dataset in JSON format that I would like to upload to IrisCouchDB.
I found the following instructions: http://kxepal.iriscouch.com/docs/1.3/api/database/common.html
But I am a newbie and it also seems to be for a single JSON document. Im afraid I will just create one huge entry and not multiple documents entries which is what I want.
I have NodeJS but I don't have Cradle. Will I need it in order to perform this function? Any help is greatly appreciated!
Oh, no! That's bad idea to reference on docs at my Iris host - that's was preview dev build. Please follow the official docs: http://docs.couchdb.org/en/latest/api/index.html
To update multiple document's you need to use /db/_bulk_docs resource:
curl -X POST http://localhost:5984/db/_bulk_docs \
-H "Content-Type:application/json" \
-d '{"docs":[{"name":"Yoda"}, {"name":"Han Solo"}, {"name":"Leia"}]}'
So you see the format of data you should send to CouchDB? Now it's all depending from you JSON file format. If whole file is an array of objects: just wrap his content into {"docs":..} object. If not: you have to write some small library to convert this file data into the required format.