What's wrong with creating a dictionary in ClickHouse? - json

I need to create a dictionary in ClickHouse, which will request data via the API. The dictionary is being created, but when I open it I get errors.
I create a dictionary like this:
CREATE DICTIONARY IF NOT EXISTS dbName.AppDict
(
Name String,
AppType Int32
)
PRIMARY KEY Name
SOURCE(HTTP(
url 'http://my_service.ru/api/Services/list?Offset=0'
format 'JSONCompactStringsEachRowWithNamesAndTypes'
credentials(user 'user' password 'xyz')
))
LAYOUT(COMPLEX_KEY_HASHED())
LIFETIME(60);
When I open the dictionary, I get the following error:
ClickHouse exception, code: 1002, host: localhost, port: 8123; Code: 27. DB::Exception: Cannot parse input: expected '[' before: '{"result":{"totalCount":2,"services":[{"version": "1.0.1","platformOS": "windows","name": "my_service1",: While executing JSONCompactEachRowRowInputFormat. (CANNOT_PARSE_INPUT_ASSERTION_FAILED) (version 21.11.2.2 (official build))
by url from http://my_service.ru/api/Services/list?Offset=0 I get the following response:
{
"result": {
"totalCount": 2,
"services": [
{
"version": "1.0.1",
"platformOS": "windows",
"name": "my_service1",
"appType": 0
},
{
"version": "2.0.0",
"platformOS": "windows",
"name": "my_service2",
"appType": 1
}
]
}
}
What needs to be done so that when I open a dictionary I get?
Name
AppType
my_service1
1.0.1
my_service2
2.0.0

Related

Can Filebeat parse JSON fields instead of the whole JSON object into kibana?

I am able to get a single JSON object in Kibana:
By having this in the filebeat.yml file:
output.elasticsearch:
hosts: ["localhost:9200"]
How can I get the individual elements in the JSON string. So say if I wanted to compare all the "pseudorange" fields of all my JSON objects. How would I:
Select "pseudorange" field from all my JSON messages to compare them.
Compare them visually in kibana. At the moment I can't even find the message let alone the individual fields in the visualisation tab...
I have heard of people using logstash to parse the string somehow but is there no way of doing this simply with filebeat? If there isn't then what do I do with logstash to help filter the individual fields in the json instead of have my message just one big json string that I cannot interact with?
I get the following output from output.console, note I am putting some information in <> to hide it:
"#timestamp": "2021-03-23T09:37:21.941Z",
"#metadata": {
"beat": "filebeat",
"type": "doc",
"version": "6.8.14",
"truncated": false
},
"message": "{\n\t\"Signal_data\" : \n\t{\n\t\t\"antenna type:\" : \"GPS\",\n\t\t\"frequency type:\" : \"GPS\",\n\t\t\"position x:\" : 0.0,\n\t\t\"position y:\" : 0.0,\n\t\t\"position z:\" : 0.0,\n\t\t\"pseudorange:\" : 20280317.359730639,\n\t\t\"pseudorange_error:\" : 0.0,\n\t\t\"pseudorange_rate:\" : -152.02620448094211,\n\t\t\"svid\" : 18\n\t}\n}\u0000",
"source": <ip address>,
"log": {
"source": {
"address": <ip address>
}
},
"input": {
"type": "udp"
},
"prospector": {
"type": "udp"
},
"beat": {
"name": <ip address>,
"hostname": "ip-<ip address>",
"version": "6.8.14"
},
"host": {
"name": "ip-<ip address>",
"os": {
<ubuntu info>
},
"id": <id>,
"containerized": false,
"architecture": "x86_64"
},
"meta": {
"cloud": {
<cloud info>
}
}
}
In Filebeat, you can leverage the decode_json_fields processor in order to decode a JSON string and add the decoded fields into the root obejct:
processors:
- decode_json_fields:
fields: ["message"]
process_array: false
max_depth: 2
target: ""
overwrite_keys: true
add_error_key: false
Credit to Val for this. His answer worked however as he suggested my JSON string had a \000 at the end which stops it being JSON and prevented the decode_json_fields processor from working as it should...
Upgrading to version 7.12 of Filebeat (also ensure version 7.12 of Elasticsearch and Kibana because mismatched versions between them can cause issues) allows us to use the script processor: https://www.elastic.co/guide/en/beats/filebeat/current/processor-script.html.
Credit to Val here again, this script removed the null terminator:
- script:
lang: javascript
id: trim
source: >
function process(event) {
event.Put("message", event.Get("message").trim());
}
After the null terminator was removed the decode_json_fields processor did its job as Val suggested and I was able to extract the individual elements of the JSON field which allowed Kibana visualisation to look at the elements I wanted!

Getting Error: Failed to parse Dockerrun JSON file: json: invalid use of ,string struct tag, trying to unmarshal unquoted value into int

I used JSON linter to check the validity of this json and it says ok. But when I deploy this to AWS Elastic Beanstalk, it errors out with message:
Error: Failed to parse Dockerrun JSON file: json: invalid use of
,string struct tag, trying to unmarshal unquoted value into int
Below is Dockerrun.aws.json:
{
"AWSEBDockerrunVersion": 2,
"containerDefinitions": [
{
"name": "delivery-slot-notifier",
"image": "akshaylokur/delivery-slot-notifier:latest",
"essential": true,
"portMappings": [
{
"hostPort": 8080,
"containerPort": 8080
}
],
"memory": 128
}
]
}
Any clues?
Thanks
You should be seeing that error in your logs because your Dockerrun.aws.json has a bunch of options are not supported by Beanstalk (looks like you are using multi container docker-run for single container?). Also, for single docker env, Version has to be 1, whereas 2 is used for multi container env. Here's a Dockerrun.aws.json that will work:
{
"AWSEBDockerrunVersion": "1",
"Image": {
"Name": "akshaylokur/delivery-slot-notifier:latest",
"Update": "true"
},
"Ports": [
{
"hostPort": 8080,
"containerPort": 8080
}
]
}
Here are all the supported options for Dockerrun.aws.json

JSON int key issue data e2e (invalid character '1' looking for beginning of object key string)

My app uses aerospike to store Map in one of the bins,
I am use endly for e2e testing, which uses JSON for data representation:
How do to populate datastore with with JSON where key needs to be an int ?
Since json does not allowed int key I am getting the following error: invalid character '1' looking for beginning of object key string
Here is my data workflow (invoked by regression workflow)
#data.yaml
defaults:
datastore: db1
pipeline:
register:
action: dsunit:register
config:
driverName: aerospike
descriptor: "tcp([host]:3000)/[namespace]"
parameters:
dbname: db1
namespace: test
host: 127.0.0.1
port: 3000
dateFormat: yyyy-MM-dd hh:mm:ss
prepare:
data:
action: nop
init:
- key = data.db.mydaaset
- mydaaset = $AsTableRecords($key)
setup:
action: dsunit:prepare
URL: regression/db1/data/
data: $mydaaset
Here is my use case level data:
#mydaaset.json
[
{
"Table": "myDataset",
"Value": [{
"id": "$id",
"created": "$timestamp.yesterday",
"fc":{
1191: "$timestamp.yesterday",
1192: "$timestamp.now",
}
}],
"AutoGenerate": {
"id": "uuid.next"
},
"Key": "${tagId}_myDataset"
}
]
In your example #mydaaset.json file is invalid JSON, thus you getting
'invalid character '1' looking for beginning of object key string'
parsing error
In order to pre-seed your use case test data in aerospike with map[int]int bin you can use AsInt UDF
#mydaaset.json
[
{
"Table": "myDataset",
"Value": [{
"id": "$id",
"created": "$timestamp.yesterday",
"fc":{
"$AsInt(1191)": "$timestamp.yesterday",
"$AsInt(1192)": "$timestamp.now",
}
}],
"AutoGenerate": {
"id": "uuid.next"
},
"Key": "${tagId}_myDataset"
}
]

MySQL LOAD JSON with Line Breaks produces error

I have a MySQL (version 5.7.21) DB with 2 columns:
id is a simple auto-increment field INT(11)
and jsondata is a JSON object field.
The JSON I need to import looks like this:
{
"archive": "0iHBvqAX-2017-12-10T23:15:06",
"barchive": "0iHBvqAX-2017-12-10T23:15:06",
"id": "698fd4fd641a211fb569f704ee114f9829505a567ebc85502ea030705038600b",
"name": "0iHBvqAX-2017-12-10T23:15:06",
"start": "2017-12-10T23:15:08.000000",
"time": "2017-12-10T23:15:08.000000"
}
This is my MySQL Import:
LOAD DATA LOCAL INFILE 'myfile.json' INTO TABLE thetable(jsondata);
Error:
ERROR 3140 (22032): Invalid JSON text: "Missing a name for object member." at position 1 in value for column 'thetable.jsondata'.
Now, if we remove all the returns like this:
{ "archive": "0iHBvqAX-2017-12-10T23:15:06", "barchive": "0iHBvqAX-2017-12-10T23:15:06", "id": "698fd4fd641a211fb569f704ee114f9829505a567ebc85502ea030705038600b", "name": "0iHBvqAX-2017-12-10T23:15:06", "start": "2017-12-10T23:15:08.000000", "time": "2017-12-10T23:15:08.000000" }
The import works perfect. Is there a way to tell MySQL to ignore the \n in the JSON file?

Inserting json file into Cassandra table

I am currently using the Cassandra-Ruby driver to insert data from a JSON file into an existing table in my database.
the JSON file looks like this:
[
{
"id": "123",
"destination": "234",
"type": "equipment",
"support": "type 1",
"test": "test1"
},
{
"id": "234",
"destination": "123",
"type": "equipment",
"support": "type 1",
"test": "test1"
}
]
I am reading in the file like this:
file = File.read('itemType.json')
data_hash = JSON.parse(file) #return an array of hashes
Iterate through the array and get each hash
and insert each hash onto the table
data_hash.each do |has|
#check the type of each object
#puts has.class #return hash
insert_statement = session.prepare('INSERT INTO keyspace.table JSON ?')
session.execute(insert_statement, [has]) #error occurs here
end
After running this code, I get this error message
in `assert_instance_of': options must be a Hash
I checked that each object being inserted in the table is a hash, so I'm not sure why I'm getting this issue.
You are saying that you are inserting a JSON but you are not, you are trying to insert an object. See this example from the documentation:
INSERT INTO cycling.cyclist_category JSON '{
"category" : "Sprint",
"points" : 700,
"id" : "829aa84a-4bba-411f-a4fb-38167a987cda"
}';
You have to give it a json format if you do it like that.
using .to_json add \ escape character. This gave me error
INSERT INTO organization_metadata JSON '{\"id\":9150,\"destroyed\":false,\"name\":\"ABC\",\"timestamp\":1510541801000000000}';
and the following worked.
INSERT INTO organization_metadata JSON '{"id":9150,"destroyed":false,"name":"ABC","timestamp":1510541801000000000}';