I have a MySQL (version 5.7.21) DB with 2 columns:
id is a simple auto-increment field INT(11)
and jsondata is a JSON object field.
The JSON I need to import looks like this:
{
"archive": "0iHBvqAX-2017-12-10T23:15:06",
"barchive": "0iHBvqAX-2017-12-10T23:15:06",
"id": "698fd4fd641a211fb569f704ee114f9829505a567ebc85502ea030705038600b",
"name": "0iHBvqAX-2017-12-10T23:15:06",
"start": "2017-12-10T23:15:08.000000",
"time": "2017-12-10T23:15:08.000000"
}
This is my MySQL Import:
LOAD DATA LOCAL INFILE 'myfile.json' INTO TABLE thetable(jsondata);
Error:
ERROR 3140 (22032): Invalid JSON text: "Missing a name for object member." at position 1 in value for column 'thetable.jsondata'.
Now, if we remove all the returns like this:
{ "archive": "0iHBvqAX-2017-12-10T23:15:06", "barchive": "0iHBvqAX-2017-12-10T23:15:06", "id": "698fd4fd641a211fb569f704ee114f9829505a567ebc85502ea030705038600b", "name": "0iHBvqAX-2017-12-10T23:15:06", "start": "2017-12-10T23:15:08.000000", "time": "2017-12-10T23:15:08.000000" }
The import works perfect. Is there a way to tell MySQL to ignore the \n in the JSON file?
Related
I need to create a dictionary in ClickHouse, which will request data via the API. The dictionary is being created, but when I open it I get errors.
I create a dictionary like this:
CREATE DICTIONARY IF NOT EXISTS dbName.AppDict
(
Name String,
AppType Int32
)
PRIMARY KEY Name
SOURCE(HTTP(
url 'http://my_service.ru/api/Services/list?Offset=0'
format 'JSONCompactStringsEachRowWithNamesAndTypes'
credentials(user 'user' password 'xyz')
))
LAYOUT(COMPLEX_KEY_HASHED())
LIFETIME(60);
When I open the dictionary, I get the following error:
ClickHouse exception, code: 1002, host: localhost, port: 8123; Code: 27. DB::Exception: Cannot parse input: expected '[' before: '{"result":{"totalCount":2,"services":[{"version": "1.0.1","platformOS": "windows","name": "my_service1",: While executing JSONCompactEachRowRowInputFormat. (CANNOT_PARSE_INPUT_ASSERTION_FAILED) (version 21.11.2.2 (official build))
by url from http://my_service.ru/api/Services/list?Offset=0 I get the following response:
{
"result": {
"totalCount": 2,
"services": [
{
"version": "1.0.1",
"platformOS": "windows",
"name": "my_service1",
"appType": 0
},
{
"version": "2.0.0",
"platformOS": "windows",
"name": "my_service2",
"appType": 1
}
]
}
}
What needs to be done so that when I open a dictionary I get?
Name
AppType
my_service1
1.0.1
my_service2
2.0.0
I have a copy data activity in Azure data factory which reads in a csv file. This csv file is produced by a 3rd party so I cannot change it. One of the headings has a full stop (or period) in it: 'foo.bar'. When I run the activity I get the error message:
Failure happened on 'Source' side. ErrorCode=JsonInvalidDataFormat,
'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,
Message=Error occurred when deserializing source JSON file 'foo;bar'.
Check if the data is in valid JSON object format.,Source=Microsoft.DataTransfer.ClientLibrary,'
The csv look like this
state,sys_updated_on,foo.bar,sys_id
New,03/06/2021 12:42:18,S Services,xxx
Resolved,03/06/2021 12:35:06,MS Services,yyy
New,03/06/2021 12:46:18,S Services,zzz
The source dataset looks like this:
{
"name": "my_dataset",
"properties": {
"linkedServiceName": {
"referenceName": "my_linked_service",
"type": "LinkedServiceReference"
},
"annotations": [],
"type": "DelimitedText",
"typeProperties": {
"location": {
"type": "AzureBlobStorageLocation",
"fileName": "i.csv",
"folderPath": "Temp/exports/widgets",
"container": "blah"
},
"columnDelimiter": ",",
"escapeChar": "\\",
"firstRowAsHeader": true,
"quoteChar": "\""
},
"schema": []
}
}
You can skip the header like this:
Don't set first row as header:
Set Skip line count: 1 at source settings:
Some other way, you also could use Data Flow with Derived column to create a new schema to replace the column foo.bar:
I am on Windows 10. I recently obtained a large JSON file (200 MB) via webscraping, and I am now trying to import the file to MongoDB using Compass Community via the import data button. However, whenever I try to import the file, I get the following error:
Unexpected token l in JSON at position 0 while parsing near 'l
Here are the first few lines of the JSON file I am trying to import:
{
"bands": [{
"activity": "Split-up",
"bandMembers": ["https://www.metal-archives.com/artists/Jon_Powlowski/760544", "https://www.metal-archives.com/artists/Ruben_Martinez/760545", "https://www.metal-archives.com/artists/Greg_Eickmier/416646", "https://www.metal-archives.com/artists/Nedwob/471955"],
"bandName": "A // Solution",
"country": "United States",
"dateAdded": "2018-08-04",
"genre": "Crust Punk/Thrash Metal",
"label": {
"labelName": "Voltic Records",
"labelUrl": "https://www.metal-archives.com/labels/Voltic_Records/47794"
},
"location": "California",
"lyricalThemes": "N/A",
"releases": [{
"numReviews": 0,
"releaseName": "Butterfly",
"reviewAverage": null,
"type": "EP",
"url": "https://www.metal-archives.com/albums/A_--_Solution/Butterfly/723154",
"year": "1989"
}, {
"numReviews": 0,
"releaseName": "Things to Come",
"reviewAverage": null,
"type": "EP",
"url": "https://www.metal-archives.com/albums/A_--_Solution/Things_to_Come/723155",
"year": "1995"
}
],
"similarArtists": null,
"url": "https://www.metal-archives.com/bands/A_--_Solution/3540442600",
"yearFormed": "N/A",
"yearsActive": "N/A"
}, {
"activity": "Active",
Does anyone have an idea on how I can fix this error?
EDIT: I ran the import again after restarting Compass and got this:
Unexpected token : in JSON at position 0 while parsing near ': null,
Is this error related at all to the other one?
The import data button needs the object to be inlined according to https://docs.mongodb.com/compass/master/import-export/#import-data-into-a-collection.
Apart from that, I had issues with the "Unexpected token : in JSON at position 0", and even tho I could not figure out the cause yet, I tried creating a new .json and copying the content into it, and surprisingly, it worked.
Also, remember to leave a line break at the end of the file.
To convert the json into a 1 line format, you could use the following python script:
import json
import sys
import codecs
import os
def read_file(name):
with open(name, encoding='utf8') as f:
return f.read()
def write_file(name, text):
os.makedirs(os.path.dirname(name), exist_ok=True)
with codecs.open(name, "w", "utf-8-sig") as temp:
temp.writelines(text)
text = read_file(sys.argv[1])
data = json.loads(text)
result = json.dumps(text, ensure_ascii=False) + "\n"
write_file(sys.argv[2], result)
Our Go server outputs logs to stdout in JSON, each line looking something like the following:
{"time": "2017-06-01 14:00:00", "message": "Something happened", "level": "DEBUG"}
Our docker-compose uses the standard json-file logger, which wraps each line in a log field as an escaped string, like so:
{
"log": "{\"time\": \"2017-06-01 14:00:00\", \"message\": \"Something happened\", \"level\": \"DEBUG\"}\"",
"timestamp": "<the time>",
...more fields...
}
But we don't want the log field to be escaped as a string, we want it embedded as JSON at the same level:
{
"log": {
"time": "2017-06-01 14:00:00",
"message": "Something happened",
"level": "DEBUG"
},
"timestamp": "<the time>",
...more fields...
}
Can this be achieved?
Looks like this can't be done. But I can convert the JSON string to actual JSON in Filebeat, which we are using to pass logs to Kibana and Elastalert. To do that I used the decode_json_fields option under processors in filebeat.yml.
I am currently using the Cassandra-Ruby driver to insert data from a JSON file into an existing table in my database.
the JSON file looks like this:
[
{
"id": "123",
"destination": "234",
"type": "equipment",
"support": "type 1",
"test": "test1"
},
{
"id": "234",
"destination": "123",
"type": "equipment",
"support": "type 1",
"test": "test1"
}
]
I am reading in the file like this:
file = File.read('itemType.json')
data_hash = JSON.parse(file) #return an array of hashes
Iterate through the array and get each hash
and insert each hash onto the table
data_hash.each do |has|
#check the type of each object
#puts has.class #return hash
insert_statement = session.prepare('INSERT INTO keyspace.table JSON ?')
session.execute(insert_statement, [has]) #error occurs here
end
After running this code, I get this error message
in `assert_instance_of': options must be a Hash
I checked that each object being inserted in the table is a hash, so I'm not sure why I'm getting this issue.
You are saying that you are inserting a JSON but you are not, you are trying to insert an object. See this example from the documentation:
INSERT INTO cycling.cyclist_category JSON '{
"category" : "Sprint",
"points" : 700,
"id" : "829aa84a-4bba-411f-a4fb-38167a987cda"
}';
You have to give it a json format if you do it like that.
using .to_json add \ escape character. This gave me error
INSERT INTO organization_metadata JSON '{\"id\":9150,\"destroyed\":false,\"name\":\"ABC\",\"timestamp\":1510541801000000000}';
and the following worked.
INSERT INTO organization_metadata JSON '{"id":9150,"destroyed":false,"name":"ABC","timestamp":1510541801000000000}';