Load JSON data into BigQuery table - json

Question

I'm trying to load simple JSON data into BigQuery table the following way:
$ bq load \
--apilog \
--source_format=NEWLINE_DELIMITED_JSON \
my_dataset.my_table \
./input.json ./schema.json
but get the following error message:
Upload complete.
Waiting on bqjob_xxxx_xxx ... (3s) Current status: DONE
BigQuery error in load operation: Error processing job 'my_project_id:bqjob_xxxx_xxx': CSV table encountered too many errors, giving up. Rows: 1; errors: 1.
Failure details:
- file-00000000: Error detected while parsing row starting at
position: 0. Error: Data between close double quote (") and field
separator.
It complains about some CSV error, but I'm trying to load JSON (--source_format=NEWLINE_DELIMITED_JSON)
My input.json contains this data:
{"domain":"stackoverflow.com","key":"hello","value":"world"}
My schema.json is the following:
[
{
"name": "domain",
"type": "string",
"mode": "nullable"
},
{
"name": "key",
"type": "string",
"mode": "nullable"
},
{
"name": "value",
"type": "string",
"mode": "nullable"
}
]
bq version 2.0.25:
$ gcloud version | grep ^bq
bq 2.0.25

Answer 1

The problem here is that the flag apilog expects a string as input. This command should work for you:
bq load \
--apilog '' \
--source_format=NEWLINE_DELIMITED_JSON \
my_dataset.my_table \
./input.json ./schema.json
Empty string sends output to stdout. If you want to save the log to a local file then you can just send a non-empty string, such as --apilog 'localfile_name'.

Answer 2

BQ command says:
USAGE: bq.py [--global_flags] <command> [--command_flags] [args]
As you see there are global_flags and command_flags
For the global_flags that have values you need to use the equal sign:
--flag=value
The command_flags are either boolean:
--[no]replace
Or they take arguments that must follow the flag:
--source_format NEWLINE_DELIMITED_JSON
Also do not mix global and command flags: apilog is a global flag.
I would rewrite your command into:
$ bq --apilog load \
--source_format NEWLINE_DELIMITED_JSON \
my_dataset.my_table \
./input.json ./schema.json