I have exported JSON files from aws dynamodb in this format:
[
{
"__typename": "Article",
<snip>
}
<snip>
]
This results in "Invalid JSON" error:
aws dynamodb batch-write-item --request-items file://Article.json
Error parsing parameter '--request-items': Invalid JSON:
The correct format is (https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/SampleData.LoadData.html)
How to export the DynamoDB so that I could easily import it to tables? Or is there anything I could do to transform the JSON files that I receive from third party?
The json seems to be invalid. you could use https://jsonlint.com/ to validate the JSON.
A sample valid JSON messages are available at https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/samples/sampledata.zip
To import data to DynamoDB Table
aws dynamodb batch-write-item --request-items file://ProductCatalog.json
{
"UnprocessedItems": {}
}
you could export data from DynamoDB using the following command
aws dynamodb scan --table-name ProductCatalog > ProductCatalog_export.json
DynamoDB can import data in three formats: CSV, DynamoDB JSON, and Amazon Ion.
CSV
DynamoDB Json
Amazon Ion
More information at https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/S3DataImport.Format.html
Related
I tried: vault kv put -format=json secrets/path #file.json
Getting this error:
Failed to parse K=V data: invalid key/value pair "#file.json": json cannot unmarshal array into Go value of type map[string] interface {}
When trying vault kv put -format=json secrets/path file.json I get:
Failed to parse K=V data: invalid key/value pair "file.json": format must be key=value
Not sure what I'm doing wrong.
The first form is more correct if you're attempting to pass a JSON file as an argument. The second form is not referencing a file, it's just invalid syntax.
The error message on the first form suggests that the JSON file you have is formatted incorrectly. The required format will depend on your KV engine version. If you are working with a KV v2 engine, it is required to put your key:value pairs into a data top level map. If you are working with a KV v1 engine, each key:value pair needs to itself be a top level object.
KV v1:
{
"key": "value",
"foo": "bar",
"bar": "baz"
}
KV v2:
{
"data": {
"key": "value",
"foo": "bar",
"bar": "baz"
},
"options": {}
}
The -output-curl-string flag is great for inspecting what the vault CLI tool is doing under the hood, try adding it and see what transformations the binary applies to your commands.
I downloaded .json file from amazon s3, but its content is just a value of the first key/value pair.
original json file is like this:
{
"_1": [
{
"Name": "name",
"Type": "type"
}
]
}
but downloaded json file is not even json file, it only has list inside.
[
{
"Name": "name",
"Type": "type"
}
]
I tried aws s3 sync / aws s3 copy / aws s3api get-object and all of its result is same.
I only want to download original file from the s3 bucket.
is there any solution?
Updates
I just copied the original content on the s3 select from preview and saved it as a file.
I found out its md5 checksum and file size is totally different with the object overview.
It seems the original file on the s3 bucket is corrupted, but I'm not sure how its preview is still same as the original content.
I found out aws s3api select-object-content can give me same result as the S3 select from preview but without indents.
For indents, I decided to reindent after I receive the uncorrupted results.
I used below command to retrieve my json files.
aws s3api select-object-content \
--bucket $BUCKET \
--key $KEY --expression "select * from s3object" \
--expression-type 'SQL' \
--input-serialization '{"JSON": {"Type": "LINES"}, "CompressionType": "NONE"}' \
--output-serialization '{"JSON": {}}' \
/dev/stdout | python -mjson.tool > $KEY
I have file.csv with similar to this structure
loremipsum; machine, metal
As i understand, succeful import will looks like
{
text: "loremipsum", << string
tags: ["machine","metal"] << object with two fields
}
The best result i get
{
text: "loremipsum", << string
tags: "machine, metal" << string
}
If i understand it correctly then please tell me how to do succeful import. Thanks.
Edit: because "tags" object should contain ~16 urls, so tell me how it should be stored correctly.
Ideally, below command should be used to import csv file to mongoDb (Maybe you are using the same):
mongoimport --db users --type csv --headerline --file /filePath/fileName.csv
I think ,Your Problem is with array type of data (if I understood correctly...!!).
Then, You need to first add one document in collection and export it as CSV file. This will give you the format which is expected by above command to import your data correctly. And then Arrange your data as per exported CSV file.
link Answer Here Explained it Very well
I had this data in Excel
I wanted like this in MongoDB
{
"name":"John",
"age":30,
"cars":[ "Ford", "BMW", "Fiat" ]
}
I did replaced the headings like cars.0 cars.1 cars.2
Like this
I used the tool mongoimport and ran this command
mongoimport.exe --uri "mongodb+srv:/localhost/<MYDBName>" --username dbuser --collection ----drop --type csv --useArrayIndexFields --headerline --file 1.csv
here my csv is 1.csv
I'd like to import a json file containing an array of jsons into a mysql table.
Using the LOAD DATA LOCAL INFILE '/var/lib/mysql-files/big_json.json' INTO TABLE test(json); give the error messageInvalid JSON text: "Invalid value." at position 1 in value for column
My json looks like the following:
[
{
"id": "6defd952",
"title": "foo"
},
{
"id": "98ee3d8b",
"title": "bar"
}
]
How can I parameterize the load command to iterate through the array?
For whatever reason, MySQL errors out when you try and load pretty-printed JSON, both in 5.7 and in 8.0+, so you'll need the JSON to be in compact form, i.e.:
[{"id":"6defd952","title":"foo"},{"id":"98ee3d8b","title":"bar"}]
This can be accomplished by reading the json object to jq and using the -c flag to output in compact form to re-format the text:
Windows:
type input.json | jq -c . > infile.json
Mac/Linux:
cat input.json | jq -c . > infile.json
If you're reading from a std_out instead of a file, just replace the cat/type section and pipe to jq in the same manner. The resultant "infile.json" should then work for the LOAD DATA INFILE statement.
If you wanted to then iterate through the loaded array, use JSON_EXTRACT(JSON_FIELD, CONCAT('$[', i, ']')) with ordinal variable i to traverse the root document (array) in a single query.
Is there any way to dump mongo collection into json format? Either on the shell or using java driver.I am looking for the one with best performance.
Mongo includes a mongoexport utility (see docs) which can dump a collection. This utility uses the native libmongoclient and is likely the fastest method.
mongoexport -d <database> -c <collection_name>
Also helpful:
-o: write the output to file, otherwise standard output is used (docs)
--jsonArray: generates a valid json document, instead of one json object per line (docs)
--pretty: outputs formatted json (docs)
Use mongoexport/mongoimport to dump/restore a collection:
Export JSON File:
mongoexport --db <database-name> --collection <collection-name> --out output.json
Import JSON File:
mongoimport --db <database-name> --collection <collection-name> --file input.json
WARNING
mongoimport and mongoexport do not reliably preserve all rich BSON data types because JSON can only represent a subset of the types supported by BSON. As a result, data exported or imported with these tools may lose some measure of fidelity.
Also, http://bsonspec.org/
BSON is designed to be fast to encode and decode. For example,
integers are stored as 32 (or 64) bit integers, so they don't need to
be parsed to and from text. This uses more space than JSON for small
integers, but is much faster to parse.
In addition to compactness, BSON adds additional data types
unavailable in JSON, notably the BinData and Date data types.
Here's mine command for reference:
mongoexport --db AppDB --collection files --pretty --out output.json
On Windows 7 (MongoDB 3.4), one has to move the cmd to the place where mongod.exe and mongo.exe file resides =>
C:\MongoDB\Server\3.4\bin else it won't work saying it does not recongnize mongoexport command.
From the Mongo documentation:
The mongoexport utility takes a collection and exports to either JSON or CSV. You can specify a filter for the query, or a list of fields to output
Read more here: http://www.mongodb.org/display/DOCS/mongoexport
Here is a little node script that I write to dump all collections in a specific database to the specified output directory...
#!/usr/bin/env node
import { MongoClient } from 'mongodb';
import { spawn } from 'child_process';
import fs from 'fs';
const DB_URI = 'mongodb://0.0.0.0:27017';
const DB_NAME = 'database-name';
const OUTPUT_DIR = 'output-directory';
const client = new MongoClient(DB_URI);
async function run() {
try {
await client.connect();
const db = client.db(DB_NAME);
const collections = await db.collections();
if (!fs.existsSync(OUTPUT_DIR)) {
fs.mkdirSync(OUTPUT_DIR);
}
collections.forEach(async (c) => {
const name = c.collectionName;
await spawn('mongoexport', [
'--db',
DB_NAME,
'--collection',
name,
'--jsonArray',
'--pretty',
`--out=./${OUTPUT_DIR}/${name}.json`,
]);
});
} finally {
await client.close();
console.log(`DB Data for ${DB_NAME} has been written to ./${OUTPUT_DIR}/`);
}
}
run().catch(console.dir);