DynamoDB&NodeJS: search table from JSON array - json

Question

I have a DynamoDB table with only two columns "EmailId" and "SubscriptionId". "EmailId" is Primary sort key and "SubscriptionId" is Primary partition key. I have to insert a record into it but before that I need to make sure that the record does not exist. I get the records from a third party API endpoint in JSON array format. So, I will have to search in the table and the records that do not exist in the will have to be inserted.
The records I get are in the following format. This is a sample response and I can get maybe 1000 of records in the array.
[{
"emailId": "abc1#abc1.com",
"subscriptionId": "A1"
}, {
"emailId": "abc2#abc2.com",
"subscriptionId": "A2"
}, {
"emailId": "abc3#abc3.com",
"subscriptionId": "A3"
}]
I don't want to pick each record from the array above, search the table and if not found, insert it because this table is going to get huge. Is there any other way we can do that? I am using this with NodeJS. Though I can not change the JSON array but I can make changes to DynamoDB table. Any suggestions?

Answer 1

The batchWrite item API can be used to put multiple items in a batch. The maximum number of requests in the batch is 25.
The best part is that if the key in the PutRequest is already present in the table, it updates the item rather than throwing some error or exception (i.e. key is not unique).
The disadvantage of this approach is that the latest update will overwrite all the attributes of the existing item in the table. For example, if the existing item in the table has 5 attributes and the latest update has only 3 attributes, the table will have only 3 attributes (as present in the latest PutRequest) after the latest batch execution.
var docClient = new AWS.DynamoDB.DocumentClient();
var params = {
RequestItems: {
"subscription": [
{
PutRequest: {
Item: {
"emailId": "abc1#abc1.com",
"subscriptionId": "A1"
}
}
},
{
PutRequest: {
Item: {
"emailId": "abc2#abc2.com",
"subscriptionId": "A2"
}
}
},
{
PutRequest: {
Item: {
"emailId": "abc3#abc3.com",
"subscriptionId": "A3"
}
}
}
]
}
};
docClient.batchWrite(params, function (err, data) {
if (err) {
console.error("Unable to write item. Error JSON:", JSON.stringify(err,
null, 2));
} else {
console.log("Write Item succeeded:", JSON.stringify(data, null, 2));
}
});