Dynamodb batchgetItem API behavior for non existent keys - exception

What is the behavior of DynamoDb BatchGetItem API if none of the keys exist in dynamodb?
Does it returns an empty list or throws an exception?
I am not sure about this after reading their doc: link
but I may be missing something.

BatchGetItem will not throw an exception. The results for those items will not be present in the Responses map in the response. This is also stated in the BatchGetItem documentation:
If a requested item does not exist, it is not returned in the result.
Requests for nonexistent items consume the minimum read capacity units
according to the type of read. For more information, see Capacity
Units Calculations in the Amazon DynamoDB Developer Guide.
This behavior is also easy to verify. This is for a Table with a hash key attribute named customer_id (the full example I am using is here):
dynamoDB.batchGetItem(new BatchGetItemSpec()
.withTableKeyAndAttributes(new TableKeysAndAttributes(EXAMPLE_TABLE_NAME)
.withHashOnlyKeys("customer_id", "ABCD", "EFGH")
.withConsistentRead(true)))
.getTableItems()
.entrySet()
.stream()
.forEach(System.out::println);
dynamoDB.batchGetItem(new BatchGetItemSpec()
.withTableKeyAndAttributes(new TableKeysAndAttributes(EXAMPLE_TABLE_NAME)
.withHashOnlyKeys("customer_id", "TTTT", "XYZ")
.withConsistentRead(true)))
.getTableItems()
.entrySet()
.stream()
.forEach(System.out::println);
Output:
example_table=[{ Item: {customer_email=jim#gmail.com, customer_name=Jim, customer_id=ABCD} }, { Item: {customer_email=garret#gmail.com, customer_name=Garret, customer_id=EFGH} }]
example_table=[]

Related

How can I query for multiple values after a wildcard?

I have a json object like so:
{
_id: "12345",
identifier: [
{
value: "1",
system: "system1",
text: "text!"
},
{
value: "2",
system: "system1"
}
]
}
How can I use the XDevAPI SearchConditionStr to look for the specific combination of value + system in the identifier array? Something like this, but this doesn't seem to work...
collection.find("'${identifier.value}' IN identifier[*].value && '${identifier.system} IN identifier[*].system")
By using the IN operator, what happens underneath the covers is basically a call to JSON_CONTAINS().
So, if you call:
collection.find(":v IN identifier[*].value && :s IN identifier[*].system")
.bind('v', '1')
.bind('s', 'system1')
.execute()
What gets executed, in the end, is (simplified):
JSON_CONTAINS('["1", "2"]', '"2"') AND JSON_CONTAINS('["system1", "system1"]', '"system1"')
In this case, both those conditions are true, and the document will be returned.
The atomic unit is the document (not a slice of that document). So, in your case, regardless of the value of value and/or system, you are still looking for the same document (the one whose _id is '12345'). Using such a statement, the document is either returned if all search values are part of it, and it is not returned if one is not.
For instance, the following would not yield any results:
collection.find(":v IN identifier[*].value && :s IN identifier[*].system")
.bind('v', '1')
.bind('s', 'system2')
.execute()
EDIT: Potential workaround
I don't think using the CRUD API will allow to perform this kind of "cherry-picking", but you can always use SQL. In that case, one strategy that comes to mind is to use JSON_SEARCH() for retrieving an array of paths corresponding to each value in the scope of identifier[*].value and identifier[*].system i.e. the array indexes and use JSON_OVERLAPS() to ensure they are equal.
session.sql(`select * from collection WHERE json_overlaps(json_search(json_extract(doc, '$.identifier[*].value'), 'all', ?), json_search(json_extract(doc, '$.identifier[*].system'), 'all', ?))`)
.bind('2', 'system1')
.execute()
In this case, the result set will only include documents where the identifier array contains at least one JSON object element where value is equal to '2' and system is equal to system1. The filter is effectively applied over individual array items and not in aggregate, like on a basic IN operation.
Disclaimer: I'm the lead developer of the MySQL X DevAPI Connector for Node.js

How to aggregate an ObjectSet after SearchAround

I have 2 phonograph objects, each one having millions of rows, which I have linked by using the Search Around methods.
On the example below, I filter to an Object Set of Flights based on the departure code, then I Search Around to the Passengers on those flights and then I filter again based on an attribute of Passengers Object.
const passengersDepartingFromAirport = Objects.search()
.flights()
.filter(flight => flight.departureAirportCode.exactMatch(airportCode))
.searchAroundPassengers()
.filter(passenger => passenger.passengerAttribute.exactMatch(value));
The result of the above code is:
LOG [2022-04-19T14:25:58.182Z] { osp: {},
objectSet:
{ objectSetProvider: '[Circular]',
objectSet: { type: 'FILTERED', filter: [Object], objectSet: [Object] } },
objectTypeIds: [ 'passengers' ],
emptyOrderByStep:
{ objectSet: '[Circular]',
orderableProperties:
{ attributeA: [Object],
attributeB: [Object],
attributeB: [Object],
...
Now, when I am trying to use take() or takeAsync() or to aggregate the result using groupBy(), I receive the below error:
RemoteError: INVALID_ARGUMENT ObjectSet:ObjectSetTooLargeForSearchAround with instance ID xxx.
Error Parameters: {
"RemoteError.type": "STATUS",
"objectSetSize": "2160870",
"maxAllowedSize": "100000",
"relationSide": "TARGET",
"relationId": "flights-passengers"
}
SafeError: RemoteError: INVALID_ARGUMENT ObjectSet:ObjectSetTooLargeForSearchAround with instance ID xxx
What could be the way to aggregate or to reduce the result of the above ObjectSet?
The current object storage infrastructure has a limit on the size of the "left side" or "starting object set" for a search around of 100,000 objects.
You can define and object set that uses a search around, which is what you're seeing as the result when you execute the Function before attempting any further manipulations.
Using take() or groupBy "forces" the resolution of the object set definition. I.e. you no longer need the pointer to the objects, but you need to actually materialize some data from each individual object to do that operation.
It's in this materialization step that the limit comes into play - the object sets are resolved and, if the object set at the search around step is larger than 100,000 objects, the request will fail with the above message.
There is ongoing work for Object Storage v2, which will eventually support much larger search-around requests, but for now it's necessary create a query pattern that results in less than 100,000 objects before making a search around.
In some cases it's possible to create an "intermediate" object type that represents a different level of granularity in your data or two invert the direction of your search around to find a way to address these limits.

What is the json body in post request in order to fetch all rows from an operation list with key?

I have defined a list for operational data in yang model as:
list listener-state {
key “listener-name”;
config false;
description
“common statistics for given listener (i.e sent messages)”;
uses listener-state-info;
…
}
I use opendaylight api (org.opendaylight.yangtools.yang.data.codec.gson.JsonParserStream) which will convert the json body in request to org.opendaylight.yangtools.yang.data.api.schema.NormalizedNode, in order to finally generate the XML rpc for confd server.
In my case, I want to fetch all rows from this operation list, then I try to make the json as :
“command”: {“service” : {“server” : {“listener-state” : {}}}},
I will get exception that : “Input is missing some of the keys of listener-state”
Then I can add the key value to the json body :
“command”: {“service” : {“server” : {“listener-state” : {“listener-name”: “first”}}}},
This case, I can only get one row. I also try to leave the key value as blank:
“command”: {“service” : {“server” : {“listener-state” : {“listener-name”: “”}}}},
Then the response will be all key values instead of all rows. So now my question is what the json will be in order to get all rows in the list without knowing the key values ?
This should be feasible since I figure out XML request can do that. But I can't figure out what the matching json will be.
Thanks.
I did bunch of investigation. Unfortunately, I don't think there is a way to fetch the whole table

Format for storing json in Amazon DynamoDB

I've got JSON file that looks like this
{
"alliance":{
"name_part_1":[
"Ab",
"Aen",
"Zancl"
],
"name_part_2":[
"aca",
"acia",
"ythrae",
"ytos"
],
"name_part_3":[
"Alliance",
"Bond"
]
}
}
I want to store it in dynamoDB.
The thing is that I want a generator that would take random elements from fields like name_part_1, name_part_2 and others (number of name_parts_x is unlimited and overalls number of items in each parts might be several hundreds) and join them to create a complete word. Like
name_part_1[1] + name_part_2[10] + name_part[3]
My question is that what format I should use to do that effectively? Or NoSQL shouldn't be used for that? Should I refactor JSON for something like
{
"name": "alliance",
"parts": [ "name_part_1", "name_part_2", "name_part_3" ],
"values": [
{ "name_part_1" : [ "Ab ... ] }, { "name_part_2": [ "aca" ... ] }
]
}
This is a perfect use case for DynamoDB.
You can structure like this,
NameParts (Table)
namepart (partition key)
range (hash key)
namepart: name_part_1 range: 0 value: "Ab"
This way each name_part will have its own range and scalable. You can extend it to thousands or even millions.
You can do a batch getitem from the sdk of your choice and join those values.
REST API reference:
https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchGetItem.html
Hope it helps.
You can just put the whole document as it is in DynamoDB and then use document path to access the elements you want.
Document Paths
In an expression, you use a document path to tell DynamoDB where to
find an attribute. For a top-level attribute, the document path is
simply the attribute name. For a nested attribute, you construct the
document path using dereference operators.
The following are some examples of document paths. (Refer to the item
shown in Specifying Item Attributes.)
A top-level scalar attribute: ProductDescription A top-level list
attribute. (This will return the entire list, not just some of the
elements.) RelatedItems The third element from the RelatedItems list.
(Remember that list elements are zero-based.) RelatedItems[2] The
front-view picture of the product. Pictures.FrontView All of the
five-star reviews. ProductReviews.FiveStar The first of the five-star
reviews. ProductReviews.FiveStar[0] Note The maximum depth for a
document path is 32. Therefore, the number of dereferences operators
in a path cannot exceed this limit.
Note that each document requires a unique Partition Key.

Accessing attributes of a json

I have this JSON output
{
"status":1,
"complete":1,
"list":{
"792489954":{
"item_id":"792489954",
"resolved_id":"792489954",
"given_url":"http:\/\/www.liveathos.com\/?gclid=Cj0KEQiAzb-kBRDe49qh9s75m-wBEiQATOxgwZcJ5_ws34o4PUSUYDGqs8HEbLF-LyjxrTPOwn6AYV8aAmMk8P8HAQ",
"given_title":"Athos - Wearable Technology for Fitness",
"favorite":"0",
"status":"0",
"time_added":"1418754744",
"time_updated":"1418754746",
"time_read":"0",
"time_favorited":"0",
"sort_id":0,
"resolved_title":"Wearable Technology for Fitness",
"resolved_url":"http:\/\/www.liveathos.com\/?gclid=Cj0KEQiAzb-kBRDe49qh9s75m-wBEiQATOxgwZcJ5_ws34o4PUSUYDGqs8HEbLF-LyjxrTPOwn6AYV8aAmMk8P8HAQ",
"excerpt":"Thank you for reserving Athos. You will receive a confirmation email with reservation details and a referral link where you get $10 off your next order.",
"is_article":"0",
"is_index":"0",
"has_video":"0",
"has_image":"0",
"word_count":"25"
},
"692647226":{
"item_id":"692647226",
"resolved_id":"692647226",
"given_url":"http:\/\/www.terrafugia.com\/news",
"given_title":"News | Terrafugia",
"favorite":"0",
"status":"0",
"time_added":"1418754204",
"time_updated":"1418754204",
"time_read":"0",
"time_favorited":"0",
"sort_id":1,
"resolved_title":"News",
"resolved_url":"http:\/\/www.terrafugia.com\/news",
"excerpt":"",
"is_article":"0",
"is_index":"1",
"has_video":"0",
"has_image":"0",
"word_count":"0"
},
`...etc` "since":1419641101
};
It's a JSON with a list that should have article items on it. I want to access the properties of these smaller items like given_url.
I'm using
for key in (BIGJSONRESPONSE).list
do etc...
When I try printing the key, I get only the id that comes before an item. Any idea how to access more?
Thanks!
You asked for the keys, you got the keys. To get the item associated with that key, use BIGJSONRESPONSE.list[key].
This should be correct:
for in returns the keys of an object. Since the key refers to a property of the object one can access the property using [key].
for (key in (BIGJSONRESPONSE).list)
{
console.log(BIGJSONRESPONSE.list[key]) // log the entry to console for debugger.
}