BSON document size exceeds 16 MB in mongo document - json

I would like to store ticket details in an array in a mongo document. it works fine till the document size reaches 16MB, after that I get the exception (Resulting document after update is larger than 16777216) and program terminates abruptly. I cannot split this document coz it stores all the ticket details falls under that year 2016.
here goes my document structure.
{
year:2016,
purpose: ticketdetail,
tickets:[
{ticketid:001, desc:"xyz", created:20161231},
{ticketid:002, desc:"xyz", created:20161231},
{ticketid:003, desc:"xyz", created:20161231},
.......
.......
{ticketid:00N, desc:"xyz", created:20161231},
}]
}

You will need to split your document into separate documents, perhaps in a different collection. I would not recommend GridFS, because you cannot query data within a GridFS blob.
Here is a suggested document structure:
{
_id: ObjectId("85bf0ef0b9692c0010978359"),
"ticketid" : "001",
"desc" : "xyz",
"created" : ISODate("2016-12-31T00:00:00.000Z")
}
,
{
_id: ObjectId("85bed4257726f90010d4e21f"),
"ticketid" : "002",
"desc" : "xyz",
"created" : ISODate("2016-12-31T00:00:00.000Z")
}
Notes on this structure:
Each ticket is in a different document - this makes it scalable, because there is no limit on the number of documents in a collection.
The "created" field is now in a proper date field. This gives you more accurate queryability.
You said that your original document was needed to store all tickets for 2016. A suitable query on this new collection will return you all tickets for 2016, so you don't need to store them all in a single document:
db.tickets.find({
"created" : {
$gte: ISODate("2016-01-01"),
$lt: ISODate("2017-01-01")
}
}
});

Related

Is it possible to query JSON data in DynamoDB?

Let's say my JSON looks like this (example provided here) -
{
"year" : 2013,
"title" : "Turn It Down, Or Else!",
"info" : {
"directors" : [
"Alice Smith",
"Bob Jones"
],
"release_date" : "2013-01-18T00:00:00Z",
"rating" : 6.2,
"genres" : [
"Comedy",
"Drama"
],
"image_url" : "http://ia.media-imdb.com/images/N/O9ERWAU7FS797AJ7LU8HN09AMUP908RLlo5JF90EWR7LJKQ7##._V1_SX400_.jpg",
"plot" : "A rock band plays their music at high volumes, annoying the neighbors.",
"rank" : 11,
"running_time_secs" : 5215,
"actors" : [
"David Matthewman",
"Ann Thomas",
"Jonathan G. Neff"
]
}
}
I would like to query all movies where genres contains Drama.
I went through all of the examples but it seems that I can query only on hash key and sort key. I can't have JSON document as key itself as that is not supported.
You cannot. DynamoDB requires that all attributes you are filtering for have an index.
As you want to query independently of your main index, you are limited to Global Secondary Indexes.
The documentation lists on what kind of attributes indexes are supported:
The index key attributes can consist of any top-level String, Number, or Binary attributes from the base table; other scalar types, document types, and set types are not allowed.
Your type would be an array of Strings. So this query operation isn't supported by DynamoDB at this time.
You might want to consider other NoSQL document based databases which are more flexible like MongoDB Atlas, if you need this kind of querying functionality.
String filterExpression = "coloumnname.info.genres= :param";
Map valueMap = new HashMap();
valueMap.put(":param", "Drama");
ItemCollection scanResult = table
.scan(new ScanSpec().
withFilterExpression(filterExpression).
withValueMap(valueMap));
One example that I took from AWS Developer Forums is as follows.
We got some hints for you from our team. Filter/condition expressions for maps have to have key names at each level of the map specified separately in the expression attributeNames map.
Your expression should look like this:
{
"TableName": "genericPodcast",
"FilterExpression": "#keyone.#keytwo.#keythree = :keyone",
"ExpressionAttributeNames": {
"#keyone": "attributes",
"#keytwo": "playbackInfo",
"#keythree": "episodeGuid"
},
"ExpressionAttributeValues": {
":keyone": {
"S": "podlove-2018-05-02t19:06:11+00:00-964957ce3b62a02"
}
}
}

JSON structure for a ticket reservation system with Firebase?

I'm currently developing a ticket reservation system where the user should be able to choose an event, see a list of sections, a section contains a list of seats, thats either free or occupied/booked by another user. I've chosen to use Firebase as my backend, but got very little experience with databases and zero using JSON. How would I go about structuring a system like this?
This is what I got so far:
{
"events" : {
"e2017" : {
"name" : "event 2017",
"date" : "1490567256550"
}
},
"eventSections" : {
"e2017" : {
"e2017-A" : {
"isFull" : false,
"totalSeats": 40,
"bookedSeats": 20
}
}
},
"sectionSeats" : {
"e2017-A" : {
"A1": {
"isBooked" : true,
"bookedBy" : "userId"
},
"A2": {
"isBooked" : false,
"bookedBy" : false
}
}
}
}
The following structure is an example structure for a very specific use case of your question. You had the following criteria:
1) user should be able to choose an event
2) see a list of sections
3) a section contains a list of seats
4) (seats are) either free or occupied/booked by another user
and a structure that meets that criteria
events
event_0
sections
section_0
seats
seat_0: "free"
seat_1: "occupied"
section_1
seats
seat_0: "occupied"
seat_1: "occupied"
event_1
sections
section_0
seats
seat_0: "free"
seat_1: "occupied"
section_1
seats
seat_0: "occupied"
seat_1: "occupied"
While this is an answer, it's not the only one. For example. What if you wanted to know which events had free seats? You really can't query this structure for that data (unless you filter in code).
However, if you present a list of events to a user where they can click on the event to get more data, this structure would work fine as it could be used to present sections and seats available.
If you have a different specific use case, update your question with more data.

Best way to handle data list of REST web service with foreign key(one to many)

I am going to implement the REST base CRUD modal in my my app.I wan to display the list of product data with edit and delete link
Product
id, title, unit_id, product_type_id, currency_id,price
Q1: what should be json response look like?
There are two formats comes in my mind to place the data in Json as a response of REST Get call
[
{
id:1,
title:"T-Shirt",
unit_id:20,
unit_title: "abc"
product_type_id:30,
product_type_title:"xyz"
currency_id: 10,
currency_name: "USD"
min_price:20
},
{...}
]
and the another one is
[
{
id:1,
title:"T-Shirt",
unit: {
id: 20,
title: "abc"
},
product_type: {
id: 30,
title: "xyz"
},
currency_id: {
id:10,
name: "USD"
},
min_price:20
},
{...}
]
what is the better and standard way to handle the above scenario?
Furthermore, let suppose I have 10 more properties in product table which will never display on list page. but i needed it when user going to edit the specific item.
Q2: Should I the load all data once at the time of displaying product list and pass the data to edit component.
or
Load only the needed propeties of product table and pass the id to produt edit component and a new REST GET call with id to get the properties of product.
I am using React + Redux for my front end
Typically, you would create additional methods for API consumers to retrieve the values that populate the lists of currency, product_type and unit when editing in a UI.
I wouldn't return more data than necessary for an individual Product object.

REST POST json body to support complex query advice

I'm fairly new to REST. All of our legacy webservices were SOAP based with enterprise (ORACLE or DB2) databases. We are now moving to REST/couchbase.
Our team is looking into implementing a complex query method. We already have implemented simple query methods using GET, for example GET returns all entries and a GET/067e6162-3b6f-4ae2-a171-2470b63dff00 would return the entry for 067e6162-3b6f-4ae2-a171-2470b63dff00.
We want to support a query method that would support receiving several query parameters such a list of Ids and date ranges. The number of Ids can number into a few thousand and because of this, we realize we cannot pass these query parameters in a GET HTTP header since there is a limit on header size.
We are starting to look into passing our query parameters into the JSON body of a POST request. For example, we could have client pass in a few thousand Ids as an array and also pass in a date range, so we'd have each query param/filter be an object. The JSON body would then be an array of objects. For example:
{
"action" : "search",
"queryParameters" : {
[
{
“operation”: “in”,
"key" : "name.of.attribute.Id",
"value" : "[{ "id: "067e6162-3b6f-4ae2-a171-2470b63dff00"}, {"id": "next id"....}],
},
{
“operation”: “greater”,
"key" : "name.of.attribute “,
"value" : "8/20/2016"
},
{
“operation”: “less”,
"key" : "name.of.attribute “,
"value" : "8/31/2016"
}
]
}
The back end code would then receive POST and read the body. It would see action is a search and then look for any entries in the list that are in the list of Ids that are in the date range of > 8/20/2016 and < 8/31/2016.
I've been trying to look online for tips/best practices on how best to structure the JSON body for complex queries but have not found much. So any tips, guidance or advice would be greatly appreciated.
thanks.

How to display 'c' array values alone from the given JSON document below using MongoDB?

I am a newbie to MongoDB. I am experimenting the various ways of extracting fields from a document inside collection.
Here in the below JSON document, I am finding it difficult to get extract it according to my need
{
"_id":1,
"dependencies":{
"a":[
"hello",
"hi"
],
"b":[
"Hmmm"
],
"c":[
"Vanilla",
"Strawberry",
"Pista"
],
"d":[
"Carrot",
"Cauliflower",
"Potato",
"Cabbage"
]
},
"productid":"25",
"date":"Thu Jul 30 11:36:49 PDT 2015"
}
I need to display the following output:
c:[
"Vanilla",
"Strawberry",
"Pista"
]
Can anyone please help me in solving it?
MongoDB Aggregation comes into rescue to get the result you are looking for :
$Project--> Passes along the documents with only the specified fields to the next stage in the pipeline. The specified fields can be existing fields from the input documents or newly computed fields.
db.collection.aggregate( [
{ $project :
{ c: "$dependencies.c", _id : 0 }
}
]).pretty();
As per the output you required, we just need to project ( display) the field "dependencies.c" , so we are creating a new field "c" and assigining the value of the "dependencies.c" into it.
Also by defalut "_id" field will be display along with the result, since you dont need it, so we are suppressing of the _id field by assigining "_id" : <0 or false>, so that it will not display the _id field in the output.
The above query will fetch you the result as below :
"c" : [
"Vanilla",
"Strawberry",
"Pista"
]