I am new to mongodb. I am developing a social network website, for which I have entries in my collection "users". I have the following entry of particular user.
{
"_id" : ObjectId("56a9fd3a5b4e12369150d6f0"),
"name" : "Amit",
"venue" : {
"state" : "Madhya Pradesh",
"city" : "Gwalior",
"ll" : [
"26.2215",
"78.1780"
]
},
"lsv" : [
0.05,
0.17,
0.05,
0.14,
0.18,
0.24,
0.17
],
"username" : "amit",
"frnds" : [
{
"id" : "udit",
"type" : 1
},
{
"id" : "nakul",
"type" : 1
}
]
}
For each user I have to do operation on "lsv" array by matching some values of "lsv" of another user.
I am not able to get how to write the query for that.
Related
I found the solution for json to csv conversion. Below is the sample json and solution.
{
"took" : 111,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 2,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "alerts",
"_type" : "_doc",
"_id" : "1",
"_score" : 1.0,
"_source" : {
"alertID" : "639387c3-0fbe-4c2b-9387-c30fbe7c2bc6",
"alertCategory" : "Server Alert",
"description" : "Successfully started.",
"logId" : null
}
},
{
"_index" : "alerts",
"_type" : "_doc",
"_id" : "2",
"_score" : 1.0,
"_source" : {
"alertID" : "2",
"alertCategory" : "Server Alert",
"description" : "Successfully stoped.",
"logId" : null
}
}
]
}
}
The solution :
jq -r '.hits.hits[]._source | [ "alertID" , "alertCategory" , "description", "logId" ], ([."alertID",."alertCategory",."description",."logId" // "null"]) | #csv' < /root/events.json
The problem with this solution is that I have to hard code the column names. What If my json gets a few additions under _source tag later? I need a solution which can handle the dynamic data under _source. I am open to any other tool or command in shell.
Simply use keys_unsorted (or keys if you want them sorted). See e.g. Convert JSON array into CSV using jq or How to convert arbitrary simple JSON to CSV using jq? for two SO examples. There are many others too.
I am new to elasticserach. Below is the Sample data on which elastic query needs to run. I am trying to get those docs in which account_type is "credit card" and source_name is 'SOMEVALUE'
{
"took" : 0,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "bureau_data",
"_type" : "_doc",
"_id" : "bda57e01-c564-4cdc-bb8d-79bd2db9d2f8",
"_score" : 1.0,
"_source" : {
"userid" : "bda57e01-c564-4cdc-bb8d-79bd2db9d2f8",
"raw_derived" : {
"gender" : "MALE",
"firstname" : "trsqlsz",
"middlename" : "rgj",
"lastname" : "ggksb",
"mobilephone" : "2125954664",
"dob" : "1988-06-28 00:00:00",
"applications" : [
{
"applicationid" : "c7fb0147-22fd-4a5e-8851-98241de6aa50",
"createdat" : "2019-06-07 19:28:54",
"updatedat" : "2019-06-07 19:28:55",
"source" : "4",
"source_name" : "EXPERIAN",
"applicationcreditreportid" : "b67f9180-9bb6-485c-9cfc-e7ccf9a70a69",
"accounts" : [
{
"applicationcreditreportaccountid" : "c5de28c4-cac9-4390-852a-96f143cb0b62",
"currentbalance" : 418288,
"institutionid" : "021d58b4-aba5-42c9-8d39-304a78d34aea",
"accounttypeid" : "5",
"institution_name" : "HDFC BANK",
"account_type_name" : "Personal Loan"
}
]
}
]
}
}
}
I have tried the below query and its working fine. I need if we have any optimized way to query the multiple nested fields
GET /my_index/_search
{
"query": {
"bool": {
"must": [
{
"nested": {
"path": "raw_derived.applications.accounts",
"query": {
"bool": {
"must": [
{"match": {
"raw_derived.applications.accounts.account_type_name": "Credit Card"
}}
]
}
}
}
},
{
"nested": {
"path": "raw_derived.applications",
"query": {
"bool": {
"must": [
{"match": {
"raw_derived.applications.source_name": "CIBIL"
}}
]
}
}
}
}
]
}
}
}
If I will query on the multiple nested fields it will become very long Please suggest any other way to query nested fields or multiple AND
Well your optimizations should always start with your data model / mapping since it's mostly the cause of performance issues and not your queries.
That being said, you can avoid the nested query by flattening your data. A flattened data model would lead to one document per application and account element.
Since elasticsearch is a non-relational data store, it is completely fine to index "redundant" data. This is not a lazy appraoch but a common way to handle these type of data structures.
Sample document #1:
{
"_index" : "bureau_data",
"_type" : "_doc",
"_id" : "bda57e01-c564-4cdc-bb8d-79bd2db9d2f8",
"_score" : 1.0,
"_source" : {
"userid" : "bda57e01-c564-4cdc-bb8d-79bd2db9d2f8",
"gender" : "MALE",
"firstname" : "trsqlsz",
"middlename" : "rgj",
"lastname" : "ggksb",
"mobilephone" : "2125954664",
"dob" : "1988-06-28 00:00:00",
"applicationid" : "c7fb0147-22fd-4a5e-8851-98241de6aa50",
"createdat" : "2019-06-07 19:28:54",
"updatedat" : "2019-06-07 19:28:55",
"source" : "4",
"source_name" : "EXPERIAN",
"applicationcreditreportid" : "b67f9180-9bb6-485c-9cfc-e7ccf9a70a69",
"applicationcreditreportaccountid" : "c5de28c4-cac9-4390-852a-96f143cb0b62",
"currentbalance" : 418288,
"institutionid" : "021d58b4-aba5-42c9-8d39-304a78d34aea",
"accounttypeid" : "5",
"institution_name" : "HDFC BANK",
"account_type_name" : "Personal Loan"
}
}
If the same user creates another account you would send the very same ("redundant") data, except for that other account element/data like so:
{
"_index" : "bureau_data",
"_type" : "_doc",
"_id" : "another, from es generated id",
"_score" : 1.0,
"_source" : {
"userid" : "bda57e01-c564-4cdc-bb8d-79bd2db9d2f8",
"gender" : "MALE",
"firstname" : "trsqlsz",
"middlename" : "rgj",
"lastname" : "ggksb",
"mobilephone" : "2125954664",
"dob" : "1988-06-28 00:00:00",
"applicationid" : "c7fb0147-22fd-4a5e-8851-98241de6aa50",
"createdat" : "2019-06-07 19:28:54",
"updatedat" : "2019-06-07 19:28:55",
"source" : "4",
"source_name" : "EXPERIAN",
"applicationcreditreportid" : "b67f9180-9bb6-485c-9cfc-e7ccf9a70a69",
"applicationcreditreportaccountid" : "the new id",
"currentbalance" : 4711,
"institutionid" : "foo",
"accounttypeid" : "bar",
"institution_name" : "foo bar",
"account_type_name" : "foo baz"
}
}
With that kind of data model, you can run simple queries to get your results:
GET /my_index/_search
{
"query": {
"bool": {
"must": [
{
"match":{
"account_type_name": "Credit Card"
}
},
{
"match":{
"source_name": "CIBIL"
}
}
]
}
}
}
How can I push new key and value in JSON array?
I tried I used push keyword in update query but I got a different output. I used:
db.users.updateOne({"name":"viki"},{$push{"address.district":"thambaram"}})
I have this document:
{ "_id" : ObjectId("58934f10c7592b1494fd9a4d"), "name" : "viki", "age" : 100, "subject" : [ "c", "node.js", "java" ], "address" : { "city" : "chennai", "state" : "tamilnadu", "pincode" : "123" } }
I want to add "district":"thambaram" in address json array
I need like:
{ "_id" : ObjectId("58934f10c7592b1494fd9a4d"), "name" : "viki", "age" : 100, "subject" : [ "c", "node.js", "java" ], "address" : { "city" : "chennai", "state" : "tamilnadu", "pincode" : "123","district":"thambaram"} }
Use $set
db.users.updateOne({"name":"viki"},{$set:{"address.district":"thambaram"}})
This should work.
The $push operator appends a specified value to an array. In your case you should use $set
I have 2 Json values that are similar, but there are some differences.
Json1:
{
"id": "1",
"people" : [
{
"Id" : 1421,
"Name" : "Jackson",
"Age" : 21,
"Status" : "Available"
},
{
"Id" : 5916,
"Name" : "Steven",
"Age" : 22,
"Status" : "Available"
}
],
"totalRecords" : 2
}
Json2:
{
"id": "1",
"people" : [
{
"Id" : 1421,
"Name" : "Jackson",
"Age" : 21,
"Status" : "Available"
},
{
"Id" : 5916,
"Name" : "Steven",
"Age" : 22,
"Status" : "Unavailable"
},
{
"Id" : 1337,
"Name" : "Alice",
"Age" : 19,
"Status" : "Available"
}
],
"totalRecords" : 3
}
I'd like to know if there's a way to compare the two Jsonvalues. At the moment I de-serialize the data into a type and then use the Id's and the status' to see if anythings changed. I then pick out the parts that are different (In the example it'd be Steven and Alice) and add them to a sequence for later.
I'd like to reverse a few of the steps. I'd like too compare the json, find the differences, deserialize them and then add them to the sequence, or add them to the sequence then de-serialize the whole sequence. Either way, same result.
Any ideas?
I want update a array value that is nested within an array value: i.e. set
status = enabled
where alerts.id = 2
{
"_id" : ObjectId("5496a8ed49847b6cd7c7b350"),
"name" : "joe",
"locations" : [
{
"name": "my location",
"alerts" : [
{
"id" : 1,
"status" : null
},
{
"id" : 2,
"status" : null
}
]
}
]
}
I would have used the position $ character, but cannot use it twice in a statement - multi positional operators are not supported yet: https://jira.mongodb.org/browse/SERVER-831
How do I issue a statement to only update the status field of an alert matching an id of 2?
UPDATE
If I change the schema as follows:
{
"_id" : ObjectId("5496ab2149847b6cd7c7b352"),
"name" : "joe",
"locations" : {
"my location" : {
"alerts" : [
{
"id" : 1,
"status" : "enabled"
},
{
"id" : 2,
"status" : "enabled"
}
]
},
"my other location" : {
"alerts" : [
{
"id" : 3,
"status" : null
},
{
"id" : 4,
"status" : null
}
]
}
}
}
I can then use:
update({"locations.my location.alerts.id":1},{$set: {"locations.my location.alerts.$.status": "enabled"}});
Problem is I cannot create indexes on the alert id :-(
it may be better of modelled as such, specially if an index on location and,or alerts.id is needed.
{
"_id" : ObjectId("5496a8ed49847b6cd7c7b350"),
"name" : "joe",
"location" : "myLocation",
"alerts" : [{
"id" : 1,
"status" : null
},
{
"id" : 2,
"status" : null
}
]
}
{
"_id" : ObjectId("5496a8ed49847b6cd7c7b350"),
"name" : "joe",
"location" : "otherLocation",
"alerts" : [{
"id" : 1,
"status" : null
},
{
"id" : 2,
"status" : null
}
]
}
I think you are having a wrong tool for the job. What you have in your example is relational data and it's much easier to handle with relational database. So I would suggest to use SQL-database instead of mongo.
But if you really want to do it with mongo, then I guess the only option is to fetch the document and modify it and put it back.