MongoDB - Dynamically update an object in nested array - json

I have a document like this:
{
Name : val
AnArray : [
{
Time : SomeTime
},
{
Time : AnotherTime
}
...arbitrary more elements
}
I need to update "Time" to a Date type (right now it is string)
I would like to do something psudo like:
foreach record in document.AnArray { record.Time = new Date(record.Time) }
I've read the documentation on $ and "dot" notation as well as a several similar questions here, I tried this code:
db.collection.update({_id:doc._id},{$set : {AnArray.$.Time : new Date(AnArray.$.Time)}});
And hoping that $ would iterate the indexes of the "AnArray" property as I don't know for each record the length of it. But am getting the error:
SyntaxError: missing : after property id (shell):1
How can I perform an update on each member of the arrays nested values with a dynamic value?

There's no direct way to do that, because MongoDB doesn't support an update-expression that references the document. Moreover, the $ operator only applies to the first match, so you'd have to perform this as long as there are still fields where AnArray.Time is of $type string.
You can, however, perform that update client side, in your favorite language or in the mongo console using JavaScript:
db.collection.find({}).forEach(function (doc) {
for(var i in doc.AnArray)
{
doc.AnArray[i].Time = new Date(doc.AnArray[i].Time);
}
db.outcollection.save(doc);
})
Note that this will store the migrated data in a different collection. You can also update the collection in-place by replacing outcollection with collection.

Related

How to access the key of a jsoncpp Value

I kind of feel stupid for asking this, but haven't been able to find a way to get the key of a JSON value. I know how to retrieve the key if I have an iterator of the object. I also know of operator[].
In my case the key is not a known value, so can't use get(const char *key) or operator[]. Also can't find a getKey() method.
My JSON looks like this:
{Obj_Array: [{"122":{"Member_Array":["241", "642"]}}]}
For the piece of code to parse {"122":{"Member_Array":["241", "642"]}} I want to use get_key()-like function just to retrieve "122" but seems like I have to use an iterator which to me seems to be overkill.
I might have a fundamental lack of understanding of how jsoncpp is representing a JSON file.
First, what you have won't parse in JsonCPP. Keys must always be enclosed in double quotes:
{"Obj_Array": [{"122":{"Member_Array":["241", "642"]}}]}
Assuming that was just an oversight, if we add whitespace and tag the elements:
{
root-> "Obj_Array" : [
elem0-> {
key0-> "122":
val0-> {
key0.1-> "Member_Array" :
val0.1-> [
elem0.1.0-> "241",
elem0.1.1-> "642" ]
}
}
]
}
Assuming you have managed to read your data into a Json::Value (let's call it root), each of the tagged values can be accessed like this:
elem0 = root[0];
val0 = elem0["122"]
val0_1 = val0["Member_Array"];
elem0_1_0 = val0_1[0];
elem0_1_1 = val0_1[1];
You notice that this only retrieves values; the keys were known a priori. This is not unusual; the keys define the schema of the data; you have to know them to directly access the values.
In your question, you state that this is not an option, because the keys are not known. Applying semantic meaning to unknown keys could be challenging, but you already came to the answer. If you want to get the key values, then you do have to iterate over the elements of the enclosing Json::Value.
So, to get to key0, you need something like this (untested):
elem0_members = elem0.getMemberNames();
key0 = elem0_members[0];
This isn't production quality, by any means, but I hope it points in the right direction.

How can I create an EMR cluster resource that uses spot instances without hardcoding the bid_price variable?

I'm using Terraform to create an AWS EMR cluster that uses spot instances as core instances.
I know I can use the bid_price variable within the core_instance_group block on a aws_emr_cluster resource, but I don't want to hardcode prices as I'd have to change them manually every time the instance type changes.
Using the AWS Web UI, I'm able to choose the "Use on-demand as max price" option. That's exactly what I'm trying to reproduce, but in Terraform.
Right now I am trying to solve my problem using the aws_pricing_product data source. You can see what I have so far below:
data "aws_pricing_product" "m4_large_price" {
service_code = "AmazonEC2"
filters {
field = "instanceType"
value = "m4.large"
}
filters {
field = "operatingSystem"
value = "Linux"
}
filters {
field = "tenancy"
value = "Shared"
}
filters {
field = "usagetype"
value = "BoxUsage:m4.large"
}
filters {
field = "preInstalledSw"
value = "NA"
}
filters {
field = "location"
value = "US East (N. Virginia)"
}
}
data.aws_pricing_product.m4_large_price.result returns a json containing the details of a single product (you can check the response of the example here). The actual on-demand price is buried somewhere inside this json, but I don't know how can I get it (image generated with http://jsonviewer.stack.hu/):
I know I might be able solve this by using an external data source and piping the output of an aws cli call to something like jq, e.g:
aws pricing get-products --filters "Type=TERM_MATCH,Field=sku,Value=8VCNEHQMSCQS4P39" --format-version aws_v1 --service-code AmazonEC2 | jq [........]
But I'd like to know if there is any way to accomplish what I'm trying to do with pure Terraform. Thanks in advance!
Unfortunately the aws_pricing_product data source docs don't expand on how it should be used effectively but the discussion in the pull request that added it adds some insight.
In Terraform 0.12 you should be able to use the jsondecode function to nicely get at what you want with the following given as an example in the linked pull request:
data "aws_pricing_product" "example" {
service_code = "AmazonRedshift"
filters = [
{
field = "instanceType"
value = "ds1.xlarge"
},
{
field = "location"
value = "US East (N. Virginia)"
},
]
}
# Potential Terraform 0.12 syntax - may change during implementation
# Also, not sure about the exact attribute reference architecture myself :)
output "example" {
values = jsondecode(data.json_query.example.value).terms.OnDemand.*.priceDimensions.*.pricePerUnit.USD
}
If you are stuck on Terraform <0.12 you might struggle to do this natively in Terraform other than the external data source approach you've already suggested.
#cfelipe put that ${jsondecode(data.aws_pricing_product.m4_large_price.value).terms.OnDemand.*.priceDimensions.*.pricePerUnit.USD}" in a Locals

What's the correct JsonPath expression to search a JSON root object using Newtonsoft.Json.NET?

Most examples deal with the book store example from Stefan Gössner, however I'm struggling to define the correct JsonPath expression for a simple object (no array):
{ "Id": 1, "Name": "Test" }
To check if this json contains Id = 1.
I tried the following expression: $..?[(#.Id == 1]), but this does find any matches using Json.NET?
Also tried Manatee.Json for parsing, and there it seems the jsonpath expression could be like $[?($.Id == 1)] ?
The path that you posted is not valid. I think you meant $..[?(#.Id == 1)] (some characters were out of order). My answer assumes this.
The JSON Path that you're using indicates that the item you're looking for should be in an array.
$ start
.. recursive search (1)
[ array item specification
?( item-based query
#.Id == 1 where the item is an object with an "Id" with value == 1 at the root
) end item-based query
] end array item specification
(1) the conditions following this could match a value no matter how deep in the hierarchy it exists
You want to just navigate the object directly. Using $.Id will return 1, which you can validate in your application.
All of that said...
It sounds to me like you want to validate that the Id property is 1 rather than to search an array for an object where the Id property is 1. To do this, you want JSON Schema, not JSON Path.
JSON Path is a query language for searching for values which meet certain conditions (e.g. an object where Id == 1.
JSON Schema is for validating that the JSON meet certain requirements (your data's in the right shape). A JSON Schema to validate that your object has a value of 1 could be something like
{
"properties": {
"Id": {"const":1}
}
}
Granted this isn't very useful because it'll only validate that the Id property is 1, which ideally should only be true for one object.

Azure tables unable to store flattened JSON

I am using the npm flat package, and arrays/objects are flattened, but object/array keys are surrounded by '' , like in 'task_status.0.data' using the object below.
These specific fields do not get stored into AzureTables - other fields go through, but these are silently ignored. How would I fix this?
var obj1 = {
"studentId": "abc",
"task_status": [
{
"status":"Current",
"date":516760078
},
{
"status":"Late",
"date":1516414446
}
],
"student_plan": "n"
}
Here is how I am using it - simplified code example: Again, it successfully gets written to the table, but does not write the properties that were flattened (see further below):
var flatten = require('flat')
newObj1 = flatten(obj1);
var entGen = azure.TableUtilities.entityGenerator;
newObj1.PartitionKey = entGen.String(uniqueIDFromMyDB);
newObj1.RowKey = entGen.String(uniqueStudentId);
tableService.insertEntity(myTableName, newObj1, myCallbackFunc);
In the above example, the flattened object would look like:
var obj1 = {
studentId: "abc",
'task_status.0.status': 'Current',
'task_status.0.date': 516760078,
'task_status.1.status': 'Late',
'task_status.1.date': 516760078,
student_plan: "n"
}
Then I would add PartitionKey and RowKey.
all the task_status fields would silently fail to be inserted.
EDIT: This does not have anything to do with the actual flattening process - I just checked a perfectly good JSON object, with keys that had 'x.y.z' in it, i.e. AzureTables doesn't seem to accept these column names....which almost completely destroys the value proposition of storing schema-less data, without significant rework.
. in column name is not supported. You can use a custom delimiter to flatten your objects instead.
For example:
newObj1 = flatten(obj1, {delimiter: '__'});

Append value to JSON decode array parameter stored in MySQL

Im trying to work out how to append a zero to a specific JSON decoded array value for multiple records stored in a MySQL table according to some conditions.
for example, for table 'menu', column 'params'(text) have records containing JSON decoded arrays of this format:
{"categories":["190"],"singleCatOrdering":"","menu-anchor_title":""}
and column 'id' has a numeric value of 90.
my goal is to add a zero to 'categories' value in menu.params whenever (for example) menu.id is under 100.
for this records the result being
{"categories":["1900"],"singleCatOrdering":"","menu-anchor_title":""}
so im looking for a SQL Query that will search and find the occurrences of "categories": ["999"] in the Database and update the record by adding a zero to the end of the value.
this answer is partially helpful by offering to use mysql-udf-regexp but its referring to REPLACE a value and not UPDATE it.
perhaps the REGEXP_REPLACE? function will do the trick. i have never used this library and am not familiar with it, perhaps there is an easier way to achieve what i need ?
Thanks
If I understand your question correctly, you want code that does something like this:
var data = {
"menu": {
"id": 90,
"params": {
"categories": ["190"],
"singleCatOrdering": "",
"menu-anchor_title": ""
}
}
};
var keys = Object.keys(data);
var columns;
for (var ii = 0, key; key = keys[ii]; ii++) {
value = data[key];
if (value.id < 100) {
value.params.categories[0] += "0";
alert(value.params.categories[0]);
}
}
jsFiddle
However, I am not using a regular expression at all. Perhaps if you reword the question, the necessity of a regex will become clearer.