I am using the npm flat package, and arrays/objects are flattened, but object/array keys are surrounded by '' , like in 'task_status.0.data' using the object below.
These specific fields do not get stored into AzureTables - other fields go through, but these are silently ignored. How would I fix this?
var obj1 = {
"studentId": "abc",
"task_status": [
{
"status":"Current",
"date":516760078
},
{
"status":"Late",
"date":1516414446
}
],
"student_plan": "n"
}
Here is how I am using it - simplified code example: Again, it successfully gets written to the table, but does not write the properties that were flattened (see further below):
var flatten = require('flat')
newObj1 = flatten(obj1);
var entGen = azure.TableUtilities.entityGenerator;
newObj1.PartitionKey = entGen.String(uniqueIDFromMyDB);
newObj1.RowKey = entGen.String(uniqueStudentId);
tableService.insertEntity(myTableName, newObj1, myCallbackFunc);
In the above example, the flattened object would look like:
var obj1 = {
studentId: "abc",
'task_status.0.status': 'Current',
'task_status.0.date': 516760078,
'task_status.1.status': 'Late',
'task_status.1.date': 516760078,
student_plan: "n"
}
Then I would add PartitionKey and RowKey.
all the task_status fields would silently fail to be inserted.
EDIT: This does not have anything to do with the actual flattening process - I just checked a perfectly good JSON object, with keys that had 'x.y.z' in it, i.e. AzureTables doesn't seem to accept these column names....which almost completely destroys the value proposition of storing schema-less data, without significant rework.
. in column name is not supported. You can use a custom delimiter to flatten your objects instead.
For example:
newObj1 = flatten(obj1, {delimiter: '__'});
Related
I'm creating an AWS Step Function definition in Dhall. However, I don't know how to create a common structure they use for Choice states such as the example below:
{
"Not": {
"Variable": "$.type",
"StringEquals": "Private"
},
"Next": "Public"
}
The Not is pretty straightforward using mapKey and mapValue. If I define a basic Comparison:
{ Type =
{ Variable : Text
, StringEquals : Optional Text
}
, default =
{ Variable = "foo"
, StringEquals = None Text
}
}
And the types:
let ComparisonType = < And | Or | Not >
And adding a helper function to render the type as Text for the mapKey:
let renderComparisonType = \(comparisonType : ComparisonType )
-> merge
{ And = "And"
, Or = "Or"
, Not = "Not"
}
comparisonType
Then I can use them in a function to generate the record halfway:
let renderRuleComparisons =
\( comparisonType : ComparisonType ) ->
\( comparisons : List ComparisonOperator.Type ) ->
let keyName = renderComparisonType comparisonType
let compare = [ { mapKey = keyName, mapValue = comparisons } ]
in compare
If I run that using:
let rando = ComparisonOperator::{ Variable = "$.name", StringEquals = Some "Cow" }
let comparisons = renderRuleComparisons ComparisonType.Not [ rando ]
in comparisons
Using dhall-to-json, she'll output the first part:
{
"Not": {
"Variable": "$.name",
"StringEquals": "Cow"
}
}
... but I've been struggling to merge that with "Next": "Sup". I've used all the record merges like /\, //, etc. and it keeps giving me various type errors I don't truly understand yet.
First, I'll include an approach that does not type-check as a starting point to motivate the solution:
let rando = ComparisonOperator::{ Variable = "$.name", StringEquals = Some "Cow" }
let comparisons = renderRuleComparisons ComparisonType.Not [ rando ]
in comparisons # toMap { Next = "Public" }
toMap is a keyword that converts records to key-value lists, and # is the list concatenation operator. The Dhall CheatSheet has a few examples of how to use both of them.
The above solution doesn't work because # cannot merge lists with different element types. The left-hand side of the # operator has this type:
comparisons : List { mapKey : Text, mapValue : Comparison.Type }
... whereas the right-hand side of the # operator has this type:
toMap { Next = "Public" } : List { mapKey : Text, mapValue : Text }
... so the two Lists cannot be merged as-is due to the different types for the mapValue field.
There are two ways to resolve this:
Approach 1: Use a union whenever there is a type conflict
Approach 2: Use a weakly-typed JSON representation that can hold arbitrary values
Approach 1 is the simpler solution for this particular example and Approach 2 is the more general solution that can handle really weird JSON schemas.
For Approach 1, dhall-to-json will automatically strip non-empty union constructors (leaving behind the value they were wrapping) when translating to JSON. This means that you can transform both arguments of the # operator to agree on this common type:
List { mapKey : Text, mapValue : < State : Text | Comparison : Comparison.Type > }
... and then you should be able to concatenate the two lists of key-value pairs and dhall-to-json will render them correctly.
There is a second solution for dealing with weakly-typed JSON schemas that you can learn more about here:
Dhall Manual - How to convert an existing YAML configuration file to Dhall
The basic idea is that all of the JSON/YAML integrations recognize and support a weakly-typed JSON representation that can hold arbitrary JSON data, including dictionaries with keys of different shapes (like in your example). You don't even need to convert the entire the expression to this weakly-typed representation; you only need to use this representation for the subset of your configuration where you run into schema issues.
What this means for your example, is that you would change both arguments to the # operator to have this type:
let Prelude = https://prelude.dhall-lang.org/v12.0.0/package.dhall
in List { mapKey : Text, mapValue : Prelude.JSON.Type }
The documentation for Prelude.JSON.Type also has more details on how to use this type.
Im trying to work out how to append a zero to a specific JSON decoded array value for multiple records stored in a MySQL table according to some conditions.
for example, for table 'menu', column 'params'(text) have records containing JSON decoded arrays of this format:
{"categories":["190"],"singleCatOrdering":"","menu-anchor_title":""}
and column 'id' has a numeric value of 90.
my goal is to add a zero to 'categories' value in menu.params whenever (for example) menu.id is under 100.
for this records the result being
{"categories":["1900"],"singleCatOrdering":"","menu-anchor_title":""}
so im looking for a SQL Query that will search and find the occurrences of "categories": ["999"] in the Database and update the record by adding a zero to the end of the value.
this answer is partially helpful by offering to use mysql-udf-regexp but its referring to REPLACE a value and not UPDATE it.
perhaps the REGEXP_REPLACE? function will do the trick. i have never used this library and am not familiar with it, perhaps there is an easier way to achieve what i need ?
Thanks
If I understand your question correctly, you want code that does something like this:
var data = {
"menu": {
"id": 90,
"params": {
"categories": ["190"],
"singleCatOrdering": "",
"menu-anchor_title": ""
}
}
};
var keys = Object.keys(data);
var columns;
for (var ii = 0, key; key = keys[ii]; ii++) {
value = data[key];
if (value.id < 100) {
value.params.categories[0] += "0";
alert(value.params.categories[0]);
}
}
jsFiddle
However, I am not using a regular expression at all. Perhaps if you reword the question, the necessity of a regex will become clearer.
Im new to angularJS and web designing as a whole. Im trying to get a data field(or element) from a JSON. For example, this is what the JSON looks like
{
"Name":"Raymond Eugene Monce",
"Dateofbirth":"1924-0308T00:00:00Z",
"Ethnicity":"Caucasian",
"Languages":["{English}"],
},
and I'm trying to get the "Name" data field. This is what my .js file looks like,
var profile = angular.module('profile', ['ui.bootstrap','ngResource']);
profile.controller("profileController", ["$scope","$resource", function($scope, $resource) {
// get the user id
$scope.userid = sessionStorage["cerestiuserid"];
// json we get from server
$scope.apicall = sessionStorage["cerestihome"]; // NEED TO CHANGE API
// grabs the user we want
$scope.userResource = $resource($scope.apicall + "/api/userprofile/",
{Userid:21},
{'get':{method: 'POST'}}
);
// fetch JSON
$scope.userResource.get(function(result) {
// get the name field
$scope.name = result;
sessionStorage["name"] = JSON.stringify(result);
});
and my .html file,
<div ng-controller = "profileController" style="float:left">
<!-- profile pic -->
<div class="pull-left">
<div class="container-fluid">
<div class="profile">
<div class="row">
<div class="center-block">
<div class="profilePic">
<img ng-src="{{profilePic()}}" class="img-responsive">
<!-- name field -->
<label class="caption">
<h4>{{name.name}}</h4>
</label>
</div>
Again, Im not having problems with the Database or API calls. I just want to know how I can get and display the name field of the JSON. Thanks.
strelok2010's comment above should work although that depends on if your result really looks like the one defined at the top of your question.
Your result seems to be a normal javascript object not JSON. (yeah they are different, and that confused me when I learned it.) I assume that because you stringify the result from a javascript object into JSON. Therefore if that is working right your result is either a javascript object or an array of javascript objects. I'm assuming an array. You might want to check though.
I noticed your earlier post had a related problem.
In that one you were asking to access a property of an object that was in an array. In that case it was result as well. Here was the answer from your previous question
var result = [{"name": "Jason"
"date of birth": "february 23, 2985"
....
}];
var firstResultsName = result[0].name;
There are two things I am unsure of due to the inconsistency between this and your last question.
First your name property in your results object is spelled with a capital N here as opposed to a lower case n in your last question.
Keep in mind that capitilization matters in javascript.
Second your result in your last question was an array of objects and in this it seems to be just an object.
So depending on which one it is will determine your solution. So instead of writing every possible solution I'll show you how to determine the solution.
Remember we are dealing with a normal array of javascript objects. I'll try to go into detail so it's extra clear (sorry I heard you were new to web developement, I'm assuming JavaScript too.), but sorry if it's a little too detailed. I will also be breaking it into parts to go deeper into the array of objects that I'll use in my example, but traversing into the data structure can all be done in a single line as I will show.
You can only do actions on the 'outermost-form' (by the way 'outermost-form' is just a term I'll use for clarification it's not really a technical term.) and work your way into the collection (object/array/string)
As an example we have an array of people, with the array being the 'outermost-form'
var people = [
{
"name": "Bob",
"occupation": "Architect",
"date of birth": "01/23/83"
},
{
"name": "Timothy",
"Occupation": "Accountant",
"date of birth": "02/23/78"
}
];
If we saw the value of people at this moment it not surprisingly be.
[
{
"name": "Bob",
"occupation": "Architect",
"date of birth": "01/23/83"
},
{
"name": "Timothy",
"Occupation": "Accountant",
"date of birth": "02/23/78"
}
]
Start with the Array
Since it's an array as the 'outermost-form' we can get one of its values using an index. Just like any other array. Just for a bit of contrast I'll show you how what we are doing is similar to any other array by showing an example of an array by itself
// simple array example
var array = ["foo", "bar", "baz"];
array[0] // returns "foo"
// more simple array example, but less practical (it's more just for showing how javascript can work.)
["foo", "bar", "baz"][2] // returns "baz"
Back to our main example. Let's make a variable person and store our first person in the people array in that value.
var person = people[0];
Now if saw our person variable it would equal the following
{
"name": "Bob",
"occupation": "Architect",
"date of birth": "01/23/83"
}
You can see just like the normal array it grabs the first item in the array. You can see how we are slowly traversing into our people data structure. (that being an array of objects.)
Enter the Object
Okay so now we have the person object, but we want the name of that person so since we are dealing with an object we have to access its properties we can do this with either 'dot notation', e.g. <object>.<property>, or 'bracket notation' which can be done with either a variable or a string for the property name. e.g. <object>.["<property>"] or <object>.[<variable>]
So just as a side example I will show you what it normally takes to get the value of a property of an object just so you can compare and see there's no 'magic' going on. Keep in mind javascript is case-sensitive. Also javascript objects properties can go with or without surrounding quotes unlike JSON. One last thing having a space in the property name forces us to use quotes, and also forces us to access that property via bracket notation.
var result;
var obj = { foo: 1, Bar: 2, "foo bar": 3 };
var randomVarName = "Bar"; // notice the capital B in Bar is important since it was declared that way.
result = obj.foo; // result equals 1
result = obj[randomVarName]; // result equals 2
result = obj["foo bar"]; // result equals 3
Back again to our main train of thought. So we have traversed into our people array to find the person object now let's get their name.
var name = person.name;
The value of name would be.
"Bob"
You can do with that what you wish. You could have also used any of the previous ways to get an objects property including bracket notation.
Do Everything we just did in a Single Line
So to write that all in one line you would just write
people[0].name
Apply to your Question
So to apply to your question if your result looks like this
var result = [
{
"name": "Jason"
"date of birth": "february 23, 2985"
....
}
];
Then you need this to get the name
result[0].name
If it's just this
var result = {
"name": "Jason"
"date of birth": "february 23, 2985"
....
}
Then you just need
result.name
As asked in the comment if you want to get the date of birth property out of the object you need to use bracket notation to get the element out of an object. Bracket notation is one of the two object property accessors the other being dot notation. I covered both at the enter the object section. It can be used at anytime, but is usable in some cases that dot notation does not work.
An example and quote from MDN:
get = object[property_name];
object[property_name] = set;
property_name is a string. The string does not have to be a valid identifier; > it can have any value, e.g. "1foo", "!bar!", or even " " (a space).
So since certain character like spaces can't be used in dot notation bracket notation must be used in those special cases when those characters are present.
Below is the bracket notation of the date of birth.
result["date of birth"]
Like I said before it can be used anywhere, but generally dot notation is preferred for its brevity. So just to show that, we will show the name field being accessed using bracket notation:
result["name"]
One additional reason you may want to use bracket notation is for its ability to use variables like so.
var prop_name = "date of birth";
result[prop_name];
which actually if you understand the principle of that example the MDN example might make more sense.
If you have a question feel free to leave me a comment.
I have a document like this:
{
Name : val
AnArray : [
{
Time : SomeTime
},
{
Time : AnotherTime
}
...arbitrary more elements
}
I need to update "Time" to a Date type (right now it is string)
I would like to do something psudo like:
foreach record in document.AnArray { record.Time = new Date(record.Time) }
I've read the documentation on $ and "dot" notation as well as a several similar questions here, I tried this code:
db.collection.update({_id:doc._id},{$set : {AnArray.$.Time : new Date(AnArray.$.Time)}});
And hoping that $ would iterate the indexes of the "AnArray" property as I don't know for each record the length of it. But am getting the error:
SyntaxError: missing : after property id (shell):1
How can I perform an update on each member of the arrays nested values with a dynamic value?
There's no direct way to do that, because MongoDB doesn't support an update-expression that references the document. Moreover, the $ operator only applies to the first match, so you'd have to perform this as long as there are still fields where AnArray.Time is of $type string.
You can, however, perform that update client side, in your favorite language or in the mongo console using JavaScript:
db.collection.find({}).forEach(function (doc) {
for(var i in doc.AnArray)
{
doc.AnArray[i].Time = new Date(doc.AnArray[i].Time);
}
db.outcollection.save(doc);
})
Note that this will store the migrated data in a different collection. You can also update the collection in-place by replacing outcollection with collection.
I see a lot of references to "compressed JSON" when it comes to different serialization formats. What exactly is it? Is it just gzipped JSON or something else?
Compressed JSON removes the key:value pair of json's encoding to store keys and values in seperate parallel arrays:
// uncompressed
JSON = {
data : [
{ field1 : 'data1', field2 : 'data2', field3 : 'data3' },
{ field1 : 'data4', field2 : 'data5', field3 : 'data6' },
.....
]
};
//compressed
JSON = {
data : [ 'data1','data2','data3','data4','data5','data6' ],
keys : [ 'field1', 'field2', 'field3' ]
};
This method of usage i found here
Content from link (http://www.nwhite.net/?p=242)
rarely find myself in a place where I am writing javascript applications that use AJAX in its pure form. I have long abandoned the ‘X’ and replaced it with ‘J’ (JSON). When working with Javascript, it just makes sense to return JSON. Smaller footprint, easier parsing and an easier structure are all advantages I have gained since using JSON.
In a recent project I found myself unhappy with the large size of my result sets. The data I was returning was tabular data, in the form of objects for each row. I was returning a result set of 50, with 19 fields each. What I realized is if I augment my result set I could get a form of compression.
// uncompressed
JSON = {
data : [
{ field1 : 'data1', field2 : 'data2', field3 : 'data3' },
{ field1 : 'data4', field2 : 'data5', field3 : 'data6' },
.....
]
};
//compressed
JSON = {
data : [ 'data1','data2','data3','data4','data5','data6' ],
keys : [ 'field1', 'field2', 'field3' ]
};
I merged all my values into a single array and store all my fields in a separate array. Returning a key value pair for each result cost me 8800 byte (8.6kb). Ripping the fields out and putting them in a separate array cost me 186 bytes. Total savings 8.4kb.
Now I have a much more compressed JSON file, but the structure is different and now harder to work with. So I implement a solution in Mootools to make the decompression transparent.
Request.JSON.extend({
options : {
inflate : []
}
});
Request.JSON.implement({
success : function(text){
this.response.json = JSON.decode(text, this.options.secure);
if(this.options.inflate.length){
this.options.inflate.each(function(rule){
var ret = ($defined(rule.store)) ? this.response.json[rule.store] : this.response.json[rule.data];
ret = this.expandData(this.response.json[rule.data], this.response.json[rule.keys]);
},this);
}
this.onSuccess(this.response.json, text);
},
expandData : function(data,keys){
var arr = [];
var len = data.length; var klen = keys.length;
var start = 0; var stop = klen;
while(stop < len){
arr.push( data.slice(start,stop).associate(keys) );
start = stop; stop += klen;
}
return arr;
}
});
Request.JSON now has an inflate option. You can inflate multiple segments of your JSON object if you so desire.
Usage:
new Request.JSON({
url : 'url',
inflate : [{ 'keys' : 'fields', 'data' : 'data' }]
onComplete : function(json){}
});
Pass as many inflate objects as you like to the option inflate array. It has an optional property called ’store’ If set the inflated data set will be stored in that key instead.
The ‘keys’ and ‘fields’ expect strings to match a location in the root of your JSON object.
Based in Paniyar's answer, we can convert a List of Objects in "compressed" Json format using C# like this:
var JsonString = serializer.Serialize(
new
{
cols = new[] { "field1", "field2", "field3"},
items = data.Select(x => new object[] {x.field1, x.field2, x.field3})
});
I used an array of object because the fields can be int, bool, string...
More Reduction:
If the field is repeated very often and it is a string type, you can get compressed a little be more if you add a distinct list of that field... for instance, a field name job position, city, etc are excellent candidate for this. You can add a distinct list of this items and in each item change the value for a reference number. That will make your Json more lite.
Compressed:
[["KeyA", "KeyB", "KeyC", "KeyD", "KeyE", "KeyF"],
["ValA1", "ValB1", "ValC1", "ValD1", "ValE1", "ValF1"],
["ValA2", "ValB2", "ValC2", "ValD2", "ValE2", "ValF2"],
["ValA3", "ValB3", "ValC3", "ValD3", "ValE3", "ValF3"],
["ValA4", "ValB4", "ValC4", "ValD4", "ValE4", "ValF4"]]
Uncompressed:
[{KeyA: "ValA1", KeyB: "ValB1", KeyC: "ValC1", KeyD: "ValD1", KeyE: "ValE1", KeyF: "ValF1"},
{KeyA: "ValA2", KeyB: "ValB2", KeyC: "ValC2", KeyD: "ValD2", KeyE: "ValE2", KeyF: "ValF2"},
{KeyA: "ValA3", KeyB: "ValB3", KeyC: "ValC3", KeyD: "ValD3", KeyE: "ValE3", KeyF: "ValF3"},
{KeyA: "ValA4", KeyB: "ValB4", KeyC: "ValC4", KeyD: "ValD4", KeyE: "ValE4", KeyF: "ValF4"}]
The most likely answer is that it really is just gzipped JSON. There is no other standard meaning to this phrase.
Re-organizing a homogenous array of JSON objects into a pair of arrays is a very useful technique to make the payload smaller and to speed up encoding and decoding, it is not commonly called "compressed JSON". I haven't run across it ever in open source or any open API, but we use this technique internally and call it "jsontable".