I have a storage for a community set up in Firebase. As I have a class defined in my Swift project I need to to know whether it´s an Array or a Dictionary when generating an object from it.
I downloaded the json File and it looks like this - 2 users are stored in different data formats in firebase for the same "table":
[{"user": {
"User1": {
"adminOf": {
"3": true,
"5": true
},
"alias": "borussenpeter",
"communities": {
"3": true,
"5": true
}
},
"User2": {
"adminOf": [null, true, true, null, true],
"alias": "burkart",
"communities": [null, true, true, null, true]
}
}}]
I tried downloading the file, edit it so both users are looking the same and uploaded it again, but Firebase saves it again this way.
Of course initialising the object fails when using the wrong data type. Any thoughts on that? Thanks
The answer is in the Firebase documentation on arrays:
However, to help developers that are storing arrays in a Firebase database,... if the data looks like an array, Firebase clients will render it as an array. In particular, if all of the keys are integers, and more than half of the keys between 0 and the maximum key in the object have non-empty values, then Firebase clients will render it as an array.
In the JSON for User1 you have 2 values (3 and 5) for 5 indices. That is less than half of the keys, so Firebase doesn't render it as an array.
In the JSON for User2 you have 3 values (2, 3 and 5) for 5 indices. That is more than half of the keys, so Firebase renders it as an array.
A few ways to deal with this:
prefix the integers with a string, e.g. "group3": true.
add a dummy non-integer to the "array": e.g. "NOTUSED": false.
store values for all indices: e.g. "group0": false
In general:
[ ] = array
{ } = dictionary
How the data appears in Firebase is directly related to how it was stored in Firebase.
My guess is that when user 2 was written to firebase, the keys were missing so it wrote them as an array.
Related
I have a requirement to convert the json into csv(or a SQL table) or any other flatten structure using Data Flow in Azure Data Factory. I need to take the property names at some hierarchy and values of the child properties at lower of hierrarchy from the source json and add them both as column/row values in csv or any other flatten structure.
Source Data Rules/Constraints :
Parent level data property names will change dynamically (e.g. ABCDataPoints,CementUse, CoalUse, ABCUseIndicators names are dynamic)
The hierarchy always remains same as in below sample json.
I need some help in defining Json path/expression to get the names ABCDataPoints,CementUse, CoalUse, ABCUseIndicators etc. I am able to figure out how to retrieve the values for the properties Value,ValueDate,ValueScore,AsReported.
Source Data Structure :
{
"ABCDataPoints": {
"CementUse": {
"Value": null,
"ValueDate": null,
"ValueScore": null,
"AsReported": [],
"Sources": []
},
"CoalUse": {
"Value": null,
"ValueDate": null,
"AsReported": [],
"Sources": []
}
},
"ABCUseIndicators": {
"EnvironmentalControversies": {
"Value": false,
"ValueDate": "2021-03-06T23:22:49.870Z"
},
"RenewableEnergyUseRatio": {
"Value": null,
"ValueDate": null,
"ValueScore": null
}
},
"XYZDataPoints": {
"AccountingControversiesCount": {
"Value": null,
"ValueDate": null,
"AsReported": [],
"Sources": []
},
"AdvanceNotices": {
"Value": null,
"ValueDate": null,
"Sources": []
}
},
"XYXIndicators": {
"AccountingControversies": {
"Value": false,
"ValueDate": "2021-03-06T23:22:49.870Z"
},
"AntiTakeoverDevicesAboveTwo": {
"Value": 4,
"ValueDate": "2021-03-06T23:22:49.870Z",
"ValueScore": "0.8351945854483925"
}
}
}
Expected Flatten structure
Background:
After having multiple calls with ADF experts at Microsoft(Our workplace have Microsoft/Azure partnership), they concluded this is not possible with out of the box activities provided by ADF as is, neither by Dataflow(need not to use data flow though) nor Flatten feature. Reasons are Dataflow/Flatten only unroll the Array objects and there are no mapping functions available to pick the property names - Custom expression are in internal beta testing and will in PA in near future.
Conclusion/Solution:
We concluded with an agreement based on calls with Microsoft emps ended up to go multiple approaches but both needs the custom code - with out custom code this is not possible by using out of box activities.
Solution-1 : Use some code to flatten as per requirement using a ADF Custom Activity. The downside of this you need to use an external compute(VM/Batch), the options supported are not on-demand. So it is little bit expensive but works best if have continuous stream workloads. This approach also continuously monitor if input sources are of different sizes because the compute needs to be elastic in this case or else you will get out of memory exceptions.
Solution-2 : Still needs to write the custom code - but in a function app.
Create a Copy Activity with source as the files with Json content(preferably storage account).
Use target as Rest Endpoint of function(Not as a function activity because it has 90sec timeout when called from an ADF activity)
The function app will takes Json lines as input and parse and flatten.
If you use the above way so you can scale the number of lines cane be send in each request to function and also scale the parallel requests.
The function will do the flatten as required to one file or multiple files and store in blob storage.
The pipeline will continue from there as needed from there.
One problem with this approach is if any of the range is failed the copy activity will retry but it will run the whole process again.
Trying something very similar, is there any other / native solution to address this?
As mentioned in the response above, has this been GA yet? If yes, any reference documentation / samples would be of great help!
Custom expression are in internal beta testing and will in PA in near future.
I'm building an app (Swift) where the user can select a CSV file, containing bank transactions.
I want to parse this CSV to my Firebase Realtime database.
The input CSV file would be:
amount, label
111, Uber Eats
1678, iTunes
The output on Realtime database would be:
{
"user ID 1" : {
"-M5wUNXgmTuBgZpvT0v-" : {
"amount" : 111,
"label" : "Uber Eats"
},
"-M5wUQk4wihb3OxcQ7SX" : {
"amount" : 1678,
"label" : "iTunes"
}
},
"user ID 2" : {
"-M5wUNXgmTuBgZpvT0k-" : {
"amount" : 111,
"label" : "Deliveroo"
}
}
}
In this example, I am "user ID 1" and I uploaded two transactions from the CSV file.
I can't figure out how to mass-write these lines into Firebase in one shot.
I've tried to parse the CSV into multiple dictionaries and to write in Firebase as an array:
let myParsedCSVasArray = [
["amount": 111,
"label": "Uber Eats"],
["amount": 1678,
"label": "iTunes"]
]
self.ref.child(user!.uid).childByAutoId().setValue(myParsedCSVasArray)
But the result doesn't fit my needs, as it creates an array inside the JSON:
Result of the previous code into Firebase realtime database
Any idea how I could upload multiple dictionaries at once, and add a childByAutoId to each of them?
You can solve it using a for loop by enumerating through the collection like below
let myParsedCSVasArray = [
["amount": 111,
"label": "Uber Eats"],
["amount": 1678,
"label": "iTunes"]
]
for value in myParsedCSVasArray {
ref.child(user!.uid).childByAutoId().setValue(value)
}
For reference on loops in Swift refer this link
I use Dart not Swift so I cannot easily show you correct code. I can describe the principle though.
In order to guarantee that all the data gets written, you have to do the set or update as a single database action. Therefore you need to build the dictionary (map in my terms) that contains all your records. Each record has to have the unique key generated by Firebase.
You need to use a loop to build this dictionary. In the loop, for each record you need to get a unique key by using code similar to this let key = ref.child("posts").childByAutoId().key.
Each call to this will return a new unique key.
When the dictionary is complete you add it as one atomic update (it either all works or all fails).
This Swift code seems to do that for one record so you should be able to use it as the basis for the loop:
guard let key = ref.child("posts").childByAutoId().key else { return }
let post = ["uid": userID,
"author": username,
"title": title,
"body": body]
let childUpdates = ["/posts/\(key)": post,
"/user-posts/\(userID)/\(key)/": post]
ref.updateChildValues(childUpdates)
Hope that helps.
See this link for more info: https://firebase.google.com/docs/database/ios/read-and-write
I am getting JSON returned in this format:
{
"status": "success",
"data": {
"debtor": {
"debtor_id": 1301,
"key": value,
"key": value,
"key": value
}
}
}
Somehow, my RESTAdapter needs to provide my debtor model properties from "debtor" section of the JSON.
Currently, I am getting a successful call back from the server, but a console error saying that Ember cannot find a model for "status". I can't find in the Ember Model Guide how to deal with JSON that is nested like this?
So far, I have been able to do a few simple things like extending the RESTSerializer to accept "debtor_id" as the primaryKey, and also remove the pluralization of the GET URL request... but I can't find any clear guide to reach a deeply nested JSON property.
Extending the problem detail for clarity:
I need to somehow alter the default behavior of the Adapter/Serializer, because this JSON convention is being used for many purposes other than my Ember app.
My solution thus far:
With a friend we were able to dissect the "extract API" (thanks #lame_coder for pointing me to it)
we came up with a way to extend the serializer on a case-by-case basis, but not sure if it really an "Ember Approved" solution...
// app/serializers/debtor.js
export default DS.RESTSerializer.extend({
primaryKey: "debtor_id",
extract: function(store, type, payload, id, requestType) {
payload.data.debtor.id = payload.data.debtor.debtor_id;
return payload.data.debtor;
}
});
It seems that even though I was able to change my primaryKey for requesting data, Ember was still trying to use a hard coded ID to identify the correct record (rather than the debtor_id that I had set). So we just overwrote the extract method to force Ember to look for the correct primary key that I wanted.
Again, this works for me currently, but I have yet to see if this change will cause any problems moving forward....
I would still be looking for a different solution that might be more stable/reusable/future-proof/etc, if anyone has any insights?
From description of the problem it looks like that your model definition and JSON structure is not matching. You need to make it exactly same in order to get it mapped correctly by Serializer.
If you decide to change your REST API return statement would be something like, (I am using mock data)
//your Get method on service
public object Get()
{
return new {debtor= new { debtor_id=1301,key1=value1,key2=value2}};
}
The json that ember is expecting needs to look like this:
"debtor": {
"id": 1301,
"key": value,
"key": value,
"key": value
}
It sees the status as a model that it needs to load data for. The next problem is it needs to have "id" in there and not "debtor_id".
If you need to return several objects you would do this:
"debtors": [{
"id": 1301,
"key": value,
"key": value,
"key": value
},{
"id": 1302,
"key": value,
"key": value,
"key": value
}]
Make sense?
I am new to sails.js and mongodb. I found something strange. When i use rest api to update the record in mongodb, after record updated, the json format changed.
For example. Originally I have a record like this:
{
creator: "John",
taskname: "test",
id: "53281a5d709602dc17b000cd"
}
After clicking http://127.xxx:1337/testtask/update/53281a5d709602dc17b000cd?creator=default%20creator, following json returned.
The json field is sorted in alphabetic order.
How can i keep the origin format of the json file? Is it a bug? Is there any workaround?
{
createdAt: "2014-03-18T10:05:17.052Z",
creator: "default creator",
taskname: "test",
updatedAt: "2014-03-18T10:08:53.067Z",
id: "53281a5d709602dc17b000cd"
}
Thanks.
The problem is fields in JSON objects don't have any concept of order. A JSON object is a dictionary, or in other words just some key/value pairs. This means that this JSON:
{ "a" : "some string", "b" : "other string" }
is logically equivalent to this JSON:
{ "b" : "other string", "a" : "some string" }
If you want to preserve ordering in your JSON data there are other ways to do it. For example JSON arrays do preserve order so something like this would work:
[ { "a" : "some string" }, { "b" : "other string" } ]
Internally MongoDB may actually preserve the ordering, but that's an implementation detail and you can't depend on it.
More detail on what Mongo is doing here.
Much like the "other" framework that inspired this, there is some automatic time-stamp generation happening in your models when things are updated or created. You wouldn't be the first. Ruby people have been trapped by this for years.
There are options you can define on your collection objects to remove these fields. This comes from the Waterline documentation, which should be the manager in use.
So in addition to attributes:
autoCreatedAt: false,
autoUpdatedAt: false,
attributes: {
// normal things here
},
Of course you will need to remove any of these properties that have been created in your documents manually. See the $unset operator for doing this
I have a couple of documents in my couchdb database which all are like:
{
"_id":"1234567890",
"name": [
"category1": 5,
"category2": 8
]
}
I want to have a map function which gives me "category1" as key and 5 as value.
I just can't separate "category" and 5.
I tried the following:
function(doc){
if(doc.name){
doc.name.forEach(function(sth) {
emit(sth, null);
});
}
}
Maybe someone can help me find a solution?
Mind that:
CouchDB function emit() takes two parameters, first is a key (eg. "category1"), second is a value (eg. 5). See the view API documented in the CouchDB wiki.
[key:val, ...] is not correct JSON field syntax, did you mean list of single field records ([{"key1": val1}, {"key2": val2}, ...]) or single multi-field record ({"key1": val1, "key2": val2, ...})?
In case of "name": [{ "category1": 5}, {"category2": 8 }] argument of forEach continuation is {"category1": 5}, and you should get somehow "category1" and 5 separately (forEach() or for (var key in sth) {} once more??).
In case of "name": { "category1": 5, "category2": 8 }, remember the JS object ({"field": val, ...}) does not have forEach method, array ([val1, val2, ...]) prototype does.
Assuming you meant "name": { "category1": 5, "category2": 8 } as the name, consider trying the following map:
function (data) {
if (date.name) {
for (var sth in name) {
emit(sth, name[sth]);
}
}
}
If you are JavaScript beginer, try writing proper code in console first. You can use the one found in the web-browser (Firefox, Chrome have build-in, I advise FireBug addon for Firefox). Good luck!