Importing JSON into Firebase Firestore - json

BEFORE ANYONE MARKS THIS AS DUPLICATE, I AM AWARE OF:
How to import CSV or JSON to firebase cloud firestore
As in the question above, I am trying to send JSON to Firestore using a Google Cloud Function, allowing me to convert my realtime database to Firestore. I have been using the script provided as an answer by Maciej Kaputa in the link above. The script he provides is as follows, which is exactly the same as that provided in the link above:
const admin = require('../functions/node_modules/firebase-admin');
const serviceAccount = require("./service-key.json");
admin.initializeApp({
credential: admin.credential.cert(serviceAccount),
databaseURL: "https://<your-database-name>.firebaseio.com"
});
const data = require("./fakedb.json");
/**
* Data is a collection if
* - it has a odd depth
* - contains only objects or contains no objects.
*/
function isCollection(data, path, depth) {
if (
typeof data != 'object' ||
data == null ||
data.length === 0 ||
isEmpty(data)
) {
return false;
}
for (const key in data) {
if (typeof data[key] != 'object' || data[key] == null) {
// If there is at least one non-object item then it data then it cannot be collection.
return false;
}
}
return true;
}
// Checks if object is empty.
function isEmpty(obj) {
for(const key in obj) {
if(obj.hasOwnProperty(key)) {
return false;
}
}
return true;
}
async function upload(data, path) {
return await admin.firestore()
.doc(path.join('/'))
.set(data)
.then(() => console.log(`Document ${path.join('/')} uploaded.`))
.catch(() => console.error(`Could not write document ${path.join('/')}.`));
}
/**
*
*/
async function resolve(data, path = []) {
if (path.length > 0 && path.length % 2 == 0) {
// Document's length of path is always even, however, one of keys can actually be a collection.
// Copy an object.
const documentData = Object.assign({}, data);
for (const key in data) {
// Resolve each collection and remove it from document data.
if (isCollection(data[key], [...path, key])) {
// Remove a collection from the document data.
delete documentData[key];
// Resolve a colleciton.
resolve(data[key], [...path, key]);
}
}
// If document is empty then it means it only consisted of collections.
if (!isEmpty(documentData)) {
// Upload a document free of collections.
await upload(documentData, path);
}
} else {
// Collection's length of is always odd.
for (const key in data) {
// Resolve each collection.
await resolve(data[key], [...path, key]);
}
}
}
resolve(data);
However, this does not seem to work for me. I have set up my firebase cli in the usual way and checked if functions can be sent to my project by running the standard "Hello world" function with success. It is not a problem with my JSON, as I have used an online validator. Through some research, I found that eslint ecmaVersion 6 does not allow async functions, which can be resolved by changing the ecmaVersion in the .eslintrc.json file to "ecmaVersion": 2017. Next, you have to delete some rules from the .eslintrc.json, these are // Require the use of === and !==, // Disallow null comparisons without type-checking operators and // Disallow await inside of loops. Now, with these modifications, when I type firebase deploy --only functions into my functions folder in the terminal, I am told that the function was deployed and provided with a link to my Firebase console. However, my function doesn't appear in the functions tab of firebase, nor is my data uploaded to the cloud firestore. Does anyone know what i am doing wrong? Could it be that I am not calling exports on the function? Or is this used only for prebuilt Firebase functions?

That script is not a Cloud Function, it looks like it is desired to simply run as a Node.js standalone script with data been given whatever data from RTDB you want in Cloud Firestore.
If you want to use it as a Cloud Function, you'll need to set some exports and use the data from them as the value of data. Note - that will only allow you to transfer the data changed/read. You'll likely want to run this against the entire DB at least a first (hence why it's a standalone script).

Related

Firestore data() method does not exist on JSON parsed document after JSON stringified

I am building a FlashCard website using NextJs and firebase. I have a homepage which I want to render server side and so I am using getServerSideProps. InsidegetServerSideProps function, I am fetching all the documents of the current user from firestore and is stored in an array and is returned as props as below:
export const getServerSideProps = async(ctx: GetServerSidePropsContext) {
let notes: DocumentData[];
// fetch documents here and populate notes array like so [doc,doc,doc,..]
// data() method works here and returns document fields
console.log(notes[0].data());
// NextJs throws error "`object` ("[object Object]") cannot be serialized as JSON. Please only return JSON serializable data types.", so I have to JSON.stringify() the notes
return {
props: {
notes: JSON.stringify(notes),
}
}
}
Below, I have my homepage where I parse the JSON string and have access to the notes, but now the data() method on the document does not exist/works and throws method does not exist errors. If I have to access the document fields, I have to use the dot operator on every property of the document till I reach the fields property which is nested deep down in the object as follows:
export default function Home({ notes }) {
let docs = JSON.parse(notes); // can access notes
// data() method throws function does not exist error
console.log(docs[0].data());
// I am only able to access document fields as below
console.log(docs[0]._document.data.value.mapValue.fields);
return (
<Layout>
<HomeContent notes={docs}/>
</Layout>
);
}
I have searched everywhere and found nothing that helped me why data() method won't work. If I directly fetch the documents inside the page component on client side, the data() method on the document returns its' fields. I don't know how using JSON serializations affect it. I would always prefer to use data() method to access fields than to go that deep plus I am planning to fetch data on server on other pages as well.
I would really appreciate if you can shed some light on it. It took all of my days time.
EDIT: The code that gets notes from firestore:
// inside getServerSideProps
let notes: DocumentData[] = null;
const getNotes = async(ref: DocumentReference < DocumentData > , uid: string) => {
let tempNotes = [];
const categoriesSnapshot = await getDoc < DocumentData > (ref);
const categoriesObject = categoriesSnapshot.data();
// return if user doesn't have any documents
if (!categoriesObject) {
return;
}
const promises: [] = categoriesObject.categories.map((category: string) => {
const userCategoriesRef = collection(database, 'CategoryCollection', uid, category);
return getDocs(userCategoriesRef); // returns a promise
});
const allQuerySnapshots = await Promise.all < DocumentData > (promises);
allQuerySnapshots.forEach((querySnapshot) => {
tempNotes.push(...querySnapshot.docs);
});
notes = tempNotes;
}
const categoryDocRef = doc(database, "CategoryCollection", uid);
await getNotes(categoryDocRef, uid);

Angular 6 HttpClient.get Observable does not assign value

I suppose that the answer will be very obvious, but still it evades me. I'm new on working with observables, and now I'm facing issues assigning a value from one. I had success if I define it (this._apps) as an Observable and asking from the view to the service using subscribe (But for my taste is was way convoluted (three levels inside a map just to return another observable with the array and then another function to subscribe the previous to assign the variable and another subscription in the view to finally show the information), inefficient and on top of that I could not get it "right" again). The task is very simple. Given the class Application
export class Application {
name: string;
baseUrl: string;
deprecated: boolean;
}
And the service (just the relevant code)
private _apps: Application[] = [];
constructor(private _http: HttpClient) {
this.getAllApplications().subscribe(apps => {
console.log('Apps subscriber');
this._apps = apps;
console.log('Apps subscriber Ends ' + apps);
},
err => {
console.log(err.status); // 401
console.log(err.error.error); // undefined
console.log(JSON.parse(err.error).error); // unauthorized
});
}
private getAllApplications() {
return this._http.get<Application[]>('http://development:4300/api/v1/apps');
}
From the constructor the function which gets the information from WebAPI is triggered, and the remote call is successful, but the variable this._apps is an empty array if I try to call it from anywhere in the code. I could not determine the type of the parameter "apps" in the subscribe function, but for some reason it cannot be assigned and the best answer given is that it is a function (See my first update) in one of my tries. Currently it returns in the console "[object Object]", but apps[0] gives undefined, so it is an empty Array.
This is the console output, just starting the application:
Angular is running in the development mode. Call enableProdMode() to enable the production mode.
Refreshing apps cache calling http://development:4300/api/v1/atbc-apps
Apps subscriber
Apps subscriber Ends [object Object]
I was trying this solution among many others that I forget (to use the more modern HttpClient instead the Http I used before), so what I'm doing wrong?
Update 1
I changed the constructor to this:
constructor(private _http: HttpClient) {
this.getAllApplications().subscribe(apps => {
console.log('apps length ' + apps.length);
this._apps = apps; // Remember private _apps: Application[] = [];
console.log('Apps subscriber Ends ' + apps.toString);
},
err => {
console.log(err.status); // 401
console.log(err.error.error); // undefined
console.log(JSON.parse(err.error).error); // unauthorized
});
}
and the declaration of the function called into this:
private getAllApplications(): Observable<Application[]> {
// the exactly the same as before
}
And now I got from the console this:
apps length undefined
Apps subscriber Ends
function () {
if (this instanceof Promise) {
return PROMISE_OBJECT_TO_STRING;
}
return originalObjectToString.apply(this, arguments);
}
That is the function I was talking about. Any ideas about why even though there is no errors (nor at compile time, neither at runtime), the returning object is not a real Application array?
Change this line:
private _apps: Application[] = [];
to:
_apps: Application[] = [];
Which will default to making it public. Then this line will see it:
this._apps = apps;
At the end I suppose is a mindset to work with Observables, and I tried to build a kind of cache, so the only way I could do it (let me know if there is a better way) was using the view to fill-out the cache. I could not do it from the service itself because the calling the function from the view is synchronous and to fill out the array is async. So I had to create a public setApplicationCache procedure which is filled out after calling the service from the view, it call the setApplicationCache( Application[] ) function and the rest works because it takes just the cache to do filtering and other operations or use it from other pages w/o calling the database again and again.
This is the code from the first view called (main page)
ngOnInit() {
this._myService.getAllApplications().subscribe(arrObjApps => {
this._myService.setApplicationsCache(arrObjApps)
this.listApps = this._myService.getApplications(true);
});
And the service has this functions:
private _apps: Application[] = [];
getAllApplications(): Observable<Application[]> {
return this._http.get('http://development:4300/api/v1/atbc-apps').pipe(
map( (response: Response) => {
let results = response.json().data.map( app => {
return new Application(app.name, app.baseUrl, app.deprecated);
});
return results;
})
);
}
getApplication(appName: string): Application {
return this._apps.find(app => app.name == appName);
}
getApplications(onlyActives: boolean): Application[] {
if (onlyActives) {
return this._apps.filter(app => app.deprecated == false);
} else {
return this._apps;
}
}
And as I stated the solution should be obvious. Just again the async mindset required to work with observables.

ReactJS Fixed-Data-Table and Async JSON for DataListStore

I am trying to learn ReactJS with ES6 along with setting up an instance of Fixed-Data-Table. I'm using the ObjectDataExample example from the github repo, but instead of the faker() values fed to the DataListStore, I want to use a DataListStore that gets its cache from a remote JSON resource. This is how I have defined my DataListStore:
class MyDataListStore {
constructor(/* url string */ url) {
this.url = url || 'http://localhost:8080/default-json';
this._cache = [];
this.pageSize = 1;
this.size = 0;
this.getRemoteData(url);
}
getRemoteData() {
/**
* Fetch remote JSON to be used in the store.
*/
var that = this;
fetch(this.url).then(function(response) {
return response.json();
}).then(function(j) {
console.log(j);
//this.pageSize = j["pages"];
that.size = j["total"];
that._cache = j["table"];
if (that._cache) {
// do something here?
}
});
}
getObjectAt(/*number*/ index) /*?object*/ {
if (index < 0 || index > this.size){
return undefined;
}
if (this._cache[index] === undefined) {
//this._cache[index] = this.createFakeRowObjectData(index);
}
return this._cache[index];
}
getSize() {
return this.size;
}
}
module.exports = MyDataListStore;
As you can see I'm following the FakeObjectDataListStore provided with the example from fixed-data-table more or less. The JSON is fetched properly, the _cache is populated with an array of objects, and when you output getSize once getRemoteData has executed, you do get the size of the _cache. However, I haven't figured out how my fixed-data-table Table component should be updated once the data has been fetched. Currently the Table is rendered but is simple blank with no rows.
class ObjectDataExample extends React.Component {
constructor(props) {
super(props);
this.state = {
dataList: new MyDataListStore()
};
}
render() {
var {dataList} = this.state;
return <Table
rowHeight={70} rowsCount={dataList.getSize()} width={1170} height={500} headerHeight={30}>
<Column
header={<Cell>ID</Cell>}
cell={<TextCell data={dataList} col="id" />}
width={50}
fixed={true}
/>
<Column
header={<Cell>Email</Cell>}
cell={<TextCell data={dataList} col="email" />}
width={300}
fixed={true}
/>
</Table>
}
}
module.exports = ObjectDataExample;
I think the main issue is that I don't have any code meant to populate the table once MyDataListStore is populated with the data from the async call. However, I can't find any help from the examples given in the Fixed-Data-Table github repo or the docs. Any idea how to get this done? I assume I need to set up some sort of event listener, but I'm not sure where/how to do this, as I'm still new to both ReactJS and Fixed-Data-Table.
Edit: I should also add that when the page loads, I get the following error:
Uncaught TypeError: Cannot read property 'id' of undefined
once I set the initial this.size to more than 0. So of course the table doesn't have the available data when it's first loading.
Edit 2: After looking into this further, it looks like if I run the fetch in componentDidMount of my ObjectDataExample and use this.setState(); to reset the dataList object, then I get the table updated. However, this looks a little messy and I'd assume there's a better way to do this directly from my MyDataListStore object.
Thanks,
One design issue with the current implementation of MyDataListStore is that it does not provide a way to notify the caller when the data has been loaded.
One possible way you might do this is to implement some sort of factory function (in the example below, I'm pretending that one exists called MyDataListStore.of) that returns a Promise that eventually resolves the MyDataListStore instance once the data loads:
// In the ObjectData component constructor, we call the MyDataListStore
// factory function and once it resolves, we assign it to our
// state. This will cause our component to re-render.
constructor() {
MyDataListStore.of(myDataListStoreUrl).then(store => {
this.setState({ dataList: store });
});
}
Now, once the data in the data list store resolves, our template (specified in your render function) will render correctly.
The DataListStore.of function we used earlier might look something like this:
class MyDataListStore {
static of(url) {
const dataListStore = new MyDataListStore(url);
return dataListStore.getRemoteData().then(() => return dataListStore);
}
/* ... other MyDataListStore properties/methods ... */
}
And finally we need to update the getRemoteData to return a promise. This is what will allow any clients of our MyDataListStore class to be notified that the data has loaded:
getRemoteData() {
/**
* Fetch remote JSON to be used in the store.
*/
var that = this;
// Return the chained promise! This promise will resolve
// after our last callback is called.
return fetch(this.url).then(function(response) {
return response.json();
}).then(function(j) {
console.log(j);
//this.pageSize = j["pages"];
that.size = j["total"];
that._cache = j["table"];
if (that._cache) {
// do something here?
}
});
}

How to pass API url JSON http.get res.json(data) using express.js

I am trying to send a JSON object back to my client using a websites API and am getting the following error.
var body = JSON.stringify(obj, replacer, spaces);
TypeError: Converting circular structure to JSON
at Object,stringify (native)
Here is my code
app.get('/api/test', function(req, res){
http.get('http://api.biblia.com/v1/bible/content/LEB.txt.json?passage=John3.16&key=fd37d8f28e95d3be8cb4fbc37e15e18e', function(data) {
res.json(data);
});
});
If I replace data with a simple JSON object {"test":"test"}. Everything works fine. Any help with understanding what is even occuring would be helpful. I am using an Express.js Node.js Angular.js stack. Thank you!
The data variable in your callback is actually an instance of http.IncomingMessage, which is much more complex than just data. The error you're getting is because it has circular references, so you need to filter it out. There is an answer here that outlines this process using the code below:
var cache = [];
JSON.stringify(o, function(key, value) {
if (typeof value === 'object' && value !== null) {
if (cache.indexOf(value) !== -1) {
// Circular reference found, discard key
return;
}
// Store value in our collection
cache.push(value);
}
return value;
});
cache = null; // Enable garbage collection

Convert Mongoose docs to json

I returned mongoose docs as json in this way:
UserModel.find({}, function (err, users) {
return res.end(JSON.stringify(users));
}
However, user.__proto__ was also returned. How can I return without it? I tried this but not worked:
UserModel.find({}, function (err, users) {
return res.end(users.toJSON()); // has no method 'toJSON'
}
You may also try mongoosejs's lean() :
UserModel.find().lean().exec(function (err, users) {
return res.end(JSON.stringify(users));
});
Late answer but you can also try this when defining your schema.
/**
* toJSON implementation
*/
schema.options.toJSON = {
transform: function(doc, ret, options) {
ret.id = ret._id;
delete ret._id;
delete ret.__v;
return ret;
}
};
Note that ret is the JSON'ed object, and it's not an instance of the mongoose model. You'll operate on it right on object hashes, without getters/setters.
And then:
Model
.findById(modelId)
.exec(function (dbErr, modelDoc){
if(dbErr) return handleErr(dbErr);
return res.send(modelDoc.toJSON(), 200);
});
Edit: Feb 2015
Because I didn't provide a solution to the missing toJSON (or toObject) method(s) I will explain the difference between my usage example and OP's usage example.
OP:
UserModel
.find({}) // will get all users
.exec(function(err, users) {
// supposing that we don't have an error
// and we had users in our collection,
// the users variable here is an array
// of mongoose instances;
// wrong usage (from OP's example)
// return res.end(users.toJSON()); // has no method toJSON
// correct usage
// to apply the toJSON transformation on instances, you have to
// iterate through the users array
var transformedUsers = users.map(function(user) {
return user.toJSON();
});
// finish the request
res.end(transformedUsers);
});
My Example:
UserModel
.findById(someId) // will get a single user
.exec(function(err, user) {
// handle the error, if any
if(err) return handleError(err);
if(null !== user) {
// user might be null if no user matched
// the given id (someId)
// the toJSON method is available here,
// since the user variable here is a
// mongoose model instance
return res.end(user.toJSON());
}
});
First of all, try toObject() instead of toJSON() maybe?
Secondly, you'll need to call it on the actual documents and not the array, so maybe try something more annoying like this:
var flatUsers = users.map(function() {
return user.toObject();
})
return res.end(JSON.stringify(flatUsers));
It's a guess, but I hope it helps
model.find({Branch:branch},function (err, docs){
if (err) res.send(err)
res.send(JSON.parse(JSON.stringify(docs)))
});
I found out I made a mistake. There's no need to call toObject() or toJSON() at all. The __proto__ in the question came from jquery, not mongoose. Here's my test:
UserModel.find({}, function (err, users) {
console.log(users.save); // { [Function] numAsyncPres: 0 }
var json = JSON.stringify(users);
users = users.map(function (user) {
return user.toObject();
}
console.log(user.save); // undefined
console.log(json == JSON.stringify(users)); // true
}
doc.toObject() removes doc.prototype from a doc. But it makes no difference in JSON.stringify(doc). And it's not needed in this case.
Maybe a bit astray to the answer, but if anyone who is looking to do the other way around, you can use Model.hydrate() (since mongoose v4) to convert a javascript object (JSON) to a mongoose document.
An useful case would be when you using Model.aggregate(...). Because it is actually returning plain JS object, so you may want to convert it into a mongoose document in order to get access to Model.method (e.g. your virtual property defined in the schema).
PS. I thought it should have a thread running like "Convert json to Mongoose docs", but actually not, and since I've found out the answer, so I think it is not good to do self-post-and-self-answer.
You can use res.json() to jsonify any object.
lean() will remove all the empty fields in the mongoose query.
UserModel.find().lean().exec(function (err, users) {
return res.json(users);
}
It worked for me:
Products.find({}).then(a => console.log(a.map(p => p.toJSON())))
also if you want use getters, you should add its option also (on defining schema):
new mongoose.Schema({...}, {toJSON: {getters: true}})
Try this options:
UserModel.find({}, function (err, users) {
//i got into errors using so i changed to res.send()
return res.send( JSON.parse(JSON.stringify(users)) );
//Or
//return JSON.parse(JSON.stringify(users));
}
Was kinda laughing at how cumbersome this was for a second, given that this must be extremely common.
Did not bother digging in the docs and hacked this together instead.
const data = await this.model.logs.find({ "case_id": { $regex: /./, $options: 'i' }})
let res = data.map(e=>e._doc)
res.forEach(element => {
//del unwanted data
delete element._id
delete element.__v
});
return res
First i get all docs which have any value at all for the case_id field(just get all docs in collection)
Then get the actual data from the mongoose document via array.map
Remove unwanted props on object by mutating i directly