I am working on a social network app from scratch,
I designed my database (choosing Mysql) to store and retrieve data
The problem starts when I am posting videos and photos and when data become larger and larger when I am waiting to view news feeds (depending on sql query that brings the last posts from friends ans Pages).
The question is :
How can I manage and handle the big data, and how can I make the news feeds service more efficient ?
What do you think of using (Real-time) Databases from Firebase?
If you're thinking about putting up Firebase as your choice for your database, then there's nothing wrong with that. Firebase realtime database is a very secure and great option.
Firebase gives you functionality like analytics, databases, messaging and crash reporting so you can move quickly and focus on your users.
Firebase is built on Google infrastructure and scales automatically, for even the largest apps.
And on top of that, many great apps do use Firebase as their backend.
I'd say to know more, you can just start with a demo project on Firebase to learn how the things happen with Firebase, and after learning, start using it as backend for your social app.
Just go to Firebase Console and start with your own project. To learn how to use Firebase, do refer Firebase docs and if stuck, StackOverflow is always there.
If you want to get some inner details, I have some repositories on GitHub, that might help you in understanding about Firebase and the security rules and other important things.
Demo App
Firebase Security Rules
Related
I am currently developing a full-stack web application for the first time. It is a store that needs to give users the ability to upload "books", edit, delete, and manage them. As of now, I have a React front-end, that calls an Express API using Axios that queries a MySQL database. This currently can manage the product details, titles, and simple labels and relations.
However, I now need to store images and .json files dynamically as well. So, I have researched and need to use Azure Storage to store these images and allow access to them by the end-user. I have researched it and the client would like to use Azure storage as well.
I have gotten quite overwhelmed looking into the Azure documentation for Javascript image uploading, and every "tutorial" starts with create a web application, storage account, and app service.
All I would like to do is store images from the user in Azure storage, so that when I eventually deploy the website, the data is available to be accessed and then my front end or API can call Azure to get the images for the user. I apologize if the question is confusing or vague, I am really just overwhelmed by Azure's documentation and it seems like such a simple and common problem. Any guidance or references would be greatly appreciated.
Yes, it's quite possible. You simply create an Azure Storage account and upload your files as blobs via that account. Then they will be available publicly on the Internet, and your web application can reference them from wherever it is hosted.
It is possible as I believe it is quite a common practice to have a Content Delivery Network to deliver the images. I am not very familiar with Azure but I am with AWS and I can tell you that you can use an AWS S3 bucket to store the images and JSON. It comes with many different configuration options to allow content to be protected or open to the general internet.
I want to consume kafka stream into mysql using Python; on top if which I want to build a realtime web based (web app) dashboard that will automatically be refreshed (ajax) on each data insert in the database.
After some searching, found a suggestion that ajax is not good for this purpose.
This post said websockets are better than ajax in terms of Performance.
Because I am not sure on whats the best way to achieve this So your expert advice is needed.
Thanks.
"I want to consume kafka stream into mysql using Python; on top if which I want to build a realtime web based (web app) dashboard that will automatically be refreshed (ajax) on each data insert in the database."
... (wince!) ...
Pretty-please find someone among your peers who can save you from yourself. (And is there any possible way that I can say this to you, such that you can "save face?" I can't think of any.) I'm not-at-all trying to have public fun at your expense. Please – "talk immediately to your manager. (S)He, surely, can help you."
I am certainly not an expert in this field, however my company uses Elastic/Kibana to read in from Kafka topics and display the data on a dashboard. This is just one of many routes you can take, but it works very well for us. You can read a little more into it here:
https://www.elastic.co/blog/just-enough-kafka-for-the-elastic-stack-part1
Is it possible to use two databases, example, Firebase and an online webhosting server? I am developing an app that uses the webhosting server for the data and the firebase only for the chats.
Absolutely. Firebase is very flexible when it comes to picking and choosing parts which you want to use.
You don't need to migrate away from whatever backend you already have if you just need to add in something that is unrelated to your site data, such as a support chat or something.
Instead, see it as an opportunity to discover Firebase by trying out something isolated like a chat using Firebase Database. You'll find your best fit by trying out the strengths and limits of the platform.
Yes it is Possible but, you have to manage the back-end accordingly. I have created the app in which api is hosted in aws-EC2 and few part is in the Firebase. For example for some live changes I'm using the Firebase and for other things like getting feeds and all those thing i used aws EC2.
I'm struggling to find a solution to what I thought would be a common requirement so I'm hoping someone can help me with some pointers on what to search for/areas to explore.
Background
I'm building an iOS mobile app. I'm storing data locally using realm.io. The app is preinstalled with a snapshot of the content of a Wordpress mySQL database (it uses custom types). The content of the WP database is only written via the Wordpress install, the mobile app cannot write data.
Objective
So, I want to be able to check for changes since a given date (whenever the local database was last updated) and send the changed records to the mobile app (via the wp JSON api?).
I think I can fetch "posts since a date" but I need a full list of all create, update and delete operations since a given date.
Since the app is read-only I thought this type of one-way sync would be pretty straight forward but I can't find a common solution.
Any ideas to point me in the right direction would be great. Obviously, if anyone has any experience of doing this sort if thing with realm.io then that would be amazing :-)
Realm doesn't support yet any sort of synchronization mechanism across different files. We have an issue about that though, but you're likely searching rather for a solution in the immediate future.
Update: Realm launched the Realm Mobile Platform. This offers synchronization functionalities and would greatly simplify the solution for this use case.
You could use e.g. the server-side Node.js binding to pull new data from your MySQL Wordpress installation and push them to a global Realm served by the Realm Object Server. This can be read-only synchronized from the mobile apps, which would automatically receive the deltas and provide updated data to your users.
Whatever mechanism you come up yourself though in the meantime, it would require that you have read-write access from your iOS application to the realm database, so that you can update it with new data.
Pushing changed records as you describe is likely not going to work.
Apple's Push Notification service (APNS), which is the only back communication channel that works when your app is in the background or suspended, allows you to send very small payloads. You would use that to signalize your iOS app, that something changed on the server-side and there is new data to load. You would then initiate a request to a JSON-based API, wait for the response, map the returned JSON to Realm objects and store them in your database.
You want probably read more in the "Downloading Content in the Background" section of background execution chapter in the official App Programming Guides for iOS.
While pre-seeding the database from the app bundle seems like a nice idea, because the user wouldn't need to wait initially after downloading the app, that will enlarge the app itself with data, which might become in the future completely irrelevant.
At first, this question appeared to be too trivial to me to actually require a Stackoverflow post. However, after executing many Google searches for the information, I am at a lost when trying to figure this out about Couchbase.
In Couchbase (I am using the 2.2 Community version), how do I share views among developers? Is there some sort if import/export functionality available? If not, then how does Couchbase intend for developers to share the views that they are using without needing to do manual copying/pasting? It is obvious that the code that a development team would write for querying Couchbase will require accurate view names. Without having a way to send a developer a view file, to accurately setup a Couchbase DB, how can it even be possible to develop with Couchbase locally as a team?
I'm sorry if I sound a little desperate or harsh here, but if it isn't possible to share views among multiple developers, then I don't see how Couchbase can be a viable DB solution for a team of developers trying to share database configuration, similar to how a team using an SQL DB would share schema files to set up the DB.
Several ways you can approach this:
1) Create views programmatically as demonstrated here in java:
http://tugdualgrall.blogspot.com.es/2012/12/couchbase-101-create-views-mapreduce.html
or here in node.js:
http://www.tuicool.com/articles/RvYbQn
2) Store all your views in your version control system (This is the option I use). If you are developing locally then only you need your personal view code, once they are working and your tests are all passing then you can check them in.
I assume you'd then be developing on an testing environment so yes sadly here you'd have to update the views either by hand or by using option 1.
You could also take a look at perhaps using this tool but only for views: http://www.couchbase.com/communities/q-and-a/how-bulk-import-design-docs-and-views-couchbase-server
This functionality currently is not available in the admin UI.
There is a defect/enhancement open Ability to import/export views MB-8436. You can leave there your feedback and vote for it so it will be included in the next release.
In the meantime you can use Design Document REST API
Also there is a workaround blog