I am currently building a price comparison serverless web application on Firebase Hosting with more than 300K products (documents) in a Firestore DB.
Firestore has still some limitation when it comes to filtering (see here) so im trying to find an alternative to Firestore DB that includes some sort of authentication.
Note: being able to filter for multiple keys (e.g. all blue cars sorted by ranking index and cheaper than 25K is currently not possible in Firestore) is more important than scalability
Question: is it possible to use the Firebase Anonymous Authentication method to limit access to a Google MySQL Instance?
Google Cloud SQL is not integrated with Firebase Authentication. So you can't securely access your Google Cloud SQL database directly from client-side application code. This applies to all supported Cloud SQL databases (MySQL, PostgreSQL, and SQL Server at the moment).
Related
I extracted data from an API using Airflow.
The data is extracted from the API and saved on cloud storage in JSON format.
The next step is to insert the data into an SQL DB.
I have a few questions:
Should I do it on Airflow or using another ETL like AWS Glue/Azure Data factory?
How to insert the data into the SQL DB? I google "how to insert data into SQL DB using python"?. I found a solution that loops all over JSON records and inserts the data 1 record at a time.
It is not very efficient. Any other way I can do it?
Any other recommendations and best practices on how to insert the JSON data into the SQL server?
I haven't decided on a specific DB so far, so feel free to pick the one you think fits best.
thank you!
You can use Airflow just as a scheduler to run some python/bash scripts in defined time with some dependencies rules, but you can also take advantage of the operators and the hooks provided by Airflow community.
For the ETL part, Airflow isn't an ETL tool. If you need some ETL pipelines, you can run and manage them using Airlfow, but you need an ETL service/tool to create them (Spark, Athena, Glue, ...).
To insert data in the DB, you can create your own python/bash script and run it, or use the existing operators. You have some generic operators and hooks for postgress, MySQL and the different databases (MySQL, postgres, oracle, mssql), and there are some other optimized operators and hooks for each cloud service (AWS RDS, GCP Cloud SQL, GCP Spanner...), if you want to use one of the managed/serverless services, I recommend using its operators, and if you want to deploy your service on a VM or K8S cluster, you need to use the generic ones.
Airflow supports almost all the popular cloud services, so try to choose your cloud provider based on cost, performance, team knowledge and the other needs of your project, and you will surly find a good way to achieve your goal with Airlfow.
You can use Azure Data Factory or Azure Synapse Analytics to move data in Json file to SQL server. Azure Data Factory supports 90+ connectors as of now. (Refer MS doc on Connector overview - Azure Data Factory & Azure Synapse for more details about connectors that are supported by Data Factory).
Img:1 Some connectors which are supported by ADF.
Refer MS docs on pre-requisites and Required Permissions to connect Google cloud storage with ADF
Take source connector as Google Cloud storage in copy activity. Reference: Copy data from Google Cloud Storage - Azure Data Factory & Azure Synapse | Microsoft Learn
Take SQL DB connector for sink.
ADF supports Auto create table option when there is no table created in Azure SQL database. Also, you can map the source and sink columns in mapping settings.
I have just created a Project in Google Cloud, and attached a Cloud SQL Database instance to that project. I was able to deploy a Django app that is connected to that DB just fine.
However, I would like to create a separate Django app/Project that is attached to the same Cloud SQL Database that my first Django app is attached to.
Is this possible?
One Django app is responsible for web scraping and supplying constant data to the database while my second Django app (the one I have already deployed) analyzes and returns json on that data. It would be advantageous to separate the two apps because if I ever needed to revise my web scraping algorithm, the whole app would not be down.
You can use cloudsql proxy for both apps. Also as long as you authorize both of your applications with service account that have access to the cloudsql it should be fine.
You can use database that you want, in your current project or in an external one. If you use Cloud SQL proxy, the service account of your app
Either Default AppENgine service account if on App Engine
Or Compute Engine default service account
except is you have defined a specific service account on your component (Cloud Run, Compute engine)
I recommend strongly to use a specific service account on the component if it's possible (not possible with App Engine)
The role to grant on the service account is the following: roles/cloudsql.client
However, I recommend you to smartly think to your design. The current trend is to lock 1 database to 1 service (or microservice).
Think about the schema update: a synchronous update will be required between to 2 services when you update the schema or one, or the other app will fail.
Same thing in case of rollback, both apps need to be rollbacked.
If there is 2 teams, one on each app, their release planning must be sync, and you will lost in velocity and agility.
Maybe, it fits your requirement, or you can duplicate the data, inside the database (other schema). As you wish.
I have read many example and articles on this,till now I got to know cloud functions uses firebase database either firestore or real time database.But ,I want to know if it is possible to use mySQL database and use cloud functions with node.js script as backend?
And then I will use this cloud function in my flutter app.
Any help is appreciated.
You can do most things in Cloud Functions that you can do in server-side Node.js, Python, or Go (the languages supported for Cloud Functions)
You can access Cloud SQL (MySQL or PostgreSQL) but you can also access any other server exposed to the Internet from Cloud Functions.
To be able to access servers outside Google Cloud, you need to use have paid Firebase plan though.
We would like to hook up an iPhone app directly to a database in the cloud. From the iPhone app, we would like to use some REST API to retrieve data from the cloud database. We prefer using MySQL or Mongo as the data store.
We prefer not setting up our database system on AWS or another hosted provider. We would like a cloud service that offers the database layer out of the box.
What database-as-a-service companies support this, and what are the pros/cons of each?
We looked around, and it seems like MongoLab is the only service offering a REST API.
Mongo is a great database for API interaction as the query language uses javascript expressions. Here are the open source API libraries that are implemented in Python
Sleepy Mongoose
Eve
Primary advantage
JavaScript can handle the querying part of data retrieval
Primary disadvantage
The API libraries above have difficulty capturing complex queries
Can i use mysql database connected to django application on for appengine paltform ??
A relational database is only available on their business version, google appengine uses google bigtable by default.
Since you're using django on appengine you might want to check out http://www.allbuttonspressed.com/projects/django-nonrel for easier database handling if you find it too intimidating