I've written a script in Google Colab that performs some queries via API to a server where I host some information, let's say weather data from several weather stations we own, that data is downloaded in a JSON format then I save those files in my Google Drive as I'll use them later in the same script to generate some condensed tables by country and by month, those summary files are also stored in my Google Drive account. After that, I send the summary files via email to some people.
I need to run this script every other day and, well, it is no big deal to do that, but I'd like to make it happen without human intervention, mainly because of scalability. One of the main points is that some libraries are not easy to install locally and it is way easier to make it work in Colab, that's why I don't use some solutions like crontab or windows scheduler to perform this, and I need to make the Colab notebook to run periodically.
I've tried solutions like PythonAnywhere, but I've spent too much time trying to modify the script to work with PyDrive and, so far, haven't achieved it. I've also tried to run it using GCP but haven't found how to access my Google Drive files yet using Google Cloud Functions. I've been also working on building an image using Docker and if I deploy it to someone else, it'll be working, but the idea is to make the running schedulable not only deployable. I've also read the Colabctl option but haven't been able to make it run yet.
Please, I'm open to receiving any suggestions in order to achieve my goal.
Thank you,
Billy.
there is a paid scheduler https://cloud.google.com/scheduler
or you can use https://github.com/TensorTom/colabctl with your own cron
Related
I have a database file which is located on my own google drive (private) and it's updated on a daily basis.
I wrote a python script in order to track the data from the database file, however I need to download the DB file to my pc, and then running my script locally, every single day.
I am wondering if there are any better options, so I wouldn't have to download the DB file and move it manually.
From a slight searching on the web I found that there is no way to run the script in the google drive folder (obviously due to security issues), and using google cloud platform is not a real option since it's not a free service (and as I understood there is no free trial).
Anyways, any method that would make my life easier would be happily accepted.
Sorry
That's not possible AFAIK. At least in the way you have asked the question.
It may be that you are looking for a Database hosting service, which, as a rule, are not free. I remember seeing a SQL viewer around, I don't know if it is still available and I don't think it was accessible via a local Python script.
Google Cloud Platform, as other services, do however offer a free tier of services - though this depends on how much use you need to give it and also on your location. Though this can get quite involved quite quickly. There are also various options to choose from. For example Big Query may be something you are interested in.
A Possible Workaround
I assume that the reason you don't want to download it is because it is tedious, and not because your network connection or hard drive can't handle it, if so:
The simple workaround may just be to find a way to automatically download the database via the (Google Drive API)[https://developers.google.com/drive] with your Python Script, modify it, or read it, and then upload it again if you need to. This can be done all within one script using the API. So all you would need to do is run the Python script and you wouldn't need to manually download and move it into a folder. You would also not need to worry about running over the free tier and getting billed for it.
You could even use the Google Drive Desktop software to keep the database file synced locally. Since it is local, I doubt the software would be able to block a Python script on the file system.
If you don't even want to think about it, you could set up the Python script with a CRON job or a Scheduled task if you are on Windows.
Just curious if there's a standard method to take one VM instance and migrate it to a completely different Google Cloud account. If not, I guess I could download all the site files and server configuration files (virtual hosts, apache config, php.ini, etc) but am hoping there's a more streamlined approach.
Just to be clear I have john#gmail.com and john2#gmail.com, both completely separate accounts each with VMs on their respective Google Cloud accounts. I want to move a VM instance from john#gmail.com to john2#gmail.com
I'm open to any method that will get the job done. I like Google's snapshot feature but I have doubts I'll be able to move the specific snapshot over. I'm thinking maybe it'll be possible by creating an image but even that I'm not 100% sure.
Thanks!!
Create a custom image, then share it, then import that image into the target project.
I see many tutos to download GA data to a google spreadsheet and refresh it automatically
but none to export data into mysql table(s)
Is it possible ?
If yes where can i find tutos or a freelancer with skills on this subject
If yes is it possible to store the data of several UA in the same table ?
Thx
Yes, you can do that with the Google Analytics core reporting api and you can use their query explorer to see what data you can pull from it.
Also here is all their libraries and sample code that you can use as a tutorial. As far as hiring someone, I would check out the job board here or odesk.com.
if you don't want to play around with code and development, just go ahead and use Analytics Canvas that provides easy data import via API from Google Analytics, which you can then easily export to SQL / MySQL / any other database. All using simple and intuitive graphic interface.
I usually use another solution that does not require coding — Skyvia tool: Google Analytics and MySQL Integration. It allows me to create a copy of Google Analytics report data in MySQL and keep it up-to-date with little to no configuration efforts. I don’t even need to prepare the schema — Skyvia can automatically create a table for my report data. You can load 10000 records per month for free — this is enough for me.
We are creating a web application using MySQL as our database, is there a way that some files from the hosting site of our application can be sync to the user's google drive?
There's no direct way of synchronizing your local data with Drive.
However, you can pseudo-sync with little load on your server by using Changes. Basically, you can get list of file changes since the time you specify. If I were you, I make a cron job of checking file changes from Drive and Files.get specific files that have changed.
Besides what JunYoung Gwak suggests -> to ask google by polling them you will also have to keep a last edited date in your app and there might be cases when the local file is newer. will have to keep the same time zone as google to make it work for <24 hour changes.
So both sides will need to have a give changes since -date-time-in-timezone and a way to take a file for one place and update to the other.
Might have conflicts that need to be resolved by a diff tool.
I have an android app which successfully reads .json files from my localsever, however when the application runs on a mobile it is unable to read the files. I have had a nightmare trying to resolve the issue so I've given up.
I figured that perhaps uploading the .json files to an online web server would be an easier option. I have no previous experience with online web servers and wondered if someone could recommend a website?
I would only be using it to upload 5 .json files that are no more than 5KB and would probably only need it to be hosted online for about a month, I'm not sure if there is an alternative to a monthly subscription.
There are a lot of free web services. However, if you want to make it fast and easiest, try google sites. It gives you an evaluation time of about a month and it should be enough for you.
However, if you just want few files, you can go with any cheap web host such as bluehost and if you just want subscription on monthly basis.
You can also use amazon EC2 which provide you services on hourly basis so if you do not want to use it, just turn it off.
http://aws.amazon.com/ec2/pricing/