The goal is to set up a simple relational database in Google Cloud SQL (or BigQuery) that automatically receives/retrieves daily emailed reports. The data source, in this case, is Doubleclick but regardless of the source, I'd like to better understand how scheduled email reports (as attachments) can be sent to or ingested by Google Cloud SQL.
Is there some other app or service out there to make this connection? Is there a Google product like MS Visual Studio to run jobs? Sorry for the very beginner questions but none of the Google support articles are very helpful!
You will need to use Google Apps Scripts to tie the process together.
App Scripts can DoubleClick data and potentially read from a gmail account.
Once you have access to the data, you should be able to use BigQuery service in script to load the data
You can then create a scheduled project trigger within script to poll periodically for new files
Related
I've been developing an apps script project for my company that tracks our time/expenses. I've structured the project like so:
The company has a paid Gsuite account that owns all the spreadsheets hosted on the company's google drive.
Each employee has their own "user" spreadsheet which is shared from the company Gsuite account with the employee's personal gmail account.
Each of the user spreadsheets has a container-bound script that accesses a central library script.
The library script allows us to update the script centrally and the effects are immediate for each user. It also prevents users from seeing the central script and meddling with it.
Each of the user container-bound scripts have installable triggers that are authorized by the company account so that the code being run has full authority to do what it needs to to the spreadsheets.
This setup has been working quite well for us with about 40 users. The drawback to this setup is that since all the script activity is run by the company account via the triggers, the activity of all our users is logged under the single company account and therefore capped by the apps script server quotas for a single user. This hasn't been much of an issue for us yet as long as our script is efficient in how it runs. I have looked into deploying this project as a web-app for our company, but there doesn't seem to be a good way to control/limit user access to the central files. In other words, if this project was running as a web app installed by each user, each user would need to have access to all the central spreadsheets that the project uses behind the scenes. And we don't want that.
SO with that background, here is my question. How do I efficiently track apps script activity to see how close we are to hitting our server quota, and identify which of my functions need to be optimized?
I started doing this by writing a entry into a "activity log" spreadsheet every time the script was called. It tracked what function was called, and who the user was and it had a start time entry and and end time entry so I can see how long unique executions took and which ones failed. This was great because I had a live view into the project activity and could graph it using the spreadsheet graphs tools. Where this began to break down was the fact that every execution of the script required two write-actions: one for initialization and another for completion. Since the script is being executed every time a user made an edit to their spreadsheet, during times of high traffic, the activity log spreadsheet became inaccessible and errors would be thrown all over the place.
So I have since transitioned to tracking activity by connecting each script file to a single Google Cloud Platform (GCP) project and using the Logger API. Writing logs is a lot more efficient than writing an entry to a spreadsheet, so the high traffic errors are all but gone. The problem now is that the GCP log browser isn't as easy to use as a spreadsheet and I can't graph the logs or sum up the activity to see where we stand with our server quota.
I've spent some time now trying to figure out how to automatically export the logs from the GCP so I can process the logs in real-time. I see how to download the logs as csv files, which I can then import into a google spreadsheet and do the calcs and graphing I need, but this is a manual process, and doesn't show live data.
I have also figured out how to stream the logs from GCP by setting up a "sink" that transfers the logs to a "bucket" which can theoretically be read by other services. This got me excited to try out Google Data Studio, which I saw is able to use Google Cloud Storage "buckets" as a data source. Unfortunately though, Google Data Studio can only read csv files in cloud storage, and not the json files that my "sink" is generating in my "bucket" for the logs.
So I've hit a wall. Am I missing something here? I'm just trying to get live data showing current activity on our apps script project so I can identify failed executions, see total processing time, and sort the logs by user or function so I can quickly identify where I need to optimize my script.
You've already referenced using GCP side of your Apps Script.
Have a look at Metric explorer, it lets you see quota usage per resource and auto generates graph for you.
But long term I think re-building your solution may be a better idea. At minimum switching to submitting data via Google Forms will save you on operation.
I have a Google Apps Script application that currently accesses a GCP SQL database with JDBC.
Using the normal SQL database doesn't cut it, so I decided to try BigQuery.
Is there a way to access BigQuery from Google Apps Script without connecting with an account connected to the GCP project? I want guests who use my script to be able to get data.
I'm looking for either a general way (as in: IP, database, username, password and I manage the connection) or a client library way I can use with Apps Script.
Note, the BigQuery Apps Script plugin seem to only allow access to my own databases, so guests will get denied access.
As per this link , you can share your BigQuery dataset with specific users even if they're not part of the GCP project. Since they need to be able to retrieve data, they will need "Viewer" permission on the dataset. The steps to do this are described in here and this link will show you with an example how you can query your BigQuery table from Apps Script.
We have successfully deployed Google Apps Script based web applications to teams of users, those applications allow the users to log their daily activities via a simple GUI. The log data is collected per team using Google spreadsheets or ScriptDBs, depending on the size of the team.
We now want to go one step further and do analysis and reports on the user activity data across the teams. Given the amount of data, BigQuery looks like a good technology to do that. We are thinking about using Google Apps Scripts to push the data automatically on a regular (e.g. daily basis) to the BigQuery table(s). We are wondering what the best practices are to do that with the data originating from Google spreadsheets and ScriptDBs.
Unlike in previous cases, just from simply reading through the BigQuery API documentation and code snippets it does not become obvious to us what the recommended approach is.
The hint we found so far:
Write Data from Google Spreadsheets to a BigQuery Table
We don't have any examples for it, but you can use Apps Script's built-in BigQuery service to upload data directly.
BigQuery.Jobs.insert(resource, mediaData, optionalArgs)
The mediaData parameter was recently added, and allows you to pass in the CSV data as a blob which gets sent directly to BigQuery.
I am looking to try and combine Google's Cloud SQL service with google drive. Essentially I want to use Google forms for the user to easily input data, and then have that data feed into the Cloud SQL environment (from which I can do reporting and analysis).
My question is, has anyone done this already, or have any ideas of how this might be accomplished? I already have google forms writing to spreadsheets, and that works fine. I am familiar with SQL so creating reports and pulling data from the Cloud SQL environment shouldn't be hard.....but I don't know how to connect the two.
Ideally I would like something to run on a schedule (maybe on a nightly basis) to pull the data from a google spreadsheet, update the Cloud SQL database, and clear the old data from the spreadsheet.
I think the best way to do this would be to use Google Apps Script.
Apps Script allows you, for a given form, to define an onSubmit() method that will be called when someone submitted an answer to the form.
Then you can use the JDBC service to connect to Cloud SQL.
I am creating an application where I will be inserting spreadsheet data to Cloud SQL using jdbc service. I would like to know answers of following :
Q1. What will happen if the insert statement is executing and someone closes the spreadsheet/script?
Q2. is there any possibility of inserting the data to Cloud SQL using Google Apps script in offline mode?
The answer to both questions is actually a similar one - since both Cloud SQL and Apps Script are entirely always online cloud systems - it doesn't matter if the file is closed or if a user goes offline. The script should run to completion.