A crucial difference between Firestore and Storage is that Firestore stores data (objects, arrays, strings, etc.) when Storage stores files. How we handle files in Cloud Functions is very different from how we handle data.
I use Node to call APIs, get files, and write the files to Storage. Node is hard. Are there easy methods to write files to Storage?
I found an easy way to write files to Storage from Cloud Functions. [Sounds of thin ice creaking.]
index.ts
import { getStorage, ref, uploadBytes, uploadString, connectStorageEmulator } from "firebase/storage";
let file = await got('https://audio.oxforddictionaries.com/en/mp3/winter__us_2.mp3');
await uploadBytes(storageRef, file['rawBody'], metadata);
The ['rawBody'] parameter overrides the requirement that uploadBytes only uploads files.
That's cringy, right? Is there a better way?
Google says
Cloud Client Libraries are the recommended option for accessing Cloud
APIs programmatically, where available.
In terms of Cloud Storage (and every other GCP product), they provide a whole bunch of working examples that you can practically copy/paste into your code.
Node.js Cloud Client Libraries
Specifically, see Google Cloud Storage: Node.js Client
The methods for accessing Cloud Storage in Cloud Functions for Firebase comes from the Firebase Admin SDK for Node.js and are a thin wrapper around the Cloud Storage SDK for Node.js. There is no friendly Firebase wrapper for these, so my usual approach is to search for (non Cloud Functions specific, non Firebase specific) Node.js example.
Related
How to process files which are uploaded in google cloud storage (Java)
I have to unzip all the files which are uploaded in cloud storage in google cloud function.
Basically i need to check file formats of the files within zip folder.
Is there any way to download it in cloud function ?Is it right way to do
Thanks in advance
Using Cloud Functions triggered by Cloud Storage (GCS) events is a common and good practice:
https://cloud.google.com/functions/docs/calling/storage#functions-calling-storage-java
When an object is uploaded to GCS, one or more Cloud Functions is triggered, uses the Cloud Storage client library:
https://cloud.google.com/storage/docs/reference/libraries
To:
unzip the object
enumerate its content
process (check file types etc.)
Do something with the result
NOTE events only arise on changes so you'll likely want to consider a mechanism that reconciles the source bucket's content for zipped files that e.g. existed prior to the trigger's deployment
I'm following the Design Automation API v3 tutorial for Revit.
When doing a workitem post I'm a little unclear about the "rvtFile" and "result" arguments. Can the rvtFile url be in an aws bucket? Also what are the restrictions for the result website? It states that it needs to be a signed url, but can this just be another aws bucket? Or do I need to create a website? (Note: I've never done any web development. Everything I know i learned from this tutorial)
Since Design Automation for Revit runs on cloud (and not your local machine), it needs a way to download your input files. You may put your files on any of the storage service providers (say Amazon S3) and provide direct download links to it. For Design Automation to have access to it, you will either need to make those files be public urls or keep them private and generate a signed url for it. When DA4R runs your workitem, the direct download urls provided in the workitem payload will be called to download your files to the worker machine.
Design Automation also does not store any of your result files. So, you will have to generate a signed url for uploading them to appropriate cloud location(s) (say a location in Amazon S3 bucket).
While Amazon S3 is just an example, there are several other storage providers. I also recommend reading Autodesk Forge's Data management APIs:
https://forge.autodesk.com/api/data-management-cover-page/
EDIT:
Useful links
Tutorials: https://learnforge.autodesk.io/
AU Class: https://www.autodesk.com/autodesk-university/class/Revit-Data-Forge-How-Can-Design-Automation-Revit-API-Help-Me-2018
We currently use Jive Cloud N which can use the Rest API and allows the use of Custom Apps. Our UI devs have created an app which uses a JS GET to pull data from a JSON file for our "Birthdays and Anniversaries" tile.
At the moment, the JSON file is hosted on our UI dev's Google Cloud Apps account, but we wish to host it internally so we don't have to keep contacting them for changes.
I uploaded the file to our OneDrive for Business storage and created a public URL with full read permissions but the Jive platform is throwing an error trying to load the custom app.
The error is that the file
has been blocked by CORS policy: No "Access-Control-Allow-Origin"
header is present
Our dev said that to get it working on his Google Cloud App storage, he had to specify the allow-control-allow-origin field in the server's server app.yaml file. I don't know what this is and if there is an equivalent for ODfB/SharePoint.
To get to my question: How can I host this JSON file on ODfB or even somewhere on our Azure tenancy so that it can be used? Or am I better off trying to setup a Google Cloud App storage location and replicate our dev's setup? FYI - I'd prefer the former because we're using M$ for a number of cloud hosted services already.
Thanks in advance
To get to my question: How can I host this JSON file on ODfB or even somewhere on our Azure tenancy so that it can be used?
FYI - I'd prefer the former because we're using M$ for a number of cloud hosted services already.
Per my understanding, you could leverage Azure Blob Storage to store your JSON file, and you could use Microsoft Azure Storage Explorer to easily manage/share your files.
Moreover, You could manage anonymous read access to your containers and blobs, refer to this tutorial for more details. Also, you could leverage SAS to grant limited access to your storage account for other clients, you could follow this tutorial for getting started with SAS.
For a simple way, you could create your storage account and leverage Microsoft Azure Storage Explorer to manage/share your file as follows:
For cross domain accessing, you need to configure CORS Setting:
For sharing your file(blob), you could Set Container Public Access Level or leverage SAS to grant limited access to your file for other clients as follows:
Right click your container, select "Set Public Access Level":
Sample file for share: https://brucechen.blob.core.windows.net/brucechen/index.json
Also, you could right click your JSON file, click "Get Shared Access Signature":
Sample file for share: https://brucechen.blob.core.windows.net/brucechen/index.json?st=2017-02-28T08%3A04%3A00Z&se=2017-09-01T08%3A04%3A00Z&sp=r&sv=2015-12-11&sr=b&sig=rVkorHeNOd4j2YhkmmxZ6DfXVLf1FoN2smY6mNRIoWs%3D
I am trying to programmatically retrieve my company's app data from the Google Developer's Console, specifically the daily installs. I have found that Google recommends the gsutil tool to access the data programmatically through the Google Cloud Storage SDK. However, I beleive they charge for this service. I want a free way to programmatically retrieve the data, preferably as a JSON stream to avoid dealing with file downloads. I have found the "direct reporting" links, but I have problems authenticating when I try to use them, and I also have to do something with the actual files then.
Is there a way to get a JSON version of the data through OAuth2 or something without downloading an Excel file? Has anyone had to do this?
You should look into use the Core Reporting API.
There are client libraries available in a number of languages.
You should work through the Hello Analytics APIs to get started.
Java Script
PHP
Python
Java
A quick solution for building a dashboard would also be the Embed API.
Using the gsutil tool to access the company's storage bucket that are provided by google is a free service. I wrote a code that will run the gsutil code as a process through the command line and parsed the downloaded .csv files into a database for storage. OAuth2 was not necessary.
I want to develop an offline chrome application.
As in offline app SQL is not available , so what API can serve the following purpose.
=>Large Storage
=>Efficient method to set and get values
=>Fast
=>Secured (user cannot temper the data)
Confused between IndexDB and File System API
I have knowledge of web languages and how online apps can store data on server. But don't know much about how to save data offline.
It all depends on your needs.
The Chrome apps have couple of limitations. Because they must to be very fast some web API's are disabled. You can't use localStorage and webSql for example.
However in apps you have different set of storage options:
chrome.storage.local - equivalent for localStorage but asynchronous. you can also save/read many objects at once
chrome.storage.sync - same as above but data are shared between different app instances (on other browser's profiles or machines)
web filesystem API - well known web filesystem API that can keep any kind of file in protected, browser storage. User's do not have direct access to this files, only the app have
extension to the above: chrome.syncFileSystem - it works similar to the above but files saved using this API are synced between app's instances (e.g. different machines) using Google Drive as a back-end. However user's can't see synced files in Drive UI because they are hidden.
chrome.fileSystem API - another extension to the web filesystem API and it gives you access to the user's sandboxed local filesystem. You can read from and write to selected by the user locations.
IndexedDB - quoting the docs: IndexedDB is an API for client-side storage of significant amounts of structured data, which also enables high performance searches of this data using indexes.
other custom solutions saving data on some server and syncing changes in all instances
You can choose one of above. As I can see you'll probably want to use IndexedDB API. It is not SQL and it is different approach to saving data. If you never use it before try some sample app first. However it's fast, efficient and combining with unlimitedStorage permission also can set large amount of data.
I also suggesting you to read Offline First page in Chrome Apps documentation where are examples of solutions for making an app offline.