First, the system architecture:
Server: Running IIS ASP and delivering data to a hundred or so WinXP+ clients in the field upon automated requests from those clients. Data sent from the server is large graphic or video files. If a file is placed on the server by a user, these remote clients will "discover" it and download the file.
Clients: As stated above, the clients are remote unattended boxes that fetch content from the server. The end purpose is digital signage.
Problem: All clients hitting the server at the same time makes for slow transfers of large files - not enough bandwidth.
Solution (I think): Use Google Cloud Storage or Google Drive to hold the files and have the clients request (automated and unattended) those files. I think Google would have a higher available bandwidth (at least the NSA thinks so).
Questions:
Which is a better solution between Google Cloud Storage and Google Drive?
Is it possible to use Windows PowerShell or WScript to run scripts to interact with Google? Reason is that I need to avoid installing new software on the client machines that might require user interaction.
Yes you can use powershell as long as you can urlfetch https data. The oauth flow might be tricky to get working, follow examples for installed apps.
100% use cloud storage instead of drive. Drive is not meant to scale with simultaneous downloads and has several quotas so you will need to implement exponential backoff etc with drive.
Yes you can use Drive or Cloud Storage. I would go for Drive over Cloud Storage, because :-
It's free, Cloud Storage will cost you and so you have to worry about your credit card expiring
It's easier to program since it's a simple http GET to retrieve your files
You need to think about your security model. With Drive you could (nb not should), make the files public. Provided your clients can be informed of the URL, then there is no OAuth to worry about. If you need better security, install a Refresh Token on each client. Before each download, your client will make a call to Google to convert the refresh token to an access token. I suggest prototype without OAuth to begin with. Then if (a) it fits, and (b) you need more security, add OAuth.
The Drive web app gives you your management console for the downloadable files. If you use Cloud Storage, you'll need to write your own.
The quota issue is discussed here Google Drive as a video hosting/streaming platform?
Because the quota isn't documented, we can only guess at what the restrictions are. It seems to be bandwidth for a given file, so the larger the file, the fewer the number of downloads. A simple workaround is to use the copy API https://developers.google.com/drive/v2/reference/files/copy to make multiple copies of the file.
You have other options too. Since these are simply static files, you could host them on Google Sites or Google App Engine. you could also store them within App Engine datastore which has a free quota.
Finally, you could even consider a BitTorrent approach.
Related
I have a PC connected to the network in my small company, and the folder related to important data are synchronized via Google Drive Sync, if unfortunately a ransomware or other virus should encrypt or damage the data, Google Drive would synchronize the new corrupt data?
Yes, all synchronizers spread the infection. If you want to have the files safe from viruses, look for a solution in the cloud that does not have the files locally. Thus a hypothetical virus could not reach them. Take a look to Dataprius, Sharepoint or Documentum.
My problem is that I need to build a cloud storage for my customers/clients/users, who can log in to my Cloud Storage Service.
I need to understand how they work in the back end, or how they’re developed, or how can I build my solution using a server, where I can thin provision my hard drive, let users see their data, etc. What resources and articles, along with the required skills, can I use? Or is there a software like WordPress is for websites?
Some additional points to better understand the problem:
How does Google Drive or Dropbox work in the background? Do they create a folder directory or a disk drive partition for each user?
Some of what I have in my mind: I develop a website where users purchase a plan of say 10 GB. The site then sends the userId, password, plan information to my Cloud Server, where I can assign storage to him.
At first, I thought to solve the problem with a root folder, where each new user will have a folder of his own. But that's where I found my first stumbling block: how to assign a size limit to a folder?
I also need to use the free storage (that the user is not using) to allocate to other users. And I don't think that can be done in directories (correct me if I'm wrong).
So far, I've searched about cloud storage, folder sizing, thin provisioning, public cloud, private cloud, etc. Most of the courses I've found so far teach about Amazon, Google, etc. However, I need to host my own cloud service.
Someone suggested to use Nextcloud or Syncthing, but they are not what I'm looking for (according to my understanding).
1- Syncthing works off of a peer-to-peer architecture rather than a client-server architecture.
2- NextCloud, from what I get, offers cloud storage for myself.
I apologize for the long post, but I'm in a real bind here. Any help will be much appreciated.
Nextcloud does what you want. You can create group folders and give permissions to registered users or groups. You can share files or folders with external users. It is not only for single private users. You have NC instances with 1000's of users or more
We are trying to upload file to GCP console, but the performance is impacting our data transfer , being a cloud hosted its taking much longer to upload as compared to a Azure or AWS services. Any suggestion?
There could be multiple reasons why you are experiencing slow uploads. You must verify and troubleshoot on the following lines:
Is this purely related a case fo network bandwidth/congestion? Try upgrading your network bandwidth. If your case is more enterprise in nature, have you explored about Dedicated Interconnect
Try disabling, versioning/encryption and other miscellaneous object-store features before the upload - they influence upload speed
Are you copying data to the closest region where your bucket is launched? try changing your bucket relocation if this is not the case
Have you considered a multi-file upload or compressed file upload strategy? This is result in faster upload speeds also.
I'm developing a Google Apps Script Web App. This web app is the next version of FlipVideo, a script that I previously developed in a Google Sheet (see version at https://sites.google.com/view/flipvideo/)
This is a school oriented web app. A lot of students will access at the same time from around the world (thousands probably). I'm worried about scalability and quota limits.
I will publish the web app using "Execute the apps as: user accessing the web app" and "Who has access the app: anyone". Could this configuration allow thousands of concurrently executions or there is a hard quota impossible to scale?
Unfortunately, what you require is not currently possible. GAS Web Apps have a max of 30 concurrent users (see simultaneous executions under current limitations on the quotas page). You'll need to leverage a different solution to operate at such a large scale. If you want to stick with Google's Cloud Platform I would recommend using Cloud Functions. However, its not a free service.
There are too many scripts running simultaneously for this Google user
account. This indicates that you have too many scripts executing at
once, although not necessarily the same script. Like the exception
above, this most commonly occurs for custom functions that are called
repeatedly in a single spreadsheet.
This above limitation means a single user cannot have the same script executing 30 times. For example, you can achieve this limit by opening the same web app in 30 different tabs. So I guess the limit on the number of users does not exist, or else App Script would lose all meaning as an IDE.
I'm developing a web application that's going to start with 200gb of data to be storaged. Over the years, the same application possibly can reach 1tb, perhaps 2tb in 5 years.
What I want from this application is the clients to upload files to the server and the server then upload files do Google Drive, persisting the webviewlink on database. It's working this way on localhost.
I know two options for authentication for Google Drive API: client account and service account.
Service Account's option fits better for me because I want the server to have control of the files, not the client have control.
But Service Account can storage too few data and the storage limit can't be increased. The limit is something around 15gb I guess, not sure.
If the Service Account will not help me, what options would I have to storage 2tb of data or more? Should I find another way to storage the files?
I'd like to stay using Google. If there's not any option using Google Drive API, please, suggest anything else for this scenario.
You have a couple of options.
Use a regular account instead of a Service Account. You will still need to pay for the storage, but it will work and you'll have everything in a single account. From your question "I want the server to have control of the files, not the client have control" I suspect you have looked at the OAuth quickstart examples and concluded that only end users can grant access. That's not the case. It's perfectly valid, and really quite simple, for your server app to grant access to an account it controls. See How do I authorise an app (web or installed) without user intervention? for how to do this.
Use multiple Service Accounts and shard your data across them. The various accounts could all share their folders to a kinda master account which would then have a coherent view of the entire corpus.
Personally I'd go with option 1 because it's the easiest to set up and manage.
Either way, make sure you understand how Google will want to charge you for the storage. For example, although each Service Account has a free quota, it is ultimately owned by the regular user that created it and the standard user quota limits and charges probably apply to that user.