I am hosting my server for my website on AWS Elastic beanstalk. On my server I am storing files that are getting uploaded by end users (or myself) in an "Images" folder. Thus new files are getting created in the folder every time an image gets uploaded to the website.
How can i download the latest file of my server on E.B. with these new images. I can download the original zip file I uploaded but it doesn't have the new data in the folders.
TY
You should not be storing any files worth anything on the EB node.
if user uploads content, you should be uploading that in turn to S3, or your DB, or any kind of file storage. That is usually decided during architecture.
So while the actual answer is "this should never happened in the first place", I must precise the main reason is that auto-scaling can kill your nodes without you knowing. which would destroy the uploads. or bring new nodes, spreading your content through multiple nodes.
While I also understand this answer might not be helping you if you already done the mistake have content to be transfered out of the node. I would
disable autoscaling
enable termination protection on the node
transfer data via rsync/ssh/s3 or favorite different options by SSH
automate transfer method to be done a subsequent time.
implement new upload method for the future of your app
deploy new method so no new content is uploaded to previous storage location
re-transfer your date from old to new storage location.
disable termination protection and reenable autoscaling
make sure new nodes and receiving traffic and why not kill that previous node
remember servers are cattle not pets
Related
My goal is to make a website (hosted on Google's App Engine through a Bucket) that includes a upload button much like
<p>Directory: <input type="file" webkitdirectory mozdirectory /></p>
that prompts users to select a main directory.
The main directory will first generate a subfolder and have discrete files being written every few seconds, up to ~4000 per subfolder, upon which the machine software will create another subfolder and continue, and so on.
I want Google Bucket to automatically create a Bucket folder based on metadata (e.g. user login ID and time) in the background, and the website should monitor the main directory and subfolders, and automatically upload every file, sequentially from the time they are finished being written locally, into the Cloud Bucket folder. Each 'session' is expected to run for ~2-5 days.
Creating separate Cloud folders is meant to separate user data in case of multiple parallel users.
Does anyone know how this can be achieved? Would be good if there's sample code to adapt into existing HTML.
Thanks in advance!
As per #JohnHanely, this is not really feasible using a application. I also do not understand the use case entirely but I can provide some insight into monitoring Cloud Buckets.
GCP provides Cloud Functions:
Respond to change notifications emerging from Google Cloud Storage. These notifications can be configured to trigger in response to various events inside a bucket—object creation, deletion, archiving and metadata updates.
The Cloud Storage Triggers will help you avoid having to monitor the buckets yourself and can instead leave that to GCF.
Maybe you could expand on what you are trying to achieve with that many folders? Are you trying to create ~4,000 sub-folders per user? There may be a better path forward should we know more about the intended use of the data? Is seems you want hold data and perhaps a DB is better suited?
- Application
|--Accounts
|---- User1
|-------Metadata
|----User2
|------Meatadata
I am working on a production web site on a Google Compute instance.
I want to set up a staging site, and read that the quickest way to do that is to clone the production instance.
When attempting to clone it, I get the error:
Required 'compute.images.get' permission for 'projects/wordpress-sites-170807/global/images/SANITISED-template'
I've not been able to find any useful reference to Required 'compute.images.get' permission in any Google search.
Questions:
1. I only have Editor level permissions on this particular Cloud Platform console. Is this error specific to me as a user? (I am now an "Owner" of the project, so we've eliminated the likelihood of my personal permissions being an issue)
2. If this permissions issue is related to the instance itself, how do I go about changing the permissions so that it has the "compute.image.get" permission?
As discussed in this thread, using the clone button "Create Similar" button copies the configuration to a new template. It does not make a new identical instance with the exact content of your persistent disk. In your case, the configuration included the source image from a different project. Thus, Compute Engine tried to access that project and since you have no access, it threw the error.
If the goal is to clone the instance including the persistent disk you need to create a new snapshot or image from the persistent disk, and then to retain other configs, you may use the clone button, but change the source image to the snapshot or image you have created earlier.
If the goal is creating a new instance from that original image in the other project, you need IAM roles in that project. For further information about the subject check this document
UPDATE:
The Google Cloud Console interface was updated several weeks ago, and the "CLONE" button was replaced by "CREATE SIMILAR".
I'm trying to use and modify the extract.autodesk.io (thanks to Cyrille Fauvel) but not yet successful. In a nut shell, this is what I want to do:
user drag-drop the design file (i'm ok with this)
I've removed the submit button - so right after uploading, extraction should begin in autodesk's server. (i've added a .done to trigger the auto-extraction : uploadFile (uri).done(function(){SubmitProjectDirect();}); )
no need to load a temp viewer for view/test
automatically download the bubble in zip file into our local server folder.
Delete uploaded model right away as our projects are mostly strictly confidential.
I'm encountering a 405 'Method not allowed' on 'api/file' sub folder, which I believe it should be autodesk's folder in the server.
Can anyone point the root urn of api/file?
I seem to get stuck on item 2 above due to the 405 error. But if get passed that one, I still need to solve 3, 4 and 5.
Appreciate any help...
In light of the additional comment above, the issue is a bit more complicated than I thought originally. In order to upload a file on the Autodesk cloud storage, you need to use specific endpoints, with a PUT verb and provide an oAuth Access Token.
It should be possible to setup the Flow.js to use all the above, but since it is a javascript library running on your client, it means anyone can steal your access token and use it illegitimately to either access your data, or consume your cloud credit to do action on your behalf.
Another issue is that the OSS minimum chunk is 5Mb - see this article, so you need to control this as well as providing OSS the byte assembly range information.
I would not recommend uploading to OSS from the client directly for security reason, but if you do not want to store on your server as a temporary storage, we can either proxy the Flow.js upload on the OSS storage or pipe the uploaded chunk on the Autodesk cloud storage. Both of the solution will be secured with no storage on your server, but traffic will continue to go via your server. I will create a branch on the github repo in a few days to demonstrate both approach.
I have some data for a webapp that I would like to store on the server. What would be a good location to put those files?
I have a couple of static HTML pages that contain instance specific information. They need to survive a re-deploy of the webapp. They need to be editable by the server's administrator. They are included in other HTML pages using the html object tag.
I want to store preferences on the server, but cannot use a database. I am using JSP to write and read the preferences. There is no sensitive data in the preferences. Currently I am using the log directory. But obviously that is not a great choice.
I am using Tomcat. I thought of creating an appdata/myapp directory under the webapp directory. Is that good or bad?
If the server's administrator can also deploy the app, I would add the data file itself into the source control for the app, and deploy it all together. This way you get revision control of the data, and you get the ability to revert to known good data if the server fails.
If the administrator can't deploy the app, but can only edit the file, then you need plans to back up that file in the case that the server or server filesystem dies.
A third solution would be a hybrid: put the app in one source code repository. Put the data in a second source code repository. The administrator can edit the data and deploy the data. The developer can edit the app source code, and deploy the source code. This way, both are revision controlled, but you've separated responsibility for who maintains what.
I am getting ready to develop an HTML5 application. It's not a game but for sake of discussion let's call it a simple game.
Let's say I host the app/game on my web server.
User clicks to my web server, downloads the game (may take several seconds, to a minute?), sends a high score to the server, is put on a leaderboard.
Tomorrow the user comes back and wants to play again. Does the user need to download the game again?
Or, is this where the cache manifest can be used to prevent a re-download?
And, if they don't have to download the whole thing everytime, how can I force a download if I want an update to the game?
Browser Cache
If you do nothing the browser will download and cache copies of any static files your application uses the first time it encounters them. This is automatic and the files are stored in the browser cache.
The next time the file is requested by your application the browser will send a request to the server and include the date and time of the copy it already has. If the file has changed on the server a new copy will be provided. If not, the server will respond with a 304 code - Not modified and the browser will use its existing copy. In this case a request is always sent to the server, but the file is only downloaded if it's changed.
You can configure your server to add an expiry date to static files. The browser still caches the file locally as before, but on the next access the browser checks the expiry date sent with the copy it already has. If that date hasn't passed no request is sent to the server. Now, you've saved the round-trip time of every request that refers to a cached file.
A manifest
A manifest works differently. When the initial file is downloaded the cache maifest is read and every file listed in it (apart from some specific exceptions dependent on the content of the manifest) is downloaded and store in the application store. Often, this requires the user to give permission for it to happen.
On subsequent access the manifest file is requested from the server. If the manifest hasn't changed nothing more is done and the application is loaded and run from files stored in the application store.
if the manifest has changed in any way then the new manifest is used to download and update the existing application store files.
Pros and cons
Browser cache
For
Automatic & transparent
Largely maintenance free
Many files need not be downloaded or checked
Against
Distribution of new files may be delayed
Application must run online
Manifest
For
Application can run offline
Manifest gives good control over update distribution
Against
User permission often required to use the application store
Additional maintenance is required to update the manifest appropriately.