I am working on a project that let users to create their own web pages. I have two options in my mind store the html code in mysql db, or create html doc in the file systems of the server.
I am looking for speed and efficiently as the html codes will be loaded a lot of times.
my question is what would be the best approach? I will use AWS as cloud. Is there any feature in AWS that can help with caching of html docs? would using CDN can help in this case?
Use S3 for static files(Static web hosting) and use cloudfront for cache.
The best approach would be to use AWS S3 to store your html files and use AWS CloudFront CDN to cache the content at edge locations.
For more details check Using CloudFront with Amazon S3.
If you have your own domain and would like to have a valid certificate you could use https://www.cloudflare.com/ssl next you could just store your static page in an AWS S3 bucket, follow this guide: http://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosting.html
so at the end, you have something like:
(internet) ---> cloudflare(CDN + SSL) ---> AWS S3
More info also can be found here: https://support.cloudflare.com/hc/en-us/articles/200168926-How-do-I-use-Cloudflare-with-Amazon-s-S3-Service-
Related
I have set up an S3 bucket to host static files.
When using the website endpoint (http://.s3-website-us-east-1.amazonaws.com/): it forces me to set an index file. When the file isn't found, it throws an error instead of listing directory contents.
When using the s3 endpoint (.s3.amazonaws.com): I get an XML listing of the files, but I need an HTML listing that users can click the link to the file.
I have tried setting the permissions of all files and the bucket itself to "List" for "Everyone" in the AWS Console, but still no luck.
I have also tried some of the javascript alternatives, but they either don't work under the website url (that redirects to the index file) or just don't work at all. As a last resort, a collapsible javascript listing would be better than nothing, but I haven't found a good one.
Is this possible? If so, do I need to change permissions, ACL or something else?
I've created a simple bit of JS that creates a directory index in HTML style that you are looking for: https://github.com/rgrp/s3-bucket-listing
The README has specific instructions for handling Amazon S3 "website" buckets: https://github.com/rgrp/s3-bucket-listing#website-buckets
You can see a live example of the script in action on this s3 bucket (in website mode): http://data.openspending.org/
There is also this solution: https://github.com/caussourd/aws-s3-bucket-listing
Similar to https://github.com/rgrp/s3-bucket-listing but I couldn't make it work with Internet Explorer. So https://github.com/caussourd/aws-s3-bucket-listing works with IE and also add the possibility to order the files by names, size and date. On the downside, it doesn't follow folders: only the files at one level are displayed.
This might solve your problem. Security settings for Everyone group:
(you need the bucketexplorer.com software for this)
If you are sharing files of HTTP, you may or may not want people to be able to list the contents of a bucket (folder.) If you want the bucket contents to be listed when someone enters the bucket name (http://s3.amazonaws.com/bucket_name/), then edit the Access Control List and give the Everyone group the access level of Read (and do likewise with the contents of the bucket.) If you don’t want the bucket contents list-able but do want to share the file within it, disable Read access for the Everyone group for the bucket itself, and then enable Read access for the individual files within the bucket.
I created a much simpler solution. Just place the index.html file in root of your folder and it will do the job. No configuration required. https://github.com/prabhatsharma/s3-directorylisting
I had a similar problem and created a JavaScript-and-iframe solution that works pretty well for listing directories in S3 website files. You just have to drop a couple of .html files into the directory you want to list. You can find it here:
https://github.com/adam-p/s3-file-list-page
I found s3browser, which allowed me to set up a directory on the main web site that allowed browsing of the s3 bucket. It worked very well and was very easy to set up.
Using another approach base in pure JavaScript and AWS SDK JavaScript API. Not need PHP or other engine just pure web site (Apache or even IIS).
https://github.com/juvs/s3-bucket-browser
Not intent for deploy on your own bucket (for me, no make sense).
Using the new IAM Users from AWS you can provide more specific and secure access to your buckets. No need to publish your bucket to website and make all public.
If you want secure the access, you can use the conventional methods to authenticate users for your current web site.
Hope this help too!
I am working on Highlevel multipart fileupload on aws sdk using java.But i need few clarification on below points .please help me!!
1 : What happens if there was network failure during uploading file parts?How do we re-upload ? will re-upload start from the beginning or will it get resume?
2 : Which provide better performance comparatively between low level and high level api
3 : any websites that provide Production ready code.
Thanks in advance..............
Resuming completly depends on how you have handled your implementation where you hold upload id and last part number that was uploaded. This is all you need to know to start resuming uploads in s3. That is the whle benfit of AWS S3 multipart Uploads.
Obviously High level API are good for your application if you really look forward to get progress of every upload you make. But that again depends on your application. for java sdk AWS Documentaion is too good.
Please go through AWS Java SDK Examples :https://docs.aws.amazon.com/AmazonS3/latest/dev/usingHLmpuJava.html
If you are implementaing in Java AWS Docs will be sufficient
I have an S3 account, on which are stored videos which I need to play using an HTML5 Video element inside a rails application.
Is it possible for me to be able to load the videos into the player without making the video public and directly linking to it?
I have the gem aws-sdk and I read through the documentation and didn't find any information about this.
Excuse me? How is it unclear what I'm asking? It's not really possible to make it clearer so I guess I'll basically restate what I already said.
I need to be able to this:
<video src="<url of file on S3>"> without making the file public in S3 so anybody can download it.
Look ar presigned URLs. Use the Presigner in the ruby sdk on the server to generate a presigned URL (for get method) for the key and insert that into the html.
There is an article page about doing this using the aws-sdk.
http://docs.aws.amazon.com/AmazonS3/latest/dev/UploadObjectPreSignedURLRubySDK.html
I've only ever used Paperclip to do so.
https://github.com/thoughtbot/paperclip/wiki/Restricting-Access-to-Objects-Stored-on-Amazon-S3
To access private content in S3 from your rails application, there are multiple approaches available
Create a Sign URL using Ruby AWS SDK from your Rails Backend and return the URL to the browser so that your HTML5 player can access them
Using AWS STS and AWS Ruby SDK at Rails Backend, generate a temporarily access token (Which has grants to s3 bucket) and return it to the browser, where browser can use AWS JS SDK + Token to request the file
For better performance use AWS Cloufront RTMP distributions for the Videos which serves the files from S3 with Signed URLs and Origin Access Identity limiting public access
I am creating a Virtual reality 360-degree video website using the krpano html5 player.
This was going great until testing on safari and I realised it didn't work. The reason for this is because safari does not support CORS (cross-origin resource sharing) for videos going through WebGL.
To clarify, if my videos where on the same server with my application files it would work, but because I have my files hosted on Amazon s3 , they are CORS. Now I'm unsure what to do because I have built my application on digital ocean which connects to my amazon s3 bucket, but I cannot afford to up my droplet just to get the storage I need(which is around 100GB to start and will increase in the future to Terrabytes and my video collection gets bigger).
So does anyone know a way I can get around this to make it seem like the video is not coming from a different origin or alternatively anything I can do to get past this obstacle?
Is there any way that I could set up amazon s3 and amazon EC2 so that they dont see each other as cross-origin resource sharing?
EDIT:
I load my videos like this:
<script>
function showVideo(){
embedpano({
swf:"/krpano/krpano.swf",
xml:"/krpano/videopano.xml",
target:"pano",
html5:"only",
});
}
</script>
This then calls my xml file which calls the video file:
<krpano>
<!-- add the video sources and play the video -->
<action name="add_video_sources">
videointerface_addsource(‘medium', 'https://s3-eu-west-1.amazonaws.com/myamazonbucket/Shoots/2016/06/the-first-video/videos/high.mp4|https://s3-eu-west-1.amazonaws.com/myama…ideos/high.webm');
videointerface_play(‘medium');
</action>
</krpano>
I don't know exactly how krpano core works, I assume it the javascript gets the URLs from the XML file and then makes a request to pull them in.
#datasage mentions in comments that CloudFront is a common solution. I don't know if this is what he was thinking of but it certainly will work.
I described using this solution to solve a different problem, in detail, on Server Fault. In that case, the question was about integrating the main site and "/blog/*" from a different server under a single domain name, making a unified web site.
This is exactly the same thing you need, for a different reason.
Create a CloudFront distribution, setting the alternate domain name to your site's name.
Create two (or more) origin servers pointing to your dynamic and static content origin servers.
Use one of them as default, initially handling all possible path patterns (*, the default cache behavior) and then carve out appropriate paths to point to the other origin (e.g. /asset/* might point to the bucket, while the default behavior points to the application itself).
In this case, CloudFront is being used other than for its primary purpose as a CDN and instead, we're leveraging a secondary purpose, using it as a reverse proxy that can selectively route requests to multiple back-ends, based on the path of the request, without the browser being aware that there are in fact multiple origins, because everything sits behind the single hostname that points to CloudFront (which, obviously, you'll need to point to CloudFront in DNS.)
The caching features can be disabled if you don't yet want/need/fully-understand them, particularly on requests to the application itself, where disabling caching is easily done by selecting the option to forward all request headers to the origin, in any cache behavior that sends requests to the application itself. For your objects in S3, be sure you've set appropriate Cache-Control headers on the objects when you uploaded them, or you can add them after uploading, using the S3 console.
Side bonus, using CloudFront allows you to easily enable SSL for the entire site, with a free SSL certificate from Amazon Certificate Manager (ACM). The certificate needs to be created in the us-east-1 region of ACM, regardless of where your bucket is, because that is the region CloudFront uses when fetching the cert from ACM. This is a provisioning role only, and has no performance implications if your bucket is in another region.
You need to allow your host in CORS Configuration of your AWS-S3 bucket .
Refer to Add CORS Configuration in Editing Bucket Permissions.
Hence after that, every request you make to the S3 bucket files, will have CORS headers set.
In case you need to serve the content via AWS-CDN CloudFront then follow these steps, ignore if you server content directly via S3 :
Go to AWS CloudFront Console.
Select your CloudFront Distribution.
Go to Behaviors Tab.
Create a Behavior(for the files which needs to be served with CORS Header).
Enter Path Pattern, Select Protocol & Methods.
Select All in Forward Headers option.
Save the behavior.
If needed, Invalidate the CloudFront Edge Caches by running an Invalidation request for the Files you just allowed for CORS.
Just getting going with Jekyll - running it on Debian, base install, base config.yml.
I am serving some static resources from a url ( and folder named the same )
/assets
How would I go about setting some cache headers on stuff in there ?
Done much searching and reading the docs and not much finding. Config setting ??
Hope I can be pointed in the right direction.
Cheers!
Rob
Headers, gzipping, and such would be handled by your web server, and the exact configuration mechanism and details will differ from server to server.
One great alternative is the s3_website script (https://github.com/laurilehmijoki/s3_website). It syncs your static files with S3 from AWS and can set headers and also gzip selected files. It works really nice and does all with one command.