Can't access arrayBuffer on RangeRequest - google-chrome

Trying to solve the problem referenced in this article: https://philna.sh/blog/2018/10/23/service-workers-beware-safaris-range-request/
and here:
PWA - cached video will not play in Mobile Safari (11.4)
The root problem is that we aren't able to show videos on Safari. The article says it has the fix for the issue but seems to cause another problem on Chrome. A difference in our solution is that we aren't using caching. Currently we just want to pass through the request in our service worker. Implementation looks like this:
self.addEventListener('fetch', function (event){
if (event.request.cache === 'only-if-cached' && event.request.mode !== 'same-origin') {
return;
}
if (event.request.headers.get('range')) {
event.respondWith(returnRangeRequest(event.request));
} else {
event.respondWith(fetch(event.request));
}
});
function returnRangeRequest(request) {
return fetch(request)
.then(res => {
return res.arrayBuffer();
})
.then(function(arrayBuffer) {
var bytes = /^bytes\=(\d+)\-(\d+)?$/g.exec(
request.headers.get('range')
);
if (bytes) {
var start = Number(bytes[1]);
var end = Number(bytes[2]) || arrayBuffer.byteLength - 1;
return new Response(arrayBuffer.slice(start, end + 1), {
status: 206,
statusText: 'Partial Content',
headers: [
['Content-Range', `bytes ${start}-${end}/${arrayBuffer.byteLength}`]
]
});
} else {
return new Response(null, {
status: 416,
statusText: 'Range Not Satisfiable',
headers: [['Content-Range', `*/${arrayBuffer.byteLength}`]]
});
}
});
}
We do get an array buffer returned on the range request fetch but it has a byteLength of zero and appears to be empty. The range header actually contains "bytes=0-" and subsequent requests have a start value but no end value.
Maybe there is some feature detection we can do to determine that it's chrome and we can just call fetch regularly? I'd rather have a solution that works everywhere though. Also res is showing type:"opaque" so maybe that has something to do with it? Not quite sure what to look at next. If we can't solve the problem for Chrome I might need a different solution for Safari.

It seems that it was the opaque response. I didn't realize that fetch was 'nocors' by default. Adding 'cors' mode and overwriting the range header seems to have allowed the rewrite to work on chrome. Sadly, it still doesn't work on Safari, but I was able to access the arrayBuffer after setting the cors values properly.
Here is the change I had to make:
var myHeaders = {};
return fetch(request, { headers: myHeaders, mode: 'cors', credentials: 'omit' })
.then(res => {
return res.arrayBuffer();
})
It's important that the server respond with allowed headers. e.g.
access-control-allow-methods: GET
access-control-allow-origin: *

Related

Verify an image exists at a URL when HEAD is not allowed

Using HttpClient in C#, I'm trying to verify that an image exists at a given URL without downloading the actual image. These images can come from any publicly accessible address. I've been using the HEAD HTTP verb which seems to work for many/most. Google drive images are proving difficult.
Given a public share link like so:
https://drive.google.com/file/d/1oCmOEJp0vk73uYhzDTr2QJeKZOkyIm6v/view?usp=sharing
I can happily use HEAD, get a 200 OK and it appears to be happy. But, that's not the image. It's a page where one can download the image.
With a bit of mucking around, you can change the URL to this to actually get at the image, which is what I really want to check:
https://drive.google.com/uc?export=download&id=1oCmOEJp0vk73uYhzDTr2QJeKZOkyIm6v
But, hitting that URL with HEAD results in a 405 MethodNotAllowed
Luckily, if the URL truly doesn't exist, you get back a 404 NotFound
So I'm stuck at the 405. What is my next step (NOT using Google APIs) when HEAD is not allowed? I can't assume it's a valid image if it simply doesn't 404. I check the Content-type to verify it's an image, which has issues outside the scope of this question.
HttpClient allows us to issue an http request where you can specify that you are interested about only the headers.
The trick is to pass an HttpCompletionOption enum value to the SendAsync or any other {HttpVerb}Async method:
| Enum name | Value | Description |
|---------------------|-------|---------------------------------------------------------------------------------------------------------------------|
| ResponseContentRead | 0 | The operation should complete after reading the entire response including the content. |
| ResponseHeadersRead | 1 | The operation should complete as soon as a response is available and headers are read. The content is not read yet. |
await client.GetAsync(targetUrlWhichDoesNotSupportHead, HttpCompletionOption.ResponseHeadersRead);
Here is an in-depth article that details how does this enum changes the behavior and performance of the HttpClient.
The related source code fragments:
in case of .NET Framework
in case of .NET Core
Brilliant, Peter! Thank you.
Here's my full method for anyone who may find it useful:
public async Task<bool> ImageExists(string urlOrPath)
{
try
{
var uri = new Uri(urlOrPath);
if (uri.IsFile)
{
if (File.Exists(urlOrPath)) return true;
_logger.LogError($"Cannot find image: [{urlOrPath}]");
return false;
}
using (var result = await Get(uri))
{
if (result.StatusCode == HttpStatusCode.NotFound)
{
_logger.LogError($"Cannot find image: [{urlOrPath}]");
return false;
}
if ((int)result.StatusCode >= 400)
{
_logger.LogError($"Error: {result.ReasonPhrase}. Image: [{urlOrPath}]");
return false;
}
if (result.Content.Headers.ContentType == null)
{
_logger.LogError($"No 'ContentType' header returned. Cannot validate image:[{urlOrPath}]");
return false;
}
if(new[] { "image", "binary"}.All(v => !result.Content.Headers.ContentType.MediaType.SafeTrim().Contains(v)))
{
_logger.LogError($"'ContentType' {result.Content.Headers.ContentType.MediaType} is not an image. The Url may point to an HTML download page instead of an actual image:[{urlOrPath}]");
return false;
}
var validTypes = new[] { "jpg", "jpeg", "gif", "png", "bmp", "binary" };
if(validTypes.All(v => !result.Content.Headers.ContentType.MediaType.SafeTrim().Contains(v)))
{
_logger.LogError($"'ContentType' {result.Content.Headers.ContentType.MediaType} is not a valid image. Only [{string.Join(", ", validTypes)}] accepted. Image:[{urlOrPath}]");
return false;
}
return true;
}
}
catch (Exception e)
{
_logger.LogError($"There was a problem checking the image: [{urlOrPath}] is not valid. Error: {e.Message}");
return false;
}
}
private async Task<HttpResponseMessage> Get(Uri uri)
{
var response = await _httpCli.SendAsync(new HttpRequestMessage(HttpMethod.Head, uri));
if (response.StatusCode != HttpStatusCode.MethodNotAllowed) return response;
return await _httpCli.SendAsync(new HttpRequestMessage() { RequestUri = uri }, HttpCompletionOption.ResponseHeadersRead);
}
Edit: added a Get() method which still uses HEAD and only uses ResponseHeadersRead if it encounters MethodNotAllowed. Using a live scenario I found it was much quicker. Not sure why. YMMV

GDrive API v3 files.get download progress?

How can I show progress of a download of a large file from GDrive using the gapi client-side v3 API?
I am using the v3 API, and I've tried to use a Range request in the header, which works, but the download is very slow (below). My ultimate goal is to playback 4K video. GDrive limits playback to 1920x1280. My plan was to download chunks to IndexedDB via v3 API and play from the locally cached data. I have this working using the code below via Range requests, but it is unusably slow. A normal download of the full 438 MB test file directly (e.g. via the GDrive web page) takes about 30-35s on my connection, and, coincidentally, each 1 MB Range requests takes almost exactly the same 30-35s. It feels like the GDrive back-end is reading and sending the full file for each subrange?
I've also tried using XHR and fetch to download the file, which fails. I've been using the webContent link (which typically ends in &export=download) but I cannot get access headers correct. I get either CORS or other odd permission issues. The webContent links work fine in <image> and <video> src tags. I expect this is due to special permission handling or some header information I'm missing that the browser handles specifically for these media tags. My solution must be able to read private (non-public, non-sharable) links, hence the use of the v3 API.
For video files that are smaller than the GDrive limit, I can set up a MediaRecorder and use a <video> element to get the data with progress. Unfortunately, the 1920x1080 limit kills this approach for larger files, where progress feedback is even more important.
This is the client-side gapi Range code, which works, but is unusably slow for large (400 MB - 2 GB) files:
const getRange = (start, end, size, fileId, onProgress) => (
new Promise((resolve, reject) => gapi.client.drive.files.get(
{ fileId, alt: 'media', Range: `bytes=${start}-${end}` },
// { responseType: 'stream' }, Perhaps this fails in the browser?
).then(res => {
if (onProgress) {
const cancel = onProgress({ loaded: end, size, fileId })
if (cancel) {
reject(new Error(`Progress canceled download at range ${start} to ${end} in ${fileId}`))
}
}
return resolve(res.body)
}, err => reject(err)))
)
export const downloadFileId = async (fileId, size, onProgress) => {
const batch = 1024 * 1024
try {
const chunks = []
for (let start = 0; start < size; start += batch) {
const end = Math.min(size, start + batch - 1)
const data = await getRange(start, end, size, fileId, onProgress)
if (!data) throw new Error(`Unable to get range ${start} to ${end} in ${fileId}`)
chunks.push(data)
}
return chunks.join('')
} catch (err) {
return console.error(`Error downloading file: ${err.message}`)
}
}
Authentication works fine for me, and I use other GDrive commands just fine. I'm currently using drives.photos.readonly scope, but I have the same issues even if I use a full write-permission scope.
Tangentially, I'm unable to get a stream when running client-side using gapi (works fine in node on the server-side). This is just weird. If I could get a stream, I think I could use that to get progress. Whenever I add the commented-out line for the responseType: 'stream', I get the following error: The server encountered a temporary error and could not complete your request. Please try again in 30 seconds. That’s all we know. Of course waiting does NOT help, and I can get a successful response if I do not request the stream.
I switched to using XMLHttpRequest directly, rather than the gapi wrapper. Google provides these instructions for using CORS that show how to convert any request from using gapi to a XHR. Then you can attach to the onprogress event (and onload, onerror and others) to get progres.
Here's the drop-in replacement code for the downloadFileId method in the question, with a bunch of debugging scaffolding:
const xhrDownloadFileId = (fileId, onProgress) => new Promise((resolve, reject) => {
const user = gapi.auth2.getAuthInstance().currentUser.get()
const oauthToken = user.getAuthResponse().access_token
const xhr = new XMLHttpRequest()
xhr.open('GET', `https://www.googleapis.com/drive/v3/files/${fileId}?alt=media`)
xhr.setRequestHeader('Authorization', `Bearer ${oauthToken}`)
xhr.responseType = 'blob'
xhr.onloadstart = event => {
console.log(`xhr ${fileId}: on load start`)
const { loaded, total } = event
onProgress({ loaded, size: total })
}
xhr.onprogress = event => {
console.log(`xhr ${fileId}: loaded ${event.loaded} of ${event.total} ${event.lengthComputable ? '' : 'non-'}computable`)
const { loaded, total } = event
onProgress({ loaded, size: total })
}
xhr.onabort = event => {
console.warn(`xhr ${fileId}: download aborted at ${event.loaded} of ${event.total}`)
reject(new Error('Download aborted'))
}
xhr.onerror = event => {
console.error(`xhr ${fileId}: download error at ${event.loaded} of ${event.total}`)
reject(new Error('Error downloading file'))
}
xhr.onload = event => {
console.log(`xhr ${fileId}: download of ${event.total} succeeded`)
const { loaded, total } = event
onProgress({ loaded, size: total })
resolve(xhr.response)
}
xhr.onloadend = event => console.log(`xhr ${fileId}: download of ${event.total} completed`)
xhr.ontimeout = event => {
console.warn(`xhr ${fileId}: download timeout after ${event.loaded} of ${event.total}`)
reject(new Error('Timout downloading file'))
}
xhr.send()
})

How to handle multiple redirection in puppeteer?

I am trying to open a page after a form post inside evaluate. There are 2 redirections after form post which can be any number and then I find a final page.
I tried to handled it by putting below (2 times for 2 redirections) after evaluate in which form post happened.
await page.waitForNavigation({'waitUntil':'domcontentloaded'});
await page.waitForNavigation({'waitUntil':'domcontentloaded'});
The above worked properly but I have to handle the situations when any number of redirections can happen.
I won't have any specific selector on DOM as page might be different many times.
Puppeteer version: 1.4.0
Platform / OS version: Linux
URLs (if applicable): NA
Node.js version: 8.10.0
The below is part of code which I am using:
const formPost = await page.evaluate(a => {
var form = formBuilder("payment_post", "post", acsUrl);
for (var i in a) {
form.add(i, i, 'hidden', a[i]);
}
form.generate("pareqFormContainer");
form.submit();
return document.querySelector('#pareqFormContainer').innerHTML;
}, jsonData)
.then(function () {
logger.info("form submitted with pareq and MD for txnId : " + jsonData.txnId)
});
await page.waitForNavigation({'waitUntil' : 'domcontentloaded', 'timeout' : waitTimeOut});
await page.waitForNavigation({'waitUntil' : 'domcontentloaded', 'timeout' : waitTimeOut});

Sending nested object via post request

I'm running this little node express server, which is supposed to check if the voucher is valid later and then send an answer back to the client
this is my code
app.post('/voucher', function (request, response) {
response.setHeader('Access-Control-Allow-Origin', '*');
response.setHeader('Access-Control-Request-Method', '*');
response.setHeader('Access-Control-Allow-Methods', 'OPTIONS, GET');
response.setHeader('Access-Control-Allow-Headers', 'authorization, content-type');
if ( request.method === 'OPTIONS' ) {
response.writeHead(200);
response.end();
return;
}
console.log(request)
let results;
let body = [];
request.on('data', function(chunk) {
body.push(chunk);
}).on('end', function() {
results = Buffer.concat(body).toString();
// results = JSON.parse(results);
console.log('#### CHECKING VOUCHER ####', results)
let success = {success: true, voucher: {name: results,
xxx: 10}}
success = qs.escape(JSON.stringify(success))
response.end(success)
} )
}
);
It is obviously just an example and the actual check is not implemented yet. So far so good.
Now on the client side where I work with REACT, I can not seem to decode the string I just send there.
there I'm doing this
var voucherchecker = $.post('http://localhost:8080/voucher', code , function(res) {
console.log(res)
let x = JSON.parse(res)
console.log(x)
console.log(qs.unescape(x))
It gives me the error
Uncaught SyntaxError: Unexpected token % in JSON at position 0
When I do it the other way arround
let x = qs.unescape(res)
console.log(x)
console.log(JSON.parse(x))
Than it tells me
Uncaught TypeError: _querystring2.default.unescape is not a function
Maybe you can help me? I don't know what the issue is here. Thank you.
Also another question on this behalf, since I'm only a beginner. Is there smarter ways to do such things than I'm doing it now? I have react which renders on the client and I have a mini express server which interacts a few times with it during the payment process.
The both run on different ports.
What would be the standard way or best practice to do such things?
I'm a bit perplexed as to why your backend code has so much going on in the request.
Since you asked for if there is a different way to write this, I will share with you how I would write it.
Server
It seems that you want your requests to enable CORS, it also seems that you originally wanted to parse a JSON in your request body.
This is how I would recommend you re-write your endpoint
POST /voucher to take a request with body JSON
{
code: "xxxxx"
}
and respond with
{
success: true,
voucher: {
name: results,
xxx: 10
}
}
I would recommend you use express's middleware feature as you will probably use CORS and parse JSON in most your requests so in your project I would.
npm install body-parser
npm install cors
then in your app initialization
var bodyParser = require('body-parser')
var cors = require('cors')
var app = express()
// parse application/x-www-form-urlencoded
app.use(bodyParser.urlencoded({ extended: false }))
// parse application/json you can choose to just pars raw text as well
app.use(bodyParser.json())
// this will set Access-Control-Allow-Origin * similar for all response headers
app.use(cors())
You can read more about body-parser and cors in their respective repos, if you don't want to use them I would still recommend you use your own middleware in order to reduse future redundancy in your code.
So far this will substitute this part of your code
response.setHeader('Access-Control-Allow-Origin', '*');
response.setHeader('Access-Control-Request-Method', '*');
response.setHeader('Access-Control-Allow-Methods', 'OPTIONS, GET');
response.setHeader('Access-Control-Allow-Headers', 'authorization, content-type');
if ( request.method === 'OPTIONS' ) {
response.writeHead(200);
response.end();
return;
}
console.log(request)
let results;
let body = [];
request.on('data', function(chunk) {
body.push(chunk);
}).on('end', function() {
results = Buffer.concat(body).toString();
// results = JSON.parse(results);
Now your route definition can just be
app.post('/voucher', function (request, response) {
var result = request.body.code // added by body-parser
console.log('#### CHECKING VOUCHER ####', result)
// express 4+ is smart enough to send this as json
response.status(200).send({
success: true,
voucher: {
name: results,
xxx: 10
}
})
})
Client
your client side can then be, assuming $ is jquery's post function
var body = {
code: code
}
$.post('http://localhost:8080/voucher', body).then(function(res) {
console.log(res)
console.log(res.data)
return res.data
})

FineUploader OnComplete method not firing

So, I'm using FineUploader 3.3 within a MVC 4 application, and this is a very cool plugin, well worth the nominal cost. Now, I just need to get it working correctly.
I'm pretty new to MVC and absolutely new to passing back JSON, so I need some help getting this to work. Here's what I'm using, all within doc.ready.
var manualuploader = $('#files-upload').fineUploader({
request:
{
endpoint: '#Url.Action("UploadFile", "Survey")',
customHeaders: { Accept: 'application/json' },
params: {
//variables are populated outside of this code snippet
surveyInstanceId: (function () { return instance; }),
surveyItemResultId: (function () { return surveyItemResultId; }),
itemId: (function () { return itemId; }),
imageLoopCounter: (function () { return counter++; })
},
validation: {
allowedExtensions: ['jpeg', 'jpg', 'gif', 'png', 'bmp']
},
multiple: true,
text: {
uploadButton: '<i class="icon-plus icon-white"></i>Drop or Select Files'
},
callbacks: {
onComplete: function(id, fileName, responseJSON) {
alert("Success: " + responseJSON.success);
if (responseJSON.success) {
$('#files-upload').append('<img src="img/success.jpg" alt="' + fileName + '">');
}
}
}
}
EDIT: I had been using Internet Explorer 9, then switched to Chrome, Firefox and I can upload just fine. What's required for IE9? Validation doesn't work, regardless of browser.
Endpoint fires, and file/parameters are populated, so this is all good! Validation doesn't stop a user from selecting something outside of this list, but I can work with this for the time being. I can successfully save and do what I need to do with my upload, minus getting the OnComplete to fire. Actually, in IE, I get an OPEN/SAVE dialog with what I have currently.
Question: Are the function parameters in onComplete (id, filename, responseJSON) getting populated by the return or on the way out? I'm just confused about this. Does my JSON have to have these parameters in it, and populated?
I don't do this (populate those parameters), and my output method in C# returns JsonResult looking like this, just returning 'success' (if appropriate):
return Json(new { success = true });
Do I need to add more? This line is after the saving takes place, and all I want to do is tell the user all is good or not. Does the success property in my JSON match up with the responseJSON.success?
What am I missing, or have wrong?
Addressing the items in your question:
Regarding restrictions inside of the "select files" dialog, you must also set the acceptFiles validation option. See the validation option section in the readme for more details.
Your validation option property in the wrong place. It should not be under the request property/option. The same is true for your text, multiple, and callbacks options/properties. Also, you are not setting your callbacks correctly for the jQuery plug-in.
The open/save dialog in IE is caused by your server not returning a response with the correct "Content-Type" header. Your response's Content-Type should be "text/plain". See the server-side readme for more details.
Anything your server returns in it's response will be parsed by Fine Uploader using JSON.parse when handling the response client-side. The result of invoking JSON.parse on your server's response will be passed as the responseJSON parameter to your onComplete callback handler. If you want to pass specific information from your server to your client-side code, such as some text you may want to display client-side, the new name of the uploaded file, etc, you can do so by adding appropriate properties to your server response. This data will then be made available to you in your onComplete handler. If you don't have any need for this, you can simply return the "success" response you are currently returning. The server-side readme, which I have linked to, provides more information about all of this.
To clarify what I have said in #2, your code should look like this:
$('#files-upload').fineUploader({
request: {
endpoint: '#Url.Action("UploadFile", "Survey")',
customHeaders: { Accept: 'application/json' },
params: {
//variables are populated outside of this code snippet
surveyInstanceId: (function () { return instance; }),
surveyItemResultId: (function () { return surveyItemResultId; }),
itemId: (function () { return itemId; }),
imageLoopCounter: (function () { return counter++; })
}
},
validation: {
allowedExtensions: ['jpeg', 'jpg', 'gif', 'png', 'bmp']
},
text: {
uploadButton: '<i class="icon-plus icon-white"></i>Drop or Select Files'
}
})
.on('complete', function(event, id, fileName, responseJSON) {
alert("Success: " + responseJSON.success);
if (responseJSON.success) {
$('#files-upload').append('<img src="img/success.jpg" alt="' + fileName + '">');
}
});