For my program I have to include huge index and data files in the program bundle. Because it is an universal app, I have included these files in a folder named "Data" within the "Shared" Project.
Now I try to read:
StorageFile file = await ApplicationData.Current.LocalFolder.GetFileAsync("Data/"+fileName);
Stream stream = (await file.OpenReadAsync()).AsStreamForRead();
BinaryReader reader = new BinaryReader(stream);
Windows.Storage.FileProperties.BasicProperties x = await file.GetBasicPropertiesAsync();
I get a System.ArgumentException "mscorlib.ni.dll" at the first line. What's wrong?
If somebody can help me and I get the file, I want to find the filesize. I hope, I can find this Information within the FileProperties (last line of code).
Then I want to set a FilePointer within that file and to read a defined number of binary data. Can I do that without reading the whole file in memory?
What you are trying to do is to access LocalFolder, which is not the same as Package.Current.InstalledLocation.
If you want to access files that are included with your package, you can do for example like this - by using URI schemes:
StorageFile file = await StorageFile.GetFileFromApplicationUriAsync(new Uri(#"ms-appx:///Data/"+fileName));
using (Stream stream = (await file.OpenReadAsync()).AsStreamForRead())
using (BinaryReader reader = new BinaryReader(stream))
{
Windows.Storage.FileProperties.BasicProperties x = await file.GetBasicPropertiesAsync();
}
or like this - by getting file from your Package, which you can access as StorageFolder - also pay attention here to use correct slashes (as it may be a source of exception):
StorageFile file = await Windows.ApplicationModel.Package.Current.InstalledLocation.GetFileAsync(#"Data\" + fileName);
using (Stream stream = (await file.OpenReadAsync()).AsStreamForRead())
using (BinaryReader reader = new BinaryReader(stream))
{
Windows.Storage.FileProperties.BasicProperties x = await file.GetBasicPropertiesAsync();
}
Note also that I've put your Stream and BinaryReader into using, as they are IDisposable and it's suitable to release those resources as they are no longer needed.
Note also that when your shared project has a name MySharedProject, you will have to modify the Path of above URI:
StorageFile file = await StorageFile.GetFileFromApplicationUriAsync(new Uri(#"ms-appx:///MySharedProject/Data/"+fileName));
or obtain the suitable StorageFolder:
StorageFile file = await Windows.ApplicationModel.Package.Current.InstalledLocation.GetFileAsync(#"MySharedProject\Data\" + fileName);
One remark after discussion:
When you add a file with .txt extension to your project, its Build Action by default is set to Content. But when you add file with .idx extension, as I've checked, its Build Action is set to None by default. To include those files in your package, change them to Content.
After Romasz has brought me to the right path, I can see the problem is quite different there.
My data files were involved in the correct place in the project, but Visual Studio does not bind all what you want.
In my project I need large data files to be firmly integrated into the program. These are between 13 KB and 41 MB in size and have file types .idx and .dat. These names are part of the problem.
What I know so far:
I may add .txt files with seemingly arbitrary size. Tested with 41 MB - no problem.
The same file with changed file type .idx is not added. The file is simply not included in the compiled project. No error message.
Of course I can rename the .idx files to another file type (tested with .id), but I want to know why idx files are treated differently. And why I got no error indication.
Related
Over the years on snapchat I have saved lots of photos that I would like to retrieve now, The problem is they do not make it easy to export, but luckily if you go online you can request all the data (thats great)
I can see all my photos download link and using the local HTML file if I click download it starts downloading.
Here's where the tricky part is, I have around 15,000 downloads I need to do and manually clicking each individual one will take ages, I've tried extracting all of the links through the download button and this creates lots of Urls (Great) but the problem is, if you past the url into the browser then ("Error: HTTP method GET is not supported by this URL") appears.
I've tried a multitude of different chrome extensions and none of them show the actually download, just the HTML which is on the left-hand side.
The download button is a clickable link that just starts the download in the tab. It belongs under Href A
I'm trying to figure out what the best way of bulk downloading each of these individual files is.
So, I just watched their code by downloading my own memories. They use a custom JavaScript function to download your data (a POST request with ID's in the body).
You can replicate this request, but you can also just use their method.
Open your console and use downloadMemories(<url>)
Or if you don't have the urls you can retrieve them yourself:
var links = document.getElementsByTagName("table")[0].getElementsByTagName("a");
eval(links[0].href);
UPDATE
I made a script for this:
https://github.com/ToTheMax/Snapchat-All-Memories-Downloader
Using the .json file you can download them one by one with python:
req = requests.post(url, allow_redirects=True)
response = req.text
file = requests.get(response)
Then get the correct extension and the date:
day = date.split(" ")[0]
time = date.split(" ")[1].replace(':', '-')
filename = f'memories/{day}_{time}.mp4' if type == 'VIDEO' else f'memories/{day}_{time}.jpg'
And then write it to file:
with open(filename, 'wb') as f:
f.write(file.content)
I've made a bot to download all memories.
You can download it here
It doesn't require any additional installation, just place the memories_history.json file in the same directory and run it. It skips the files that have already been downloaded.
Short answer
Download a desktop application that automates this process.
Visit downloadmysnapchatmemories.com to download the app. You can watch this tutorial guiding you through the entire process.
In short, the app reads the memories_history.json file provided by Snapchat and downloads each of the memories to your computer.
App source code
Long answer (How the app described above works)
We can iterate over each of the memories within the memories_history.json file found in your data download from Snapchat.
For each memory, we make a POST request to the URL stored as the memories Download Link. The response will be a URL to the file itself.
Then, we can make a GET request to the returned URL to retrieve the file.
Example
Here is a simplified example of fetching and downloading a single memory using NodeJS:
Let's say we have the following memory stored in fakeMemory.json:
{
"Date": "2022-01-26 12:00:00 UTC",
"Media Type": "Image",
"Download Link": "https://app.snapchat.com/..."
}
We can do the following:
// import required libraries
const fetch = require('node-fetch'); // Needed for making fetch requests
const fs = require('fs'); // Needed for writing to filesystem
const memory = JSON.parse(fs.readFileSync('fakeMemory.json'));
const response = await fetch(memory['Download Link'], { method: 'POST' });
const url = await response.text(); // returns URL to file
// We can now use the `url` to download the file.
const download = await fetch(url, { method: 'GET' });
const fileName = 'memory.jpg'; // file name we want this saved as
const fileData = download.body; // contents of the file
// Write the contents of the file to this computer using Node's file system
const fileStream = fs.createWriteStream(fileName);
fileData.pipe(fileStream);
fileStream.on('finish', () => {
console.log('memory successfully downloaded as memory.jpg');
});
We are using Firebase Google Cloud Storage Bucket to store our files.
When the logged in user wants the download the file kept inside certain folder
Eg: 123/admin/1469611803143/123.xlsx
The url generated will be
https://firebasestorage.googleapis.com/v0/b/MYWEBSITE.appspot.com/o/123%2Fadmin%2F1469611803143%2F123.xlsx?alt=media&token=whatever_alpa_numeric_token
As I download this file the file name will be 123%2Fadmin%2F1469611803143%2F123.xlsx
and not 123.xlsx
We have tried using download attribute to change the file name
but this did not change the file name to 123.xlsx
Please HELP
I'm pretty new with firebase but I achieved this with the following code :
var storageRef = firebase.storage().ref();
var child = storageRef.child("your path");
var uploadTask = child.put(<file>);
uploadTask.on(firebase.storage.TaskEvent.STATE_CHANGED,
function(snapshot){
// HANDLE TASK PROGRESS
},
function(error){
// HANDLE ERROR
},
function(){
// UPLOAD SUCCESSFULL
var newMetadata = {
contentDisposition : "attachment; filename=" + fileName
}
child.updateMetadata(newMetadata)
})
This is (fortunately or unfortunately) intended behavior. Technically, files in Firebase Storage are stored with the full path (so 123%2Fadmin%2F1469611803143%2F123.xlsx is actually the file name--the slashes and percent escaping are part of the name, and are only represented as path separators in the UI), which is how we get this behavior.
We're likely to modify how downloads work in the future (in that we'll truncate the name), but we've been busy fixing other bugs and polishing higher priority pieces.
In my WinRT application I have the following code:
resultingFile = await downloadFolder.CreateFileAsync(filename, CreationCollisionOption.OpenIfExists);
var downloader = new BackgroundDownloader();
var operation = downloader.CreateDownload(new Uri(rendition.Url), resultingFile);
await operation.StartAsync();
After the CreateFileAsync call I can verify that I do have a 0byte file at the filename path (and double verified by pulling the location out of the resultingFile itself.
However, when operation.StartAsync() is called I get a FileNotFoundException claiming the system could not find the file specified. Unfortunately, that's all it tells me and there is no inner exception.
I have also verified that rendition.Url gives me a valid url that downloads the content I'm expecting to be downloading.
Am I doing something wrong here?
Apparently this code isn't what is throwing the error but it's some code the BackgroundDownloader uses to coordinate things that can't find it's own file.
Uninstalling the application and redeploying it fixed it.
Good waste of 3 hours :(
I have recently contributed some code to Moodle which uses some of the capabilities of HTML5 to allow files to be uploaded in forms via drag and drop from the desktop (the core part of the code is here: https://github.com/moodle/moodle/blob/master/lib/form/dndupload.js for reference).
This is working well, except for when a user drags a folder / directory instead of a real file. Garbage is then uploaded to the server, but with the filename matching the folder.
What I am looking for is an easy and reliable way to detect the presence of a folder in the FileList object, so I can skip it (and probably return a friendly error message as well).
I've looked through the documentation on MDN, as well as a more general web search, but not turned up anything. I've also looked through the data in the Chrome developer tools and it appears that the 'type' of the File object is consistently set to "" for folders. However, I'm not quite convinced this is the most reliable, cross-browser detection method.
Does anyone have any better suggestions?
You cannot rely on file.type. A file without an extension will have a type of "". Save a text file with a .jpg extension and load it into a file control, and its type will display as image/jpeg. And, a folder named "someFolder.jpg" will also have its type as image/jpeg.
Instead, try to read the first byte of the file. If you are able to read the first byte, you have a file. If an error is thrown, you probably have a directory:
try {
await file.slice(0, 1).arrayBuffer();
// it's a file!
}
catch (err) {
// it's a directory!
}
If you are in the unfortunate position of supporting IE11, The file will not have the arrayBuffer method. You have to resort to the FileReader object:
// use this code if you support IE11
var reader = new FileReader();
reader.onload = function (e) {
// it's a file!
};
reader.onerror = function (e) {
// it's a directory!
};
reader.readAsArrayBuffer(file.slice(0, 1));
I also ran into this problem and below is my solution. Basically, I took have a two pronged approach:
(1) check whether the File object's size is large, and consider it to be a genuine file if it is over 1MB (I'm assuming folders themselves are never that large).
(2) If the File object is smaller than 1MB, then I read it using FileReader's 'readAsArrayBuffer' method. Successful reads call 'onload' and I believe this indicates the file object is a genuine file. Failed reads call 'onerror' and I consider it a directory. Here is the code:
var isLikelyFile = null;
if (f.size > 1048576){ isLikelyFile = false; }
else{
var reader = new FileReader();
reader.onload = function (result) { isLikelyFile = true; };
reader.onerror = function(){ isLikelyFile = false; };
reader.readAsArrayBuffer(f);
}
//wait for reader to finish : should be quick as file size is < 1MB ;-)
var interval = setInterval(function() {
if (isLikelyFile != null){
clearInterval(interval);
console.log('finished checking File object. isLikelyFile = ' + isLikelyFile);
}
}, 100);
I tested this in FF 26, Chrome 31, and Safari 6 and three browsers call 'onerror' when attempting to read directories. Let me know if anyone can think of a use case where this fails.
I proposing calling FileReader.readAsBinaryString on the File object. In Firefox, this will raise an Exception when the File is a Directory. I only do this if the File meets the conditions proposed by gilly3.
Please see my blog post at http://hs2n.wordpress.com/2012/08/13/detecting-folders-in-html-drop-area/ for more details.
Also, version 21 of Google Chrome now supports dropping folders. You can easily check if the dropped items are folders, and also read their contents.
Unfortunately, I don´t have any (client-side) solution for older Chrome versions.
One other note is that type is "" for any file that has an unknown extension. Try uploading a file named test.blah and the type will be empty. AND... try dragging and dropping a folder named test.jpg - type will be set to "image/jpeg". To be 100% correct, you can't depend on type solely (or if at all, really).
In my testing, folders have always been of size 0 (on FF and Chrome on 64-bit Windows 7 and under Linux Mint (Ubuntu essentially). So, my folder check is just checking if size is 0 and it seems to work for me in our environment. We also don't want 0-byte files uploaded either so if it's 0 byte the message comes back as "Skipped - 0 bytes (or folder)"
FYI, this post will tell you how to use dataTransfer API in Chrome to detect file type: http://updates.html5rocks.com/2012/07/Drag-and-drop-a-folder-onto-Chrome-now-available
The best option is to use both the 'progress' and 'load' events on a FileReader instance.
var fr = new FileReader();
var type = '';
// Early terminate reading files.
fr.addEventListener('progress', function(e) {
console.log('progress - valid file');
fr.abort();
type = 'file';
});
// The whole file loads before a progress event happens.
fr.addEventListener('load', function(e) {
console.log('load - valid file');
type = 'file';
});
// Not a file. Possibly a directory.
fr.addEventListener('error', function(e) {
console.log('error - not a file or is not readable by the web browser');
});
fr.readAsArrayBuffer(thefile);
This fires the error handler when presented with a directory and most files will fire the progress handler after reading just a few KB. I've seen both events fire. Triggering abort() in the progress handler stops the FileReader from reading more data off disk into RAM. That allows for really large files to be dropped without reading all of the data of such files into RAM just to determine that they are files.
It may be tempting to say that if an error happens that the File is a directory. However, a number of scenarios exist where the File is unreadable by the web browser. It is safest to just report the error to the user and ignore the item.
An easy method is the following:
Check if the file's type is an empty string: type === ""
Check if the file's size is 0, 4096, or a multiple of it: size % 4096 === 0.
if (file.type === "" && file.size % 4096 === 0) {
// The file is a folder
} else {
// The file is not a folder
}
Note: Just by chance, there could be files without a file extension that have the size of some multiple of 4096. Even though this will not happen very often, be aware of it.
For reference, please see the great answer from user Marco Bonelli to a similar topic. This is just a short summary of it.
i want to save a dropped image into the html5 filesystem. The problem is that i do not know how to write binary data into the file system with the file api. Below are parts of my code:
var reader = new FileReader();
reader.onloadend = function(e)
{
var data = e.target.result;
// Open File System code (Codes Omitted)
// Create file (Codes Omitted)
fileEntry.createWriter(function(fileWriter)
{
// This is the problem, the file is create, something is written to the file
// because the size is exact same with the dragged file (image). But when i
// try to view the file (image), it displays nothing.
var bb = new WebKitBlobBuilder
bb.append(data);
fileWriter.write(bb.getBlob(""));
}
}
reader.readAsBinaryString(cur_file);
I intentionally left out some codes like the file system and file creation. I only need help with saving binary data.
Thanks.
I've encounter a similar issue
Since you didn't parse your code here,I assume that you read the image in a wrong way,
maybe
You read the cur_file as binary string,Did your fileReader read it by readAsText method
Since the file stores binaryString,If your fileReader read it by another way,e.g readAsDataURL,that means you encode it again,so you couldn't get the right answer