I am trying to work with Firebase Cloud Functions, and ImageMagick, similar to how the thumbnail demo is done. By re-purposing the demo script, I want to execute a CLI command for ImageMagick to split PDF pages to images.
convert -density 150 presentation.pdf -quality 90 output-%3d.jpg
The snippet
exports.splitPdfPages = functions.storage.object().onFinalize(async (object) => {
const fileBucket = object.bucket; // The Storage bucket that contains the file.
const filePath = object.name; // File path in the bucket.
const contentType = object.contentType; // File content type.
const metageneration = object.metageneration; // Number of times metadata has been generated. New objects have a value of 1.
// Download file from bucket.
const bucket = admin.storage().bucket(fileBucket);
const tempFilePath = path.join(os.tmpdir(), fileName);
const tempSplitImagesPath = tempFilePath.replace(".png", "_%3d.png");
await bucket.file(filePath).download({destination: tempFilePath});
console.log('PDF downloaded locally to', tempFilePath);
// Generate split page images using ImageMagick.
await spawn('convert', ['-density', '150', tempFilePath, '-quality', '90', tempSplitImagesPath]);
console.log('pages split images created at', tempFilePath);
...
// Uploading the split images.
...
// Once the thumbnail has been uploaded delete the local file to free up disk space.
return fs.unlinkSync(tempFilePath);
});
Unfortunately, I'm encountering errors in the Cloud Functions log indicating the statement error
ChildProcessError: convert -density 150 /tmp/7eCxdDKqCb0rlYVw3AYf__foobar.pdf -quality 100 /tmp/7eCxdDKqCb0rlYVw3AYf__foobar_%3d.png failed with code 1
I searched for resolution to the error, but it only indicates that whitespaces are the main reason for the issue (which based on my statement doesn't have any). Invoking generateThumbnail function works properly, so I'm presuming its based on my changes
Am I missing something to properly call the ImageMagick command for converting PDF pages to image?
Looking forwad to hearing from you.
Related
I am following a tutorial to resize images via Cloud Functions on upload and am experiencing two major issues which I can't figure out:
1) If a PNG is uploaded, it generates the correctly sized thumbnails, but the preview of them won't load in Firestorage (Loading spinner shows indefinitely). It only shows the image after I click on "Generate new access token" (none of the generated thumbnails have an access token initially).
2) If a JPEG or any other format is uploaded, the MIME type shows as "application/octet-stream". I'm not sure how to extract the extension correctly to put into the filename of the newly generated thumbnails?
export const generateThumbs = functions.storage
.object()
.onFinalize(async object => {
const bucket = gcs.bucket(object.bucket);
const filePath = object.name;
const fileName = filePath.split('/').pop();
const bucketDir = dirname(filePath);
const workingDir = join(tmpdir(), 'thumbs');
const tmpFilePath = join(workingDir, 'source.png');
if (fileName.includes('thumb#') || !object.contentType.includes('image')) {
console.log('exiting function');
return false;
}
// 1. Ensure thumbnail dir exists
await fs.ensureDir(workingDir);
// 2. Download Source File
await bucket.file(filePath).download({
destination: tmpFilePath
});
// 3. Resize the images and define an array of upload promises
const sizes = [64, 128, 256];
const uploadPromises = sizes.map(async size => {
const thumbName = `thumb#${size}_${fileName}`;
const thumbPath = join(workingDir, thumbName);
// Resize source image
await sharp(tmpFilePath)
.resize(size, size)
.toFile(thumbPath);
// Upload to GCS
return bucket.upload(thumbPath, {
destination: join(bucketDir, thumbName)
});
});
// 4. Run the upload operations
await Promise.all(uploadPromises);
// 5. Cleanup remove the tmp/thumbs from the filesystem
return fs.remove(workingDir);
});
Would greatly appreciate any feedback!
I just had the same problem, for unknown reason Firebase's Resize Images on purposely remove the download token from the resized image
to disable deleting Download Access Tokens
goto https://console.cloud.google.com
select Cloud Functions from the left
select ext-storage-resize-images-generateResizedImage
Click EDIT
from Inline Editor goto file FUNCTIONS/LIB/INDEX.JS
Add // before this line (delete metadata.metadata.firebaseStorageDownloadTokens;)
Comment the same line from this file too FUNCTIONS/SRC/INDEX.TS
Press DEPLOY and wait until it finish
note: both original and resized will have the same Token.
I just started using the extension myself. I noticed that I can't access the image preview from the firebase console until I click on "create access token"
I guess that you have to create this token programatically before the image is available.
I hope it helps
November 2020
In connection to #Somebody answer, I can't seem to find ext-storage-resize-images-generateResizedImage in GCP Cloud Functions
The better way to do it, is to reuse the original file's firebaseStorageDownloadTokens
this is how I did mine
functions
.storage
.object()
.onFinalize((object) => {
// some image optimization code here
// get the original file access token
const downloadtoken = object.metadata?.firebaseStorageDownloadTokens;
return bucket.upload(tempLocalFile, {
destination: file,
metadata: {
metadata: {
optimized: true, // other custom flags
firebaseStorageDownloadTokens: downloadtoken, // access token
}
});
});
One of our requirements for an admin tool is to create a form that can be filled and translated to a downloadable pdf file. (A terms and condition with blank input fields to be exact).
I did some googling and tried creating a form in html and css and converted it into a canvas using the html2canvas package. Then I used the jspdf package to convert it into a pdf file. The problem is that I cannot get it to fit and resize accordingly to an a4 format with correct margins. I'm sure I can get to a somewhat working solution if I spend some time on it.
However, my real question is how would you guys solution this? Is there a 3rd party app/service that does this exact thing? Or would you do all this in the server side? Our current app is using angular 7 with firebase as our backend.
Cheers!
I was able to use the npm package pdfmake to create a dynamic pdf based on user information the user provided while interacting with my form. (I was using React) It opened the pdf in a new tab and the user is able to save the pdf. In another application (still React),
I used the same package to create a receipt so you can customize the size of the "page". We created the pdf and used the getBase64() method and sent the pdf as an email attachement.
My service function:
getEvidenceFile(id: number, getFileContent: boolean) {
return this.http.get(environment.baseUrl + ‘upload’ + ‘/’ + id , {responseType: ‘blob’ as ‘json’})
.map(res => res);
}
My component function called from the selected item of a FileDownload…
FileDownload(event: any) {
// const blob = await this.callService.getEvidenceFile(event.target.value, true);
// const url = window.URL.createObjectURL(blob);
this.callService.getEvidenceFile(event.target.value, true).subscribe(data => {
var binaryData = [];
binaryData.push(data);
var downloadLink = document.createElement(‘a’);
downloadLink.href = window.URL.createObjectURL(new Blob(binaryData));
document.body.appendChild(downloadLink);
downloadLink.click();
});
}
I have a public (anyone with the link can view) file on my Google Drive and I want to use the content of it in my Android app.
From what I could gather so far, I need the fileID, the OAuth token and the client ID - these I already got. But I can't figure out what is the exact methodology of authorising the app or fetching the file.
EDIT:
Simply reading it using file.readAsLines didn't work:
final file = new File(dogListTxt);
Future<List<String>> dogLinks = file.readAsLines();
return dogLinks;
The dogLinks variable isn't filled with any data, but I get no error messages.
The other method I tried was following this example but this is a web based application with explicit authorization request (and for some reason I was never able to import the dart:html library).
The best solution would be if it could be done seamlessly, as I would store the content in a List at the application launch, and re-read on manual refresh button press.
I found several old solutions here, but the methods described in those doesn't seem to work anymore (from 4-5 years ago).
Is there a good step-by-step tutorial about integrating the Drive API in a flutter application written in dart?
I had quite a bit of trouble with this, it seems much harder than it should be. Also this is for TXT files only. You need to use files.export() for other files.
First you need to get a list fo files.
ga.FileList textFileList = await drive.files.list(q: "'root' in parents");
Then you need to get those files based on ID (This is for TXT Files)
ga.Media response = await drive.files.get(filedId, downloadOptions: ga.DownloadOptions.FullMedia);
Next is the messy part, you need to convert your Media object stream into a File and then read the text from it. ( #Google, please make this easier.)
List<int> dataStore = [];
response.stream.listen((data) {
print("DataReceived: ${data.length}");
dataStore.insertAll(dataStore.length, data);
}, onDone: () async {
Directory tempDir = await getTemporaryDirectory(); //Get temp folder using Path Provider
String tempPath = tempDir.path; //Get path to that location
File file = File('$tempPath/test'); //Create a dummy file
await file.writeAsBytes(dataStore); //Write to that file from the datastore you created from the Media stream
String content = file.readAsStringSync(); // Read String from the file
print(content); //Finally you have your text
print("Task Done");
}, onError: (error) {
print("Some Error");
});
There currently is no good step-by-step tutorial, but using https://developers.google.com/drive/api/v3/manage-downloads as a reference guide for what methods to use in Dart/Flutter via https://pub.dev/packages/googleapis: to download or read the contents of a Google Drive file, you should be using googleapis/Drive v3, or specifically, the methods from the FilesResourceApi class.
drive.files.export(), if this is a Google document
/// Exports a Google Doc to the requested MIME type and returns the exported content. Please note that the exported content is limited to 10MB.
drive.files.get(), if this something else, a non-Gdoc file
/// Gets a file's metadata or content by ID.
Simplified example:
var drive = new DriveApi(http_client);
drive.files.get(fileId).then((file) {
// returns file
});
However, what I discovered was that this Dart-GoogleAPIs library seemed to be missing a method equivalent to executeMediaAndDownloadTo(outputStream). In the original Google Drive API v3, this method adds the alt=media URL parameter to the underlying HTTP request. Otherwise, you'll get the error, which is what I saw:
403, message: Export requires alt=media to download the exported
content.
And I wasn't able to find another way to insert that URL parameter into the current request (maybe someone else knows?). So as an alternative, you'll have to resort to implementing your own Dart API to do the same thing, as hinted by what this OP did over here https://github.com/dart-lang/googleapis/issues/78: CustomDriveApi
So you'll either:
do it through Dart with your own HttpClient implementation and try to closely follow the REST flow from Dart-GoogleAPIs, but remembering to include the alt=media
or implement and integrate your own native-Android/iOS code and use the original SDK's convenient executeMediaAndDownloadTo(outputStream)
(note, I didn't test googleapis/Drive v2, but a quick examination of the same methods looks like they are missing the same thing)
I wrote this function to get file content of a file using its file id. This is the simplest method I found to do it.
Future<String> _getFileContent(String fileId) async {
var response = await driveApi.files.get(fileId, downloadOptions: DownloadOptions.fullMedia);
if (response is! Media) throw Exception("invalid response");
return await utf8.decodeStream(response.stream);
}
Example usage:
// save file to app data folder with 150 "hello world"s
var content = utf8.encode("hello world" * 150);
driveApi.files
.create(File(name: fileName, parents: [appDataFolder]),
uploadMedia: Media(Stream.value(content), content.length))
.then((value) {
Log().i("finished uploading file ${value.id}");
var id = value.id;
if (id != null) {
// after successful upload, read the recently uploaded file content
_getFileContent(id).then((value) => Log().i("got content is $value"));
}
});
This may sound silly... but is there any way to embed all videos in a directory to a webpage? I'm hosting some videos on my website but right now you have to manually browse the directory and just click a link to a video.
I know I can just embed those videos to a html page but is there any way to make it adapt automatically when I add new videos?
How you do this will depend on how you are building your server code and web page code, but the example below which is node and angular based does exactly what you are asking:
// GET: route to return list of upload videos
router.get('/video_list', function(req, res) {
//Log the request details
console.log(req.body);
// Get the path for the uploaded_video directory
var _p;
_p = path.resolve(__dirname, 'public', 'uploaded_videos');
//Find all the files in the diectory and add to a JSON list to return
var resp = [];
fs.readdir(_p, function(err, list) {
//Check if the list is undefined or empty first and if so just return
if ( typeof list == 'undefined' || !list ) {
return;
}
for (var i = list.length - 1; i >= 0; i--) {
// For each file in the directory add an id and filename to the response
resp.push(
{"index": i,
"file_name": list[i]}
);
}
// Set the response to be sent
res.json(resp);
});
});
This code is old in web years (i.e. about 3 years old) so the way node handles routes etc is likely different now but the concepts remains the same, regardless of language:
go to the video directory
get the lit of video files in it
build them into a JSON response and send them to the browser
browser extracts and displays the list
The browser code corresponding to the above server code in this case is:
$scope.videoList = [];
// Get the video list from the Colab Server
GetUploadedVideosFactory.getVideoList().then(function(data) {
// Note: should really do some type checking etc here on the returned value
console.dir(data.data);
$scope.videoList = data.data;
});
You may find some way to automatically generate a web page index from a directory, but the type of approach above will likely give you more control - you can exclude certain file names types etc quite easily, for example.
The full source is available here: https://github.com/mickod/ColabServer
I've started using Gulp JS and must admit I'm finding it really useful.
One of the tasks I need to perform is zip up a collection of folders into individual zip files, one for each folder and then zip all this zipped files up into one single zip file. Using Gulp-Zip I've managed to get this far:
var modelFolders = [
'ELFH_Check',
'ELFH_DDP',
'ELFH_Free'
];
gulp.task('zipModels', function () {
for (var i = 0; i < modelFolders.length; i++) {
var model = modelFolders[i];
gulp.src('**/*', {cwd: path.join(process.cwd(), '/built_templates/' + model) })
.pipe(zip(model + '.zip'))
.pipe(gulp.dest('./built_templates'));
};
});
This works and outputs ELFH_Check.zip, ELFH_DDP.zip and ELFH_Free.zip. However, I then need to zip up these zip files into one zip file called "Templates.zip" and I've not managed to get this task to work:
// zip up model files
gulp.task('zipTemplate', ['zipModels'], function () {
gulp.src('*.zip', {cwd: path.join(process.cwd(), './built_templates/') })
.pipe(zip('Templates_.zip'))
.pipe(gulp.dest('./built_templates'));
});
Does anyone know if this is possible or what I'm doing wrong?
I saw the problem as well, and it seems to be related to the cwd option somehow. I'll investigate further.
After #OverZealous comment, I investigated further and found two issues:
As he said, you need to hint gulp to wait until the end of the dependency task (zipModels), by returning a stream from it. As you have multiple streams, you can use event-stream.merge to return a bundle stream.
The reason why the bundle zip wouldn't work, is because you cwd points to /built_templates/, and the second slash is causing some problem. To work properly, you need to remove the trailing slash, so it should be path.join(process.cwd(), '/built_templates').
IMPORTANT
Anyway, you should avoid temporary files. Gulp philosophy is to try using pipes to avoid IO. In that direction, what you want to do is to cut the intermediary dest steps, merge the streams, zip them, and finally, output them.
Something like that:
var es = require('event-stream');
var modelFolders = [
'ELFH_Check',
'ELFH_DDP',
'ELFH_Free'
];
gulp.task('zipModels', function () {
var zips = [],
modelZip;
for (var i = 0; i < modelFolders.length; i++) {
var model = modelFolders[i];
modelZip = gulp.src('**/*', {cwd: path.join(process.cwd(), '/built_templates/' + model) })
.pipe(zip(model + '.zip'));
// notice we removed the dest step and store the zip stream (still in memory)
zips.push(modelZip);
};
// we finally merge them (the zips), zip them again, and output.
return es.merge.apply(null, zips)
.pipe(zip('templates.zip'))
.pipe(gulp.dest('./'));
});
By the name of your folder (built_templates), it seems you have some other task that will generate the temporary built files. Preferably, you don't want these as well. You should pipe their streams directly to the ZIP stream, a finally, to the bundle-zip stream. By doing that, you would have a simple stream flow, with one disk read, and one disc write at the end, with no temporary files.
If you need them to be different tasks, consider having a function that will generate the stream up to the step before the gulp.dest pipe, and use this function on all subtasks.
Additionally, always try to hint your async tasks by returning a stream, a promise or receiving a callback function, and advise the end of the task.