Two part quersion.
Part 1:
Im uploading an image to my server and want to save it to my database.
So far:
table:
resolver:
registerPhoto: inSequence([
async (obj, { file }) => {
const { filename, mimetype, createReadStream } = await file;
const stream = createReadStream();
const t = await db.images.create({
Name: 'test',
imageData: stream ,
});
},
])
executing query:
Executing (default): INSERT INTO `images` (`Id`,`imageData`,`Name`) VALUES (DEFAULT,?,?);
But nothing is saved.
Im new to this and im probably missing something but dont know what.
Part2:
This is followed by part 1, lets say I manage to save the image, how do I read it and send it back to my FE?
An edit: Ive read alot of guides saving the an image name to the db and then tha actuall image in a folder. This is NOT what im after, want to save the image to the DB and then be able to fetch it from the DB abd present it.
This took me some time but I finaly figured it out.
First step (saving to the db):
Have to get the entire stream data and read it like this:
export const readStream = async (stream, encoding = 'utf8') => {
stream.setEncoding('base64');
return new Promise((resolve, reject) => {
let data = '';
// eslint-disable-next-line no-return-assign
stream.on('data', chunk => (data += chunk));
stream.on('end', () => resolve(data));
stream.on('error', error => reject(error));
});
};
use like this:
const streamData = await readStream(stream);
Before saving I tur the stream into a buffer:
const buff = Buffer.from(streamData);
Finaly the save part:
db.images.create(
{
Name: filename,
imageData: buff,
Length: stream.bytesRead,
Type: mimetype,
},
{ transaction: param }
);
Note that I added Length and Type parameter, this is needed if you like to return a stream when you return the image.
Step 2 (Retrieving the image).
As #xadm said multiple times you can not return an image from GRAPHQL and after some time I had to accept that fact, hopefully graphql will remedy this in the future.
S What I needed to do is set up a route on my fastify backend, send a image Id to this route, fetch the image and then return it.
I had a few diffirent approaches to this but in the end I simpy returned a binary and on the fronted I encoded it to base64.
Backend part:
const handler = async (req, reply) => {
const p: postParams = req.params;
const parser = uuIdParserT();
const img = await db.images.findByPk(parser.setValueAsBIN(p.id));
const binary = img.dataValues.imageData.toString('binary');
const b = Buffer.from(binary);
const myStream = new Readable({
read() {
this.push(Buffer.from(binary));
this.push(null);
},
});
reply.send(myStream);
};
export default (server: FastifyInstance) =>
server.get<null, any>('/:id', opts, handler);
Frontend part:
useEffect(() => {
// axiosState is the obj that holds the image
if (!axiosState.loading && axiosState.data) {
// #ts-ignore
const b64toBlob = (b64Data, contentType = '', sliceSize = 512) => {
const byteCharacters = atob(b64Data);
const byteArrays = [];
for (let offset = 0; offset < byteCharacters.length; offset += sliceSize) {
const slice = byteCharacters.slice(offset, offset + sliceSize);
const byteNumbers = new Array(slice.length);
// #ts-ignore
// eslint-disable-next-line no-plusplus
for (let i = 0; i < slice.length; i++) {
byteNumbers[i] = slice.charCodeAt(i);
}
const byteArray = new Uint8Array(byteNumbers);
byteArrays.push(byteArray);
}
const blob = new Blob(byteArrays, { type: contentType });
return blob;
};
const blob = b64toBlob(axiosState.data, 'image/jpg');
const urlCreator = window.URL || window.webkitURL;
const imageUrl = urlCreator.createObjectURL(blob);
setimgUpl(imageUrl);
}
}, [axiosState]);
and finaly in the html:
<img src={imgUpl} alt="NO" className="imageUpload" />
OTHER:
For anyone who is attempting the same NOTE that this is not a best practice thing to do.
Almost every article I found saved the images on the sever and save an image Id and other metadata in the datbase. For the exact pros and cons for this I have found the following helpful:
Storing Images in DB - Yea or Nay?
I was focusing on finding out how to do it if for some reason I want to save an image in the datbase and finaly solved it.
There are two ways to store images in your SQL database. You either store the actual image on your server and save the image path inside your mySql db OR you create a BLOB using the image and store it in db.
Here is a handy read https://www.technicalkeeda.com/nodejs-tutorials/nodejs-store-image-into-mysql-database
you should save the image in a directory and save the link of this image in the database
Related
This question already has an answer here:
How can I resize all existing images in firebase storage?
(1 answer)
Closed 9 months ago.
I have requirement to resize new and existing images stored in firebase store. For new image, I enabled firebase's resize image extension. For existing image, how can I resized the image and get the newly resized image url to update back to database via api.
Here is my firebase function to get existing image urls from database. My question is how to resize the image and get the new image url?
const functions = require("firebase-functions");
const axios =require("axios");
async function getAlbums() {
const endpoint = "https://api.mydomain.com/graphql";
const headers = {
"content-type": "application/json",
};
const graphqlQuery = {
"query": `query Albums {
albums {
id
album_cover
}
}`
};
functions.logger.info("Call API");
const response = await axios({
url: endpoint,
method: 'post',
headers: headers,
data: graphqlQuery
});
if(response.errors) {
functions.logger.info("API ERROR : ", response.errors) // errors if any
} else {
return response.data.data.albums;
}
}
exports.manualGenerateResizedImage = functions.https.onRequest(async () => {
const albums = await getAlbums();
functions.logger.info("No. of Album : ", albums.length);
});
I think the below answer from Renaud Tarnec will definitely help you.
If you look at the code of the "Resize Images" extension, you will see that the Cloud Function that underlies the extension is triggered by a onFinalize event, which means:
When a new object (or a new generation of an existing object) is
successfully created in the bucket. This includes copying or rewriting
an existing object.
So, without rewriting/regenerating the existing images the Extension will not be triggered.
However, you could easily write your own Cloud Function that does the same thing but is triggered, for example, by a call to a specific URL (HTTPS cloud Function) or by creating a new document in a temporary Firestore Collection (background triggered CF).
This Cloud Function would execute the following steps:
Get all the files of your bucket, see the getFiles() method of the
Google Cloud Storage Node.js Client API. This method returns a
GetFilesResponse object which is an Array of File instances.
By looping over the array, for each file, check if the file has a
corresponding resized image in the bucket (depending on the way you
configured the Extension, the resized images may be in a specific
folder)
If a file does not have a corresponding resized image, execute the
same business logic of the Extension Cloud Function for this File.
There is an official Cloud Function sample which shows how to create a Cloud Storage triggered Firebase Function that will create resized thumbnails from uploaded images and upload them to the database URL, (see the last lines of index.js file)
Note : If you have a lot of files to treat, you should most probably work by batch, since there is a limit of 9 minutes for Cloud Function execution. Also, depending on the number of images to treat, you may need to increase the timeout value and/or the allocated memory of your Cloud Function, see https://firebase.google.com/docs/functions/manage-functions#set_timeout_and_memory_allocation
In case someone need it. This is how I resized existing image.
const functions = require("firebase-functions");
const axios = require("axios");
const { Storage } = require("#google-cloud/storage");
const storage = new Storage();
// Don't forget to replace with your bucket name
const bucket = storage.bucket("projectid.appspot.com");
async function getAlbums() {
const endpoint = "https://api.mydomain.com/graphql";
const headers = {
"content-type": "application/json",
};
const graphqlQuery = {
query: `query Albums {
albums {
id
album_cover
}
}`,
};
const response = await axios({
url: endpoint,
method: "post",
headers: headers,
data: graphqlQuery,
});
if (response.errors) {
functions.logger.error("API ERROR : ", response.errors); // errors
if any
} else {
return response.data.data.albums;
}
}
function getFileName(url) {
var decodeURI = decodeURIComponent(url);
var index = decodeURI.lastIndexOf("/") + 1;
var filenameWithParam = decodeURI.substr(index);
index = filenameWithParam.lastIndexOf("?");
var filename = filenameWithParam.substr(0, index);
return filename;
}
function getFileNameFromFirestore(url) {
var index = url.lastIndexOf("/") + 1;
var filename = url.substr(index);
return filename;
}
const triggerBucketEvent = async () => {
bucket.getFiles(
{
prefix: "images/albums", // you can add a path prefix
autoPaginate: false,
},
async (err, files) => {
if (err) {
functions.logger.error(err);
return;
}
const albums = await getAlbums();
await Promise.all(
files.map((file) => {
var fileName = getFileNameFromFirestore(file.name);
var result = albums.find((obj) => {
return getFileName(obj.album_cover) === fileName;
});
if (result) {
var file_ext = fileName.substr(
(Math.max(0, fileName.lastIndexOf(".")) || Infinity) + 1
);
var newFileName = result.id + "." + file_ext;
// Copy each file on thumbs directory with the different name
file.copy("images/albums/" + newFileName);
} else {
functions.logger.info(file.name, " not found in album list!");
}
})
);
}
);
};
exports.manualGenerateResizedImage = functions.https.onRequest(async () => {
await triggerBucketEvent();
});
I am attempting to build a discord bot, and one of its features is that it will be able to recognize whether a user has interacted with the bot before by using a json array database. What is instead happening is that it does not read the array, and writes the username into the file even if it is already in the database.
const Discord = require('discord.js');
const fs = require('fs');
const { json } = require('stream/consumers');
const {prefix, token} = require('./config.json');
const {recognizedusers} = require('./database.json');
//Read File
let db = fs.readFileSync('./database.json');
db = JSON.parse(db);
// New discord client
const client = new Discord.Client({ intents: ["GUILDS", "GUILD_MESSAGES"] });
//Client run once
client.once('ready', () => {
console.log('Ready!');
});
//Client always run code
client.on('message', message => {
let msg_content = message.content.toLowerCase()
// Mixed message content response
if (msg_content.includes('test 1') && msg_content.includes('test 2')) {
message.channel.send('test 2: electric boogaloo')
}
if (msg_content.includes('test') && !message.author.username.includes(db.recognizedusers)) {
message.channel.send('unrecognized name')
//Edit Variable
db.recognizedusers.push(message.author.username)
// write file
fs.writeFileSync('./database.json', JSON.stringify(db));
}
});
client.login(token);
Elitezen fixed my problem.
I think you ment !db.recognizedusers.includes(message.author.username)
I want to save received base64 string in mysql table in a BLOB field. I have used sequelize version 5 as my ORM and the model is defined as follows.
sequelize.define('PLAYER', {
player_id: {
primaryKey: true,
type: DataTypes.INTEGER,
allowNull: false,
autoIncrement: true
},
player_name: DataTypes.STRING,
player_image: {
type: DataTypes.BLOB('medium'),
get () {
let data = this.getDataValue('player_image');
return data ? data.toString('base64') : '';
}
}
}
In my put method,
db.PLAYER.findByPk(req.params.playerId).then((player)=>{
if (req.body.player_image) {
const base64 = req.body.player_image.replace(/^data:image\/[a-z]+;base64,/, "");
const blob = b64toBlob(base64, 'image/jpeg');
player.player_image = blob;
}
player.player_name = req.body.player_name;
player.save().then(() => {
res.status(200).json(player);
}).catch(function (err) {
return next(err);
});
});
For Base64 to Blob conversion.
const b64toBlob = (b64Data, contentType = '', sliceSize = 512) => {
const byteCharacters = atob(b64Data);
const byteArrays = [];
for (let offset = 0; offset < byteCharacters.length; offset += sliceSize) {
const slice = byteCharacters.slice(offset, offset + sliceSize);
const byteNumbers = new Array(slice.length);
for (let i = 0; i < slice.length; i++) {
byteNumbers[i] = slice.charCodeAt(i);
}
const byteArray = new Uint8Array(byteNumbers);
byteArrays.push(byteArray);
}
const blob = new Blob(byteArrays, { type: contentType });
return blob;
}
A blob is saving in the table but my response body is as follows. I have converted BLOB to base64 before sending the response.
{
"player_id": 1032,
"player_name": "Oliver Driscoll",
"player_image": "[object Blob]"
}
Which means the saved blob is actually "[object Blob]". Not the image.
The received base64 (req.body.player_image) for the end point is confirmed as correct. I am running out of time and don't know what to do as this is the requirement (Saving the image as blob).
I Know, this is a old issue. But i had this same problem, and after so many tries, i was able to solve it. So, if anyone have this problem in the future, i will let my solution here:
In my case i was able to get a Blob already, from a fetch response.
So i have in my code like this:
await fetch(imageUrl)
.then(async (response) => {
// GET Blob Object from response; can use response.arrayBuffer() directly too
let myBlobImg = await response.blob();
// Convert Blob.ArrayBUffer to a Buffer
let myBufferImg = Buffer.from(await myBlobImg.arrayBuffer());
})
Then i just save myBufferImg on DB, as a Buffer. In your case #silent_27 could be
player.player_image = Buffer.from(await myBlobImg.arrayBuffer());
I'm using SQlite3 DataBase on NodeJS with sqlite3^5.0.1 package.
I save like this on my dataBase and works fine. I've tried save like myBlobImg.text();. But this away, i was not able to load on a <img src={}> html tag in my application.
If anyone need to know, to show this BLOB on a <img> HTML, i just do a get on my DB to get my ProductObject and convert to a base64 image:
// imageData is the Buffer saved on BD. imageData.data is an Array from Buffer
let imgBase64 = Buffer.from(product.productImage.imageData.data).toString('base64');
imgBase64 = `data:image/jpeg;base64,${imgBase64}`;
//....
<img src={imgBase64}>
I Had to use Buffer.from again otherwhise if i just use like
let imgBase64 = product.productImage.imageData.toString('base64');
the imgBase64 would be "[Object ojbect]
I have been working on a React Native project with Expo that uses a json file to store local data. I am importing the data like so
import data from '../database.json'
I am making changes (adding and removing) to the imported JSON by using data.push(new_data). These changes are not persistent when I close the app because I cannot figure out how to save them. I have looked at using the expo-file-system library as so:
import * as FileSystem from 'expo-file-system';
...
FileSystem.writeAsStringAsync(FileSystem.documentDirectory + 'database.json', data);
This is from looking at examples in the API documentations. This however always throws promise rejections and doesn't end up writing the file. Can you point me in the right direction?
Also, should I import the database.json in a different way so I will already have the uri to save it to?
The documentation doesn't give an example of it's returned props in promises, so I was overlooking it for longer than I care to admit 😅. I was really dedicated to figuring this out so I could use the Expo solution, and totally missed the return Promise for createFileAsync, so hopefully this saves someone a significant amount of time in the future.
import * as FileSystem from 'expo-file-system';
const { StorageAccessFramework } = FileSystem;
const saveFile = async () => {
const permissions = await StorageAccessFramework.requestDirectoryPermissionsAsync();
// Check if permission granted
if (permissions.granted) {
// Get the directory uri that was approved
let directoryUri = permissions.directoryUri;
let data = "Hello World";
// Create file and pass it's SAF URI
await StorageAccessFramework.createFileAsync(directoryUri, "filename", "application/json").then(async(fileUri) => {
// Save data to newly created file
await FileSystem.writeAsStringAsync(fileUri, data, { encoding: FileSystem.EncodingType.UTF8 });
})
.catch((e) => {
console.log(e);
});
} else {
alert("You must allow permission to save.")
}
}
Use AsyncStorage instead. The react native package is deprecated but working, or use #react-native-community/async-storage and convert json to string (AsyncStorage can only store strings)
Set item
import AsyncStorage from '#react-native-community/async-storage';
...
await AsyncStorage.setItem('myData', JSON.stringify(data))
Get item
const data = await AsyncStorage.getItem('myData')
I found #JayMax answer very helpful however it's only for Android.
On iOS all you need to do is use Sharing.shareAsync and then you can save data to the file. Check this example:
const fileUri = FileSystem.documentDirectory + 'data.txt';
FileSystem.writeAsStringAsync(fileUri, 'here goes your data from JSON. You can stringify it :)', {
encoding: FileSystem.EncodingType.UTF8,
});
const UTI = 'public.text';
Sharing.shareAsync(fileUri, {UTI}).catch((error) => {
console.log(error);
});
If you using AsyncStorage, it only store for small data. Maybe 6mb or 10 mb.
You can use expo fileSystem
import * as FileSystem from 'expo-file-system';
...
FileSystem.writeAsStringAsync(FileSystem.documentDirectory + 'database.json', data);
Convert your data (Type json to string) Such as this:
writeData = async () => {
var persons = ''
await axios.get(`http://192.168.0.48:4000/api/sql/student`)
.then(res => {
persons = res.data
})
await FileSystem.writeAsStringAsync(FileSystem.documentDirectory + `offline_queue_stored.json`, JSON.stringify(persons));
}
#1.If the JSON File is in your Project Folder (PC/Laptop)
import data from './database.json';
#2. If the JSON File is in your Phone
import * as FileSystem from 'expo-file-system';
import * as DocumentPicker from 'expo-document-picker';
this.state = {
fileURI: null,
};
componentDidMount = () =>{
this._pickDocument();
}
_pickDocument = async () => {
let result = await DocumentPicker.getDocumentAsync({});
this.setState({
fileURI: result.uri
})
let fileData = await FileSystem.readAsStringAsync(this.state.fileURI)
console.log(fileData)
};
I am learning about streaming with nodejs, I understand the examples shown in the request npm module;
request(url).pipe(fs.createWriteStream('./filename.json'))
But there are two parts of my problem.
Case 1:
function fetchSitemaps() {
return requestAsync(url).then(data => {
const $ = cheerio.load(data);
let urls = [];
$("loc").each((i, e) => urls.push($(e).text()));
fs.writeFileSync('./sitemaps.json', JSON.stringify(urls))
})
}
I want to convert the above from writeFileSync to createWriteStream, but how do I keep appending data to an array which is in JSON format?
Case 2:
function fetchLyricUrls() {
let sitemaps = JSON.parse(fs.readFileSync('./sitemaps.json'));
sitemaps.forEach((sitemap, i) => {
let fileName = i + '.json';
if(url_pat.exec(sitemap)) {
fileName = url_pat.exec(sitemap)[1] + '.json';
}
requestAsync(url).then(data => {
const $ = cheerio.load(data);
let urls = [];
$("loc").each((i, e) => urls.push($(e).text()));
return urls;
}).then(urls => {
let allUrls = [];
urls.map(u => {
return requestAsync(u).then(sm => {
const $ = cheerio.load(sm);
$("loc").each((i, e) => allUrls.push($(e).text()))
fs.writeFileSync('./lyrics.json', JSON.stringify(allUrls))
return allUrls;
});
});
});
});
}
The first part of the problem is same, appending to a json data using writeStream, but this time, I want to parse the the html data and get some text, which I want to send using stream, not the html data as a whole.
So let's split up the answers
Case 1
First of all I'd try to keep the data as a stream and try not to accumulate it. So in essence, instead of loading the whole sitemap and then parsing it, I'd use something like the xml-nodes so that the nodes are a separate stream. Then my module scramjet would come to transform
const request = require('request');
const xmlNodes = require('xml-nodes');
const writable = fs.createWritableStream('./sitemaps.json');
const cheerio = require('cheerio');
const scramjet = require('scramjet');
writable.write('[');
let first = 0;
request('http://example.com/sitemap.xml')
// this fetches your sitemap
.on('end', () => writable.end("]"))
// when the stream ends, this will end the sitemaps.json
.pipe(xmlNodes('loc'))
// this extracts your "loc" nodes
.pipe(new scramjet.DataStream())
// this creates a mappable stream
.map((nodeString) => cheerio('loc', nodeString).text())
// this extracts the text as in your question
.map((url) => (first++ ? ',' : '') + JSON.stringify(url))
// this makes sure that strings are nicely escaped
// and prepends them with a comma on every node, but first one
.pipe(writable, {end: false})
// and this will push all your entries to the writable stream
Case 2
Here you'll need to do something similar, although if case 1 is an immediate step, then I'd suggest to store the files in lines of JSONs, not an array. It'd make easier to stream that way.