i've build my first Node app in which i need to use 5-10 global variables a lot. The thing is i would like to be able to change those values without restarting the server.
So what i though was setup an interval and update those files either from a ( JSON ? ) file or through a couple of queries to the database.
Now what would be my better option here ? Both mysql and read file modules are used in the app.
Security based wouldn't it be best to place the json file behind the public folder and read from that ? Although without sql injection being possible i think in the DB should be pretty safe too.
What do you guys think ?? Still a newbie in Node JS.
Thanks
With yamljs, the overhead is that you will need to install it. From your question, it seems you are already using a bunch of 3rd party modules in your app.
Why not use something that is a part of node.js itself?
Use the fs module.
Example:
var fs = require('fs');
var obj;
fs.readFile('myData.json', 'utf8', function (err, data) {
if (err) throw err;
obj = JSON.parse(data);
});
Docs: https://nodejs.org/api/fs.html#fs_fs_readfile_file_options_callback
A common technique for passing configuration variables to a server is via a YAML file. Normally this file is read once when the server starts but you could periodically query they file to see when it was last updated and if the file was changed update the configuration variables currently in use.
yamljs
YAML = require('yamljs');
config = YAML.load('myfile.yml');
then you can periodically check the last time a file was modified using the mtime property of fs.stat
fs.stat(path, [callback])
If you find that the last modified time has changed then you can re-read the YAML file and update your config with the new values. ( you will probably want to do a sanity check to make sure the file wasn't corrupted etc. )
If you don't want to write the file watching logic yourself I recommend checking out chokidar
// Initialize watcher.
var watcher = chokidar.watch('myfile.yml', {
ignored: /[\/\\]\./,
persistent: true
});
// Add event listeners.
watcher.on('change', function(path) {
// Update config
})
Related
Im trying to perform a simple download of a .docx file info a buffer so I can handle it latter inside my Cloud Function. I've been using the whole Google Platform for multiple projects but never faced the need to download in server side, and now I need to, I just cant.
The following piece of code is not working, it just sends timeout as a response (I don't even get an error If I try to catch it or something):
var bucket = admin.storage().bucket("gs://myBucket.com");
return bucket.file("001Lineales/4x3-1/1000.docx").download().then((contents)=>{
var buffer = contents[0];
//I never get into this point
}).catch((error)=>{
//No error
})
I tried in a local NodeJs script and worked as expected. Also tried to perform a readStream() download but no luck, the function gets hang up in any try of downloading the file.
return new Promise((resolve,reject)=>{
var archivo = bucket.file(selectedCategory).createReadStream();
var array = [];
//Under here, never happens
archivo.on('data', (d) => {array.push(d)}).on("end",()=>{
var newbuff = Buffer.concat(array);
resolve(newbuff)
})
})
The file permissions read/write are public. And the main problem is that debugging is difficult cause Im not able to perform this function in local emulator.
What can I do? Thanks in advance.
EDIT:
Double checking a local call with emulator, I get the following error:
Anonymous caller does not have storage.objects.get access to the Google Cloud Storage object.
Double check the service account hat you've assigned to the Cloud Function and that you've given it the permission it needs.
I think Storage Object Viewer will give you what you need to read a file into the buffer.
By default, if you haven't changed it, the AppEngine's default service account gets used, which I don't think has access to Storage.
I have this fs.writeFile code that is suppose to update the localdata.json file when a newWorkout is POST'ed to the database, this takes local state data in an attempt to write to the file..
it wont work though and throws a , TypeError: fs.writeFile is not a function error.. working on the fix now but if anyone sees anything help is appreciated.
fs.writeFile(
"./localdata.json",
JSON.stringify(newWorkout.eventDateTime),
"utf-8",
function(err) {
if (err) throw err
console.log("Done!")
}
)
Given that you are working in a Node.js environment, is seems like fs is not a proper Node.js File System object. Copied from the Node.js 10.x documentation:
To use this module:
const fs = require('fs');
Nodes File System cannot be used "browser side" these calls are meant to happen on the server side of things.
Did you include:
var fs = require("fs");
...the rest of your code...
So I am attempting to learn how to use the Google Sheets API with Node.js. In order to get an understanding, I followed along with the node.js quick start guide supplied by Google. I attempted to run it, nearly line for line a copy of the guide, just without documentation. I wind up encountering this: cmd console output that definitely didn't work.
Just in case anyone wants to see if I am not matching the guide, which is entirely possible since I am fairly new to this, here is a link to the Google page and my code.
https://developers.google.com/sheets/api/quickstart/nodejs
var fs = require('fs');
var readline = require('readline');
var google = require('googleapis');
var googleAuth = require('google-auth-library');
var SCOPES = ['https://www.googleapis.com/auth/spreadsheets.readonly'];
var TOKEN_DIR = (process.env.HOME || process.env.HOMEPATH ||
process.env.USERPROFILE) + '/.credentials/';
var TOKEN_PATH = TOKEN_DIR + 'sheets.googleapis.com-nodejs-quickstart.json';
fs.readFile('client_secret.json', function processClientSecrets(err, content) {
if (err) {
console.log('Error loading client secret file: ' + err);
}
authorize(JSON.parse(content), listMajors);
});
I have tried placing the JSON file in each and every part of the directory, but it still won't see it. I've been pulling hairs all day, and a poke in the right direction would be immensely appreciated.
From your command output:
Error loading client secret file
So your if (err) line is being triggered. But since you don't throw the error, the script continues anyway (which is dangerous in general).
SyntaxError: Unexpected token u in JSON at position 0
This means that the data you are passing to JSON.parse() is undefined. It is not a valid JSON string.
You could use load-json-file (or the thing it uses, parse-json) to get more helpful error messages. But it's caused by the fact that your content variable has nothing since the client_secret.json you tried to read could not be found.
As for why the file could not be found, there could be a typo in either the script or the filename you saved the JSON in. Or it may have to do with the current working directory. You may want to use something like this to ensure you end up with the same path regardless of the current working directory.
path.join(__dirname, 'client_secret.json')
Resources
path.join()
__dirname
I'm trying to help a friend about his nodejs application.
He use NodeMailer to send emails automatically. We both use ubuntu.
To avoid any leaks, the credential for NodeMailer are set as an environnement variable called EMAIL_CREDENTIALS.
In the app, EMAIL_CREDENTIAL is called as follow:
var emailCredentials = process.env.EMAIL_CREDENTIALS;
if (emailCredentials === 'undefined') {
throw Error('Email credentials are not present');
}
// create reusable transporter object using the default SMTP transport
var transporter = nodemailer.createTransport(emailCredentials);
To set email credential I added a line in src/environnement :
EMAIL_CREDENTIALS={host:"smtp.gmail.com", port: 587, secure: false, auth: {user: "**user**", pass: "**pass**"}}
When I run some test, NodeMailer return the following error
Error: Unsupported configuration, downgrade Nodemailer to v0.7.1 to use it
at Nodemailer.sendMail (/home/**path**/node_modules/nodemailer/lib/nodemailer.js:274:18)
This error happen because type of param === 'string' in nodemailer.createTransport(param) .
My friend has been using this code for a pretty long time with no problem.
Somehow, I understand where the error come from but I would like to know how is it possible that the environnement variable of my friends setup isn't a string and mine is ? And how should I set my variable to be able to run his code without modifying it ?
EDIT: To use JSON.parse() would be a solution but I would prefer not to modifie the app code and as I said this configuration seems to work for my friend so I would like to understand where's the difference.
It is better not to use environment variables for other than simple values. I would suggest you to use a plain JS or JSON file with those variables, that you can simple require(). You can ignore this file via .gitignore, if you do not want it to be included in a git repository.
You can take a look at my configuration module here: https://www.npmjs.com/package/mikro-config
It is designed to be used this way.
Using it, your general configuration will be stored in /config/default.js file, and your environment specific configuration will be stored in /config/env/$NODE_ENV.js file (or in /config/env/$NODE_ENV.local.js, which should be ignored by git, as I described above).
I try to import a local .json-file using d3.json().
The file filename.json is stored in the same folder as my html file.
Yet the (json)-parameter is null.
d3.json("filename.json", function(json) {
root = json;
root.x0 = h / 2;
root.y0 = 0;});
. . .
}
My code is basically the same as in this d3.js example
If you're running in a browser, you cannot load local files.
But it's fairly easy to run a dev server, on the commandline, simply cd into the directory with your files, then:
python -m SimpleHTTPServer
(or python -m http.server using python 3)
Now in your browser, go to localhost:3000 (or :8000 or whatever is shown on the commandline).
The following used to work in older versions of d3:
var json = {"my": "json"};
d3.json(json, function(json) {
root = json;
root.x0 = h / 2;
root.y0 = 0;
});
In version d3.v5, you should do it as
d3.json("file.json").then(function(data){ console.log(data)});
Similarly, with csv and other file formats.
You can find more details at https://github.com/d3/d3/blob/master/CHANGES.md
Adding to the previous answers it's simpler to use an HTTP server provided by most Linux/ Mac machines (just by having python installed).
Run the following command in the root of your project
python -m SimpleHTTPServer
Then instead of accessing file://.....index.html open your browser on http://localhost:8080 or the port provided by running the server. This way will make the browser fetch all the files in your project without being blocked.
http://bl.ocks.org/eyaler/10586116
Refer to this code, this is reading from a file and creating a graph.
I also had the same problem, but later I figured out that the problem was in the json file I was using(an extra comma). If you are getting null here try printing the error you are getting, like this may be.
d3.json("filename.json", function(error, graph) {
alert(error)
})
This is working in firefox, in chrome somehow its not printing the error.
Loading a local csv or json file with (d3)js is not safe to do. They prevent you from doing it. There are some solutions to get it working though. The following line basically does not work (csv or json) because it is a local import:
d3.csv("path_to_your_csv", function(data) {console.log(data) });
Solution 1:
Disable the security in your browser
Different browsers have different security setting that you can disable. This solution can work and you can load your files. Disabling is however not advisable. It will make you vulnerable for all kind of threads. On the other hand, who is going to use your software if you tell them to manually disable the security?
Disable the security in Chrome:
--disable-web-security
--allow-file-access-from-files
Solution 2:
Load your csv/json file from a website.
This may seem like a weird solution but it will work. It is an easy fix but can be unpractical though. See here for an example. Check out the page-source. This is the idea:
d3.csv("https://path_to_your_csv", function(data) {console.log(data) });
Solution 3:
Start you own browser, with e.g. Python.
Such a browser does not include all kind of security checks. This may be a solution when you experiment with your code on your own machine. In many cases, this may not be the solution when you have users. This example will serve HTTP on port 8888 unless it is already taken:
python -m http.server 8888
python -m SimpleHTTPServer 8888 &
Open the (Chrome) browser address bar and type the underneath. This will open the index.html. In case you have a different name, type the path to that local HTML page.
localhost:8888
Solution 4:
Use local-host and CORS
You may can use local-host and CORS but the approach is not user-friendly coz setting up this, may not be so straightforward.
Solution 5:
Embed your data in the HTML file
I like this solution the most. Instead of loading your csv, you can write a script that embeds your data directly in the html. This will allow users use their favorite browser, and there are no security issues. This solution may not be so elegant because your html file can grow very hard depending on your data but it will work though. See here for an example. Check out the page-source.
Remove this line:
d3.csv("path_to_your_csv", function(data) { })
Replace with this:
var data =
[
$DATA_COMES_HERE$
]
You can't readily read local files, at least not in Chrome, and possibly not in other browsers either.
The simplest workaround is to simply include your JSON data in your script file and then simply get rid of your d3.json call and keep the code in the callback you pass to it.
Your code would then look like this:
json = { ... };
root = json;
root.x0 = h / 2;
root.y0 = 0;
...
I have used this
d3.json("graph.json", function(error, xyz) {
if (error) throw error;
// the rest of my d3 graph code here
}
so you can refer to your json file by using the variable xyz and graph is the name of my local json file
Use resource as local variable
var filename = {x0:0,y0:0};
//you can change different name for the function than json
d3.json = (x,cb)=>cb.call(null,x);
d3.json(filename, function(json) {
root = json;
root.x0 = h / 2;
root.y0 = 0;});
//...
}