Importing local json file using d3.json does not work - json

I try to import a local .json-file using d3.json().
The file filename.json is stored in the same folder as my html file.
Yet the (json)-parameter is null.
d3.json("filename.json", function(json) {
root = json;
root.x0 = h / 2;
root.y0 = 0;});
. . .
}
My code is basically the same as in this d3.js example

If you're running in a browser, you cannot load local files.
But it's fairly easy to run a dev server, on the commandline, simply cd into the directory with your files, then:
python -m SimpleHTTPServer
(or python -m http.server using python 3)
Now in your browser, go to localhost:3000 (or :8000 or whatever is shown on the commandline).
The following used to work in older versions of d3:
var json = {"my": "json"};
d3.json(json, function(json) {
root = json;
root.x0 = h / 2;
root.y0 = 0;
});

In version d3.v5, you should do it as
d3.json("file.json").then(function(data){ console.log(data)});
Similarly, with csv and other file formats.
You can find more details at https://github.com/d3/d3/blob/master/CHANGES.md

Adding to the previous answers it's simpler to use an HTTP server provided by most Linux/ Mac machines (just by having python installed).
Run the following command in the root of your project
python -m SimpleHTTPServer
Then instead of accessing file://.....index.html open your browser on http://localhost:8080 or the port provided by running the server. This way will make the browser fetch all the files in your project without being blocked.

http://bl.ocks.org/eyaler/10586116
Refer to this code, this is reading from a file and creating a graph.
I also had the same problem, but later I figured out that the problem was in the json file I was using(an extra comma). If you are getting null here try printing the error you are getting, like this may be.
d3.json("filename.json", function(error, graph) {
alert(error)
})
This is working in firefox, in chrome somehow its not printing the error.

Loading a local csv or json file with (d3)js is not safe to do. They prevent you from doing it. There are some solutions to get it working though. The following line basically does not work (csv or json) because it is a local import:
d3.csv("path_to_your_csv", function(data) {console.log(data) });
Solution 1:
Disable the security in your browser
Different browsers have different security setting that you can disable. This solution can work and you can load your files. Disabling is however not advisable. It will make you vulnerable for all kind of threads. On the other hand, who is going to use your software if you tell them to manually disable the security?
Disable the security in Chrome:
--disable-web-security
--allow-file-access-from-files
Solution 2:
Load your csv/json file from a website.
This may seem like a weird solution but it will work. It is an easy fix but can be unpractical though. See here for an example. Check out the page-source. This is the idea:
d3.csv("https://path_to_your_csv", function(data) {console.log(data) });
Solution 3:
Start you own browser, with e.g. Python.
Such a browser does not include all kind of security checks. This may be a solution when you experiment with your code on your own machine. In many cases, this may not be the solution when you have users. This example will serve HTTP on port 8888 unless it is already taken:
python -m http.server 8888
python -m SimpleHTTPServer 8888 &
Open the (Chrome) browser address bar and type the underneath. This will open the index.html. In case you have a different name, type the path to that local HTML page.
localhost:8888
Solution 4:
Use local-host and CORS
You may can use local-host and CORS but the approach is not user-friendly coz setting up this, may not be so straightforward.
Solution 5:
Embed your data in the HTML file
I like this solution the most. Instead of loading your csv, you can write a script that embeds your data directly in the html. This will allow users use their favorite browser, and there are no security issues. This solution may not be so elegant because your html file can grow very hard depending on your data but it will work though. See here for an example. Check out the page-source.
Remove this line:
d3.csv("path_to_your_csv", function(data) { })
Replace with this:
var data =
[
$DATA_COMES_HERE$
]

You can't readily read local files, at least not in Chrome, and possibly not in other browsers either.
The simplest workaround is to simply include your JSON data in your script file and then simply get rid of your d3.json call and keep the code in the callback you pass to it.
Your code would then look like this:
json = { ... };
root = json;
root.x0 = h / 2;
root.y0 = 0;
...

I have used this
d3.json("graph.json", function(error, xyz) {
if (error) throw error;
// the rest of my d3 graph code here
}
so you can refer to your json file by using the variable xyz and graph is the name of my local json file

Use resource as local variable
var filename = {x0:0,y0:0};
//you can change different name for the function than json
d3.json = (x,cb)=>cb.call(null,x);
d3.json(filename, function(json) {
root = json;
root.x0 = h / 2;
root.y0 = 0;});
//...
}

Related

Convert HTML to PDF or PNG without headeless browser instance in NodeJS

TL;DR:
Any suggestions in NodeJS to convert an HTML to PDF or PNG without any headless browser instances.
Also anyone uses puppeteer in any production environment. I would like to know how the resource utilisations and performance of running headless browser in prod.
Longer version:
In a NodeJS server we need to convert an HTML string to a PDF or PNG based on the request params. We are using puppeteer to generate this PDF and PNG (screenshot) deployed in a google cloud function. In my local running this application in a docker and restricted memory usage to 100MB and this seems working. But in cloud function it throws memory limit exception when we set the cloud function to 250MB memory. For a temporary solution we upgraded the cloud function to 1 GB.
We would like to try any alternatives for puppeteer without any headless browser approach. Another library PDF-Kit looks good but it have canvas api kind of input. We can't directly feed html.
Any thoughts or input on this
Any suggestions in NodeJS to convert an HTML to PDF or PNG without any headless browser instances.
Yes, you can try with jsPDF. I never used it before.
The syntax is simple.
Under the hood it looks no headless browser libraries are used and it seems this is a 100% pure javascript implementation.
You can feed the library directly with and HTML string.
BUT there is no png option. For images anyway there are a lot of solution that could be combined with jsPDF (so, HTML to PDF to PNG) or also other HTML to PNG direct solutions. Take a look here.
Also anyone uses puppeteer in any production environment. I would like to know how the resource utilisations and performance of running headless browser in prod.
When you want use puppeteer, I suggest to split services: a simple http server that must just handle the HTTP communication with your clients and a separate puppeteer service. Both services must be scalable but, ofcourse, the second will require more resources to run. To optimize resorces, I suggest using puppeter-cluster to create a cluster of puppeteer workers. You can better handle errors, flow and concurrency and at the same time you can save memory by using a single istance of Chromium (with the CONCURRENCY_PAGE or CONCURRENCY_CONTEXT model)
If you can use Docker, then a great solution for you may be Gotenberg.
It's an incredible service that can convert a lot of formats (HTML, Markdown, Word, Excel, etc.) into PDF.
If your page render depends on JavaScript, then no problem, it will run it and wait (you can even configure the max wait time) for the page to be completely rendered to generate your PDF.
We are using it for an application that generates 3000 PDFs per day and never had any issue with it.
Demo:
Take a look at this sample HTML invoice: https://sparksuite.github.io/simple-html-invoice-template/
Now let's convert it to PDF:
Boom, done!
1: Gotenberg URL (here using a demo endpoint provided by Gotenberg team with some limitations like 2 requests per second per IP and 5MB body limit)
2: pass an url parameter with the URL of the webpage you want to convert
3: You get the PDF as the HTTP response with Content-Type application/pdf
Curl version:
curl --location --request POST 'https://demo.gotenberg.dev/forms/chromium/convert/url' \
--form 'url="https://sparksuite.github.io/simple-html-invoice-template/"' \
-o myfile.pdf
Node.JS version:
const fetch = require('node-fetch');
const FormData = require('form-data');
const fs = require('fs');
async function main() {
const formData = new FormData();
formData.append('url', 'https://sparksuite.github.io/simple-html-invoice-template/')
const res = await fetch('https://demo.gotenberg.dev/forms/chromium/convert/url', {
method: 'POST',
body: formData
})
const pdfBuffer = await res.buffer()
// You can do whatever you like with the pdfBuffer, such as writing it to the disk:
fs.writeFileSync('/home/myfile.pdf', pdfBuffer);
}
main()
Using your own Docker instance instead of the demo endpoint, here is what you need to do:
1. Create the Gotenberg Docker container:
docker run -p 3333:3000 gotenberg/gotenberg:7 gotenberg
2. Call the http://localhost:3333/forms/chromium/convert/url endpoint:
Curl version:
curl --location --request POST 'http://localhost:3333/forms/chromium/convert/url' \
--form 'url="https://sparksuite.github.io/simple-html-invoice-template/"' \
-o myfile.pdf
Node.JS version:
const fetch = require('node-fetch');
const FormData = require('form-data');
const fs = require('fs');
async function main() {
const formData = new FormData();
formData.append('url', 'https://sparksuite.github.io/simple-html-invoice-template/')
const res = await fetch('http://localhost:3333/forms/chromium/convert/url', {
method: 'POST',
body: formData
})
const pdfBuffer = await res.buffer()
// You can do whatever you like with the pdfBuffer, such as writing it to the disk:
fs.writeFileSync('/home/myfile.pdf', pdfBuffer);
}
main()
Gotenberg homepage: https://gotenberg.dev/
If you have access to command wkhtmltopdf, I recommended it.
We use with success in our production website to generate pdfs.
First generate file_name html file, then
wkhtmltopdf --encoding utf8 --disable-smart-shrinking --dpi 100 -s {paper_size} -O {orientation} '{file_name}'

Why does Chrome DevTools show multiple garbled versions of my source code for my Vue application?

I have a Vue application and I'm trying to debug it in Chrome DevTools. The problem is when I try to find the file I want to debug, I get a list of files with the same name plus some weird hash tacked onto the end:
When I open any one file, I get some garbled minified code:
Sometimes I can find the file I want (with the original source code) but sometimes not.
What are these weird files and how can I find the file I want (with the original source code). Is there a way of getting the DevTools to list only the original source code files?
Thanks.
What tool in dev tools are you using to get that list? Seems like a list of cached files, so it's showing all the old versions of your code.
If you go to the network tab and reload the page. You should see a list of all the resources downloaded by the browser. Choose the js filter and you should see your vue js bundle (made by webpack) somewhere in that list.
To allow chrome to display the source correctly you need to generate the Source Maps in development deployments.
I am not sure what tool you are using to build and bundle, but it is likely that you might have support for this already.
Chrome Details:
https://developer.chrome.com/docs/devtools/javascript/source-maps/
OMG - debugging my debugging environment. It's SO maddening.
I'm working with Vue v2, and I'm using vuetify in my app. Here is a complete vue.config.js configuration that solved this problem for me.
// vue.config.js file
const path = require('path')
const { defineConfig } = require('#vue/cli-service')
module.exports = defineConfig({
transpileDependencies: [
'vuetify'
],
configureWebpack: config => {
if (process.env.NODE_ENV === 'development') {
// See available sourcemaps:
// https://webpack.js.org/configuration/devtool/#devtool
config.devtool = 'eval-source-map'
// console.log(`NOTICE: vue.config.js directive: ${config.devtool}`)
config.output.devtoolModuleFilenameTemplate = info => {
let resPath = path.normalize(info.resourcePath)
let isVue = resPath.match(/\.vue$/)
let isGenerated = info.allLoaders
let generated = `webpack-generated:///${resPath}?${info.hash}`
let vuesource = `vue-source:///${resPath}`
return isVue && isGenerated ? generated : vuesource
}
config.output.devtoolFallbackModuleFilenameTemplate =
'webpack:///[resource-path]?[hash]'
}
},
})
I found a work around for this. While you can not see the source code of your file, just change the code (add console or sth.) of the file you want to see while Vue is hot reloading your changes. It occurs to me that the source code is then reachable when you check the developer console.
There is a surprising number of developers I meet on projects that have no idea there are official browser extensions for debugging Vue, Router, VueX etc.
Stumbling across this question prompted me to post this life saving link for those that land here and have missed the existence of this essential tool:
https://devtools.vuejs.org/guide/installation.html

Read user directory in HTML 5 and load images in it

I've been toying around with the FileSystem and File API, in Chrome, to try to implement a transient "instant gallery". The user chooses a directory, and all the images in it are then displayed in the webpage.
But I'm having a hard time, it seems Chrome requires some extra launching arguments to allow file access, FileSystem and File API are not W3C and not portable, I cannot instantiate certain objects...
I cannot even get the directory absolute path to open files in it (though maybe I don't need the absolute path, but I feel like it lacks a good documentation).
Anyway, I wanted to know how to implement this? Is there another API? A simpler way? Do I absolutely need to use FileSystem and File, and set Chrome's arguments?
In order to read the files in the directory you will need to create a DirectoryReader object, and use the readEntries() method to read the content of the directory:
fs.root.getDirectory('Documents', {}, function(dirEntry){<br>
var dirReader = dirEntry.createReader();
dirReader.readEntries(function(entries) {<br>
for(var i = 0; i < entries.length; i++) {
var entry = entries[i];
if (entry.isDirectory){
console.log('Directory: ' + entry.fullPath);
}
else if (entry.isFile){
console.log('File: ' + entry.fullPath);
}
}
}, errorHandler);
}, errorHandler);
Please take a look here: http://code.tutsplus.com/tutorials/toying-with-the-html5-filesystem-api--net-24719
But I think that Chrome will not be able to access an entire directory that the user has selected from his computer, only if the user has selected multiple files in an input field. If that is ok and suits your needs there is a good tutorial here:
http://www.html5rocks.com/en/tutorials/file/dndfiles/

nodejs dont recognize own extension instead of JSON

I am trying to use own .extension for .JSON files. But when I require them node don't recognize them.
For example when I do:
var users = require('users.json');
users is now a object like:
name: 'somebody', age: 27
When I do just the same file but other .extension"
require('users.myextension')
users is now empty;
=> {}
Is there a way to fix this, otherwise I have to use just JSON as extension.
You can look at globals
In short you can instruct how require should load a file in this way:
require.extensions['.extension'] = require.extensions['.json'];
Please mind that this is deprecated but as stated in the note the feature probably will not be removed since the module system is locked.

Is there any way to get command line parameters in Google Chrome extension?

I need to launch Chrome from command line with custom parameter, which
contains path to some js-file. Further this path will be used in
extension.
I browsed carefully all related documentation and clicked all nodes in
Chrome debugger, but found nothing which can resemble on command line
parameters. Is it possible anyway to get these parameters or it's need
to write more complex npapi-extension? (theoretically in such npapi-
extension we able to get self process through win-api, command line of
self process and so on).
Hack alert: this post suggests passing a fake URL to open that has all the command-line parameters as query string parameters, e.g.,
chrome.exe http://fakeurl.com/?param1=val1&param2=val2
Perhaps pass the path to your extension in a custom user agent string set via the command line. For example:
chrome.exe --user-agent='Chrome 43. My path is:/path/to/file'
Then, in your extension:
var path = navigator.userAgent.split(":");
console.log(path[1])
Basically I use the technique given in #dfrankow's answer, but I open 127.0.0.1:0 instead of a fake URL. This approach has two advantages:
The name resolution attempt is skipped. OK, if I've chosen the fake URL carefully to avoid opening an existing URL, the name resolution would fail for sure. But there is no need for it, so why not just skip this step?
No server listens on TCP port 0. Using simply 127.0.0.1 is not enough, since it is possible that a web server runs on the client machine, and I don't want the extension to connect to it accidentally. So I have to specify a port number, but which one? Port 0 is the perfect choice: according to RFC 1700, this port number is "reserved", that is, servers are not allowed to use it.
Example command line to pass arguments abc and xyz to your extension:
chrome "http://127.0.0.1:0/?abc=42&xyz=hello"
You can read these arguments in background.js this way:
chrome.windows.onCreated.addListener(function (window) {
chrome.tabs.query({}, function (tabs) {
var args = { abc: null, xyz: null }, argName, regExp, match;
for (argName in args) {
regExp = new RegExp(argName + "=([^\&]+)")
match = regExp.exec(tabs[0].url);
if (!match) return;
args[argName] = match[1];
}
console.log(JSON.stringify(args));
});
});
Console output (in the console of the background page of the extension):
{"abc":"42","xyz":"hello"}
You could try:
var versionPage = "chrome://version/strings.js";
$.post(versionPage, function(data){
data = data.replace("var templateData = ", "");
data = data.slice(0, -1);
var jsonOb = $.parseJSON(data);
alert(jsonOb.command_line);
});
This assumes you are using jQuery in your loading sequence, you could always substitute with any other AJAX method
Further to the answers above about using the URL to pass parameters in, note that only Extensions, not Apps, can do this. I've published a Chrome Extension that just intercepts the URL and makes it available to another App.
https://chrome.google.com/webstore/detail/gafgnggdglmjplpklcfhcgfaeehecepg/
The source code is available at:
https://github.com/appazur/kiosk-launcher
for Wi-Fi Public Display Screens