I would like to merge all json files in a directory in nodejs. The files are being uploaded by user and all i]I know is their names are device"count".json. Count being incrememented each time. I know of json-concat but how do I use it to merge all files in a directory?
jsonConcat({
src: [],
dest: "./result.json"
}, function (json) {
console.log(json);
});
You can use the fs modules to read the files in a directory and then pass them to json-concat.
const jsonConcat = require('json-concat');
const fs = require('fs');
// an array of filenames to concat
const files = [];
const theDirectory = __dirname; // or whatever directory you want to read
fs.readdirSync(theDirectory).forEach((file) => {
// you may want to filter these by extension, etc. to make sure they are JSON files
files.push(file);
})
// pass the "files" to json concat
jsonConcat({
src: files,
dest: "./result.json"
}, function (json) {
console.log(json);
});
if You read docs carefully You'll see this part:
The options object passed may have the following keys:
src:
(String) path pointing to a directory
(Array) array of paths pointing to files and/or directories
defaults to . (current working directory)
Note: if this is a path points to a single file, nothing will be done.
so here is fix:
1) move json files to some concrete path.
2) check this code:
jsonConcat({
src: './path/to/json/files',
dest: "./result.json"
}, function (json) {
console.log(json);
});
and here is the prove how it uses src param
Mostly it needs from developer not just to use 3-rd part packages, but also dive into it's sources.
in short: KISS (:
Related
Using Nuxt 3, I am struggling to do something that appears simple: I would like to get a list of restaurants from an api served by nuxt/nitro in order to use it on the client side. My original file, is a .csv file stored under assets/: assets/list.csv.
Here is what I have in my component .vuefile:
//...
const { restaurants } = await useFetch('/api/restaurants')
//...
And the content of server/api/restaurants.js:
import csv from 'csvtojson'
export default defineEventHandler(async(event) => {
const data = await csv().fromFile('~/assets/list.csv')
return { data }
})
But I get an error "[500] File does not exist". I've tried many variants but always get an error here or there. Could you help me figure out the solution? Thanks.
Actually solved by realizing the following:
As the doc suggests, the assets/ directory is for assets that are processed by the bundler (Vite or Webpack). Nuxt won't serve files in the assets/ directory unless nuxt.config.ts is configured with the appropriate loader (e.g. a csv loader). Hence the 500 error.
Nuxt Content, on the other hand, is useful to automatically parse a .csv file located in the content/ directory:
In nuxt.config.ts:
modules: ["#nuxt/content"]
In the component .vue file, the following will expose the parsed csv in data.body:
const { data } = await useAsyncData("list", () => queryContent("/list").findOne())
The beauty of nuxt is that you don't need to import anything, it does it for you.
I'm using the ipfs-http-client module to interact with IPFS. My problem is that I need the file extension on the link that I generate, and it seems that I can only get it with the wrapWithDirectory flag (-w with the command line). But this flag makes the result empty so far. The documentation on IPFS is only about the command line, and I've only found out a few tutorials about how to do it, but with other tool than JS, or by uploading folders manually. I need to do it from a JS script, from a single file. The motivation is that I want to generate metadata for an NFT, and a metadata field requires to point to a file with a specific extension.
Full detail: I need to add a GLB file on Opensea. GLB are like GLTF, it's a standard for 3D file. Opensea can detect the animation_url field of the metadata of an NFT and render that file. But it needs to end with .glb. Translation, my NFT needs its metadata to look like that:
{
name: <name>,
description: <description>,
image: <image>,
animation_url: 'https://ipfs.io/ipfs/<hash>.glb' // Opensea requires the '.glb' ending.
}
The way I do this so far is as follows:
import { create } from 'ipfs-http-client';
const client = create({
host: 'ipfs.infura.io',
port: 5001,
protocol: 'https',
headers: { authorization },
});
const result = await client.add(file); // {path: '<hash>', cid: CID}
const link = `https://ipfs.io/ipfs/${result.path}` // I can't add an extension here.
In that code, I can put animation_url: link in the metadata object, but OpenSea won't recognize it.
I have tried adding the option mentioned above as well:
const result = await client.add(file, {wrapWithDirectory: true}); // {path: '', cid: CID}
But then result.path is an empty string.
How can I generate a link ending with .glb?
Found out the solution. It indeed involves creating a directory, which is the returned CID, so that we can append the file name with its extension at the end. The result is https://ipfs.io/ipfs/<directory_hash>/<file_name_with_extension>.
So, correcting the code above it gives the following:
import { create } from 'ipfs-http-client';
const client = create({
host: 'ipfs.infura.io',
port: 5001,
protocol: 'https',
headers: { authorization },
});
const content = await file.arrayBuffer(); // The file needs to be a buffer.
const result = await client.add(
{content, path: file.name},
{wrapWithDirectory: true}
);
// result.path is empty, it needs result.cid.toString(),
// and then one can manually append the file name with its extension.
const link = `https://ipfs.io/ipfs/${result.cid.toString()}/${result.name}`;
I want to keep my test data in a JSON file that I need to import in cucumber-protractor custom framework. I read we can directly require a JSON file or even use protractor params. However that doesn't work. I don't see the JSON file listed when requiring from a particular folder.
testdata.json
{
"name":"testdata",
"version":"1.0.0",
"username":"1020201",
"password":"1020201"
}
Code in the Config.js
onPrepare: function() {
var data = require('./testdata.json');
},
I don't see the testdata.json file when giving path in require though its available at the location.
I wish to access JSON data using data.name, data.version etc.
Following is my folder structure:
You should make sure your json file is located in the current directory & and in the same folder where your config file resides as you are giving this path require('./testdata.json'); -
There are many ways of setting your data variables and accessing them globally in your test scripts -
1st method: Preferred method is to use node's global object -
onPrepare: function() {
global.data = require('./testdata.json');
},
Now you could access data anywhere in your scripts.
2nd Method Is to use protractor's param object -
exports.config = {
params: {
data: require('./testdata.json');
}
};
you can then access it in the specs/test scripts using browser.params.data
In my code I am calling a query from my lambda function
let featured_json_data = JSON.parse(fs.readFileSync('data/jsons/featured.json'))
This works locally because my featured.json is in the directory that I am reading from. However when I deploy with serverless, the zip it generates doesn't have those files, I get a
ENOENT: no such file directory, open...
I tried packaging by adding
package:
include:
- data/jsons/featured.json
but it just doesn't work. The only way I get this to work is manually adding the json file and then change my complied handler.js code to read from the json file in the root directory.
In this screenshot I have to add the jsons then manually upload it again and in the compiled handler.js code change the directory to exclude the data/jsons
I want to actually handle this in my servereless.yml
You can load JSON files using require().
const featured_json_data = require('./featured.json')
Or better yet, convert your JSON into JS!
For working with non-JSON files, I found that process.cwd() works for me in most cases. For example:
const fs = require('fs');
const path = require('path');
export default async (event, context, callback) => {
try {
console.log('cwd path', process.cwd());
const html = fs.readFileSync(
path.resolve(process.cwd(), './html/index.html'),
'utf-8'
);
const response = {
statusCode: 200,
headers: {
'Content-Type': 'text/html'
},
body: html
};
callback(null, response);
} catch (err) {
console.log(err);
}
};
I recommend looking at copy-webpack-plugin: https://github.com/webpack-contrib/copy-webpack-plugin
You can use it to package other files to include with your Lambda deployment.
In my project, I had a bunch of files in a /templates directory. In webpack.config.js to package up these templates, for me it looks like:
const CopyWebpackPlugin = require('copy-webpack-plugin');
module.exports = {
plugins: [
new CopyWebpackPlugin([
'./templates/*'
])
]
};
fs.readFileSync cannot find file when deploying with lambda
Check the current directory and check target directory content in deploy environment. Add appropriate code for that checking to your program/script.
In my Vue project, I have mocked some data for next step development. I already save the test data in a json file. And my vue project is typical one created with Vue-Cli, and the structure for my project goes as following:
My_project
build
config
data
service_general_info.json
node_modules
src
components
component-A
component-A.vue
as you can see, all the folders are created by the vue-cli originally. And I make a new folder data and place the test data json file inside.
And I want to read in the data by axios library in an event handling function inside the component of component-A as following:
methods: {
addData() {
console.log('add json data...');
axios.get('./../../data/service_general_info.json');
},
},
I use relative path to locate the target file.But get 404 error back. So how to set the path correctly? Currently I am running the dev mode in local host.
The error message is: GET http://localhost:8080/data/service_general_info.json 404 (Not Found)
In Vue-cli project, axios can't get data from custom folder.
You should use static folder to save test json file.
So you should change axios call like this:
axios.get('/static/service_general_info.json');
This will get data from json.
If you are doing just for sake of testing then you can save it in public folder and access it directly on http root.
e.g. I have the file results.json in public folder then I can access it using http://localhost:8080/results.json
For me it didn't work using static folder. I had to put it in public folder.
I put json folder in public & then accessed it like below.
getCountries() {
return axios.get('json/country-by-abbreviation.json', { baseURL: window.location.origin })
.then((response) => { return response.data; })
.catch((error) => {
throw error.response.data;
});
}
When the http call is made from the server, axios has no idea that you're on http://localhost:8080, you have to give the full url.
Like this:
methods: {
addData() {
console.log('add json data...');
axios.get('http://localhost:8080/data/service_general_info.json');
},
},
I had this same issue, only the above solutions wouldn't work as it is being uploaded to a subdirectory. I found I needed to put it in the public/assets folder and use:
axios.get(process.env.BASE_URL+'assets/file.json')
While in vue.config.js I have set the local and live paths
module.exports = {
publicPath: process.env.NODE_ENV === 'production'
? '/path/to/app/'
: '/'
}
You can simply read a static JSON file using import. Then assign in data.
import ServiceInfo from './../../data/service_general_info.json';
export default{
data(){
return {
ServiceInfo
}
}
}