CSV file read issue in firebase functions - json

I am trying to read a csv file in a firebase function so that I can process the file and do the rest operations using the data.
import * as csv from "csvtojson";
const csvFilePath = "<gdrive shared link>"
try{
console.log("First Method...")
csv()
.fromFile(csvFilePath)
.then((jsonObj: any)=>{
console.log("jsonObj....",JSON.stringify(jsonObj));
})
console.log("Second Method...")
const jsonArray=await csv().fromFile(csvFilePath);
console.log("jsonArray...", JSON.stringify(jsonArray))
}
catch(e){
console.log("error",JSON.stringify(e))
}
The above mentioned are the 2 methods I have tried for reading the csv but both shows the firebase error
'Error: File does not exist. Check to make sure the file path to your csv is correct.'
In case of 'csvFilePath' I have tried 2 methods
Just added the csv file in same folder of the function and added the code like
const csvFilePath = "./student.csv"
Added the same file to google drive and changed the access permissions to anyone with the link can read and edit and given the path to same
const csvFilePath = "<gdrive shared link>"
Both shows the same error. In case of google drive I don't want to use any sort of google credential because I was intented to read a simple csv file in firebase function.

I will start by proposing that you convert your csv to json locally or without the function and see if it works. This is because I see you are using ES6 imports which might be causing an issue since all the documentation uses require. You can also try CSV Parse or some solutions provided in this question as an alternative, trying them without the function to check if it actually works and discard it. Actually, you can upload the JSON once you have converted it from the csv, but that depends on what you are trying to do.
I think the best way to achieve this, is following the approach given in this question, that first uploads the file into cloud storage and using onFinalize() to trigger the conversion.
Also, will address these three questions that went through similar issues with the path. They were able to fix it by adding __dirname. Each one has some extra useful information.
Context for "relative paths" seems to change to the calling module if a module is imported
The csvtojson converter ignores my file name and just puts undefined
How to avoid the error which throws a csvtojson

Related

How to benefit from tree-shaking and code-splitting while loading JSON in Nuxt?

I have a nuxt-app, where I have a lot of posts. I recently refactored the project and I won't generate all posts anymore, as it is just taking too much time to do that.
Instead I have a page where I would fetch the matching post content via url query:
www.mypage.com/posts/?post=my-post-slug
Because the content is lying in static json files, for example in:
/static/data/posts/my-post-slug.json
/static/data/posts/my-post-slug_2.json
/static/data/posts/my-post-slug_3.json
/static/data/posts/my-post-slug_n.json
I read the post https://github.com/nuxt/nuxt.js/issues/123
about how to load json in the best way.
I decided to do something like this in the fetch() hook:
// ... simplified code
async fetch() {
let postSlug = this.$route.query.post
const post = this.$axios
.get(`/posts/posts.de.${postSlug}.json`)
.then((data) => {
return data?.data
})
.catch((error) => {
console.error('error: ', error)
const code = parseInt(error.response && error.response.status)
if (code === 404) {
this.$nuxt.error({ statusCode: 404, message: 'Post not found' })
}
})
this.activePost = post?.items?.[0] || false
}
As already said, I do not generate the actual posts, but I generate all post urls in my sitemap.xml.
When running the generate in analyze mode I have now a huuuuge bundle size (app.js), and I can't understand why...
-> Check out the attached image. (Note: app.js has a ridiculous size of 34MB!!!!😕)
I don't understand why all my post jsons appear in the static and the dist part of the bundle???
I don't understand at all why they appear in there. I want them to just lie in the static folder, but not be included in the app bundle.
(you can see that there are files like events.bundle.de.json included. I need those to fetch a list of all posts for example. I do that also within my fetch hook only.
I would be very happy if somebody could point out why those files are included (twice)!
Those files are not included "twice". You need them, so you do have them locally in your static folder.
Meanwhile, you should probably put them inside of your src directory if you don't want/need to have them exposed publicly and benefit from code-splitting thanks to Webpack as explained in the post you've linked (which is still valid even if it's a 2017 one!).
Here, since you're making an axios call and using target: 'static', it will bundle your whole thing to work even without JS and it does that ahead of time. So, in order to have all the possibilities, it includes all in the final bundle I think.
If you want to only load what is needed while not shipping a big static directory, you should import them dynamically. You can use a dynamic import: load only the needed JSON by passing the actual postSlug.
PS: style-wise, prefer using async/await (.then is deprecated) and maybe also $axios.$get.
Although I think #kissu s answer is answering my question in the title, it was not the solution for my problem. For the sake of completeness I will post what I found out after long and many hours of debugging. I still don't quite understand why this even happened, but maybe someone could comment on that as well:
In my nuxt-project I am using a utility file getData.js of which I import a function getDataServer into one of my vuex store modules.
// vuex store module: store/data.js
import { getPreviewData } from '~/api/getData'
The code looks like this:
// getData.js
// Get static JSON file (e.g. basic.de.json or posts.de.1.json)
export function getDataServer(fileProps) {
return require(`~/${fileProps.path}${fileProps.name}.${fileProps.lang}${
fileProps.page ? `.${fileProps.page}` : ''
}.json`)
}
Only by importing and not even by executing that function webpack would bundle EVERY .json file it can find in my root folder into my app.js.
That is why I had a dist folder appearing in my bundle, if not deleted. (The point I talk about in my original question, where I have things included twice).
I even created additional folders and .json files to see, and they were all bundled no matter what.
Only after removing the getData.js from my project my bundle became clean.
I understand that with the require command, webpack cannot tree-shake things, so I would have expected that some code-splitting features would not work, but what I did not expect was that this bit of code would automatically get every .json in my folder...
Does anyone know why importing that function would execute it in a way that acts as a wildcard for all .jsons?
To me it still does not make any sense.
Thanks and cheers.

How to generate a JSON file using JMeter Report Generator

I am trying to create a statistics.json file with JMeter using ReportGenerator, populated with the results of my .jmx tests. Is it possible to do this with JMeter?
I have gone through this tutorial: https://jmeter.apache.org/usermanual/generating-dashboard.html which focuses on creating an html dashboard using the Report Generator, but I have a project requirement of creating/updating a statstics.json file as well. I have already pulled the necessary data using a JSON Extractor post processor, and I can get the custom variables from that extractor to show up in my debug response, and in my CSV file (after adding some sample_variables to user.properties). Unfortunately I have been unsuccessful in finding more info about how to create a JSON file with these responses.
In my reportgenerator.properties file, the only parts I see that relate to json are:
jmeter.reportgenerator.exporter.json.classname=org.apache.jmeter.report.dashboard.JsonExporter
jmeter.reportgenerator.exporter.json.property.output_dir=report-output
I'm looking for some settings that would allow me to edit what goes into that JSON file, but I'm having trouble finding information in the docs. Do I need to be sending or setting my custom variables in another settings file? Any help clarifying this would be much appreciated!
Looking at JMeter source code you cannot efficiently control what's being exported into statistics.json file externally, you will have to either amend the JsonExporter class code or come up with your own implementation of the AbstractDataExporter and choose what, where and how to store.
private void createStatistic(Map<String, SamplingStatistic> statistics, MapResultData resultData) {
LOGGER.debug("Creating statistics for result data:{}", resultData);
SamplingStatistic statistic = new SamplingStatistic();
ListResultData listResultData = (ListResultData) resultData.getResult("data");
statistic.setTransaction((String) ((ValueResultData)listResultData.get(0)).getValue());
statistic.setSampleCount((Long) ((ValueResultData)listResultData.get(1)).getValue());
statistic.setErrorCount((Long) ((ValueResultData)listResultData.get(2)).getValue());
statistic.setErrorPct(((Double) ((ValueResultData)listResultData.get(3)).getValue()).floatValue());
statistic.setMeanResTime((Double) ((ValueResultData)listResultData.get(4)).getValue());
statistic.setMinResTime((Long) ((ValueResultData)listResultData.get(5)).getValue());
statistic.setMaxResTime((Long) ((ValueResultData)listResultData.get(6)).getValue());
statistic.setMedianResTime((Double) ((ValueResultData)listResultData.get(7)).getValue());
statistic.setPct1ResTime((Double) ((ValueResultData)listResultData.get(8)).getValue());
statistic.setPct2ResTime((Double) ((ValueResultData)listResultData.get(9)).getValue());
statistic.setPct3ResTime((Double) ((ValueResultData)listResultData.get(10)).getValue());
statistic.setThroughput((Double) ((ValueResultData)listResultData.get(11)).getValue());
statistic.setReceivedKBytesPerSec((Double) ((ValueResultData)listResultData.get(12)).getValue());
statistic.setSentKBytesPerSec((Double) ((ValueResultData)listResultData.get(13)).getValue());
statistics.put(statistic.getTransaction(), statistic);
}
An easier option would be writing your sample variables into a separate file using Flexible File Writer
I'm leaving the accepted answer because it is correct. However, I'd like to add that I was able to complete my requirement by using a JSR223 post processor to write a groovy script that creates a csv file wherever I need, and fill it with any data that I needed.

Cannot download file from Google Storage inside Cloud Function?

Im trying to perform a simple download of a .docx file info a buffer so I can handle it latter inside my Cloud Function. I've been using the whole Google Platform for multiple projects but never faced the need to download in server side, and now I need to, I just cant.
The following piece of code is not working, it just sends timeout as a response (I don't even get an error If I try to catch it or something):
var bucket = admin.storage().bucket("gs://myBucket.com");
return bucket.file("001Lineales/4x3-1/1000.docx").download().then((contents)=>{
var buffer = contents[0];
//I never get into this point
}).catch((error)=>{
//No error
})
I tried in a local NodeJs script and worked as expected. Also tried to perform a readStream() download but no luck, the function gets hang up in any try of downloading the file.
return new Promise((resolve,reject)=>{
var archivo = bucket.file(selectedCategory).createReadStream();
var array = [];
//Under here, never happens
archivo.on('data', (d) => {array.push(d)}).on("end",()=>{
var newbuff = Buffer.concat(array);
resolve(newbuff)
})
})
The file permissions read/write are public. And the main problem is that debugging is difficult cause Im not able to perform this function in local emulator.
What can I do? Thanks in advance.
EDIT:
Double checking a local call with emulator, I get the following error:
Anonymous caller does not have storage.objects.get access to the Google Cloud Storage object.
Double check the service account hat you've assigned to the Cloud Function and that you've given it the permission it needs.
I think Storage Object Viewer will give you what you need to read a file into the buffer.
By default, if you haven't changed it, the AppEngine's default service account gets used, which I don't think has access to Storage.

Trouble with getting script to recognize a JSON file in the directory (Google API)

So I am attempting to learn how to use the Google Sheets API with Node.js. In order to get an understanding, I followed along with the node.js quick start guide supplied by Google. I attempted to run it, nearly line for line a copy of the guide, just without documentation. I wind up encountering this: cmd console output that definitely didn't work.
Just in case anyone wants to see if I am not matching the guide, which is entirely possible since I am fairly new to this, here is a link to the Google page and my code.
https://developers.google.com/sheets/api/quickstart/nodejs
var fs = require('fs');
var readline = require('readline');
var google = require('googleapis');
var googleAuth = require('google-auth-library');
var SCOPES = ['https://www.googleapis.com/auth/spreadsheets.readonly'];
var TOKEN_DIR = (process.env.HOME || process.env.HOMEPATH ||
process.env.USERPROFILE) + '/.credentials/';
var TOKEN_PATH = TOKEN_DIR + 'sheets.googleapis.com-nodejs-quickstart.json';
fs.readFile('client_secret.json', function processClientSecrets(err, content) {
if (err) {
console.log('Error loading client secret file: ' + err);
}
authorize(JSON.parse(content), listMajors);
});
I have tried placing the JSON file in each and every part of the directory, but it still won't see it. I've been pulling hairs all day, and a poke in the right direction would be immensely appreciated.
From your command output:
Error loading client secret file
So your if (err) line is being triggered. But since you don't throw the error, the script continues anyway (which is dangerous in general).
SyntaxError: Unexpected token u in JSON at position 0
This means that the data you are passing to JSON.parse() is undefined. It is not a valid JSON string.
You could use load-json-file (or the thing it uses, parse-json) to get more helpful error messages. But it's caused by the fact that your content variable has nothing since the client_secret.json you tried to read could not be found.
As for why the file could not be found, there could be a typo in either the script or the filename you saved the JSON in. Or it may have to do with the current working directory. You may want to use something like this to ensure you end up with the same path regardless of the current working directory.
path.join(__dirname, 'client_secret.json')
Resources
path.join()
__dirname

Meteor: reading simple JSON file

I am trying to read a JSON file with Meteor. I've seen various answers on stackoverflow but cannot seem to get them to work. I have tried this one which basically says:
Create a file called private/test.json with the following contents:
[{"id":1,"text":"foo"},{"id":2,"text":"bar"}]
Read the file contents when the server starts (server/start.js):
Meteor.startup(function() {
console.log(JSON.parse(Assets.getText('test.json')));
});
However this seemingly very simple example does not log anything to the console. If I trye to store it in a variable instead on console.logging it and then displaying it client side I get
Uncaught ReferenceError: myjson is not defined
where myjson was the variable I stored it in. I have tried reading the JSON client side
Template.hello.events({
'click input': function () {
myjson = JSON.parse(Assets.getText("myfile.json"));
console.log("myjson")
});
}
Which results in:
Uncaught ReferenceError: Assets is not defined
If have tried all of the options described here: Importing a JSON file in Meteor with more or less the same outcome.
Hope someone can help me out
As per the docs, Assets.getText is only available on the server as it's designed to read data in the private directory, to which clients should not have access (thus the name).
If you want to deliver this information to the client, you have two options:
Use Assets.getText exactly as you have done, but inside a method on the server, and call this method from the client to return the results. This seems like the best option to me as you're rationing access to your data via the method, rather than making it completely public.
Put it in the public folder instead and use something like jQuery.getJSON() to read it. This isn't something I've ever done, so I can't provide any further advice, but it looks pretty straightforward.
The server method is OK, just remove the extra semi-colon(;). You need a little more in the client call. The JSON data comes from the callback.
Use this in your click event:
if (typeof console !== 'undefined'){
console.log("You're calling readit");
Meteor.call('readit',function(err,response){
console.log(response);
});
}
Meteor!