Convert CSV to JSON client side with React DropZone - json

From React dropzone, i receive a File object with a File.preview property whose value is a blob:url. i.e. File {preview: "blob:http://localhost:8080/52b6bad4-58f4-4ths-a2f5-4ee258ba864a"
Is there a way to convert this to json on the client? The file isnt need to be stored in a database (the convert JSON will be). I've attempted to use csvtojson but it's unable to use the file system as it uses node to power it. Ideally would like to convert this in the client if possible, once user has uploaded it. Any suggestions welcomed.
<Dropzone
name={field.name}
onDrop={(acceptedFiles, rejectedFiles) => {
acceptedFiles.forEach(file => {
console.log(file)
let tempFile = file.preview
csv()
.fromSteam(tempFile) // this errors with fs.exists not a function as its not running serverside
.on('end_parsed',(jsonArrObj)=>{
console.log(jsonArrObj)
})
})
}}
>

Yes, its possible with FileReader and csv:
import csv from 'csv';
// ...
const onDrop = onDrop = (e) => {
const reader = new FileReader();
reader.onload = () => {
csv.parse(reader.result, (err, data) => {
console.log(data);
});
};
reader.readAsBinaryString(e[0]);
}
// ...
<Dropzone name={field.name} onDrop={onDrop} />
FileReader API: https://developer.mozilla.org/en/docs/Web/API/FileReader
csv package: https://www.npmjs.com/package/csv

Related

How to save imported JSON file with Expo Filesystem

I have been working on a React Native project with Expo that uses a json file to store local data. I am importing the data like so
import data from '../database.json'
I am making changes (adding and removing) to the imported JSON by using data.push(new_data). These changes are not persistent when I close the app because I cannot figure out how to save them. I have looked at using the expo-file-system library as so:
import * as FileSystem from 'expo-file-system';
...
FileSystem.writeAsStringAsync(FileSystem.documentDirectory + 'database.json', data);
This is from looking at examples in the API documentations. This however always throws promise rejections and doesn't end up writing the file. Can you point me in the right direction?
Also, should I import the database.json in a different way so I will already have the uri to save it to?
The documentation doesn't give an example of it's returned props in promises, so I was overlooking it for longer than I care to admit 😅. I was really dedicated to figuring this out so I could use the Expo solution, and totally missed the return Promise for createFileAsync, so hopefully this saves someone a significant amount of time in the future.
import * as FileSystem from 'expo-file-system';
const { StorageAccessFramework } = FileSystem;
const saveFile = async () => {
const permissions = await StorageAccessFramework.requestDirectoryPermissionsAsync();
// Check if permission granted
if (permissions.granted) {
// Get the directory uri that was approved
let directoryUri = permissions.directoryUri;
let data = "Hello World";
// Create file and pass it's SAF URI
await StorageAccessFramework.createFileAsync(directoryUri, "filename", "application/json").then(async(fileUri) => {
// Save data to newly created file
await FileSystem.writeAsStringAsync(fileUri, data, { encoding: FileSystem.EncodingType.UTF8 });
})
.catch((e) => {
console.log(e);
});
} else {
alert("You must allow permission to save.")
}
}
Use AsyncStorage instead. The react native package is deprecated but working, or use #react-native-community/async-storage and convert json to string (AsyncStorage can only store strings)
Set item
import AsyncStorage from '#react-native-community/async-storage';
...
await AsyncStorage.setItem('myData', JSON.stringify(data))
Get item
const data = await AsyncStorage.getItem('myData')
I found #JayMax answer very helpful however it's only for Android.
On iOS all you need to do is use Sharing.shareAsync and then you can save data to the file. Check this example:
const fileUri = FileSystem.documentDirectory + 'data.txt';
FileSystem.writeAsStringAsync(fileUri, 'here goes your data from JSON. You can stringify it :)', {
encoding: FileSystem.EncodingType.UTF8,
});
const UTI = 'public.text';
Sharing.shareAsync(fileUri, {UTI}).catch((error) => {
console.log(error);
});
If you using AsyncStorage, it only store for small data. Maybe 6mb or 10 mb.
You can use expo fileSystem
import * as FileSystem from 'expo-file-system';
...
FileSystem.writeAsStringAsync(FileSystem.documentDirectory + 'database.json', data);
Convert your data (Type json to string) Such as this:
writeData = async () => {
var persons = ''
await axios.get(`http://192.168.0.48:4000/api/sql/student`)
.then(res => {
persons = res.data
})
await FileSystem.writeAsStringAsync(FileSystem.documentDirectory + `offline_queue_stored.json`, JSON.stringify(persons));
}
#1.If the JSON File is in your Project Folder (PC/Laptop)
import data from './database.json';
#2. If the JSON File is in your Phone
import * as FileSystem from 'expo-file-system';
import * as DocumentPicker from 'expo-document-picker';
this.state = {
fileURI: null,
};
componentDidMount = () =>{
this._pickDocument();
}
_pickDocument = async () => {
let result = await DocumentPicker.getDocumentAsync({});
this.setState({
fileURI: result.uri
})
let fileData = await FileSystem.readAsStringAsync(this.state.fileURI)
console.log(fileData)
};

Batching with useQuery react hooks getting back undefined

I am currently working on a project which requires me to make multiple queries/mutations. I tried setting up my apollo client with BatchHttpLink and I can see the data I am requesting in the network tab in the browser. It is coming back at an array of objects instead of JSON.
But the issue is when I try to grab the data in my component data is undefined. I tried using HttpLink instead of BatchHttpLink and I can get the data back from the hook.
My suspicion is the shape of the object that comes back from the response is different, I tried looking into documentation but I can't find much about batching.
Currently using "#apollo/client#^3.0.2"
Here's my client set up.
import { ApolloClient, InMemoryCache, ApolloLink, from } from '#apollo/client'
import { BatchHttpLink } from '#apollo/client/link/batch-http'
import { onError } from '#apollo/client/link/error'
const BASE_URL = 'http://localhost:4000'
const httpLink = new BatchHttpLink({
uri: BASE_URL,
credentials: 'include',
})
const csrfMiddleware = new ApolloLink((operation, forward) => {
operation.setContext(({ headers = {} }) => ({
headers: {
...headers,
'X-CSRF-Token': getCSRFToken(),
},
}))
return forward(operation)
})
const errorMiddleware = onError(({ networkError }) => {
if (networkError && 'statusCode' in networkError && networkError.statusCode === 401) {
window.location.assign('/accounts/login')
}
})
const client = new ApolloClient({
link: from([errorMiddleware, csrfMiddleware, httpLink]),
cache: new InMemoryCache(),
})
This is the react hook I'm trying to console log.
const {data} = useQuery(GET_USER_PERMISSIONS_AND_PREFERENCES)
Figured it out. You need to add another middleware to return the data that the useQuery hook can recognize. The data that comes back in the batch call is an array of objects shaped
{
payload: {
data: { ... }
}
}
So something like this did the trick for me
const batchParseMiddleware = new ApolloLink((operation, forward) => {
return forward(operation).map((data: any) => data.payload)
})
I have been having a similar issue, and have so far only been able to solve it by breaking batching and converting to a normal HttpLink

csv from URL to json file in node.js

"New to programming"
I have a CSV file at
http://vhost11.lnu.se:20090/assig2/data1.csv
I am trying to convert it to a local json file. My code below.
I am getting {"X":"153","Y":"21","time":"21438"}} value in my data1.json.
const request=require('request')
const csv=require('csvtojson')
const fs = require('fs')
csv()
.fromStream(request.get('http://vhost11.lnu.se:20090/assig2/data1.csv'))
.on("json",function(jsonObj){ //single json object will be emitted for each csv line
console.log(jsonObj);
fs.writeFile("./data1.json", JSON.stringify(jsonObj), (err) => {
if (err) {
console.error(err);
return;
};
});
});
Where did I go wrong?
The callback function in the on event is called for each line. You'll want to initialize an empty list in the outer most scope and push jsonObj to it from the callback in on. You can then write your list to a file when the input file is done being read by handling the done event.

Understanding the streaming concept in node.js

I am trying to convert a CSV data to JSON data while streaming from a HTTP url by using "csvtojson" package.
const csv = require("csvtojson");
const request = require('request');
let options = {
uri: '',
****
};
let tempArr = [];
csv()
.fromStream(request(options))
.on("json", (jsonObj) => {
if (JSON.parse(jsonObj.Response).intents[0].intent == "None")
tempArr.push(JSON.parse(jsonObj.Response));
})
.on('done', (error) => {
callback(null, tempArr)
})
This is calling under an API. When I starts the server and call this api to convert csv to json, it works perfectly.
And If I call the same API again, the "json" event is not getting triggered, Instead "done" event is triggered directly.
i.e., the streaming is not done from the second time. why is it behaving like this?
and what should I do to solve this problem?

CSV to JSON using NodeJS

I'd like to convert a CSV file to a JSON object using NodeJS. The problem is that my CSV file is hosted on a special URL.
URL : My CSV here
var fs = require("fs");
var Converter = require("csvtojson").Converter;
var fileStream = fs.createReadStream("myurl");
var converter = new Converter({constructResult:true});
converter.on("end_parsed", function (jsonObj) {
console.log(jsonObj);
});
fileStream.pipe(converter);
Issue :
Error: ENOENT, open 'C:\xampp\htdocs\GestionDettes\http:\www.banque-france.fr\fileadmin\user_upload\banque_de_france\Economie_et_Statistiques\Changes_et_Taux\page3_quot.csv'
at Error (native)
Edit #1 :
Request.get(myurl, function (error, Response, body) {
var converter = new Converter({
constructResult: true,
delimiter: ';'
});
converter.fromString(body,function(err, taux){
console.log(taux); // it works
});
});
I did just that in a module reading and writing on different protocol in different data formats. I used request to get http resources.
If you want take a look at alinex-datastore. With this module it should work like:
const DataStore = require('#alinex/datastore').default;
async function transform() {
const ds = new DataStore('http://any-server.org/my.csv');
await ds.load();
await ds.save('file:/etc/my-app.json');
}
That should do it.