I am trying to convert a CSV data to JSON data while streaming from a HTTP url by using "csvtojson" package.
const csv = require("csvtojson");
const request = require('request');
let options = {
uri: '',
****
};
let tempArr = [];
csv()
.fromStream(request(options))
.on("json", (jsonObj) => {
if (JSON.parse(jsonObj.Response).intents[0].intent == "None")
tempArr.push(JSON.parse(jsonObj.Response));
})
.on('done', (error) => {
callback(null, tempArr)
})
This is calling under an API. When I starts the server and call this api to convert csv to json, it works perfectly.
And If I call the same API again, the "json" event is not getting triggered, Instead "done" event is triggered directly.
i.e., the streaming is not done from the second time. why is it behaving like this?
and what should I do to solve this problem?
Related
I have been working on a React Native project with Expo that uses a json file to store local data. I am importing the data like so
import data from '../database.json'
I am making changes (adding and removing) to the imported JSON by using data.push(new_data). These changes are not persistent when I close the app because I cannot figure out how to save them. I have looked at using the expo-file-system library as so:
import * as FileSystem from 'expo-file-system';
...
FileSystem.writeAsStringAsync(FileSystem.documentDirectory + 'database.json', data);
This is from looking at examples in the API documentations. This however always throws promise rejections and doesn't end up writing the file. Can you point me in the right direction?
Also, should I import the database.json in a different way so I will already have the uri to save it to?
The documentation doesn't give an example of it's returned props in promises, so I was overlooking it for longer than I care to admit 😅. I was really dedicated to figuring this out so I could use the Expo solution, and totally missed the return Promise for createFileAsync, so hopefully this saves someone a significant amount of time in the future.
import * as FileSystem from 'expo-file-system';
const { StorageAccessFramework } = FileSystem;
const saveFile = async () => {
const permissions = await StorageAccessFramework.requestDirectoryPermissionsAsync();
// Check if permission granted
if (permissions.granted) {
// Get the directory uri that was approved
let directoryUri = permissions.directoryUri;
let data = "Hello World";
// Create file and pass it's SAF URI
await StorageAccessFramework.createFileAsync(directoryUri, "filename", "application/json").then(async(fileUri) => {
// Save data to newly created file
await FileSystem.writeAsStringAsync(fileUri, data, { encoding: FileSystem.EncodingType.UTF8 });
})
.catch((e) => {
console.log(e);
});
} else {
alert("You must allow permission to save.")
}
}
Use AsyncStorage instead. The react native package is deprecated but working, or use #react-native-community/async-storage and convert json to string (AsyncStorage can only store strings)
Set item
import AsyncStorage from '#react-native-community/async-storage';
...
await AsyncStorage.setItem('myData', JSON.stringify(data))
Get item
const data = await AsyncStorage.getItem('myData')
I found #JayMax answer very helpful however it's only for Android.
On iOS all you need to do is use Sharing.shareAsync and then you can save data to the file. Check this example:
const fileUri = FileSystem.documentDirectory + 'data.txt';
FileSystem.writeAsStringAsync(fileUri, 'here goes your data from JSON. You can stringify it :)', {
encoding: FileSystem.EncodingType.UTF8,
});
const UTI = 'public.text';
Sharing.shareAsync(fileUri, {UTI}).catch((error) => {
console.log(error);
});
If you using AsyncStorage, it only store for small data. Maybe 6mb or 10 mb.
You can use expo fileSystem
import * as FileSystem from 'expo-file-system';
...
FileSystem.writeAsStringAsync(FileSystem.documentDirectory + 'database.json', data);
Convert your data (Type json to string) Such as this:
writeData = async () => {
var persons = ''
await axios.get(`http://192.168.0.48:4000/api/sql/student`)
.then(res => {
persons = res.data
})
await FileSystem.writeAsStringAsync(FileSystem.documentDirectory + `offline_queue_stored.json`, JSON.stringify(persons));
}
#1.If the JSON File is in your Project Folder (PC/Laptop)
import data from './database.json';
#2. If the JSON File is in your Phone
import * as FileSystem from 'expo-file-system';
import * as DocumentPicker from 'expo-document-picker';
this.state = {
fileURI: null,
};
componentDidMount = () =>{
this._pickDocument();
}
_pickDocument = async () => {
let result = await DocumentPicker.getDocumentAsync({});
this.setState({
fileURI: result.uri
})
let fileData = await FileSystem.readAsStringAsync(this.state.fileURI)
console.log(fileData)
};
I am currently working on a project which requires me to make multiple queries/mutations. I tried setting up my apollo client with BatchHttpLink and I can see the data I am requesting in the network tab in the browser. It is coming back at an array of objects instead of JSON.
But the issue is when I try to grab the data in my component data is undefined. I tried using HttpLink instead of BatchHttpLink and I can get the data back from the hook.
My suspicion is the shape of the object that comes back from the response is different, I tried looking into documentation but I can't find much about batching.
Currently using "#apollo/client#^3.0.2"
Here's my client set up.
import { ApolloClient, InMemoryCache, ApolloLink, from } from '#apollo/client'
import { BatchHttpLink } from '#apollo/client/link/batch-http'
import { onError } from '#apollo/client/link/error'
const BASE_URL = 'http://localhost:4000'
const httpLink = new BatchHttpLink({
uri: BASE_URL,
credentials: 'include',
})
const csrfMiddleware = new ApolloLink((operation, forward) => {
operation.setContext(({ headers = {} }) => ({
headers: {
...headers,
'X-CSRF-Token': getCSRFToken(),
},
}))
return forward(operation)
})
const errorMiddleware = onError(({ networkError }) => {
if (networkError && 'statusCode' in networkError && networkError.statusCode === 401) {
window.location.assign('/accounts/login')
}
})
const client = new ApolloClient({
link: from([errorMiddleware, csrfMiddleware, httpLink]),
cache: new InMemoryCache(),
})
This is the react hook I'm trying to console log.
const {data} = useQuery(GET_USER_PERMISSIONS_AND_PREFERENCES)
Figured it out. You need to add another middleware to return the data that the useQuery hook can recognize. The data that comes back in the batch call is an array of objects shaped
{
payload: {
data: { ... }
}
}
So something like this did the trick for me
const batchParseMiddleware = new ApolloLink((operation, forward) => {
return forward(operation).map((data: any) => data.payload)
})
I have been having a similar issue, and have so far only been able to solve it by breaking batching and converting to a normal HttpLink
Trying my first angular exercise. Receiving undefined value on 1st time from http post, but 2nd time getting proper response (Edge, Firefox). Thanks!
LoginService (Calls Http post method and returns observable)
login(loginRequest: LoginRequest){
console.log("LoginService.login - userName " + loginRequest.username);
let options = new RequestOptions({ headers: headers });
return this.http.post(this.http_url, loginRequest, options).map( res =>
res.json());
LoginFormComponent (calls service class and convert JSON to typescript object)
onSubmit() {
this.loginSvc.login(this.loginRequest).subscribe(
data => this.loginResponseStr = data,
error => alert(error),
() => console.log('Request completed')
);
var loginResponse = new LoginResponse();
loginResponse.fillFromJSON(JSON.stringify(this.loginResponseStr));
console.log(loginResponse.status);
console.log(loginResponse.statusDesc);
if(loginResponse.status == "SUCCESS"){
this.router.navigate(['/home-page']);
}
Console log
LoginService.login - userName admin main.bundle.js:370:9
undefined main.bundle.js:266:9
undefined main.bundle.js:267:9
Request completed main.bundle.js:263:181
LoginService.login - userName admin main.bundle.js:370:9
SUCCESS main.bundle.js:266:9
VALID USER main.bundle.js:267:9
Request completed main.bundle.js:263:181
Angular server calls are asynchronous, that mean the code wont wait for the server to respond before executing the rest of the code. Such as PHP. So you would not see a blank page waiting for the server to send data. When you want to deal with the respose come from a server call you have to add all the code within the subscribe; that means if this information needed to be passed to another service.
Your code should look like this.
onSubmit() {
this.loginSvc.login(this.loginRequest).subscribe(
data => {
this.loginResponseStr = data
var loginResponse = new LoginResponse();
loginResponse.fillFromJSON(JSON.stringify(data));
console.log(loginResponse.status);
console.log(loginResponse.statusDesc);
if (loginResponse.status == "SUCCESS") {
this.router.navigate(['/home-page']);
}
},
error => alert(error),
() => console.log('Request completed')
);
}
From React dropzone, i receive a File object with a File.preview property whose value is a blob:url. i.e. File {preview: "blob:http://localhost:8080/52b6bad4-58f4-4ths-a2f5-4ee258ba864a"
Is there a way to convert this to json on the client? The file isnt need to be stored in a database (the convert JSON will be). I've attempted to use csvtojson but it's unable to use the file system as it uses node to power it. Ideally would like to convert this in the client if possible, once user has uploaded it. Any suggestions welcomed.
<Dropzone
name={field.name}
onDrop={(acceptedFiles, rejectedFiles) => {
acceptedFiles.forEach(file => {
console.log(file)
let tempFile = file.preview
csv()
.fromSteam(tempFile) // this errors with fs.exists not a function as its not running serverside
.on('end_parsed',(jsonArrObj)=>{
console.log(jsonArrObj)
})
})
}}
>
Yes, its possible with FileReader and csv:
import csv from 'csv';
// ...
const onDrop = onDrop = (e) => {
const reader = new FileReader();
reader.onload = () => {
csv.parse(reader.result, (err, data) => {
console.log(data);
});
};
reader.readAsBinaryString(e[0]);
}
// ...
<Dropzone name={field.name} onDrop={onDrop} />
FileReader API: https://developer.mozilla.org/en/docs/Web/API/FileReader
csv package: https://www.npmjs.com/package/csv
I'm running this little node express server, which is supposed to check if the voucher is valid later and then send an answer back to the client
this is my code
app.post('/voucher', function (request, response) {
response.setHeader('Access-Control-Allow-Origin', '*');
response.setHeader('Access-Control-Request-Method', '*');
response.setHeader('Access-Control-Allow-Methods', 'OPTIONS, GET');
response.setHeader('Access-Control-Allow-Headers', 'authorization, content-type');
if ( request.method === 'OPTIONS' ) {
response.writeHead(200);
response.end();
return;
}
console.log(request)
let results;
let body = [];
request.on('data', function(chunk) {
body.push(chunk);
}).on('end', function() {
results = Buffer.concat(body).toString();
// results = JSON.parse(results);
console.log('#### CHECKING VOUCHER ####', results)
let success = {success: true, voucher: {name: results,
xxx: 10}}
success = qs.escape(JSON.stringify(success))
response.end(success)
} )
}
);
It is obviously just an example and the actual check is not implemented yet. So far so good.
Now on the client side where I work with REACT, I can not seem to decode the string I just send there.
there I'm doing this
var voucherchecker = $.post('http://localhost:8080/voucher', code , function(res) {
console.log(res)
let x = JSON.parse(res)
console.log(x)
console.log(qs.unescape(x))
It gives me the error
Uncaught SyntaxError: Unexpected token % in JSON at position 0
When I do it the other way arround
let x = qs.unescape(res)
console.log(x)
console.log(JSON.parse(x))
Than it tells me
Uncaught TypeError: _querystring2.default.unescape is not a function
Maybe you can help me? I don't know what the issue is here. Thank you.
Also another question on this behalf, since I'm only a beginner. Is there smarter ways to do such things than I'm doing it now? I have react which renders on the client and I have a mini express server which interacts a few times with it during the payment process.
The both run on different ports.
What would be the standard way or best practice to do such things?
I'm a bit perplexed as to why your backend code has so much going on in the request.
Since you asked for if there is a different way to write this, I will share with you how I would write it.
Server
It seems that you want your requests to enable CORS, it also seems that you originally wanted to parse a JSON in your request body.
This is how I would recommend you re-write your endpoint
POST /voucher to take a request with body JSON
{
code: "xxxxx"
}
and respond with
{
success: true,
voucher: {
name: results,
xxx: 10
}
}
I would recommend you use express's middleware feature as you will probably use CORS and parse JSON in most your requests so in your project I would.
npm install body-parser
npm install cors
then in your app initialization
var bodyParser = require('body-parser')
var cors = require('cors')
var app = express()
// parse application/x-www-form-urlencoded
app.use(bodyParser.urlencoded({ extended: false }))
// parse application/json you can choose to just pars raw text as well
app.use(bodyParser.json())
// this will set Access-Control-Allow-Origin * similar for all response headers
app.use(cors())
You can read more about body-parser and cors in their respective repos, if you don't want to use them I would still recommend you use your own middleware in order to reduse future redundancy in your code.
So far this will substitute this part of your code
response.setHeader('Access-Control-Allow-Origin', '*');
response.setHeader('Access-Control-Request-Method', '*');
response.setHeader('Access-Control-Allow-Methods', 'OPTIONS, GET');
response.setHeader('Access-Control-Allow-Headers', 'authorization, content-type');
if ( request.method === 'OPTIONS' ) {
response.writeHead(200);
response.end();
return;
}
console.log(request)
let results;
let body = [];
request.on('data', function(chunk) {
body.push(chunk);
}).on('end', function() {
results = Buffer.concat(body).toString();
// results = JSON.parse(results);
Now your route definition can just be
app.post('/voucher', function (request, response) {
var result = request.body.code // added by body-parser
console.log('#### CHECKING VOUCHER ####', result)
// express 4+ is smart enough to send this as json
response.status(200).send({
success: true,
voucher: {
name: results,
xxx: 10
}
})
})
Client
your client side can then be, assuming $ is jquery's post function
var body = {
code: code
}
$.post('http://localhost:8080/voucher', body).then(function(res) {
console.log(res)
console.log(res.data)
return res.data
})