Writing JSON data to a JSON file - json

Hello in my application I am currently trying to create my own custom log files in .json format. Reason for this is because I want a well structured and accurate log file which can be easily read and would not depend on some special code in my application to read the data.
I have been able to create a json file: activities.json
I have been able to write and append to that file using File::append($path, $json)
This is a sample of the file contents:
{"controller":"TestController","function":"testFunction()","model":"Test","user":"Kayla","description":"Something happened!! Saving some JSON data here!","date":"2016-06-15"}
{"controller":"TestController","function":"testFunction()","model":"Test","user":"Jason","description":"Something happened!! Saving some JSON data here!","date":"2016-06-15"}
{"controller":"UserController","function":"userFunction()","model":"User","user":"Jason","description":"Another event occurred","date":"2016-06-15"}
Now my issue is the above is not a valid JSON. How do I get it in this format:
[
{"controller":"TestController","function":"testFunction()","model":"Test","user":"Kayla","description":"Something happened!! Saving some JSON data here!","date":"2016-06-15"},
{"controller":"TestController","function":"testFunction()","model":"Test","user":"Jason","description":"Something happened!! Saving some JSON data here!","date":"2016-06-15"},
{"controller":"UserController","function":"userFunction()","model":"User","user":"Jason","description":"Another event occurred","date":"2016-06-15"}
]
Is there a way of writing and appending to a json file in laravel? As much as possible I want to avoid reading the entire file before the append and doing a search and replace since the file may contain hundreds to thousands of records.
I will not use the default Laravel Log function.

Related

How to move data from response of Http (JSON data) to a JSON file

I am trying to build an Azure Data Factory pipeline where it copies the Response of Output (JSON format) from Azure Function and move its data to a JSON file.
I am not using REST API.
Initially, I built a pipeline where it used Variable to copy the data into a Text file.
Now, I am trying to move data into a JSON file so that I could map JSON file's columns to SQL table.
What is best architecture of this data pipeline?
Do I need a Text file from moving data of "output.Response" (from 1st step) to JSON file?
I tried to use Text file as 'Source' and JSON file as 'Sink' (inside "Copy data" step), but I am not able to map 'Source' and 'Sink'.
I also tried having the third step with JSON files as 'Source' and 'Sink', but I am not sure how #variables('myVariable') should be carried into this step as it is not a text file.
Thank you.
I was able to find the solution by having the data type of JSON on next "Copy data" pointing to the text file (csv) that I used. Even though the file was originally created with DelimitedText (csv) file, when I pointed that same file, but using JSON format, it took it.

How to save response data in to CSV in Jmeter

I am trying to save Jmeter response in CSV file. I have tried some of the answers from StackOver Flow but those are not working.
Can anyone suggest me how can I achieve this?
Note: I have tried this and this.
Most probably it is something not feasible when it comes to CSV files as response data can have delimiter characters which will break your CSV file structure. For example if you use comma as delimiter and your response data contains commas - it will be not possible to save the response into a single CSV "cell".
If you need to save the response data I would recommend using XML format of the .jtl results file for this. You can amend JMeter's default result file configuration as follows:
Add next lines to user.properties file:
jmeter.save.saveservice.output_format=xml
jmeter.save.saveservice.response_data=true
That's it, next time you run JMeter in command-line non-GUI mode as:
jmeter -n -t test.jmx -l result.xml
the result.xml file will contain the response data which can be examined using either View Results Tree listener or XML viewer/editor of your choice.
More information: How to Save Response Data in JMeter
If the response of your request can be stored in csv then you can use one of the following
'Save Responses to a file' in Listeners.
Capture the response using regular expression extractor and write the value to a file in Beanshell code.

Loading JSON data with CSV Data Set Config

I'm new to Jmeter so I hope this question is not too off the wall. I am trying to test an HTTP endpoint that accepts a large JSON payload and processes it. I have collected a few hundred JSON blobs in a file and want to use those as my input for testing. The only way that I have come across for loading the data is using the CSV config. I have a single line of the file for each request. I have attempted to use \n as a delimiter and have also tried adding a tab character \t to the end of each line. My requests all show in put of<EOF>.
Is there a way to read a file of JSON objects, line at a time, and pass them to my endpoint as the body in a POST?
You need to provide more information, to wit: example JSON file (first 2 lines), your CSV Data Set Configuration, jmeter.log file, etc. so we could help.
For the time being I can state that:
Given CSV file looking like:
{"foo":"bar"}
{"baz":"qux"}
And pretty much default CSV Data Set Config setup
JMeter normally reads the CSV data
Be aware that there are alternatives to the CSV Data Set Config, for example:
__CSVRead() function. The equivalent syntax would be ${__CSVRead(test.csv,0)}
__StringFromFile() function. The equivalent syntax would be ${__StringFromFile(test.csv,,,)}
See Apache JMeter Functions - An Introduction to get familiarized with the JMeter Functions concept.

Convert JSON to CSV in nifi

I want to convert JSON files to CSV in nifi. We can achieve this in Python and other programming languages and have multiple articles on it. I have multiple JSON files and each file has different schema(one specific file will have one schema only). I can see there are templates to convert CSV to JSON and other conversions. But I didn't see any template to convert JSON data to CSV. I have gone through the article https://community.hortonworks.com/articles/64069/converting-a-large-json-file-into-csv.html ,however here we are hard coding the schema. As I have multiple files and each file has different schema, I can't hardcode the schema. Any suggestions please.
Conversion between formats is typically done through ConvertRecord by plugging in the appropriate record reader and record writer, in this case a JSON reader and CSV writer.
To make use of the record processors you need to defined Avro schemas for your data and put them in a schema registry, NiFi provides a local one.
There are lots of examples and posts out there about the record stuff, this slide deck shows an example of CSV to JSON, but would be easy to reverse the situation for your scenario:
https://www.slideshare.net/BryanBende/apache-nifi-record-processing
This post has some other info:
https://bryanbende.com/development/2017/06/20/apache-nifi-records-and-schema-registries

Camus - Writing to multiple file types

I am quite new to using LinkedIn's camus and have successfully written data files from Kafka to Hdfs.
In general, I use JsonStringMessagdecoder to read a JSON and write the same to .dat file using StringRecordWriterProvider.
But is it possible to write to multiple file types?
Suppose the json in kafka is as follows :
{ "user":"John", "message":"Hi how are you?" }
Now I want John,Hi how are you? to be written to one file, while user,message to be written to another file(.meta) in the same location. Is it possible ?