Suppose we are getting lots of list data in json format, for every single day the api returns the same data.
Now if I apply the filter on the json then where can I store the API json result for the current day so that there is no need to call the API multiple times.
How can I store it in a txt file or in a database or maybe in cache?
It depends on your aims. You may use a text file or the DB field.
You may use a Redis as a cache.
Try to start with text file at first. Probably it will help you.
1) Draft usage of text (.json) file.
// $json = json_encode($array); // if you don't have json data
$filePath = sprintf('%s_cache.json', date('Y-m-d'));
file_put_contents($filename, $json);
2) Usage of JSON in MySQL
INSERT INTO table VALUES (JSON_OBJECT("key", "value")); // something like this
INSERT INTO table VALUES ('{"key": "value"}'); // or this one
More details about MySQL are here: https://dev.mysql.com/doc/refman/5.7/en/json.html
Related
I'm trying to load, filter and unload some json files using AWS Athena:
CREATE EXTERNAL TABLE IF NOT EXISTS
json_based_table(file_line string)
LOCATION 's3://<my_bucket>/<some_path/';
UNLOAD
(SELECT file_line from json_based_table limit 10)
TO 's3://<results_bucket>/samples/'
WITH (format = 'JSON');
Problem is the output is a set of files containing a json per line that has a single key "file_line" who's value is a json line from the original file as a string.
How do I UNLOAD such a table values only? (ignoring the column name I had to create to load the files)
It seems that by choosing
WITH (format = 'TEXTFILE');
I can get what I want.
Choosing JSON as a format is good for preserving the tabular structure of the table in a file and was a misleading name in this case.
I am currently in the process of moving my sensor reading data from my Azure Blob storage into a SQL database. I have multiple .csv files and in those files I have various columns that holds the date ( in the format: 25/4/2017), time, sensor_location and sensor_readings.
My question; If I want to store the data according to their respective columns using Logic App, what step should I take? and how do I push the second file data into the row after the first file data? Thanks
You will need to either write a script, (any high level language which has support or extensions for mysql will do, python, php, nodejs, etc) to import your data or you can use a mysql client like sequelpro https://www.sequelpro.com/ which imports csv files.
Here is a link as to how to insert data into mysql with php:
http://php.net/manual/en/pdo.prepared-statements.php
You can read the csv file with:
$contents = file_get_contents('filename.csv');
$lines = explode("\n", $contents);
foreach($lines as $line) { ...
// insert all rows to mysql here
I have json files, volume is approx 500 TB. I have loaded complete set into hive data warehouse.
How would I validate or test the data that was loaded into hive warehouse. What should be my testing strategy ?
Client want us to validate the json data. Whether the data loaded into hive is correct ot not. Is there any miss? If yes, which field it was?
Please help.
How is your data being stored in hive tables ?
One option is create a Hive UDF function that receive the JSON string and validate the data and return another string with the error message or an empty string if the JSON string is well formed.
Here is a Hve UDF tutorial: http://blog.matthewrathbone.com/2013/08/10/guide-to-writing-hive-udfs.html
With the Hive UDF function in place you can executequeries like:
select strjson, validateJson(strjson) from jsonTable where validateJson(strjson) != "";
I have a csv file that looks like this:
varCust_id,varCust_name,varCity,varStateProv,varCountry,varUserId,varUsername
When I run the HTTP Post Request to create a new customer, I get a JSON response. I am extracting the cust_id and cust_name using the json extractor. How can I enter this new value into the csv for the correct variable? For example, after creating the customer, the csv would look like this:
varCust_id,varCust_name,varCity,varStateProv,varCountry,varUserId,varUsername
1234,My Customer Name
Or once I create a user, the file might look like this:
varCust_id,varCust_name,varCity,varStateProv,varCountry,varUserId,varUsername
1234,My Customer Name,,,,9876,myusername
In my searching through the net, I have found ways and I'm able to append these extracted variables to a new line but in my case, I need to replace the value in the correct location so it is associated to the correct variable I have set up in the csv file.
I believe what you're looking to do can be done via a BeanShell PostProcessor and is answered here.
Thank you for the reply. I ended up using User Defined Variables for some things and BeanShell PreProcessors for other bits vs. using the CSV.
Well, never tried this. But what you can do is create all these variables and set them to Null / 0.
Once done, update these during your execution. At the end, you can concatenate these with any delimiter (say ; or Tab) and just push in CSV as a single string.
Once you got data in CSV, you can easily split in Ms excel.
im new to jmeter.
I'm in need to test the performance of some json requests which normally do some insertion or update in db .
I need to change some record id dynamically . For example
{"columnName":"company","newValue":"cts","oldValue":"","timeStamp":"11-05-2012 14:54:24","version":"1"}],"instruction":"contact_list","**recordId**":"8294547"}]}
i want record id should be dynamically get from csv file. so i did like
"recordid":"$recordid"}
But after HIT TO THE SERVER I found in the request THAT the one more time request id was printing after the json loop ends like
{"columnName":"company","newValue":"cts","oldValue":"","timeStamp":"11-05-2012 14:54:24","version":"1"}],"instruction":"contact_list","recordId":"**8294547**"}]}
8294547
which returns to malformed json request.
can u please tell the way to avoid extra appending recordid after the json loop .
above json is not the actual one just i copied the part i need .
You're missing curly braces. The format for inserting a variable in JMeter is ${myVar}, not $myVar.