I have inserted a file into MySQL database as a byte array:
preparedStatment.setBytes(10, inpStream.toString().getBytes());
I had already gotten the file from the database as a byte array but I'm stuck at getting the actual file's contents. What would be the correct way of doing this?
This is wrong:
preparedStatment.setBytes(10, inpStream.toString().getBytes());
You are storing the result of the InputStream#toString() as bytes in the DB. The InputStream#toString() does not return the file's contents as you seem to think, instead it returns the default classname#hashcode representation inherited from Object#toString().
You need PreparedStatement#setBinaryStream() instead:
preparedStatement.setBinaryStream(10, inpStream);
Then you can retrieve it by ResultSet#getBinaryStream():
InputStream inpStream = resultSet.getBinaryStream("columnname");
Or if you really need PreparedStatement#setBytes(), then you would need to write the InputStream to a ByteArrayOutputStream the usual way first and then get the bytes by its toByteArray() method. But this is not memory efficient per se.
Related
I have a dictionary file with 200,000 items in it.
I have a Dictionary Model which matches the SQLite db and the proper methods.
If I try to parse the whole file, it seems to hang. If I do 8000 items, it seems to do it quite quickly. Is there a size limit, or is just because there might be some corrupted data somewhere? This json was exported from the sqlite db as json pretty, so I would imagine it was done correctly. It also works fine with the first 8000 items.
String peuJson = await getPeuJson();
List<Dictionary> dicts = (json.decode(peuJson) as List)
.map((i) => Dictionary.fromJson(i))
.toList();
JSON is similar to other data formats like XML - if you need to transmit more data, you just send more data. There's no inherent size limitation to the JSON request. Any limitation would be set by the server parsing the request.
I am writing a Post Man test, using javascript, to verify that a certain part of the api response is correct. The whole api response comes back as json. But, one part of this json response is base64 encrypted, 128 bit encrypted, and also compressed. So, I store this one part of the json response in a variable. Then, I base 64 decode that and store it into a second variable. Finally, I 128 bit decrypt and store this into a third variable.
My issue now is that I want to decompress the data in this third variable. To be clear, this is now a subset of the original api response that came back from the server. It has been base64 decoded and 128 bit decrypted. Now, I need to decompress it. All of the research I have found deals with unzipping a .zip file. I cannot use that because I do not have this information stored in a file. I have this information stored in a variable.
So, has anyone been able to decompress data that is stored in a variable instead of being stored in a .zip, .jar, or other file type?
=======================================
So that it's also more tangile for you to understand, here is a close example of what I am trying:
var jsonData = pm.response.json();
//rawResponse is a subset of the json response, and the part we need to test later, but here its still encrypted and compressed
var rawResponse = jsonData.field[2].data;
//base 64 decode
var base64Decoded = atob(rawResponse);
//our key for the 128 bit decryption
var key = 1234ourmadeupkeyhere; //actual key works when used, but obviously not listed here
//Do the 128 bit decrypting
var decryptedData = CryptoJS.AES.decrypt(base64Decoded, key);
//In this variable decompressedData, this is the data we need to decompress
var decompressedData = howDoIDecompressThisHere(decryptedData );
//Assertion
pm.expect(decompressedData).to.be.equal("blahblahblah something readable and no longer compressed");```
In Tableau Public's documentation one can read that
A single JSON object cannot exceed 128 MB.
When a single object top-level array exceeds 128 MB, you must convert
it to a file where the JSON objects are defined one per line.
When trying to load a (95 MB!) JSON file into Tableau, i get an error:
Tableau encountered a single object larger than 128 MB in the JSON
file. Please try again with a smaller JSON file.
My file is one large array in the following format:
[
{"id":"1389406","updatedDate":"2018-01-31T10:17:31Z","createdDate":"2018-01-31T10:17:31Z","deleted":false,"Ids":["2466958"],"location":{"code":"bibd","name":"Main Library"},"status":{"code":"-","display":"AVAILABLE"},"barcode":"blah blah","callNumber":"TS","itemType":"In-house loan"},
{"id":"1389406","updatedDate":"2018-01-31T10:17:31Z","createdDate":"2018-01-31T10:17:31Z","deleted":false,"Ids":["2466958"],"location":{"code":"bibd","name":"Main Library"},"status":{"code":"-","display":"AVAILABLE"},"barcode":"blah blah","callNumber":"TS","itemType":"In-house loan"}
...
]
What could i do to make Tableau accept this file?
Well, you basically have to do it as the help text stipulates. One JSON object per line. Nothing more. I tried just that, but I forgot to remove the comma (,) i had at the end of every line.
Your file should look like this:
{"id":"1389406","updatedDate":"2018-01-31T10:17:31Z","createdDate":"2018-01-31T10:17:31Z","deleted":false,"Ids":["2466958"],"location":{"code":"bibd","name":"Main Library"},"status":{"code":"-","display":"AVAILABLE"},"barcode":"blah blah","callNumber":"TS","itemType":"In-house loan"}
{"id":"1389406","updatedDate":"2018-01-31T10:17:31Z","createdDate":"2018-01-31T10:17:31Z","deleted":false,"Ids":["2466958"],"location":{"code":"bibd","name":"Main Library"},"status":{"code":"-","display":"AVAILABLE"},"barcode":"blah blah","callNumber":"TS","itemType":"In-house loan"}
I have a tricky data set to parse through and have not been able to formulate a processing method for it and several failed ideas have just made me more confused...
Sample data in CSV format - before processing
C:\User1\Videos\videos
10
C:\User1\Videos\videos
22
C:\User2\Videos\videos
8
C:\User2\Videos\videos
67
C:\User3\Videos\videos
18
C:\User3\Videos\videos
12
C:\User4\Videos\videos
90
I'm trying to combine the lengths of the video files in each user's video directory and output a list of each user and the total runtime of all their files.
Result - after processing
C:\User1\Videos\videos
32
C:\User2\Videos\videos
75
C:\User3\Videos\videos
30
C:\User4\Videos\videos
90
I'm looking for pseudocode or any advice really as to how I can achieve this result. I have been unsuccessful in trying to use nested loops and am having a hard time conceptualizing other solutions. I am writing this in VBScript for convenience with file processing in Windows.
Thanks so much for the help in advance, I appreciate any advice you can offer.
First, this is a line delimited format with two lines per record.
1: directory
2: video length
Second, you need only a single loop to read each line of the file and process the data.
Steps
Dim a dic var. Set dic = CreateObject("Scripting.Dictionary").
Dim vars for filepath, userkey, and length value
Loop and read lines from file
Inside the loop, read line 1 and identify user for the record. VBScript has the ability to split the string. Based on the example, if the idea is to aggregate all lengths under User1 no matter what the remaining subfolders are then split string and grab the first path element and use this as the user key. You can check that the second element is Videos to filter, etc or use more elements as the key or as expressed in your results example use the full string as the key for exact matching. Store the user key in a local variable.
Inside the loop, read second line and parse length from line 2, store in local variable length.
Inside the loop, check if key exists in the dictionary then if so get value for key and add to length yielding the sum, add key,sum to dictionary. "dic.Item(userkey) = sum" Else if it does not exist then just add key,value "dic.Item(userkey) = value"
Repeat from step 4 until end of file
List items from dictionary by getting keys and then printing each key and the key's value from the dictionary.
The value stored in the dictionary could be an Object to store more information. Don't forget error handling.
I have a very large CSV file (8000+ items) of URLs that I'm reading with a CSV Data Set Config element. It is populating the path of an HTTP Request sampler and iterating through with a while controller.
This is fine except what I want is have each user (thread) to pick a random URL from the CSV URL list. What I don't want is each thread using CSV items sequentially.
I was able to achieve this with a Random Order Controller with multiple HTTP Request samplers , however 8000+ HTTP Samplers really bogged down jmeter to an unusable state. So this is why I put the HTTP Sampler URLs in the CSV file. It doesn't appear that I can use the Random Order Controller with the CSV file data however. So how can I achieve random CSV data item selection per thread?
There is another way to achieve this:
create a separate thread group
depending on what you want to achieve:
add a (random) loop count -> this will set a start offset for the thread group that does the work
add a loop count or forever and a timer and let it loop while the other thread group is running. This thread group will read a 'pseudo' random line
It's not really random, the file is still read sequentially, but your work thread makes jumps in the file. It worked for me ;-)
There's no random selection function when reading csv data. The reason is you would need to read the whole file into memory first to do this and that's a bad idea with a load test tool (any load test tool).
Other commercial tools solve this problem by automatically re-processing the data. In JMeter you can achieve the same manually by simply sorting the data using an arbitrary field. If you sort by, say Surname, then the result is effectively random distribution.
Note. If you ensure the default All Threads is set for the CSV Data Set Config then the data will be unique in the scope of the JMeter process.
The new Random CSV Data Set Config from BlazeMeter plugin should perfectly fit your needs.
As other answers have stated, the reason you're not able to select a line at random is because you would have to read the whole file into memory which is inefficient.
Rather than trying to get JMeter to handle this on the fly, why not just randomise the file order itself before you start the test?
A scripting language such as perl makes short work of this:
cat unrandom.csv | perl -MList::Util=shuffle -e 'print shuffle<STDIN>' > random.csv
For my case:
single column
small dataset
Non-changing CSV
I just discard using CSV and refer to https://stackoverflow.com/a/22042337/6463291 and use a Bean Preprocessor instead, something like this:
String[] query = new String[]{"csv_element1", "csv_element2", "csv_element3"};
Random random = new Random();
int i = random.nextInt(query.length);
vars.put("randomOption",query[i]);
Performance seems ok, if you got the same issue can try this out.
I am not sure if this will work, but I will anyways suggest it.
Why not divide your URLs in 100 different CSV files. Then in each thread you generate the random number and use that number to identify CSV file to read using __CSVRead function.
CSVRead">http://jmeter.apache.org/usermanual/functions.html#_CSVRead
Now the only part I am not sure if the __CSVRead function reopens the file every time or shares the same file handle across the threads.
You may want to try it. Please share your findings.
A much straight forward solution.
In CSV file, add another column (say B)
apply =RAND() function in the first cell of column B (say B1). This will create random float number.
Drag the cell (say B1) corner to apply for all the corresponding URLs
Sort column B.
your URL will be sorted randomly.
Delete column B.