In jmeter using http requests i'm posting some json bundles and from the responses i'm using jsr223 post processor to extract data and store it inside csv files, each entry in each line. now for 10 post requests i'm getting duplicate data into the csv file. Is there a way to read back csv files and remove duplicate lines using jmeter. The number of lines in csv files can be almost 200,000.
eg:csv file be like
csvFile1.csv:
line1
line2
duplicateline
...........so on
You can read the file into an ArrayList as
new File('/path/to/file').readLines()
You can remove the duplicate entries using unique() function as
def lines = file.readLines().unique()
You can write the unique lines back using Writer
Putting everything together:
def file = new File('/path/to/file')
def lines = file.readLines().unique()
file.withWriter { writer ->
lines.each {line ->
writer.writeLine(line)
}
}
Demo:
Just in case: The Groovy Templates Cheat Sheet for JMeter
Related
I have JSON data in a CVS file that I need to break apart into seperate JSON files. The data looks like this: {"EventMode":"","CalculateTax":"Y",.... There are multiple rows of this and I want each row to be a separate JSON file. I have used code provided by Jatin Grover that parses the CVS into JSON:
lcount = 0
out = json.dumps(row)
jsonoutput = open( 'json_file_path/parsedJSONfile'+str(lcount)+'.json', 'w')
jsonoutput.write(out)
lcount+=1
This does an excellent job the problem is it adds "R": " before the {"EventMode... and adds extra \ between each element as well as item at the end.
Each row of the CVS file is already valid JSON objects. I just need to break each row into a separate file with the .json extension.
I hope that makes sense. I am very new to this all.
It's not clear from your picture what your CSV actually looks like.
I mocked up a really small CSV with JSON lines that looks like this:
Request
"{""id"":""1"", ""name"":""alice""}"
"{""id"":""2"", ""name"":""bob""}"
(all the double-quotes are for escaping the quotes that are part of the JSON)
When I run this little script:
import csv
with open('input.csv', newline='') as input_file:
reader = csv.reader(input_file)
next(reader) # discard/skip the fist line ("header")
for i, row in enumerate(reader):
with open(f'json_file_path/parsedJSONfile{i}.json', 'w') as output_file:
output_file.write(row[0])
I get two files, json_file_path/parsedJSONfile0.json and json_file_path/parsedJSONfile1.json, that look like this:
{"id":"1", "name":"Alice"}
and
{"id":"2", "name":"bob"}
Note that I'm not using json.dumps(...), that only makes sense if you are starting with data inside Python and want to save it as JSON. Your file just has text that is complete JSON, so basically copy-paste each line as-is to a new file.
I need to create test data preparation script and capture JSON response data to CSV file.
In the actual test, I need to read parameters from CSV file.
Is there any possibilities of saving entire JSON data as filed in CSV file (or) need to extract each filed and save it to CSV file?
The main issue JSON have comma, You can overcome it by saving JSON to file and use different delimiter instead of comma separated, for example #
Then read file using CSV Data Set Config using # Delimiter
Delimiter to be used to split the records in the file. If there are fewer values on the line than there are variables the remaining variables are not updated - so they will retain their previous value (if any).
Also you can save JSON in every row and then get data using different delimiter as #
You can save entire JSON response into a JMeter Variable by adding a Regular Expression Extractor as a child of the HTTP Request sampler which returns JSON and configuring it like:
Name of created variables: anything meaningful, i.e. response
Regular Expression: (?s)(^.*)
Template: $1$
Then you need to declare this response as a Sample Variable by adding the next line to user.properties file:
sample_variables=response
And finally you can use Flexible File Writer plugin to store the response variable into a file, if you don't have any other Sample Variables you should use variable#0
I've a csv like this:
NAME;F1;F2;
test1;field1;field2
test2;field1;field2
test3;field1;field2
I would test only test1, so I would change the csv in
ID;F1;F2;
test1;field1;field2
#test2;field1;field2
#test3;field1;field2
how can I skip rows of test2 and test3 in jmeter?
There is always a way to do to something..
maybe my way is not the best and "pretty" but it worth!
Thread Group
Loop Controller
csv Data Set Config
if Controller
Http Request
Inside If Controller I added this code:
${__groovy(vars.get('ID').take(1)!='#')}
In this way when you put an # at the start of the row it will be skipped.
I hope it could be helpfull for someone.
You cannot, the only option I can think of is creating a new CSV file out of the existing one with just first 2 lines like:
Add setUp Thread Group to your Test Plan
Add JSR223 Sampler to the setUp Thread Group
Put the following code into "Script" area
new File('original.csv').readLines().take(2).each {line ->
new File('new.csv') << line << System.getProperty('line.separator')
}
Replace original.csv with path to the current CSV file and set up CSV Data Set Config to use new.csv
The above code will write first 2 lines from the original.csv into the new.csv so you will be able to access limited external data instead of the full CSV file.
More information:
File.readLines()
Collection.take()
The Groovy Templates Cheat Sheet for JMeter
I'm looking for help with some JavaScript to insert inside of a code step in Zapier. I have two inputs that are named/look like the following:
RIDS: 991,992,993
LineIDs: 1,2,3
Each of these should match in the quantity of items in the list. There can be 1, 2 or 100 of them. The order is significant.
What I'm looking for is a code step to model the data into one CSV matching up the positions of each. So using the above data, my output would look like this:
991,1
992,2
993,3
Does anyone have code or easily know how to achieve this? I am not a JavaScript developer.
Zapier doesn't allow you to create files in a code step. You can, though, use the code step to generate text which can then be used in another step. I used Python for my example (I'm not as familiar with Javascript but the strategy is the same).
Create CSV file in Zapier from Raw Data
Code Step with LindeIDs and RIDs as inputs
import csv
import io
# Convert inputs into lists
lids = input_data['LineIDs'].split(',')
rids = input_data['RIDs'].split(',')
# Create file-like CSV object
csvfile = io.StringIO()
filewriter = csv.writer(csvfile, delimiter=',', quotechar='|', quoting=csv.QUOTE_MINIMAL)
# Write CSV rows
filewriter.writerow(['LineID', 'RID'])
for x in range(len(lids)):
filewriter.writerow([lids[x], rids[x]])
# Get CSV object value as text and set to output
output = {'text': csvfile.getvalue()}
Use a Google Drive step to Create File from Text
File Content = Text from Step 1
Convert to Document = no
This will create a *.txt document
Use a CloudConvert step to Convert File from txt to csv.
Is it possible for each thread select the same row from the CSV file?
eg. I have 5 users and only 5 records (rows) in my CSV file. In each iteration, the 1st value from CSV should be assigned to User1, similarly for all users.
User1: myID1,pass1,item1,product1
User2: myID2,pass2,item2,product2
User3: myID3,pass3,item3,product3
User4: myID14,pass4,item4,product4
User5: myID15,pass5,item5,product5
.
.
Any solution, please?
If you have only 5 threads and 5 lines in CSV I would suggest considering switching to User Parameters instead of working with CSV.
If your CSV file can have > 5 lines and your test can have > 5 virtual users and requirement like "user 1 takes line 1" is a must, you will have to pre-load the CSV file into memory with a scripting test element like Beanshell Sampler like:
Add setUp Thread Group to your Test Plan (with 1 thread and 1 iteration)
Add Beanshell Sampler and put the following code into "Script" area:
import org.apache.commons.io.FileUtils;
List lines = FileUtils.readLines(new File("test.csv"));
bsh.shared.lines = lines;
The above code will read the contents of test.csv file (replace it with relative or full path to your CSV file) and store it into bsh.shared namespace
Add Beanshell PreProcessor as a child of the request where you need to use the values from the CSV file and put the following code into "Script" area:
int user = ctx.getThreadNum();
String line = bsh.shared.lines.get(user);
String[] tokens = line.split(",");
vars.put("ID", tokens[0]);
vars.put("pass", tokens[1]);
vars.put("item", tokens[2]);
vars.put("product", tokens[3]);
The above code will fetch the line from the list, stored in the bsh.shared namespace basing on current virtual user number, split it by comma and store the values into the JMeter Variables so you will be able to access them as:
${ID}
${pass}
${item}
${product}
See How to Use BeanShell: JMeter's Favorite Built-in Component guide for more information on using Beanshell scripting in JMeter tests.