I use CSV Data Set Config in JMeter to provide username/password data to testsuite. In some cases it reads %3CEOF%3E from file instead of data. File is located in /bin folder.
Structure of file:
username1,password1
username2,password2
There isn't any empty lines at the end of the file.
Recycle on EOF: True
Stop Thread on EOF: False
Although you should not be seeing this issue normally you can work it around by putting your request under the If Controller and setting the following condition using __groovy() function:
${__groovy(!vars.get('foo').equals('<EOF>'),)}
Replace foo with the variable reference name from the CSV Data Set Config.
I faced the exact same problem and ran into a solution by accident. Although the txt had no empty lines when opened with "notepad" when I opened the same txt file using "notepad++" I found an empty line. I removed it from notepad++ and saved and the problem was solved.
Related
I am trying to do some optimization in ADF. Setup is a third-party tool copies one JSON file per object to a BLOB storage container. These feed to a Mapping Data Flow. The individual files written by the third party tool work great. If I copy these files to a different BLOB folder using an Azure Copy Data activity, the MDF can no longer parse the files and gives an error: "JSON parsing error, unsupported encoding or multiline." I started this with a Merge Files, but outcome is same regardless of copy behavior I choose.
2ND EDIT: After another day's work, I have found that the Copy Activity Merge File from JSON to JSON definitely adds an EOL character to each single JSON object as it gets imported to the Merge file. I have also found that the MDF fails definitely with those EOL characters in the Merge file. If I remove all EOL characters from the Merge file, the same MDF will work. For me, this is a bug. The copy activity is adding a character that breaks the MDF. There seems to be a second issue in some of my data that doesn't fail as an individual file but does when concatenated that breaks the MDF when I try to pull all the files together, but I have tested the basic behavior on 1-5000 files and been able to repeat the fail/success tests.
I took the original file, and the copied file, ran them through all of sorts of test, what I eventually found when I dump into Notepad++:
Copied file:
{"CustomerMasterData":{"Customer":[{"ID":"123456","name":"Customer Name",}]}}\r\n
Original file:
{"CustomerMasterData":{"Customer":[{"ID":"123456","name":"Customer Name",}]}}\n
If I change the copied file from ending with \r\n to \n, the MDF can read the file again. What is going on here? And how do I change the file write behavior or the MDF settings so that I can concatenate or copy files without the CRLF?
EDIT: NEW INFORMATION -- It seems on further review like maybe the minification/whitespace removal is the culprit. If I download the file created by the ADF copy and format it using a JSON formatter, it works. Maybe the CRLF -> LF masked something else. I'm not sure what to do at this point, but its super frustrating.
Other possibly relevant information:
Both the source and sink JSON datasets are set to use UTF-8 (not default(UTF-8), although I tried that). Would a different encoding fix this?
I have tried remapping schemas, creating new data sets, creating new Mapping Data Flows, still get the same error.
EDITED for clarity based on comments:
In the case of a single JSON element in a file, I can get this to work -- data preview returns same success or failure as pipeline when run
In the case of multiple documents merged by ADF I get the below instead. It seems on further review like maybe the minification/whitespace removal is the culprit. If I download the file created by the ADF copy and format it using a JSON formatter, it works. Maybe the CRLF -> LF masked something else. I'm not sure what to do at this point, but its super frustrating.
Repro: Create any valid JSON as a single file, put it in blob storage, use it as a source in a mapping data flow, to do any sink operation. Create a second file with same schema, get them both to run in same flow using wildcard paths. Use a Copy Activity with Merge Files as the Sink Copy Activity and Array of Objects as the File pattern. Try to make your MDF use this new file. If it fails, download the file created by ADF, run it through a formatter (I have used both VS Code -> "Format Document" from standard VS Code JSON extension, and VS 2019 "Unminify" command) and reupload... It should work now.
don't know if you already solved the problem: I came across the exact same problem 3 days ago and after several tries I found a solution:
in the copy data activity under sink settings, use "set of objects" (instead of "array of objects") under File Pattern, so that the merged big JSON has the value of the original small JSON files written per line
in the MDF after setting up the wildcard paths with the *.json pattern, under JSON Settings select: Document per line as the Document form.
After that you should be good to go, as least it solved my problem. The automatic written CRLF in "array of objects" setting in the copy data activity should be a default setting and MSFT should provide the option to omit it in the settings in the future.
According to my test:
1.copy data activity can't change unix(LF) to windows(CRLF).
2.MDF can also parse unix(LF) file and windows(CRLF) file.
Maybe there is something else wrong.
By the way,I see there is a comma after "name":"Customer Name" in your Original file,I delete it before my test.
So I'm trying to program something using Node.js, I've got a file that is called 'profile.json', that is an object. When something happens, I need to update a value of 'name' to a new name. So I do
'profile.name = name2;'
but after I restart my program everything comes back and I have to change it again. So my problem is how would I save the json after updating it?
It is not saving because you are reading the file and updating it in the application. However, you are not changing anything in the file. Once you read the file and parse the JSON, no link exists to the original file. The JSON exists only in memory. You will want to use the NodeJS File System class to write the file. https://nodejs.org/api/fs.html First, check if the file exists, if it does delete it (or move/rename). Second, save the file using the fs.writeFile method.
I have CSV DataSet Config where I have location of my .csv file
Tried both by writing the absolute path and putting my file in the same directory. Below is the screenshot
CSV file gets stored but the next line says it must exist
Below is mine HTTP Request
HTTP REquest
By running this in command Line nothing is getting stored in the my output file
I have tried that too.
Check following things:
Name of the file is correct. In case of Unix file names are case sensitive so check the name of the file and name you have mentioned in CSV data set configuration.
Keep the CSV file in the same folder where your JMX resides, and just mention file name in CSV dataset config. This avoids confusion in future even if you move the script to any other box.
Also make sure in the file there is no new line character at the end of the file. That will result in error.
Try a fully qualified path. ~/... was not working for me either - /temp/file.csv did the trick
I am using multiple csv files in one thread for comparision purpose.
Here first CSV Data set Config returns the file names
1.csv
5.csv
1000.csv
now I want to pass above file name to second CSV Data set Config
C:\\Softwares\\Installed\\jmeter-3.0\\bin\\TestData\\files\\${filename}
Is it possible in jmeter? can any one help me to resolve the problem.
Thanks,
Vijay
CSV Data Set Config is getting initialized before any JMeter Variable therefore your ${filename} will never be resolved and you will be getting "File not found" errors.
The options are in:
Consider using __CSVRead() function instead of CSV Data Set Config
Switch to JMeter Properties instead of JMeter Variables.
Change ${filename} to ${__P(filename)} (see __P() function documentation for syntax)
Define filename property. It can be done in 2 ways:
Via user.properties file. Add the next line to the file:
filename=C:\\Softwares\\Installed\\jmeter-3.0\\bin\\TestData\\files\\1.csv
JMeter restart will be required to pick up the property
Via -J command-line argument:
jmeter -Jfilename=C:\Softwares\Installed\jmeter-3.0\bin\TestData\files\1.csv
See Apache JMeter Properties Customization Guide for more information on JMeter properties and ways of setting, getting and overriding them
In my JMeter test, I have a Loop Controller nested inside a few other modules with contains a CSV Data Set Config. I also have a CSV Data Set Config in the top of my Thread Group that reads from a CSV to get a file location. I was to use this file location in the nested CSV Data Set Confing to grab the CSV from that location and loop through that one. An error is being thrown because all the CSV files are loaded at once, in the beginning of the test. Is there a way to delay the loading of a CSV so that I can ensure the file path variable has already been set?
You will not be able to use CSV Data Set Config the way you describe. All the config elements get loaded before the test begins.
You can use Beanshell Pre/Post Processor/Sampler.
I had a requirement to load csv file in CSV Data set config of my JMeter script- Name of the csv file can be anything. I run my script with ANT. So i get the name from ANT and pass it to JMeter via a property. CSV data set config uses the property to load the CSV. Thought of sharing this as It might help you as I am not sure of your exact requirement.
EDIT:
You can have a look # __CSVRead(), __StringFromFile() functions.
http://jmeter.apache.org/usermanual/functions.html