I am trying to convert a large xlsx sheet to csv using the example given here and here . But it is throwing the below error on line
XSSFWorkbook wBook = new XSSFWorkbook(new FileInputStream(file));
Error
Caught throwable Java heap space
java.lang.OutOfMemoryError: Java heap space
at java.io.ByteArrayOutputStream.<init>(ByteArrayOutputStream.java:77)
Is there any efficient way to convert large xlsx sheet to csv without increasing heap memory?
My code is as below-
try
{
// Get the workbook object for XLSX file
// XSSFWorkbook wBook = new XSSFWorkbook(new FileInputStream(file));
XSSFWorkbook wBook = new XSSFWorkbook(new FileInputStream(file));
for(int i=0;i<wBook.getNumberOfSheets();i++)
{
XSSFSheet sheet = wBook.getSheetAt(i);
fileName = convertedFileLocation + sheet.getSheetName() + ".csv";
FileOutputStream fos = new FileOutputStream(fileName);
//System.out.println(wBook.getSheetAt(i).getSheetName());
Row row;
Cell cell;
// Iterate through each rows from first sheet
Iterator<Row> rowIterator = sheet.iterator();
while (rowIterator.hasNext())
{
row = rowIterator.next();
// For each row, iterate through each columns
Iterator<Cell> cellIterator = row.cellIterator();
while (cellIterator.hasNext())
{
cell = cellIterator.next();
data.append("\"" + cell.getStringCellValue() + "\",\r\n");
}
}
fos.write(data.toString().getBytes());
fos.close();
}
wBook.close();
}
Edited---
I am using XSSF as suggested by #axel and #gagravarr in the comment and using the class method as below. Though it is creating the converted.csv file the csv file is empty. Any idea what I am doing wrong
private boolean convertToCSV2(File file)
{
try {
OPCPackage p = OPCPackage.open(file.getPath());
String convertedFileLocation = siteConfigService.getProperty(CONVERTEDFOLDER);
String convertedFileName = convertedFileLocation+"converted.csv";
PrintStream pout=new PrintStream(convertedFileName);
XLSX2CSV xlsToCSV = new XLSX2CSV(p, pout, -1);
} catch (InvalidFormatException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return true;
}
Related
I am faced with a problem with my java-csv-mysql gui application that i am working on.
i will breakdown the application in the following functions:
1. select a CSv using a JFileChooser,
2. reading the csv
3. importing the csv to Mysql table
4. displaying the csv contents once they are imported into the Table.
I have managed to get it to do the following functions.
1. select a csv file
2. read through the csv file...-reads only one row
3. display read records
I have problems when it come to the following
1. reading 'all' the records in the csv
2. uploading to the csv.
the Error I get is an ArrayIndexOutofBoundsException:3
which is due to the reading of the csv.
the csv has the following format:
2018/01/25,58,294616/0
2018/01/27,102,298970/0
the csv needs to do the following while it reads the csv
1. read the csv,
2. separate the last column which is to be seprated by a'/'.
this will result in there being 4 columns instead of 3.
here is the Code that I have so far.
public class Payment_import_v4 extends JFrame{
private JTable table;
/**
* #param args the command line arguments
*/
public static void main(String[] args) {
EventQueue.invokeLater(new Runnable(){
public void run()
{
createAndshowGUI();
}
});
}
private static void createAndshowGUI(){
Payment_import_v4 form = new Payment_import_v4();
form.setVisible(true);
}
public Payment_import_v4(){
//form frame
super("Payment Import");
setSize(900,600);
setLocation(500,280);
getContentPane().setLayout(null);
//Label Result
final JLabel lblResult = new JLabel("Result",JLabel.CENTER);
lblResult.setBounds(150,22,370,14);
getContentPane().add(lblResult);
//Table
table = new JTable();
getContentPane().add(table);
//Table Model
final DefaultTableModel model = (DefaultTableModel)table.getModel();
model.addColumn("PayDate");
model.addColumn("Ammount");
model.addColumn("LinkId");
model.addColumn("BranchNo");
//ScrollPane
JScrollPane scroll = new JScrollPane(table);
scroll.setBounds(84,98,506,79);
getContentPane().add(scroll);
//Button Open
JButton btnOpen = new JButton("Select File");
btnOpen.setBounds(268,47,135,23);
btnOpen.addActionListener(new ActionListener(){
public void actionPerformed(ActionEvent ae){
JFileChooser fileOpen = new JFileChooser();
FileFilter filter = new FileNameExtensionFilter("CSV file","csv");
fileOpen.addChoosableFileFilter(filter);
int ret = fileOpen.showDialog(null,"Choose file");
if(ret == JFileChooser.APPROVE_OPTION){
File file = fileOpen.getSelectedFile();//gets selectedFile.
try {
BufferedReader br = new BufferedReader(new FileReader(file));
int row = 0;
//if (br.readLine() != null) {line = br.readLine();
while ((br.readLine()) != null) {
String line = br.readLine();// br string variable
String[] rawRow = line.split(",");
String lastEntry = rawRow[rawRow.length - 1];//this contains the LinkId/branchNo
String[] properLastEntry = lastEntry.split("/");//this contains the LinkId/branchNo split into two columnms
String[] oneRow = new String[rawRow.length + 1];
System.arraycopy(rawRow, 0, oneRow, 0, rawRow.length - 1);
System.arraycopy(properLastEntry, 0, oneRow, oneRow.length - properLastEntry.length, properLastEntry.length);
model.addRow(new Object[0]);
model.setValueAt(rawRow[0], row, 0);
model.setValueAt(rawRow[1], row, 1);
model.setValueAt(rawRow[2], row, 2);
model.setValueAt(rawRow[3], row, 3);
row++;
}
br.close();
//}
} catch (IOException ex) {
ex.printStackTrace();
}
lblResult.setText(fileOpen.getSelectedFile().toString());
}
}
});
getContentPane().add(btnOpen);
//btn Save
JButton btnSave = new JButton("Save");
btnSave.addActionListener(new ActionListener(){
public void actionPerformed(ActionEvent ea){
SaveData();
}
});
btnSave.setBounds(292,228,89,23);
getContentPane().add(btnSave);
}
private void SaveData(){
Connection connect = null;
Statement stmt = null;
try{
//DriverManager Loader
Class.forName("com.mysql.jdbc.Driver");
//connection string url.. the port//schema name//username//password
//this is the test Server ;oginDetails
connect = DriverManager.getConnection("jdbc:mysql://localhost:3306/riskfin", "root", "riskfin");//-------------> this is for the localhost server
stmt = connect.createStatement();
for(int i = 0;i<table.getRowCount();i++)
{
String PayDate = table.getValueAt(i,0).toString();
String Ammount = table.getValueAt(i,1).toString();
String LinkID = table.getValueAt(i,2).toString();
String BranchNo = table.getValueAt(i,3).toString();
String sql = "Insert into temp_payment_import "
+"VALUES('"+LinkID+"','"
+Ammount+"','"
+PayDate+"','"
+BranchNo+"')";
stmt.execute(sql);
}
JOptionPane.showMessageDialog(null,"Data imported Successfully");
}catch(Exception ex){
JOptionPane.showMessageDialog(null,ex.getMessage());
ex.printStackTrace();
}
try{
if(stmt!= null){
stmt.close();
connect.close();
}
}catch(SQLException e){
System.out.println(e.getMessage());
e.printStackTrace();
}
}
}
here is the exception I get.
Exception in thread "AWT-EventQueue-0" java.lang.ArrayIndexOutOfBoundsException: 3
at payment_import_v4.Payment_import_v4$2.actionPerformed(Payment_import_v4.java:120)
at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2022)
at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2348)
at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:402)
at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:259)
at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:252)
at java.awt.Component.processMouseEvent(Component.java:6533)
at javax.swing.JComponent.processMouseEvent(JComponent.java:3324)
at java.awt.Component.processEvent(Component.java:6298)
at java.awt.Container.processEvent(Container.java:2236)
at java.awt.Component.dispatchEventImpl(Component.java:4889)
at java.awt.Container.dispatchEventImpl(Container.java:2294)
at java.awt.Component.dispatchEvent(Component.java:4711)
at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4888)
at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4525)
at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4466)
at java.awt.Container.dispatchEventImpl(Container.java:2280)
at java.awt.Window.dispatchEventImpl(Window.java:2746)
at java.awt.Component.dispatchEvent(Component.java:4711)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:758)
at java.awt.EventQueue.access$500(EventQueue.java:97)
at java.awt.EventQueue$3.run(EventQueue.java:709)
at java.awt.EventQueue$3.run(EventQueue.java:703)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:80)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:90)
at java.awt.EventQueue$4.run(EventQueue.java:731)
at java.awt.EventQueue$4.run(EventQueue.java:729)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:80)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:728)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:201)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)
You went to great lengths to write very accurate array manipulation code to deal with having two separators in your CSV data. But you never actually used the oneRow array. Change this:
model.addRow(new Object[0]);
model.setValueAt(rawRow[0], row, 0);
model.setValueAt(rawRow[1], row, 1);
model.setValueAt(rawRow[2], row, 2);
model.setValueAt(rawRow[3], row, 3); // ArrayIndexOutOfBoundsException
to this:
model.addRow(new Object[0]);
model.setValueAt(oneRow[0], row, 0);
model.setValueAt(oneRow[1], row, 1);
model.setValueAt(oneRow[2], row, 2);
model.setValueAt(oneRow[3], row, 3);
By definition, rawRow will only have 3 elements in it, because the final 2 terms will still appear as a single term (the term not yet having been split again on /).
I have to add result at the last column of each row. I have to test user successfully login with correct email and password the "PASS" is append to last else "FAIL" and go with the second row and check the result of each row.
public static void main(String[] args) throws Exception {
System.setProperty("webdriver.chrome.driver", "D:\\Automation\\Selenium Drivers\\chromedriver.exe");
WebDriver driver=new ChromeDriver();
driver.get("http://www.facebook.com");
// This will load csv file
CSVReader reader = null;
try{
reader = new CSVReader(new FileReader("C:\\Users\\src\\com\\elements\\demo.csv"));
}catch (Exception e) {
e.printStackTrace();
}
String[] cell;
while ((cell=reader.readNext())!=null){
for(int i=0;i<1;i++){
String emailid=cell[i];
String password=cell[i+1];
driver.findElement(By.id("email")).sendKeys(emailid);
driver.findElement(By.id("pass")).sendKeys(password);
driver.findElement(By.id("loginbutton")).click();
String outputFile = "C:\\Users\\src\\com\\elements\\demo.csv";
try {
// use FileWriter constructor that specifies open for appending
CsvWriter csvOutput = new CsvWriter(new FileWriter(outputFile, true),',');
if(driver.getTitle().equals("Log1 in to Facebook | Facebook"))
{
csvOutput.write("Pass"); //Your selenium result.
//csvOutput.endRecord();
//csvOutput.close();
}
else if (driver.getTitle().equals("Log in to Facebook | Facebook"))
{
csvOutput.write("userName");
csvOutput.write("password");
csvOutput.write("Fail"); //Your selenium result.
csvOutput.endRecord();
csvOutput.close();
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
Try this code.
String outputFile = "test.csv";
// before we open the file check to see if it already exists
boolean alreadyExists = new File(outputFile).exists();
try {
// use FileWriter constructor that specifies open for appending
CsvWriter csvOutput = new CsvWriter(new FileWriter(outputFile, true),',');
// if the file didn't already exist then we need to write out the header line
if (!alreadyExists){
csvOutput.write("result");
csvOutput.endRecord();
}
// else assume that the file already has the correct header line
// write out a few records
csvOutput.write("userName");
csvOutput.write("password");
csvOutput.write("Pass/Fail"); //Your selenium result.
csvOutput.endRecord();
csvOutput.close();
} catch (IOException e) {
e.printStackTrace();
}
OR
If we want to use writeNext() method which take string array as a parameter then
String csv = "D:\\test.csv";
CSVWriter writer = new CSVWriter(new FileWriter(csv));
List<String[]> data = new ArrayList<String[]>();
data.add(new String[] {"India", "New Delhi"});
data.add(new String[] {"United States", "Washington D.C"});
data.add(new String[] {"Germany", "Berlin"});
writer.writeAll(data);
writer.close();
Try other option.
FileWriter writer = new FileWriter("D:/test.csv",false);
writer.append(" ");
writer.append(',');
writer.append("UserName");
writer.append(',');
writer.append("Password");
writer.append(',');
writer.append("Pass/Fail");
writer.append('\n');
//generate whatever data you want
writer.flush();
writer.close();
I am trying to append a row at the end of my csv file using the code below
public class Register {
public static void add(int k,int m,int id1) throws Exception
{
ClassLoader classLoader = Register.class.getClassLoader();
try{
FileWriter fw = new FileWriter(new File(classLoader.getResource("data/dataset.csv").getFile()),true);
BufferedWriter bw = new BufferedWriter(fw);
bw.append("\n");
bw.append(String.valueOf(id1));
bw.append(',');
bw.append(String.valueOf(m));
bw.append(',');
bw.append(String.valueOf(k));
bw.close();
}catch(IOException ioe){
System.out.println("Exception occurred:");
ioe.printStackTrace();
}
}
}
I am calling this class from a servlet using a loop as I need to add 5 lines to my csv. Everything runs fine, but nothing gets added to the csv file. Please help.
You need to close the FileWriter object to flush the content into the file as shown below:
FileWriter fw = null;
BufferedWriter bw = null;
try{
fw = new FileWriter(new File(classLoader.
getResource("data/dataset.csv").getFile()),true);
bw = new BufferedWriter(fw);
bw.append("\n");
bw.append(String.valueOf(id1));
bw.append(',');
bw.append(String.valueOf(m));
bw.append(',');
bw.append(String.valueOf(k));
bw.close();
}catch(IOException ioe){
System.out.println("Exception occurred:");
ioe.printStackTrace();
} finally {
if(bw != null) {
bw.close();
}
if(fw != null) {
fw.close();
}
}
As a side note, ensure that you are closing the resources (like the writer objects above) inside the finally block (which you are not doing).
I stored data from the database to ArrayList.and I get that data from ArrayList to write on excel file for users download purpose. when I write to excel file, it will write perfectly.but when I write more data the last line called "total" come inside the other lines.coding as below.
public String excel_missedcall(List<datefilter> result1){
Date date = new Date(System.currentTimeMillis());
String download_file="";
Properties prop = new Properties();
InputStream input = null;
String link="";
String linkfilepath="";
//read file name from properties file
try {
String filename = "config.properties";
input = MainController.class.getClassLoader().getResourceAsStream(
filename);
if (input == null) {
System.out.println("Sorry, unable to find " + filename);
}
// load a properties file from class path, inside static method
prop.load(input);
String filepath=prop.getProperty("missedcall_filePath");
download_file=filepath+"missedcall_"+date.getTime()+".xlsx";
link=prop.getProperty("missedcall_downloadfilePath");
linkfilepath=link+"missedcall_"+date.getTime()+".xlsx";
int total=0;
//create excel sheet
//Blank workbook
XSSFWorkbook workbook = new XSSFWorkbook();
//Create a blank sheet
XSSFSheet sheet= workbook.createSheet("Missedcall");
//This data needs to be written (Object[])
Map<String, Object[]> data = new TreeMap<String, Object[]>();
data.put("1", new Object[] {"Date", "TotalCalls"});
for(int i=0;i<result1.size();i++){
data.put(""+(i+2),new Object[] {result1.get(i).getMisscall_date(),Integer.parseInt(result1.get(i).getSum_misscall())});
total+=Integer.parseInt( result1.get(i).getSum_misscall());
//System.out.println("**********1*********"+(i+2));
}
data.put(""+(result1.size()+2), new Object[] {"Total", total});
//Iterate over data and write to sheet
Set<String> keyset = data.keySet();
int rownum = 0;
for (String key : keyset)
{
System.out.println("*********1**********"+key);
Row row = sheet.createRow(rownum++);
System.out.println("*********2**********"+rownum);
Object [] objArr = data.get(key);
int cellnum = 0;
for (Object obj : objArr)
{
Cell cell = row.createCell(cellnum++);
if(obj instanceof String)
cell.setCellValue((String)obj);
else if(obj instanceof Integer)
cell.setCellValue((Integer)obj);
}
}
try
{
//Write the workbook in file system
FileOutputStream out1 = new FileOutputStream(new File(download_file));
workbook.write(out1);
out1.close();
System.out.println("xlsx written successfully on disk.");
}
catch (Exception e)
{
e.printStackTrace();
}
} catch (Exception ex) {
ex.printStackTrace();
}
return linkfilepath;
}
Output of excel file is:
Date TotalCalls
2015-08-28 1895
2015-08-29 599
2015-08-30 354
2015-08-31 2028
Total 6712
2015-08-20 0
2015-08-21 0
2015-08-22 2
2015-08-23 12
2015-08-24 22
2015-08-25 324
2015-08-26 878
2015-08-27 598
Please help me.
You are using String as key , replace following line
Map<String, Object[]> data = new TreeMap<String, Object[]>();
with
Map<Integer, Object[]> data = new TreeMap<Integer, Object[]>();
I'm working on a Windows Phone 8 app.
I'm having issue appending to my JSON file.
It works fine if I keep the app open but once I close it and come back in it starts back writing from the beginning of the file.
Relevant code:
private async void btnSave_Click(object sender, RoutedEventArgs e)
{
// Create a entry and intialize some values from textbox...
GasInfoEntries _entry = null;
_entry = new GasInfoEntries();
_entry.Gallons = TxtBoxGas.Text;
_entry.Price = TxtBoxPrice.Text;
_GasList.Add(_entry);
//TxtBlockPricePerGallon.Text = (double.Parse(TxtBoxGas.Text) / double.Parse(TxtBoxPrice.Text)).ToString();
// Serialize our Product class into a string
string jsonContents = JsonConvert.SerializeObject(_GasList);
// Get the app data folder and create or open the file we are storing the JSON in.
StorageFolder localFolder = ApplicationData.Current.LocalFolder;
StorageFile textfile = await localFolder.CreateFileAsync("gasinfo.json", CreationCollisionOption.OpenIfExists); //if get await operator error add async to class (btnsave)
//open file
using (IRandomAccessStream textstream = await textfile.OpenAsync(FileAccessMode.ReadWrite))
{
//write JSON string
using (DataWriter textwriter = new DataWriter(textstream))
//using (DataWriter textwriter = new DataWriter(textstream))
{
textwriter.WriteString(jsonContents);
await textwriter.StoreAsync(); //writes buffer to store
}
}
}
private async void btnShow_Click(object sender, RoutedEventArgs e)
{
StorageFolder localfolder = ApplicationData.Current.LocalFolder;
try
{
// Getting JSON from file if it exists, or file not found exception if it does not
StorageFile textfile = await localfolder.GetFileAsync("gasinfo.json");
using (IRandomAccessStream textstream = await textfile.OpenReadAsync())
{
//read text stream
using (DataReader textreader = new DataReader(textstream))
{
//get size ...not sure what for think check the file size (lenght) then based on next 2 commands waits until its all read
uint textlength = (uint)textstream.Size;
await textreader.LoadAsync(textlength);
//read it
string jsonContents = textreader.ReadString(textlength);
// deserialize back to gas info
_GasList = JsonConvert.DeserializeObject<List<GasInfoEntries>>(jsonContents) as List<GasInfoEntries>;
displayGasInfoEntries();
}
}
}
catch
{
txtShow.Text = "something went wrong";
}
}
private void displayGasInfoEntries()
{
txtShow.Text = "";
StringBuilder GasString = new StringBuilder();
foreach (GasInfoEntries _entry in _GasList)
{
GasString.AppendFormat("Gallons: {0} \r\n Price: ${1} \r\n", _entry.Gallons, _entry.Price); // i think /r/n means Return and New line...{0} and {1} calls "variables" in json file
}
txtShow.Text = GasString.ToString();
}
Thanks
Do you call the btnShow_Click each time you've started the app? Because otherwise the _GasList will be empty; if you now call the btnSave_Click all previous made changes will be lost.
So please make sure, that you restore the previously saved json data before you add items to the _GasList.