I want to clarify that i browse similar questions about file encoding and importing csv files to Mysql. None of these worked for me.
The situation is like this:
i was in charge of the task of creating a spring boot application which generates csv files filled with random data to populate a database (each csv file correspond to a table in the DB).
the first scenario was to import these .csv files into Oracle 12c. There was no problem, i done it, there was no errors in the data.
The actual problem is using the same .csv files to import them into Mysql, it is not possible.
Clarifying both Databases (Oracle and Mysql) have both the same tables with the equivalent Data Types
I tried:
LOAD IN FILE option but the query never completes (it never shows any error message).
Exporting the data from Oracle (from SQLDeveloper) into a .csv file and then import it in Mysql, i got the same error below.
the Data Table Import Wizard from Mysql-workbench and it says:
Can't analyze file, please try to change encoding type, if that doesn't help, maybe the file is not: csv, or the file is empty.
Followed by:
Unhandled exception: 'ascii' codec can't encode character u'\xcd' in position 8: ordinal not in range(128).
I browsed questions about the encoding but none works to my case. And i'm pretty sure that i set the encoding to UTF-8 in my spring boot application.
My file writer has it:
writer = new OutputStreamWriter(fileOutputStream, StandardCharsets.UTF_8);
How is it possible that the generated .csv files work perfectly in Oracle but no in Mysql?
Aditional info:
springboot 2.0.3.RELEASE
Java version 1.8.0_162
If you are working on Java environment. I think DBIS solution can work better for you.
In this solution you only need to map csv file columns(header name or index) with column name of database table in XML file. no other code is required.
https://stackoverflow.com/a/50180272/2480620
Add one script to export table to csv and next csv to table.
<?xml version="1.0" encoding="UTF-8"?>
<config>
<connections>
<jdbc name="mysql">
<driver>com.mysql.jdbc.Driver</driver>
<url>jdbc:mysql://localhost:3306/test1</url>
<user>root</user>
<password></password>
</jdbc>
<jdbc name="oracle">
<driver>oracle.jdbc.OracleDriver</driver>
<url>jdbc:oracle:thin:#localhost:1521:orcl</url>
<user>root</user>
<password></password>
</jdbc>
</connections>
<component-flow>
<execute name="oracle2csv" success="csv2mysql" enabled="true">
<migrate>
<source>
<table connection="oracle">locations</table>
</source>
<destination>
<file delimiter="," header="false" path="D:/test_source.csv"/>
</destination>
<mapping>
<column source="location" destination="1" />
</mapping>
</migrate>
</execute>
<execute name="csv2mysql" enabled="true">
<migrate>
<source>
<file delimiter="," header="false" path="D:/test_source.csv"/>
</source>
<destination>
<table connection="mysql">locations</table>
</destination>
<mapping>
<column source="1" destination="location" />
</mapping>
</migrate>
</execute>
</component-flow>
</config>
full solution is explained
https://dbisweb.wordpress.com/
I have used it on MySql, Oracle and Postgres.
Related
In rsreportserver.config file, how can I set CSV export to have a field delimiter of none So that I can extract a fixed length file? I tried keeping it empty but it gives commas after each field value.
Also if possible, please provide the script to get fixed length file and some of the fields can have commas in between values.
I would really appreciate any input as my client is fully determined to use SSRS and not SSIS.
You can achieve this by adding a new custom extension
use the following as an example :
<Extension Name="csvnoseperator" Type="Microsoft.ReportingServices.Rendering.DataRenderer.CsvReport,Microsoft.ReportingServices.DataRendering">
<OverrideNames>
<Name Language="en-US">csvnoseperator</Name>
</OverrideNames>
<Configuration>
<DeviceInfo>
<FieldDelimiter></FieldDelimiter>
<UseFormattedValues>False</UseFormattedValues>
<NoHeader>True</NoHeader>
<Encoding>ASCII</Encoding>
<FileExtension>csv</FileExtension>
</DeviceInfo>
</Configuration>
</Extension>
You should be able to then use this as an export format from the front end.
Please ensure you backup your config file before you make this change! If you get it wrong, you will not be able to starts Reporting Service
I load data into my database from a CSV file :
<loadUpdateData encoding="UTF-8"
primaryKey="pk_id"
file="config/liquibase/roles_admin.csv"
separator=";"
tableName="role_admin">
<column name="libelle" type="STRING"/>
</loadUpdateData>
Is it possible to tell liquibase to insert new records if I add lines to my csv file?
You can modify your changeSet and add runAllways="true" this way it will run always (even if there is no change).
Using runOnChange="true" on your changeSet should do the trick.
The only way that i found to do that, is creating other csv file with the same columns like the original csv file and with the updated data, and create a new changeSet with loadUpdatedData inside and pointed to the new csv file.
I try to import a XML file into a MYSQL table using the LOAD XML function:
LOAD XML INFILE 'test.xml INTO TABLE edge_delete ROWS IDENTIFIED BY '<edge>';
the XML File is structured like this:
<netstate xmlns:xsi=....>
<timestep time="2">
<edge id="10">
<lane id="10_0">
<vehicle id="veh1" pos="4.60" speed="0.00"/>
</lane>
</edge>
</timestep>
The Problem:
All node levels start with the attribute "id". The import does not distinguish between the node levels.
Each node level should be one corresponding column in my sql table: edge | lane | vehicle id |...
Thank you for you help
A solution which worked for me:
Use a python script which replaces the "id" attributes with another attribute string (e.g. with "edge"). The script replaces the string. Maybe not the best and efficient way, but solves the problem.
Now I can import all columns of the XML file into the mySQL table.
Another workaround: Using a text editor with a find/replace function.
Did not work properly with the tools I found for large xml files > 1GB.
The script can be found here:
https://studiofreya.com/2016/11/17/replace-string-in-xml-file-with-python/#comment-86628
I want to import a XMl file into phpmyadmin.
I have a XML file with 75000 lines so inserting individually the values is a huge no, when I try to import the file as XMl to the database it gave the following message:
Import has been successfully finished, 0 queries executed.
The following structures have either been created or altered. Here you can:
View a structure's contents by clicking on its name.
Change any of its settings by clicking the corresponding "Options" link.
Edit structure by following the "Structure" link.
wp_db (Options)
(xmlfile.xml)
-----------------------------------------------------------------------
Notice in ./libraries/plugins/import/ImportXml.php#158
Undefined index: pma
Backtrace
./import.php#652: PMA\libraries\plugins\import\ImportXml->doImport(array)
The phpmyadmin is behind MAMP, a short version of the xml file looks like this:
<Report>
<Row1>
<Reference>123</Reference>
<Nature>Outside</Nature>
<Disponibility>Full</Disponibility>
<State>In progress</State>
<Person>Jon</Person>
<Seller></Seller>
</Row1>
<Row2>
<Reference>123</Reference>
<Nature>Outside</Nature>
<Disponibility>Full</Disponibility>
<State>In progress</State>
<Person>Jon</Person>
<Seller></Seller>
</Row2>
</Report>
If it's not possible at least what is the easiest way to do it?
Thanks!
phpMyAdmin supports importing XML only if it follows a special format. You can see the exact format by exporting in phpMyAdmin a table in XML. You would need to programmatically modify your existing file to adapt to the supported format.
I'm using MySQL Workbench 6.3 and got 2 Tables included. My task here is to upload XML file from a web server to my Table. One of them looks like this:
-<candidates>
<publicationDate>2016-04-26 15:00:00</publicationDate>
-<candidate>
<name>John</name>
<party>Party1</party>
</candidate>
-<candidate>
<name>Tom</name>
<party>Party2</party>
</candidate>
</candidates>
.....
and goes like this with 13 records.At the very beginning I got a notification that given XML file consists no information about styles included. Also, the other information says that default format of returning data is JSON, so to get XML in the header Accept must be included with value application/xml.
It's my very beginning with MySQL and I will be very thankful for any advises.