I'm trying to import a text file into an access database. It's not one I've written myself but the spec for the delimited text file is set up properly and the file imports properly using the wizard. When I try to use the import functions of the app itself, the ImportError table tells "Field Truncation" for one of the fields. Any help would be appreciated.
I would suggest examining each column that you're bringing in, and then measuring that against your table column properties in table "Design" mode.
Common things I've seen throw this error are:
Imported text exceeding the 255 character limit of a field (in which case you can change the field type to memo)
Date fields being set up as short date format, and then trying to import long dates/times into the field.
Text other than "yes/no/true/false" being imported into Yes/No/True/False fields.
Double-check your columns for similar names, and then check the data being imported. Sometimes when multiple people are working on a project and appending data, columns with similar names can get confused...especially if the column is collapsed so that its name is not entirely showing.
Related
When using a copy activity in Azure Data Factory to copy a typical CSV file with a header row into Parquet sink, the SINK fails with the following error due to the column names in the CSV having spaces in the header.
The column name is invalid. Column name cannot contain these
character:[,;{}()\n\t=]
The CSV is pipe delimited and displays just fine using the preview feature of the dataset with the first row marked as the header. I see no options to handle this use-case on the parquet side (sink) of the copy activity. I realize this can probably be addressed using a data flow to transform column names to remove spaces, but does that mean the native copy activity is incapable of handling this condition where a space in included in a header row?
EDIT: I should have added that dataset uses default mappings so that we can use the same dataset for any CSV to PARQUET copy. The answer provided will work for explicit mappings, but we don't see any resolution for folks who use default/dynamic mappings since we do not have access to the column names to remove spaces.
As we can note from the official Doc here
Error code: ParquetInvalidColumnName
Message: The column name is invalid. Column name cannot contain these character:[,;{}()\n\t=]
Cause: The column name contains invalid characters.
Resolution: Add or modify the column mapping to make the sink column name valid.
If you would like continue to use copy activity, there are few workarounds
1. make sure you have selected Column delimiter as Pipe(|)
2. If feasible, in mapping settings > import schema and rename the column name without spaces in destination column.
This is still an ongoing issue or request, follow here for more.
I have a csv file from a legacy dbase dbf file. The data contains a few columns which have number values with hyphens. Like this '661-'. I am trying to import the csv into MySql using 'Import External Data' in MySql Yog. The issue is, the columns that have values with hyphens are getting imported as decimals resulting in '-661.0000'.
This is odd as the column format in the csv (via MS Excel) is 'general' not 'number' and I am trying to import these values into varchar fields. Seems MySql is ignoring the settings.
Has anyone faced something like this or have any suggestions on how I can get the data in as a string not a decimal?
Thanks
ANSWER - sorry to be answering my own question but I did solve it with some group input.
The file needs to be saved with all fields (that you want to be treated as string) in quotes. MS Excel DOES NOT seem to have an option for this. Apache Open Office does. Open the file in AOO and save as text.csv. From there you can edit the filters and set all cells to be in quotes. Problem solved.
Importing csv in Rapidminer is not loading data properly in the attributes/ columns and returns errors.
I have set the parameter values correctly in the 'Data Import Wizard'.
Column Separation is set to comma and when I check the "Use Quotes" parameter I see that there are too many "?" appear in the columns even though there is data in the actual csv file.
And when I do not check the “Use Quotes” option then I notice that the content of the columns are distributed across different columns, i.e., data does not appear in the correct column. It also gives error for the date column.
How to resolve this? Any suggestions please? I saw a lot of Rapidminer videos and read about it but did not help.
I am trying to import twitter conversations data which I exported from a 3rd party SaaS tool which extracts Twitter data for us.
Could someone help me soon please? Thanks, Geeta
It's virtually impossible to debug this without seeing the data.
The use quotes option requires that each field is surrounded by double quotes. Do not use this if your data does not contain these because the input process will import everything into the first field.
When you use comma as the delimiter, the observed behaviour is likely to be because there are additional commas contained in the data. This seems likely if the data is based on Twitter. This confuses the import because it is just looking for commas.
Generally, if you can get the input data changed, try to get it produced using a delimiter that cannot appear in the raw text data. Good examples would be | or tab. If you can get quotes around the fields, this will help because it allows delimiter characters to appear in the field.
Dates formats can be handled using the data format parameter but my advice is to import the date field as a polynominal and then convert it later to date using the Nominal to Date operator. This gives more control especially when the input data is not clean.
And so we found a 3.6GB csv that we have uploaded onto S3 and now want to import into Redshift, then do the querying and analysis from iPython.
Problem 1:
This comma delimited file contains values free text that also contains commas and this is interfering with the delimiting so can’t upload to Redshift.
When we tried opening the sample dataset in Excel, Excel surprisingly puts them into columns correctly.
Problem 2:
A column that is supposed to contain integers have some records containing alphabets to indicate some other scenario.
So, the only way to get the import through is to declare this column as varchar. But then we can do calculations later on.
Problem 3:
The datetime data type requires the date time value to be in the format YYYY-MM-DD HH:MM:SS, but the csv doesn’t contain the SS and the database is rejecting the import.
We can’t manipulate the data on a local machine because it is too big, and we can’t upload onto the cloud for computing because it is not in the correct format.
The last resort would be to scale the instance running iPython all the way up so that we can read the big csv directly from S3, but this approach doesn’t make sense as a long-term solution.
Your suggestions?
Train: https://s3-ap-southeast-1.amazonaws.com/bucketbigdataclass/stack_overflow_train.csv (3.4GB)
Train Sample: https://s3-ap-southeast-1.amazonaws.com/bucketbigdataclass/stack_overflow_train-sample.csv (133MB)
Try having different delimiter or use escape characters.
http://docs.aws.amazon.com/redshift/latest/dg/r_COPY_preparing_data.html
For second issue, if you want to extract only numbers from the column after loading into char use regexp_replace or other functions.
For third issue, you can as well load it into VARCHAR field and then use substring cast(left(column_name, 10)||' '||right(column_name, 6)||':00' as timestamp)
to load it into final table from staging table
For the first issue, you need to find out a way to differentiate between the two types of commas - the delimiter and the text commas. Once you have done that, replace the delimiters with a different delimiter and use the same as delimiter in the copy command for Redshift.
For the second issue, you need to first figure out if this column needs to be present for numerical aggregations once loaded. If yes, you need to get this data cleaned up before loading. If no, you can directly load this as char/ varchar field. All your queries will still work but you will not be able to do any aggregations (sum/ avg and the likes) on this field.
For problem 3, you can use Text(date, "yyyy-mm-dd hh:mm:ss") function in excel to do a mass replace for this field.
Let me know if this works out.
I am trying to import a Text File into a MySQL through Navicat DB software.
I am struggling to import(append) a text file into a MySQL table.
The text file fields are seperated by | ;
example : |Name|Email|Address|
When i import this through the Navicat import wizard it ask for " Which delimeter seperates
the fields. So instead of selecting Tabs, ; , or any other i select | as field seperator.
But still the fields in the file do not match(sync) with the fields of the table...
Can anyone suggest any advice here?
I actually have exported the text file from another MySQL DB thru export functionality from PHPMyAdmin,,
I assume your name column is null and the values appear instead in the email column?
I suspect the problem lies in the fact that your fields are not only separated by a pipe, your rows also begin and end with a pipe.
Think of a CSV: name,email,address, not ,name,email,address,, because that would be interpreted as 5 columns, the value of the first and last field being null.
You'll have to choose a different delimiter for your rows and fields.
Beyond that, you can try importing the data into a new table and then write an insert query to map the temp fields to the ones in your database. The screen after the one where you choose the target table has a table where you can map your import fields to the target ones.
Let me know how that works out.