This error often appears when I try to CTAS from CSV data.
CSV Schema:
create or replace schema
(
`SomeCol1` BIGINT NOT NULL,
`SomeCol2` INT NOT NULL,
`SomeCol3` INT NOT NULL DEFAULT '0',
`SomeCol4` DATE NOT NULL FORMAT 'yyyy-MM-dd',
`SomeCol5` INT,
`SomeCol6` INT NOT NULL,
`SomeCol7` DECIMAL(10,2) NOT NULL,
`SomeCol8` DECIMAL (10,2) NOT NULL,
`SomeCol9` DOUBLE
)
for table "<path to CSVs>"
properties ('drill.strict' = 'true')
Then I run:
create table "<path to a parquet>" as select
<all 9 columns>
from "<path to CSVs>"
In some cases this error happens:
SYSTEM ERROR: NumberFormatException: Zero length BigInteger
Fragment 0:0
Please, refer to logs for more information. *
*) There is nothing more in the logs.
There are gigabytes of rows in the CSV and some of them are obviously in some bad shape.
1) How I'm supposed to find such error? At least on which column it happened? (except some hacks like reducing the count of columns and then rows, etc ..)
2) Is there some option to increase verbosity?
3) Or can I adjust the schema for .. better behavior?
Related
I have a table with a virtual generated column that concatenates five other columns (int and char) using CONCAT_WS(). This table contains 200-odd records and is never updated - it's just used as a lookup table. Recently, after months of untroubled processing, when I update records in a child table during which a SELECT is performed on this table, I sometimes see this error (ignore the "ITEM UPDATE FAILED" - that's me):
I am in development with a many changes every day, so it is impossible for me to determine if there is a correlating change. I have recently added "created" and "lastmodified" datetime fields to several tables with CURRENT_TIMESTAMP for DEFAULT or ON UPDATE, but not to this table.
Here's the table:
{EDIT} --- adding table definition:
CREATE TABLE `cpct_fixedfield` (
`id` int(11) UNSIGNED NOT NULL,
`name` varchar(256) NOT NULL,
`label` varchar(50) NOT NULL,
`field` int(11) NOT NULL,
`start` int(11) NOT NULL,
`rectype` int(11) NOT NULL ,
`mediatype` char(1) NOT NULL DEFAULT '' ,
`length` int(11) NOT NULL,
`userdefined` tinyint(1) NOT NULL,
`defaultval` varchar(5) NOT NULL,
`helpcode` varchar(10) NOT NULL,
`mandatory` varchar(2) NOT NULL ,
`idx` varchar(20) GENERATED ALWAYS AS (concat_ws('.',`field`,`rectype`,`mediatype`,`start`,`length`)) VIRTUAL NOT NULL)
ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
The length of the data in field never exceeds 11chars. I can view the entire table in pma or Mysql Workbench and the virtual field materialises in all records without complaint, which suggests to me that there is nothing wrong with either the expression for the virtual column or the data in the columns that expression draws on.
The error occurs in several contexts when I am updating a child table. All the updates occur in Stored Procedures/Functions. One section of code that seems to trigger the error is this:
SET idxvar = CONCAT_WS(".", SUBSTRING(tmpfldkey,3,1), rectype, ptype, position, "%") COLLATE utf8mb4_general_ci;
SELECT id INTO ffid FROM cpct_fixedfield WHERE idx LIKE idxvar AND idx != "0.0..6.2";
All the variables involved are varchars or ints. utf8mb4_general_ci is used throughout the database.
I cannot find any reference in MYSQL documentation to CONCAT or CONCAT_WS being unsafe, and none of the columns referenced has a default using a non-deterministic function. All the other questions I can find in this forum and elsewhere about this error have arisen because of the use of non-deterministic functions like CURRENT_TIMESTAMP() in the virtual field, or a component of the field.
I replaced the SELECT on the table with a (large) CASE statement and all was well, and in fact, after I did this then reverted to the SELECT I had no errors for many hours. But it just happened again (so I'm back to the case statement).
I have run out of ideas - I'm hoping someone has some knowledge/experience that can help.
Thanks
I am trying to concatenate two integers as the default value in a third field. My create table in SQL Server works fine:
CREATE TABLE MEI_Tbl
(
MEI_ID int PRIMARY KEY IDENTITY (1,1),
SRC tinyint NOT NULL DEFAULT '2',
HEI_ID AS (Cast (SRC as varchar)+ Cast (MEI_ID as varchar))
);
but when I try to create it in MySQL, I cannot find the equivalent for the concatenation of the two integers (Line 5 HEI_ID...).
**
I am aware of changing IDENTITY (1,1) to AUTO_INCREMENT for MySQL.
**
I have also tried several concat methods, but to no avail.
MySQL seems happier if I define the datatype for HEI_ID, and I have done so as varchar and int but again no success.
I have spent too much time reading about tool kits to convert SQL Server to MySQL. I just want to create the table in MySQL.
Any input would be appreciated.
MySQL does not support computed columns. Instead, you can use a view:
CREATE TABLE MEI_Tbl (
MEI_ID int PRIMARY KEY AUTO_INCREMENT,
SRC tinyint NOT NULL DEFAULT 2
);
CREATE VIEW v_MEI_Tbl as
SELECT MEI_ID, SRC,
CONCAT(src, mei_d) as HEI_ID
FROM MEI_Tbl
);
Then query from the view.
I want to create a SQL table with the following column datatypes.
Name : Varchar
Grade : Data will have “#” followed by numbers and also can be varchar. Eg : #1, #2, TML
Sizes : Can be whole numbers and fractions. Eg: 26/30, 80, 85/69
Average : Will be decimal numbers.
I have created table based on the above requirements:
CREATE TABLE [dbo].[Report_Proj](
[Name] [nvarchar](255) NOT NULL,
[Grade] [nvarchar](50) NOT NULL,
[Sizes] [float](10) NOT NULL,
[Average][decimal](10, 10) NOT NULL
But when I insert data into this table I’m getting the error “Msg 8115, Level 16, State 8, Line 1
Arithmetic overflow error converting varchar to data type numeric.
The statement has been terminated.”
Where could I possibly be going wrong.
Need the above data for reporting purposes and will not have any arithematic calculations in future.
just change the decimal value data type to (10,2)
declare #Report_Proj TABLE (
[Name] [nvarchar](255) NOT NULL,
[Grade] [nvarchar](50) NOT NULL,
[Sizes] [float](10) NOT NULL,
[Average][decimal](18, 2) NOT NULL)
insert into #Report_Proj ([Name],[Grade],[Sizes],[Average])values ('ram','#1',26/30,10.2)
select * from #Report_Proj
I'm writing a program in visual to use a database to make a PC builder type program. When I try to insert data into the PROCESSORS table, I get the following error:
ERROR:
Column name or number of supplied values does not match table definition.
Since my other inserts work fine, I don't know what's wrong with this one.
DROP TABLE PROCESSORS
DROP TABLE MOTHERBOARDS
DROP TABLE SOCKET_TYPE
DROP TABLE STORE;
CREATE TABLE PROCESSORS (
PRODCUT_ID VARCHAR(20) PRIMARY KEY,
BRAND VARCHAR(6) NOT NULL,
CORES INTEGER NOT NULL,
SPEED DECIMAL NOT NULL,
INTEGRATED_GPU VARCHAR(40) NOT NULL);
CREATE TABLE STORE(
STORE_ID VARCHAR(20) PRIMARY KEY,
PRODUCT_ID VARCHAR(20) NOT NULL,
PRODUCT_NAME VARCHAR(50) NOT NULL,
STORE VARCHAR(30) NOT NULL,
PRICE INTEGER NOT NULL);
/*Newegg Product I7-4470*/
INSERT INTO PROCESSORS VALUES('BX80646I74770','Intel',4,3.4,'None');
INSERT INTO STORE VALUES('N82E16819116900','BX80646I74770','Intel Core I7-4470 Haswell','Newegg',309.99);
Looks like you're entering a decimal in to an integer in the STORE Price column :)
Edit: Just noticed the DECIMAL type of the column is missing the impelementation syntax, e.g. DECIMAL(2,2). As it stands, according to here the default when not provided is DECIMAL(10, 0) which is a 10 digit integer basically. Provide precise parameters to the DECIMAL data type to allow the correct numbers with decimal places to be added, this will fix the INSERT problem.
I inserted some info into my mysql database and I got the following error listed below. What does this mean and how can I fix it?
1 row(s) inserted.
Inserted row id: 1
Warning: #1265 Data truncated for column 'summary' at row 1
Here is my Mysql tables structure below.
CREATE TABLE mem_articles (
id INT UNSIGNED NOT NULL AUTO_INCREMENT,
member_id INT UNSIGNED NOT NULL,
title VARCHAR(255) NOT NULL,
summary VARCHAR(255) DEFAULT NULL,
content LONGTEXT NOT NULL,
date_created DATETIME NOT NULL,
date_updated DATETIME DEFAULT NULL,
PRIMARY KEY (id)
);
I think it means that the amount of characters you attempted to insert into the summary column exceeded 255, perhaps you should alter it to be TEXT instead of VARCHAR(255).
http://dev.mysql.com/doc/refman/5.1/en/alter-table.html
It means that the data were "truncated", which in MySQL terminology means either it was truncated, or it was changed into something totally different if it was incompatible with the type.
This behaviour sucks; if you don't want it, use
SET SQL_MODE='TRADITIONAL'
Then it will behave like a sensible database (unfortunately this will probably break your entire code base if it's an existing application)
I would suggest setting the type to "longtext" or something larger.