Converting INT to Unsigned INT - ssis

I have a large table in prod with INT datatype and it is both primary key and identity column(1,1).The number of rows is 2147479257.
During the daily job run the ETL failed because of arithmetic overflow error,as it cant fit any more rows in to the destination table.
Can you please how can i change the column to unsigned int.

Change the datatype to bigint.
bigint -2^63 (-9,223,372,036,854,775,808) to 2^63-1 (9,223,372,036,854,775,807)

You've overflown an int, which means you have approximately 2B rows in the table. By switching to an unsigned int, you hope to gain another 2 billion rows of addressable space. The problem with this approach is that SQL Server does not support an unsigned int as a data type.
Your gut reaction might be to reach for Greg's approach of changing to a bigint data type. The challenge with this approach is that while your processing is dead in the water, you take the quick fix and change to bigint, any other consumer of that table is now going to fail. I went through this in 2011, by the way. We fixed the database only to have all the reporting and .NET applications fail. At that job, it'd have been far less catastrophic to have queued up the processing for a N days while we gave the appearance of normalcy than to remove all doubt by having every external facing application fail.
With the general implementation of an identity column, you can get an easy another 2B without making a single code change - simply set the identity value to the lower bound and you've bought yourself sufficient time to plan the migration to bigint. The command for this is dbcc checkident
You'll also likely want to ensure the identity column is specified as a unique value. People often set the identity column as a primary key but otherwise, you'd run code similar to the following.
SET NOCOUNT ON;
IF EXISTS
(
SELECT * FROM sys.tables AS T INNER JOIN sys.schemas AS S ON S.schema_id = T.schema_id WHERE S.name = 'dbo' AND T.name = 'IntOverfloweth'
)
BEGIN
DROP TABLE
dbo.IntOverfloweth;
END
CREATE TABLE
dbo.IntOverfloweth
(
IntOverflowethID int IDENTITY(2147483647,1) NOT NULL
, SomeValue varchar(30)
);
INSERT INTO
dbo.IntOverfloweth
(SomeValue)
OUTPUT
Inserted.*
VALUES
('Before');
BEGIN TRY
INSERT INTO
dbo.IntOverfloweth
(SomeValue)
OUTPUT
Inserted.*
VALUES
('Overflow');
END TRY
BEGIN CATCH
SELECT ERROR_MESSAGE() AS ErrorMessage;
END CATCH
-- Push the pointer back around to the begining
DBCC CHECKIDENT('dbo.IntOverfloweth', RESEED, -2147483648);
-- Ensure uniqueness
CREATE UNIQUE INDEX UQ_IntOverfloweth
ON dbo.IntOverfloweth
(
IntOverflowethID
);
INSERT INTO
dbo.IntOverfloweth
(SomeValue)
OUTPUT
Inserted.*
VALUES
('Does not Overflow');

Related

MySQL Variable Returning Incorrect Value

The Issue
I have a stored proc in a DB server that's bringing back a value of 5064803 when that record does not exist and the value should be 5064800 as per the query that builds the value of the variable.
I'm not sure if this is an issue with the value being of the FLOAT data type and the value in the record of the table ending in a double-zero or what but I cannot figure it out easily.
The table data types match those from the sensors that are set but this particular value from this sensor never actually gets set to a data type and it's usually always either a 1-8 digit INT with no decimal but I'd like to keep the data types the same as the correlated sensor just in case.
I've broke down the proc and I'm able to recreate the problem easily so I will post the detail below for those that may be able to help me figure out the issue and any workaround, etc.
The SQL Data
Create Table
delimiter $$
CREATE TABLE `number` (
`TimeInt` varchar(10) NOT NULL,
`TimeStr` datetime NOT NULL,
`IsInitValue` int(11) NOT NULL,
`Value` float NOT NULL,
`IQuality` int(11) NOT NULL,
UNIQUE KEY `uk_Times` (`TimeInt`,`TimeStr`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8$$
Insert Data
INSERT INTO `Number` (`TimeInt`,`TimeStr`,`IsInitValue`,`Value`,`IQuality`) VALUES ('1502618950','2017-08-13 10:09:10',1,5064800,0);
INSERT INTO `Number` (`TimeInt`,`TimeStr`,`IsInitValue`,`Value`,`IQuality`) VALUES ('1502618796','2017-08-13 10:06:36',0,5064800,3);
INSERT INTO `Number` (`TimeInt`,`TimeStr`,`IsInitValue`,`Value`,`IQuality`) VALUES ('1502617167','2017-08-13 09:39:27',1,5063310,0);
INSERT INTO `Number` (`TimeInt`,`TimeStr`,`IsInitValue`,`Value`,`IQuality`) VALUES ('1502613355','2017-08-13 08:35:55',0,5063310,3);
INSERT INTO `Number` (`TimeInt`,`TimeStr`,`IsInitValue`,`Value`,`IQuality`) VALUES ('1502612814','2017-08-13 08:26:54',1,0,0);
INSERT INTO `Number` (`TimeInt`,`TimeStr`,`IsInitValue`,`Value`,`IQuality`) VALUES ('1502609015','2017-08-13 07:23:35',0,0,3);
The SQL Query Breakdown
SET #bStartTime = '2017-08-13 09:24:16';
SET #bEndTime = '2017-08-13 10:06:31';
SET #LastNumber = (SELECT Value FROM Number ORDER BY TimeStr DESC LIMIT 1);
SET #NowNumber = (SELECT Value FROM Number WHERE TimeStr BETWEEN #bStartTime AND #bEndTime ORDER BY TimeStr DESC LIMIT 1);
SELECT #NowNumber;
SELECT #LastNumber;
Recreating the Issue
So based on The SQL Query Breakdown above, once all the data is in the table and then I run the queries within the SELECT queries alone within the #NowNumber and/or #LastNumber variables, I get the correct result of 5064800. However, if I run the entire SET statements for both of those to have it set the query and then just do a SELECT of those variable, it brings back the wrong result of 5064803.
So for example if I run SELECT Value FROM Number ORDER BY TimeStr DESC LIMIT 1 then the correct value is returned. If I run SET #LastNumber = (SELECT Value FROM Number ORDER BY TimeStr DESC LIMIT 1); and then run SELECT #LastNumber; I get the incorrect value returned.
Server System Specs
This particular MySQL Server is running the x86 version of 5.5.50 on Windows Server 2008 with 144 GB of RAM for some quick specs.
Question
I'd like to know what is causing this, and if there is a workaround to the problem either with or without changing the data type of the column assuming that's the issue when it's returned as a variable rather than just a straight query result.
I'll be happy to disclose more technical specs of the environment if needed but I've included what I think it important for the question. Perhaps this is a version bug or there's something obvious that causes this that I cannot see easily so I'm hoping someone can help me with this or explain why this is or is not possible with MySQL.
Sorry, declares can only be used in stored procedures in MySQL. I found this article which may help. It explains how MySQL rounds when storing digits and recommends using doubles. Try changing your floats to doubles.
MySql FLOAT datatype and problems with more then 7 digit scale

MySQL WHERE Condition on integer field returning incorrect values

I'm having a problem with MySQL returning the incorrect result when applying a WHERE condition to an integer field with a string value.
CREATE TABLE `people` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`name` varchar(255) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=4 DEFAULT CHARSET=latin1;
INSERT INTO `people` (`id`, `name`)
VALUES
(1, 'Bob'),
(2, 'Sally'),
(3, 'Jim');
Now when I run the query:
SELECT *
FROM people
WHERE id = '1-abcd';
My result set is:
id name
1 Bob
MySQL appears to be truncating the string value '1-abcd' to '1' behind the scenes as soon as it hits a non-integral character (in the conversion from a string to INT).
You're probably wondering why this matters. I'm trying to fix a site for a PCI compliance scan. The scan thinks the URI '/some/page?id=102-1' is allowing some form of sequel injection, but in reality it's showing the same content at '/some/page?id=102'.
This is not an issue in one place. It is an issue all over the place, and it's a fairly large system. Is there some way to rectify this on the MySQL end of things, so it no longer mistakenly judges the two values to be equivalent? I looked at the documentation for SQL modes, but didn't see anything regarding this circumstance.
UPDATE: I filed a dispute with the company that produced the scan, which they accepted, so I'm no longer in the woods. But it is disappointing that there's apparently no way to configure the casting behavior of MySQL from a string to INT in this case. (You can, but only for INSERTs and UPDATEs.)
What happens that MySQL type-casts the string literal value to an integer, and when it does that it starts from the left of the string and as soon as it reaches a character that cannot be considered part of a number, it strips out everything from that point on. So 1-0 gives output matching to 1. To do this you can use cast. I am not 100% sure about the syntax but it is like this:
select * from people
where id =
(
case when ISNUMERIC( '1-0' )
then cast ('1-0' as int)
else null
end )
What this will do is that if it is an numeric value then it will return the correct matching row or else not.
Edit:
The above query seems to be of MSSQL/Oracle and would not work with MySQL. For MySQL you can use RegExp. I have never use one but you can find more details here:
http://mysqlhints.blogspot.in/2012/01/how-to-find-out-if-entire-string-is.html
http://www.ash.burton.fm/blogs/2010/12/quick-tip-mysql-equivalent-of-isnumeric
http://www.justskins.com/forums/how-to-use-isnumeric-137604.html

MySql Basic table creation/handing

I'm trying to create a simple table where I insert field and I do some checks in MySql. I've used Microsoft SQL relatively easy. Instead, MySql give evrrytime query errors without even specifying what's going on. Poor MySql software design apart, here's what I'm trying to do:
1 table with 4 fields with an autoincremental autogenerated number to det an ID as primary key
CREATE TABLE `my_db`.`Patients_table` (
`ID_Patient` INT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY ,
`Patient_name` VARCHAR( 200 ) NOT NULL ,
`Recovery_Date` DATETIME NOT NULL ,
`Recovery_count` INT NOT NULL
) ENGINE = MYISAM
a simple stored procedure to insert such fields and check if something exist before inserting:
CREATE PROCEDURE nameInsert(IN nome, IN data)
INSERT INTO Patients_table (Patient_name,Recovery_Date) values (nome,data)
IF (EXISTS (SELECT Recovery_count FROM Tabella_nomi) = 0) THEN
INSERT INTO (Patients_table (Recovery_count)
ELSE
SET Recovery_count = select Recovery_count+1 from Patients_table
END
this seems wrong on many levels and MySQL useless syntax checker does not help.
How can I do this? Thanks.
There seems to be a lot wrong with this block of code. (No offense intended!)
First, Procedures need to be wrapped with BEGIN and END:
CREATE PROCEDURE nameInsert(IN nome, IN data)
BEGIN
...[actually do stuff here]
END
Second, since your table is declared with all fields as NOT NULL, you must insert all fields with an INSERT statement (this includes the Recovery_Date column, and excludes the AUTO_INCREMENT column). You can add DEFAULT CURRENT_TIMESTAMP to the date column if you want it to be set automatically.
INSERT INTO Patients_table (Patient_name,Recovery_Date) values (nome,data)
Third, what exactly is your IF predicate doing?
EXISTS (SELECT Recovery_count FROM Tabella_nomi) = 0
If you want to check if a row exists, don't put the = 0 at the end. Also, Tabella_nomi isn't declared anywhere in that procedure. Also, your SELECT statement should have a WHERE clause, since I'm assuming you want to select a specific row (this is going to select a result set of all recovery_counts).
Fourth, the second INSERT statement seems a little messy. It should look more like the first INSERT, and keep the point I made above in mind.
INSERT INTO (Patients_table (Recovery_count)
Fifth, the ELSE statement
SET Recovery_count = select Recovery_count+1 from Patients_table
Has some problems too. SET is meant for setting variables, not values in rows. I'm not 100% sure what your intent is from this statement, but it looks like you meant to increment the Recovery_count column of a certain row if it already exists. In which case, you meant to do something like this:
UPDATE Patients_table SET Recovery_count = Recovery_count+1 WHERE <conditional predicate>
Where the conditional predicate is something like this:
Patients_name = nome
Try these things, and look at the errors it gives you when you try to execute the CREATE STATEMENT. I bet they're more useful then you think!

track change in sql row for update

I've a java program to update a SQL table (id, name, status). Entire table is updated with same data or some changed data. How can I track if a row is same like it was before update or it has some modified data? id will be always same, only small typo on name. I just want to check on next update if name is modified. In this case the status field should be changed from 'same' to 'modified'. Will timestamp solve my issue? Please help.
1 - If you are looking to audit the table (inserts, updates, deletes), look at my how to prevent unwanted transactions slide deck w/code - http://craftydba.com/?page_id=880.
SEE CODE AT END!
The trigger that fills the audit table can hold information from multiple tables since the data is saved as XML. Therefore, you can un-delete if necessary. It tracks who and what made the change.
2 - If you are never going to purge the data from the audit table, why not mark the row as deleted but keep it for ever?
Many systems like people soft use effective dating to show if a record is no longer active. In the BI world this is called a type 2 dimensional table (slowly changing dimensions).
See the data warehouse institute article. http://www.bidw.org/datawarehousing/scd-type-2/
Each record has a begin and end date. All active records have a end date of null.
3 - Micorsoft SQL Server introduced the change data capture feature. While this tracks data change using a LOG reader after the fact, it lacks things like who and what made the change.
Again, all the above solutions work. I am partial to my solution!
Sincerely
John
The Crafty DBA
--
-- 7 - Auditing data changes (table for DML trigger)
--
-- Delete existing table
IF OBJECT_ID('[AUDIT].[LOG_TABLE_CHANGES]') IS NOT NULL
DROP TABLE [AUDIT].[LOG_TABLE_CHANGES]
GO
-- Add the table
CREATE TABLE [AUDIT].[LOG_TABLE_CHANGES]
(
[CHG_ID] [numeric](18, 0) IDENTITY(1,1) NOT NULL,
[CHG_DATE] [datetime] NOT NULL,
[CHG_TYPE] [varchar](20) NOT NULL,
[CHG_BY] [nvarchar](256) NOT NULL,
[APP_NAME] [nvarchar](128) NOT NULL,
[HOST_NAME] [nvarchar](128) NOT NULL,
[SCHEMA_NAME] [sysname] NOT NULL,
[OBJECT_NAME] [sysname] NOT NULL,
[XML_RECSET] [xml] NULL,
CONSTRAINT [PK_LTC_CHG_ID] PRIMARY KEY CLUSTERED ([CHG_ID] ASC)
) ON [PRIMARY]
GO
-- Add defaults for key information
ALTER TABLE [AUDIT].[LOG_TABLE_CHANGES] ADD CONSTRAINT [DF_LTC_CHG_DATE] DEFAULT (getdate()) FOR [CHG_DATE];
ALTER TABLE [AUDIT].[LOG_TABLE_CHANGES] ADD CONSTRAINT [DF_LTC_CHG_TYPE] DEFAULT ('') FOR [CHG_TYPE];
ALTER TABLE [AUDIT].[LOG_TABLE_CHANGES] ADD CONSTRAINT [DF_LTC_CHG_BY] DEFAULT (coalesce(suser_sname(),'?')) FOR [CHG_BY];
ALTER TABLE [AUDIT].[LOG_TABLE_CHANGES] ADD CONSTRAINT [DF_LTC_APP_NAME] DEFAULT (coalesce(app_name(),'?')) FOR [APP_NAME];
ALTER TABLE [AUDIT].[LOG_TABLE_CHANGES] ADD CONSTRAINT [DF_LTC_HOST_NAME] DEFAULT (coalesce(host_name(),'?')) FOR [HOST_NAME];
GO
--
-- 8 - Make DML trigger to capture changes
--
-- Delete existing trigger
IF OBJECT_ID('[ACTIVE].[TRG_FLUID_DATA]') IS NOT NULL
DROP TRIGGER [ACTIVE].[TRG_FLUID_DATA]
GO
-- Add trigger to log all changes
CREATE TRIGGER [ACTIVE].[TRG_FLUID_DATA] ON [ACTIVE].[CARS_BY_COUNTRY]
FOR INSERT, UPDATE, DELETE AS
BEGIN
-- Detect inserts
IF EXISTS (select * from inserted) AND NOT EXISTS (select * from deleted)
BEGIN
INSERT [AUDIT].[LOG_TABLE_CHANGES] ([CHG_TYPE], [SCHEMA_NAME], [OBJECT_NAME], [XML_RECSET])
SELECT 'INSERT', '[ACTIVE]', '[CARS_BY_COUNTRY]', (SELECT * FROM inserted as Record for xml auto, elements , root('RecordSet'), type)
RETURN;
END
-- Detect deletes
IF EXISTS (select * from deleted) AND NOT EXISTS (select * from inserted)
BEGIN
INSERT [AUDIT].[LOG_TABLE_CHANGES] ([CHG_TYPE], [SCHEMA_NAME], [OBJECT_NAME], [XML_RECSET])
SELECT 'DELETE', '[ACTIVE]', '[CARS_BY_COUNTRY]', (SELECT * FROM deleted as Record for xml auto, elements , root('RecordSet'), type)
RETURN;
END
-- Update inserts
IF EXISTS (select * from inserted) AND EXISTS (select * from deleted)
BEGIN
INSERT [AUDIT].[LOG_TABLE_CHANGES] ([CHG_TYPE], [SCHEMA_NAME], [OBJECT_NAME], [XML_RECSET])
SELECT 'UPDATE', '[ACTIVE]', '[CARS_BY_COUNTRY]', (SELECT * FROM deleted as Record for xml auto, elements , root('RecordSet'), type)
RETURN;
END
END;
GO
SQL Server doesn't have versioning on the table level. If you want to track difference between two field you have at least two options:
Control from your Java application - do pre-update check in your update method.
Control from SQL Server - write trigger again for pre-update check
You can also create suplimentary field where Version number will be kept
Yes you can use time stamp.
By using time stamp, you can find out the latest entry into the table and by using Order by timestamp in the query,you can get the latest and set the status based on corresponding value
It depends on how much information you need. If all you care about is whether the record has ever been modified, you can use created_when and updated_when fields. If the latter is greater than the former, it's been updated.
If you want to know what fields have been updated, you have to log the changes. The details depend on your requirements. If you need to log changes, a trigger is the best way to do it.

number of rows SQL Server

How to count or know the number of rows a table has without scaning all the table, maybe using ROW_NUMBER?
If you need a exact count, you will need to do a COUNT(*) which will scan the clustered index.
You can get a rough count using the sys.partitions schema, as shown here http://www.kodyaz.com/articles/sql-rowcount-using-sql-server-system-view-sys-partitions.aspx
Update: To get the count into a variable:
DECLARE #cnt INT;
SELECT #cnt = SUM(rows)
FROM sys.partitions
WHERE
index_id IN (0, 1)
AND object_id = OBJECT_ID('MyDB.dbo.MyTable');
SELECT #cnt;
SELECT COUNT(*) FROM Table
will return the number of rows
There is no ROW_NUMBER in SQL Server, just Oracle. Use:
SELECT COUNT(primary_key) FROM table
Where primary key the primary key column of your table.
Since its a primary key, its is already indexed, so SQL can count it without scanning the whole table (it uses a clustered index to be precise, which is much faster than a full table scan)
You could also use sys.indexes schema, but its not accurate, and you would need database admin priviledges to access, and your application database user is not supposed to have grants in that schema
A little late to the party here, but in SQL Server 2005 on, you could also use the sp_spaceused stored procedure:
DECLARE #rowCount AS INT
DECLARE #spaceUsed TABLE(
[Name] varchar(64),
[Rows] INT,
[Reserved] VARCHAR(50),
[Data] VARCHAR(50),
[Index_Size] VARCHAR(50),
[Unused] VARCHAR(50)
)
INSERT INTO #spaceUsed EXEC sp_spaceused 'MyTableName'
SET #rowCount = (SELECT TOP 1 [Rows] FROM #spaceUsed)
SELECT #rowCount AS 'Row Count'
I've gotten into the habit of using sp_spaceused in place of SELECT COUNT(*) FROM Table because it is much faster. It will most likely not be as accurate as COUNT(*), however.
MSDN: http://msdn.microsoft.com/en-us/library/ms188776.aspx
I dont believe you mean this but ill give it a try:
select count(*) from table