I'm trying to use Talend Open Studio for Data Integration for a BI school project. I have a txt file with some data like this :
NoClient;Name;Region;State;City;Postcode
24;Edna Thomas;West;California;Laguna Niguel;92677
I used jobs to transform data and insert it into tables. It works nicely.
Now, I'd like to handle SQL errors. For example, if the column length specified in the DB is 10 and if the job tries to insert a 11 length data in that column, I will get an SQL error.
How can I achieve it ? Is there a specific feature in tMysqlOutput or should I simply use triggers like this to check data before inserting :
CREATE TRIGGER my_trigger
BEFORE INSERT
ON my_table
FOR EACH ROW BEGIN
IF (SELECT LENGTH(NEW.Noclient)>255) THEN
// instructions
END IF
Hope it's clear enough ! Thanks in advance, sorry for bad english :-)
Maybe you can do a check in each variable in a tmap.
Like for example :
StringHandling.LEN(var) <= 10 ? var : StringHandling.LEFT(var,10)
And you adapt in function of the size of your field.
There's a component for this: tSchemaComplianceCheck
A nice example can be found at:
http://dwetl.com/2015/03/30/data-validation-using-tschemacompliancecheck/
Related
I have to work on a linked server. My goal: Update an entire table in mysql server(version:8.0.21) via OPENQUERY in SQL Server(version 13.0.1742.0). I tried this but it generates an error Row cannot be located for updating. Some values may have been changed since it was last read and this one The rowset was using optimistic concurrency and the value of a column has been changed after the containing row was last fetched or resynchronized.
update linkedTable
set
linkedTable.id_parent=unlinkedTable.IdCat1,
linkedTable.code=unlinkedTable.CodeFamilleFAT,
linkedTable.niveau=unlinkedTable.NiveauCategorieFAT,
linkedTable.langue=unlinkedTable.CodeLangueFAT,
linkedTable.nom=unlinkedTable.LibelleCommercialFAT,
linkedTable.descriptionA=unlinkedTable.DescriptifCom1FAT,
linkedTable.vignette=null,
linkedTable.id_categorie=unlinkedTable.id
from openquery(NAMELINKEDSERVER, 'select id_categorie, id_parent, code, niveau, langue, nom, description as descriptionA, vignette from DatabaseMySQL.Table') as linkedTable
inner join DatabaseSQLserver.dbo.Table as unlinkedTable on unlinkedTable.Id = linkedTable.id_categorie
Then I tried this:
update linkedTable
set
linkedTable.id_parent=unlinkedTable.IdCat1,
linkedTable.code=unlinkedTable.CodeFamilleFAT,
linkedTable.niveau=unlinkedTable.NiveauCategorieFAT,
linkedTable.langue=unlinkedTable.CodeLangueFAT,
linkedTable.nom=unlinkedTable.LibelleCommercialFAT,
linkedTable.descriptionA=unlinkedTable.DescriptifCom1FAT,
linkedTable.vignette=null,
linkedTable.id_categorie=unlinkedTable.id
from openquery(NAMELINKEDSERVER, 'select id_categorie, id_parent, code, niveau, langue, nom, description as descriptionA, vignette from DatabaseMySQL.Table') as linkedTable
inner join DatabaseSQLserver.dbo.Table as unlinkedTable on unlinkedTable.Id = linkedTable.id_categorie
where linkedTable.id_categorie = 1
This work but only one row is updated. So I wrote a stored procedure to update each line but it took too much time.
Can someone explain why my first query didn't work (question1) and how I can reduce the time of my stored procedure (question2)?
I use while loop (count the number of id and update each id).
Thank you in advance.
Kind Regards.
I resolve the problem by checking some option on ODBC Driver in MySQL and reading some forum. I check this box.
enter image description here
This option allows to avoid the errors quoted previously. With this option, i can update multiple values without error on join or other request. Thank you Solarflare and "Another guy" (i lost the name) for correcting me (EDIT A POST). Have nice day both.
I'm trying to build a Data Integration job uses pass through to extract data from a view in a MySQL database.
Wev'e been using pass through a lot in the project, mostly extracting data from Redshift,
however with MySQL I was not able to do make it work properly.
It keeps complaining a table is missing even though when pass through is off, view is found and data is extracted...
tried every trick I know, starting from enabling case-sensitive DBMS object names, to manually remove single/double quotes from the statement just in case MySQL confuses confuses it with something else...
No luck.
ODBC driver is [MySQL][ODBC 5.3(a) Driver][mysqld-5.5.53].
Ran on a Windows environment.
Any idea how to solve this?
Thank you in advance.
EDIT
So, first of all, one correction (even though not that important - I extract from a view, not a table).
This is the code generated by SAS Create Table transformation, pass through enabled. I only put an asterisk instead of the full list of columns:
proc sql;
connect to ODBC
(
READBUFF=10000 DATASRC="cmp.web_api" AUTHDOMAIN="MYSQL_CMP_Auth"
);
create table work."W7ZZZKOC"n as
select
*
from connection to ODBC
(
select
V_BI_ACCOUNT.ACCOUNT_NAME,
V_BI_ACCOUNT.ACQUISITION_SOURCE__C,
V_BI_ACCOUNT.ZUORA__ACTIVE__C,
V_BI_ACCOUNT.ADDRESS_LINE_1__C,
V_BI_ACCOUNT.ADDRESS_LINE_2__C,
V_BI_ACCOUNT.ADDRESS_LINE_3__C,
V_BI_ACCOUNT.AGREEMENT_DATE,
V_BI_ACCOUNT.AGREEMENT_LEGAL_CLAUSE_1__C,
V_BI_ACCOUNT.AGREEMENT_LEGAL_CLAUSE_2__C,
V_BI_ACCOUNT.PERSONBIRTHDATE,
V_BI_ACCOUNT.BLOCKED_REASON__C,
V_BI_ACCOUNT.BRAND__C,
V_BI_ACCOUNT.CPN__C,
V_BI_ACCOUNT.ACCCREATEDBYID,
V_BI_ACCOUNT.ACCCREATEDDATE,
V_BI_ACCOUNT.CURRENCY_PREFERENCE__C,
V_BI_ACCOUNT.CUSTOMER_FULL_NAME__PC,
V_BI_ACCOUNT.ACCOUNTID,
V_BI_ACCOUNT.ZUORA__CUSTOMERPRIORITY__C,
V_BI_ACCOUNT.DELIVERY_SALUTATION__C,
V_BI_ACCOUNT.DISPLAY_NAME,
V_BI_ACCOUNT.PERSONEMAIL,
V_BI_ACCOUNT.EMAILKEY__C,
V_BI_ACCOUNT.FACEBOOKKEY,
V_BI_ACCOUNT.FIRSTNAME,
V_BI_ACCOUNT.GENDER__C,
V_BI_ACCOUNT.PHONE,
V_BI_ACCOUNT.ACCLASTACTIVITYDATE,
V_BI_ACCOUNT.ACCLASTMODIFIEDDATE,
V_BI_ACCOUNT.LASTNAME,
V_BI_ACCOUNT.OTHER_EMAIL__C,
V_BI_ACCOUNT.PI_TYPE__C,
V_BI_ACCOUNT.ACCPARENTID,
V_BI_ACCOUNT.POSTCODE__C,
V_BI_ACCOUNT.PRIMARY_ACCOUNT_OF_THIS_CUSTOMER,
V_BI_ACCOUNT.ACCPRIMARY__C,
V_BI_ACCOUNT.ACCREASON_FOR_STATUS__C,
V_BI_ACCOUNT.ZUORA__SLA__C,
V_BI_ACCOUNT.ZUORA__SLASERIALNUMBER__C,
V_BI_ACCOUNT.SALUTATION,
V_BI_ACCOUNT.ACCSYSTEMMODSTAMP,
V_BI_ACCOUNT.PERSONTITLE,
V_BI_ACCOUNT.ZUORA__UPSELLOPPORTUNITY__C,
V_BI_ACCOUNT.X_CODE__C,
V_BI_ACCOUNT.ZUORA__ACCOUNT_ID__C,
V_BI_ACCOUNT.ZUORA__PAYMENTMETHODID__C,
V_BI_ACCOUNT.CITY,
V_BI_ACCOUNT.ORIGINAL_CREATED_DATE,
V_BI_ACCOUNT.SOURCE_SYSTEM_ID,
V_BI_ACCOUNT.STATUS,
V_BI_ACCOUNT.ZUORA__CONTACT_ID,
V_BI_ACCOUNT.ACCISDELETED,
V_BI_ACCOUNT.BILLING_ACCOUNT_NAME,
V_BI_ACCOUNT.ACZCREATEDDATE,
V_BI_ACCOUNT.ACZSYSTEMMODSTAMP,
V_BI_ACCOUNT.ACZLASTACTIVITYDATE,
V_BI_ACCOUNT.ZUORA__ACCOUNT__C,
V_BI_ACCOUNT.ZUORA__ACCOUNTNUMBER__C,
V_BI_ACCOUNT.ZUORA__AUTOPAY__C,
V_BI_ACCOUNT.ZUORA__BALANCE__C,
V_BI_ACCOUNT.ZUORA__CREDITCARDEXPIRATION__C,
V_BI_ACCOUNT.ZUORA__CURRENCY__C,
V_BI_ACCOUNT.ZUORA__MRR__C,
V_BI_ACCOUNT.ZUORA__PAYMENTTERM__C,
V_BI_ACCOUNT.ZUORA__PURCHASEORDERNUMBER__C,
V_BI_ACCOUNT.ZUORA__LASTINVOICEDATE__C,
V_BI_ACCOUNT.COUNTRY_NAME,
V_BI_ACCOUNT.COUNTRY_CODE,
V_BI_ACCOUNT.FAVOURITE_FOOTBALL_CLUB,
V_BI_ACCOUNT.COUNTY
from
web_api.V_BI_ACCOUNT as V_BI_ACCOUNT
);
%rcSet(&sqlrc);
disconnect from ODBC;
quit;
And again, when I extract data without pass through - works successfully,
I found out the problem was a column name exceeds 32 positions.
As SAS supports up column names up to 32,
the query fails to find PRIMARY_ACCOUNT_OF_THIS_CUSTOMER as the original column name is PRIMARY_ACCOUNT_OF_THIS_CUSTOMER__C.
EDIT
One more thing I found out is, MySQL doesn't like specifying schema name nor aliases.
Therefore,
From clause to only specify table name i.e : 'from v_bi_account' rather than 'web_api.v_bi_account'
and do not use aliases i.e use 'from v_bi_account' rather than 'from v_bi_account as v_bi_account'
Thank you guys so much for your help.
guys I need help.
I am using Mysql / phpmyadim.
I have db with table which stores name and code id of people.
+--------+---------+
| Name | code_id |
+--------+---------+
| Nazeer | MD-1 |
+--------+---------+
I have 10 contacts and ids. I am using php program which used to generate automatic code.
recently i imported more records in to db from excel file and record increase to 5000+.
My php automatic code stopped generating codes giving me syntax error on code id.
I figured out that my excel import was having code id like MD-1, MD-2, etc. and my program used automatic code for number in 3 digits since my record is over thousands which 4 digit it give syntax error.
I did some research on solving that and the answer was to change all 2 digit numbers eg. "MD-1" ~ "MD-99" TO "MD-001" ~ "MD-099" and my program will work again.
so the question is how do i do that in phpmyadmin sql to change it. I need to keep 'MD-' and add '0' then add back the corresponding number.
thanks and appreciate your help in advance.
Regrds.
this sql will update all your data, but like I said in comments, you better off fixing your php code instead.
WARNING : this sql only works assuming all your data are in the format of [MD-xxx] with 3 or less numbers in it
UPDATE your_table SET
code_id=case length(substr(code_id,4))
WHEN 1 THEN concat("MD-00",substr(code_id,4))
WHEN 2 THEN concat("MD-0",substr(code_id,4))
ELSE code_id END;
I assume that you want to update the content MD-1 to MD-001 and MD-99 to MD-099. To do that you can write a PHP code to retrieve the rows one by one and have to match patterns and then update. Here are some useful links. link 1
HINT : you can check 5 digit string and then add another 0 in the position of 3.(use [exploid] to split by "-" and then concat with "-0" 2) There are no way to do the same only by using MYSQl since it's not a programming language. And other thing is PHP is not a program. It's a programming language.
run UPDATE query and use CONCAT function :
for ($x=0; $x=<upto>; $x++){
UPDATE <table_name>
SET <columnname>= CONCAT('MD-',0,$x)
WHERE <columnname>= CONCAT('MD-',$x)
}
Below simple update command can help you.
UPDATE mytable
SET code_id=IF(LENGTH(code_id)=4,CONCAT(SUBSTRING_INDEX(code_id,'-',1),'-00',SUBSTRING_INDEX(code_id,'-',-1)),IF(LENGTH(code_id)=5,CONCAT(SUBSTRING_INDEX(code_id,'-',1),'-0',SUBSTRING_INDEX(code_id,'-',-1)),code_id));
I have two columns with mySQL:
"part_no"
"pdf_link"
I need the "pdf_link" column to automatically grab/duplicate the "part_no" value and add a .pdf extension on the end.
For example: If part_no = 00-12345-998, then pdf_link = 00-12345-998.pdf
I need this to happen every time I insert.
I appreciate the help.
Erik
you can achive this effect by using triggers I think.
http://dev.mysql.com/doc/refman/5.0/en/trigger-syntax.html
CREATE TRIGGER ins_pdf AFTER INSERT ON MY_TABLE SET #pdf_link = concat(#part_no,'.pdf')
Why store this extra computed information in the database? You can do this in the query when you pull it out, or, if needed, you could make a view that does it only as-needed.
Example pseudo query (my brain hurts right now, so this is only an example):
select concat(`part_no`, ".pdf") as `pdf_link` from `parts`;
If you really need this, you could use a trigger to duplicate the data ans add the extra string.
I'm writing a rails application on top of a legacy mysql db which also feeds a PHP production tool. Because of this setup so its not possible for me to change the databases structure.
The problem I'm having is that two table have a "time" attribute (duration) as long as the time is under 24:00:00 rails handles this, but as soon as rails comes across something like 39:00:34 I get this "ArgumentError: argument out of range".
I've looked into this problem and seen how rails handle the time type, and from my understanding it treats it like a datetime, so a value of 39:00:34 would throw this error.
I need some way of mapping / or changing the type cast so I don't get this error. Reading the value as a string would also be fine.
Any ideas would be most appreciated.
Cheers
I'm not familiar with Rails so there can be a clean, native solution to this, but if all else fails, one workaround might be writing into a VARCHAR field, then running a 2nd query to copy it over into a TIME field within mySQL:
INSERT INTO tablename (name, stringfield)
VALUES ("My Record", "999:02:02");
UPDATE tablename SET datefield = CAST(stringfield as TIME)
WHERE id = LAST_INSERT_ID();