I want to insert multiple items into a table and upsert the table on conflict. This is what I came up with the following
from sqlalchemy.dialects.postgresql import insert
meta = MetaData()
jobs_table = Table('jobs', meta, autoload=True, autoload_with=engine)
stmt = insert(jobs_table).values(jobs)
stmt.on_conflict_do_update(
index_elements=['j_id'],
set_= dict(active=True)
)
result = engine.execute(stmt)
return result.is_insert
The j_id is a unique field and I am trying to update the row if it already exists. I get the following error if the row already exists.
(psycopg2.IntegrityError) duplicate key value violates unique constraint "j_id"
DETAIL: Key (j_id)=(0ea445da-bd1d-5571-9906-0694fa85728a) already exists.
Is there something that I am missing here ?
stmt.on_conflict_do_update returns a new statement. If you change to the following it should work:
from sqlalchemy.dialects.postgresql import insert
meta = MetaData()
jobs_table = Table('jobs', meta, autoload=True, autoload_with=engine)
stmt = insert(jobs_table).values(jobs)
stmt = stmt.on_conflict_do_update(index_elements=['j_id'],
set_= dict(active=True))
result = engine.execute(stmt)
return result.is_insert
You can print(stmt) the statements to see the resulting SQL query. This can be useful to see if the statement which you are going to execute has the expected expression. Also adding echo=True to create_engine can be helpful to detect issues!
Related
I am trying to match this statement
stmt = session.query(models.Production).filter(models.Production.profile_name.regexp_match('some_name'))
results = session.execute(stmt).all()
print(results)
In profile_name column, it's saved as Some_Name. How do I get it to match ignoring capitalization?
Found an answer
from sqlalchemy import func
stmt = session.query(models.Production).filter(func.lower(models.Production.profile_name).regexp_match(func.lower('some_name')))
I am trying to fetch customer id from ":ao.mos:pro:%" and update corresponding doc ":pd:pro:%", we have thousands of ":ao.mos:pro:%" docs and corresponding ":pd:pro:%" for each mo:pro doc.
I am using below query and it works but it updates only one doc it can find indexes.
MERGE INTO `bucket1` AS d
USING `bucket1` AS p
ON d.icc = p.icc AND META(d).id LIKE ':pd:pro:%' AND META(p).id LIKE ':ao.mos:pro:%'
WHEN MATCHED THEN
UPDATE SET d.customerId = p.customerId;
Any suggestions on how to get this working for all matching docs in the bucket.
The following works.
CREATE INDEX ix11 ON `default` (icc, customerId) WHERE META().id LIKE ":ao.mos:pro:%";
CREATE INDEX ix12 ON `default` (icc, customerId) WHERE META().id LIKE ":pd:pro:%";
INSERT INTO default VALUES(":ao.mos:pro:1", {"icc":1, "customerId":100});
INSERT INTO default VALUES(":ao.mos:pro:2", {"icc":2, "customerId":101});
INSERT INTO default VALUES(":pd:pro:1", {"icc":1});
INSERT INTO default VALUES(":pd:pro:2", {"icc":1});
INSERT INTO default VALUES(":pd:pro:3", {"icc":2});
MERGE INTO `default` AS m
USING (SELECT p1.icc, p1.customerId
FROM `default` AS p1
WHERE META(p1).id LIKE ':ao.mos:pro:%' AND p1.icc IS NOT NULL) AS p
ON m.icc = p.icc AND META(m).id LIKE ':pd:pro:%'
WHEN MATCHED THEN
UPDATE SET m.customerId = p.customerId;
MERGE INTO `default` AS m
USING default AS p
ON m.icc = p.icc AND META(m).id LIKE ':pd:pro:%' AND META(p).id LIKE ':ao.mos:pro:%'
WHEN MATCHED THEN
UPDATE SET m.customerId = p.customerId;
Are you having following issue:
First source row matches 2 target rows. Second source matches same target row (by changing icc value to 1 of document key ":ao.mos:pro:2") then returns error (5320 Multiple UPDATE/DELETE of the same document (document key 'xxx') in a MERGE statement) (as with partial updated rows, You need transactions don't want update anything) as you can't update same row again. To resolve that you need to change your ON clause.
If it is a one off operation you can use Eventing Service and create a function with a bucket alias of "src_bkt" to map to bucket1 in r+w mode and the deployment feed boundary set to "From Everything" can easily do 100's of millions of items quickly without indexes.
function OnUpdate(doc,meta) {
// Filter out all non-interesting items
if (!meta.id.startsWith(":pd:pro:")) return;
// No need to do anything already updated.
if (doc.customerId)return;
var myint = meta.id.substring(8);
var otherkey = ":ao.mos:pro:" + myint;
var otherdoc = src_bkt[otherkey];
if (otherdoc) {
if (doc.icc === otherdoc.icc ) {
// update the field as we have a match and it is missing.
doc.customerId = otherdoc.customerId;
src_bkt[meta.id] = doc;
}
}
}
So now you just insert some test documents using the QWB
INSERT INTO bucket1 VALUES(":ao.mos:pro:1", {"icc":1, "customerId":100});
INSERT INTO bucket1 VALUES(":ao.mos:pro:2", {"icc":2, "customerId":101});
INSERT INTO bucket1 VALUES(":pd:pro:1", {"icc":1});
INSERT INTO bucket1 VALUES(":pd:pro:2", {"icc":1});
INSERT INTO bucket1 VALUES(":pd:pro:3", {"icc":2});
Then inspect the documents you should have
:ao.mos:pro:1{"customerId":100,"icc":1}
:ao.mos:pro:2{"customerId":101,"icc":2}
:pd:pro:1{"icc":1}
:pd:pro:2{"icc":1}
:pd:pro:3{"icc":2}
Deploy the Eventing function and look at your documents again.
:ao.mos:pro:1{"customerId":100,"icc":1}
:ao.mos:pro:2{"customerId":101,"icc":2}
:pd:pro:1{"icc":1,"customerId":100}
:pd:pro:2{"icc":1}
:pd:pro:3{"icc":2}
As expected only one doc with matching keys and matching icc properties updated.
For more performance (doing millions) you can up the workers in the functions settings to 2x the physical cores.
#necessary import goes here
engine = sqlalchemy.create_engine('mysql://root#127.0.0.1/test',echo=False)
print 'Engine created'
connection=engine.connect()
metadata=MetaData(engine)
metadata.bind=engine
Session = sessionmaker(bind=engine)
session = Session()
mapping = Table('mapping',metadata,autoload=True)
class Mapping(object):
pass
MappingMapper=mapper(Mapping,mapping)
Now i am able to write basic query for insert,update,delete,filter etc.
Q:1 I need to write complex query, where i do derive new columns based on existing columns. Ex. ColA,ColB is there on table, ColC is not part of table structure.
Select (ColA+ColB) as ColC from table where ColC > 50 order by ColC.
I am clueless how to convert above like query with SqlAlchemy. How to map, how to retrieve.
The easiest is to useHybrid Attributes.
In your case, just change the declaration of the class to the following:
from sqlalchemy.ext.hybrid import hybrid_property
class Mapping(object):
#hybrid_property
def ColC(self):
return self.ColA + self.ColB
Then the query:
qry = session.query(Mapping).filter(Mapping.ColC > 80)
will generate SQL:
SELECT mapping.id AS mapping_id, ...
FROM mapping
WHERE mapping."ColA" + mapping."ColB" > ?
Can someone tell me why this insert is failing but not giving me an error either? How do I fix this?
merge table1 as T1
using(select p.1,p.2,p.3,p.4,p.5 from #parameters p
inner join table1 t2
on p.1 = t2.1
and p.2 = t2.2
and p.3 = t2.3
and p.4 = t2.4) as SRC on SRC.2 = T1.2
when not matched then insert (p.1,p.2,p.3,p.4,p.5)
values (SRC.1,SRC.2,SRC.3,SRC.4,SRC.5)
when matched then update set t1.5 = SRC.5;
The T1 table is currently empty so nothing can match. The parameters table does have data in it. I simply need to modify this merge so that it checks all 4 fields before deciding what to do.
You can't select from a variable: from #parameters
See the following post: Using a variable for table name in 'From' clause in SQL Server 2008
Actually, you can use a variable table. Check it out:
MERGE Target_table AS [Target]
USING #parameters AS [Source]
ON (
[Target].col1 = [Source].col1
AND [Target].col2 = [Source].col2
AND [Target].col3 = [Source].col3
AND [Target].col4 = [Source].col4
)
WHEN NOT MATCHED BY TARGET
THEN INSERT (col1,col2,col3,col4,col5)
VALUES (
[Source].col1
,[Source].col2
,[Source].col3
,[Source].col4
,[Source].col5
)
WHEN MATCHED
THEN UPDATE SET [Target].col5 = [Source].col5;
I can't figure out the syntax for Mysql update with multiple concatinations. I want to be able to append a string to the end of the string stored in the database but do it to multiple columns all at once. I can do one column at a time just fine with this
UPDATE `table1`.`column1` SET `category1` = CONCAT(category1,'$value[0]',) WHERE `id`='$id';
But when I try to do it to multiple columns in the same table I get a syntax error.
UPDATE `table1`.`column1`
SET `category1` = CONCAT(category1,'5'),
`category2` = CONCAT(category2,'5'),
`category3` = CONCAT(category3,'5'),
`category4` = CONCAT(category4,'5'),
`category5` = CONCAT(category5,'5'),
`comments` = CONCAT(comments, 'jfsaklfsad')
WHERE `for_student_id`='46';
"You have an error in your SQL syntax;"
I can't find the syntax for separating each concat.
According to MySQL docs, UPDATE does not support such syntax. You must reference the table name, without the column, before the SET:
UPDATE `table1`
SET `category1` = CONCAT(category1,'5'),
`category2` = CONCAT(category2,'5'),
`category3` = CONCAT(category3,'5'),
`category4` = CONCAT(category4,'5'),
`category5` = CONCAT(category5,'5'),
`comments` = CONCAT(comments, 'jfsaklfsad')
WHERE `for_student_id`='46';