Is there an efficient way to update rows based on list of tuples in sqlalchemy?
If its a single row, then I can simply do:
session.query(table).filter(table.id == 10).update({'values': 'x'})
session.commit
however, the data i'm getting is a list of tuples
[(10, 'x'),(20,'y'),(30,'z'),(40,'p')]
table has IDs 10,20,30,40 etc.
is there efficient way to update instead of multiple individual updates?
You can convert the list of tuples to a list of dicts and then use update() with bindparam() as illustrated in the tutorial:
from pprint import pprint
import sqlalchemy as sa
engine = sa.create_engine("sqlite://")
tbl = sa.Table(
"tbl",
sa.MetaData(),
sa.Column("id", sa.Integer, primary_key=True, autoincrement=False),
sa.Column("value", sa.String(50)),
)
tbl.create(engine)
with engine.begin() as conn:
conn.execute(
tbl.insert(),
[
{"id": 10, "value": "old_10"},
{"id": 20, "value": "old_20"},
{"id": 30, "value": "old_30"},
],
)
with engine.begin() as conn:
# initial state
print(conn.execute(sa.select(tbl)).all())
# [(10, 'old_10'), (20, 'old_20'), (30, 'old_30')]
new_data = [(10, "x"), (20, "y"), (30, "z")]
params = [dict(tbl_id=a, new_value=b) for (a, b) in new_data]
pprint(params, sort_dicts=False)
"""
[{'tbl_id': 10, 'new_value': 'x'},
{'tbl_id': 20, 'new_value': 'y'},
{'tbl_id': 30, 'new_value': 'z'}]
"""
upd = (
sa.update(tbl)
.values(value=sa.bindparam("new_value"))
.where(tbl.c.id == sa.bindparam("tbl_id"))
)
print(upd)
# UPDATE tbl SET value=:new_value WHERE tbl.id = :tbl_id
conn.execute(upd, params)
# check results
print(conn.execute(sa.select(tbl)).all())
# [(10, 'x'), (20, 'y'), (30, 'z')]
Related
DECLARE #segArr NVARCHAR(max)
set #segArr = N'[1,2,3]'
DECLARE #segTb table (
k int,
v NVARCHAR(20)
);
insert into #segTb
VALUES
(0, 'a'),
(1, 'b'),
(2, 'c'),
(3, 'd'),
(4, 'e'),
(5, 'f');
select t.v from #segTb t
JOIN openjson(#segArr) a on a.[key] = t.k
for JSON auto;
I have a simple table with a key-value like structure and a JSON array that is a list of keys of the values I wanted.
The select statement can create the desired result, but the JSON format is wrong. It outputs an array of objects.
[
{
"v": "a"
},
{
"v": "b"
},
{
"v": "c"
}
]
But what I needed is an array of direct values.
[ "a", "b", "c" ]
You can use more conventional string manipulation methods to create JSON arrays in Azure SQL DB such as STRING_AGG which aggregates strings with the given separator (eg comma) and QUOTENAME which surrounds strings with the given quote character. A simple example:
SELECT QUOTENAME( STRING_AGG( QUOTENAME(v, '"' ), ',' ), '[' ) AS yourArray
FROM
(
SELECT t.v FROM #segTb t
INNER JOIN OPENJSON(#segArr) a ON a.[key] = t.k
) x;
For your particular example you could just create a user-defined table-type, with CREATE TYPE. This would have the advantage of you being able to add a primary key to the type, to guarantee duplicates cannot be added, give the optimizer a bit more info at run-time and use native relational abilities rather than bolted-on NOSQL abilities. A simple example which should run end-to-end:
IF NOT EXISTS ( SELECT * FROM sys.types st INNER JOIN sys.schemas ss ON st.schema_id = ss.schema_id WHERE st.name = N'tvpItems' AND ss.name = N'dbo')
CREATE TYPE dbo.tvpItems AS TABLE
(
k INT PRIMARY KEY
);
GO
DECLARE #items AS dbo.tvpItems
INSERT INTO #items VALUES ( 1 ), ( 2 ), ( 3 )
DECLARE #segTb TABLE (
k INT,
v NVARCHAR(20)
);
INSERT INTO #segTb
VALUES
(0, 'a'),
(1, 'b'),
(2, 'c'),
(3, 'd'),
(4, 'e'),
(5, 'f');
SELECT t.v
FROM #segTb t
INNER JOIN #items i ON i.k = t.k
FOR JSON AUTO;
I have a sql table related to discontinuous dates:
CREATE TABLE IF NOT EXISTS date_test1 ( items CHAR ( 8 ), trade_date date );
INSERT INTO `date_test1` VALUES ( 'a', '2020-03-20');
INSERT INTO `date_test1` VALUES ( 'b', '2020-03-20');
INSERT INTO `date_test1` VALUES ('a', '2020-03-21');
INSERT INTO `date_test1` VALUES ( 'c', '2020-03-22');
INSERT INTO `date_test1` VALUES ( 'd', '2020-03-22');
INSERT INTO `date_test1` VALUES ('a', '2020-03-25');
INSERT INTO `date_test1` VALUES ( 'e', '2020-03-26');
In this table, '2020-03-23' and '2020-03-24' are missed. I want to fill them by their previous data, in this table, the '2020-03-22' data.
Expected result:
The number of continues missing dates and of the records in one day are both uncertain.
So how to do this in mysql?
This solution uses Python and assumes that there aren't so many rows that they cannot be read into memory. I do not warrant this code free from defects; use at your own risk. So I suggest you run this against a copy of your table or make a backup first.
This code uses the pymysql driver.
import pymysql
from datetime import date, timedelta
from itertools import groupby
import sys
conn = pymysql.connect(db='x', user='x', password='x', charset='utf8mb4', use_unicode=True)
cursor = conn.cursor()
# must be sorted by date:
cursor.execute('select items, trade_date from date_test1 order by trade_date, items')
rows = cursor.fetchall() # tuples: (datetime.date, str)
if len(rows) == 0:
sys.exit(0)
groups = []
for k, g in groupby(rows, key=lambda row: row[1]):
groups.append(list(g))
one_day = timedelta(days=1)
previous_group = groups.pop(0)
next_date = previous_group[0][1]
for group in groups:
next_date = next_date + one_day
while group[0][1] != next_date:
# missing date
for tuple in previous_group:
cursor.execute('insert into date_test1(items, trade_date) values(%s, %s)', (tuple[0], next_date))
print('inserting', tuple[0], next_date)
conn.commit()
next_date = next_date + one_day
previous_group = group
Prints:
inserting c 2020-03-23
inserting d 2020-03-23
inserting c 2020-03-24
inserting d 2020-03-24
Discussion
With your sample data, after the rows are fetched, rows is:
(('a', datetime.date(2020, 3, 20)), ('b', datetime.date(2020, 3, 20)), ('a', datetime.date(2020, 3, 21)), ('c', datetime.date(2020, 3, 22)), ('d', datetime.date(2020, 3, 22)), ('a', datetime.date(2020, 3, 25)), ('e', datetime.date(2020, 3, 26)))
After the following is run:
groups = []
for k, g in groupby(rows, key=lambda row: row[1]):
groups.append(list(g))
groups is:
[[('a', datetime.date(2020, 3, 20)), ('b', datetime.date(2020, 3, 20))], [('a', datetime.date(2020, 3, 21))], [('c', datetime.date(2020, 3, 22)), ('d', datetime.date(2020, 3, 22))], [('a', datetime.date(2020, 3, 25))], [('e', datetime.date(2020, 3, 26))]]
That is, all the tuples with the same date are grouped together in a list so it becomes to easier to detect missing dates.
I opened and read csv file from argv to dictionary
data = open(argv[1])
reader = csv.DictReader(data)
dict_list = []
for line in reader:
dict_list.append(line)
and now when I want to access the content of the csv file like this:
for x in dict_list[0]:
print(x)
All I get is this:
"OrderedDict([('name', 'Alice'), ('AGATC', '2'), ('AATG', '8'), ('TATC', '3')])"
With this function:
for x in dict_list[0]:
print(x)
I get this result:
name
AGATC
AATG
TATC
Can you help me to access 'Alice', '2', '8' and '3'.
You can iterate through the dictionary a couple ways.
let's initialize the dictionary with your values:
from collections import OrderedDict
dict_list = OrderedDict([('name', 'Alice'), ('AGATC', '2'), ('AATG', '8'), ('TATC', '3')])
which gets us:
OrderedDict([('name', 'Alice'), ('AGATC', '2'), ('AATG', '8'), ('TATC', '3')])
you can then iterate through each key and then query the value attached:
for k in dict_list:
print(f"key={k}, value={dict_list[k]}")
and you will get:
key=name, value=Alice
key=AGATC, value=2
key=AATG, value=8
key=TATC, value=3
or, you can get both the key and the value at the same time:
for (k, v) in dict_list.items():
print(f"key={k}, value={v}")
which will get you the same output:
key=name, value=Alice
key=AGATC, value=2
key=AATG, value=8
key=TATC, value=3
I made my OrderedDict dict_list a dictionary and now I can access to the values of keys:
for x in dict_list:
temp = dict(x)
for y in types_count:
print(temp.get(y))
my test code:
local jsonc = require "jsonc"
local x = {
a = 1,
b = 2,
c = 3,
d = 4,
e = 5,
}
for k, v in pairs(x) do
print(k,v)
end
print(jsonc.stringify(x))
output:
a 1
c 3
b 2
e 5
d 4
{"a":1,"c":3,"b":2,"e":5,"d":4}
someone help:
from for pairs output, lua store table by key hash order, how can i change it?
i need output: {"a":1,"b":2,"c":3,"d":4,"e":5}
thanks
Lua tables can't preserve the order of their keys. There are two possible solutions.
You can store the keys in a separate array and iterate through that whenever you need to iterate through the table:
local keys = {'a', 'b', 'c', 'd', 'e'}
Or, instead of a hash table, you can use an array of pairs:
local x = {
{'a', 1},
{'b', 2},
{'c', 3},
{'d', 4},
{'e', 5},
}
I have a dataframe and I am trying to turn the column into a comma separated list. The end goal is to pass this comma seperated list as a list of filtered items in a SQL query.
How do I go about doing this?
> import pandas as pd
>
> mydata = [{'id' : 'jack', 'b': 87, 'c': 1000},
> {'id' : 'jill', 'b': 55, 'c':2000}, {'id' : 'july', 'b': 5555, 'c':22000}]
df = pd.DataFrame(mydata)
df
Expected solution - note the quotes around the ids since they are strings and the items in column titled 'b' since that is a numerical field and the way in which SQL works. I would then eventually send a query like
select * from mytable where ids in (my_ids) or values in (my_values):
my_ids = 'jack', 'jill','july'
my_values = 87,55,5555
I encountered a similar issue and solved it in one line using values and tolist() as
df['col_name'].values.tolist()
So in your case, it will be
my_ids = my_data['id'].values.tolist() # ['jack', 'jill', 'july']
my_values = my_data['b'].values.tolist()
Let's use apply with argument 'reduce=False' then check the dtype of the series and apply the proper argument to join:
df.apply(lambda x: ', '.join(x.astype(str)) if x.dtype=='int64' else ', '.join("\'"+x.astype(str)+"\'"), reduce=False)
Output:
b 87, 55, 5555
c 1000, 2000, 22000
id 'jack', 'jill', 'july'
dtype: object