I'm trying to convert an application that uses MySQLdb to use Peewee instead. Most of the SELECTs and INSERTs are no problem, but one class of query has me puzzled.
The original code contains:
sql = "SELECT * FROM {tbl} WHERE tail='{tail}' AND flight="+\
"'{flight}' AND dest='{dest}' AND orig='{orig}' AND "+\
"oooi='{oooi}' AND report_time > ('{time}' - INTERVAL 2 HOUR) "+\
"AND report_time < ('{time}' + INTERVAL 2 HOUR)"
cmd = sql.format(tbl = self.table, tail=tail, flight=flight, dest=dest,
orig=orig, time = report_time, oooi=oooi)
c.execute(cmd)
return c.fetchone()
Trying to rewrite that to use Peewee I've come up with:
oooi_rec = Oooi_rec.select().where(Oooi_rec.tail == self.tail,
Oooi_rec.flight == self.flight,
Oooi_rec.dest == self.dest, Oooi_rec.orig == self.orig,
Oooi_rec.oooi=self.oooi,
Oooi_rec.report_time.between(low, high))
The bit that replaces "low" and "high" is what has me mystified for now. I'm trying to puzzle out how to use Peewee's fn() but it's slow going.
Perhaps:
low - SQL('interval 2 hour')
Also, so much sql injection in the way you had previously...yeesh.
Related
I need to
1. run a select query on MYSQL DB and fetch the records.
2. Records are processed by python script.
I am unsure about the way I should proceed. Is xcom the way to go here? Also, MYSQLOperator only executes the query, doesn't fetch the records. Is there any inbuilt transfer operator I can use? How can I use a MYSQL hook here?
you may want to use a PythonOperator that uses the hook to get the data,
apply transformation and ship the (now scored) rows back some other place.
Can someone explain how to proceed regarding the same.
Refer - http://markmail.org/message/x6nfeo6zhjfeakfe
def do_work():
mysqlserver = MySqlHook(connection_id)
sql = "SELECT * from table where col > 100 "
row_count = mysqlserver.get_records(sql, schema='testdb')
print row_count[0][0]
callMYSQLHook = PythonOperator(
task_id='fetch_from_testdb',
python_callable=mysqlHook,
dag=dag
)
Is this the correct way to proceed?
Also how do we use xcoms to store the records for the following MySqlOperator?'
t = MySqlOperator(
conn_id='mysql_default',
task_id='basic_mysql',
sql="SELECT count(*) from table1 where id > 10",
dag=dag)
I was really struggling with this for the past 90 minutes, here is a more declarative way to follow for newcomers:
from airflow.hooks.mysql_hook import MySqlHook
def fetch_records():
request = "SELECT * FROM your_table"
mysql_hook = MySqlHook(mysql_conn_id = 'the_connection_name_sourced_from_the_ui', schema = 'specific_db')
connection = mysql_hook.get_conn()
cursor = connection.cursor()
cursor.execute(request)
sources = cursor.fetchall()
print(sources)
...your DAG() as dag: code
task = PythonOperator(
task_id = 'fetch_records',
python_callable = fetch_records
)
This returns to the logs the contents of your DB query.
I hope this is of use to someone else.
Sure, just create a hook or operator and call the get_records() method: https://airflow.apache.org/docs/apache-airflow/stable/_modules/airflow/hooks/dbapi.html
I have a table that has 2 columns "date_from" and "date_to". I have to make a query which should return results where "date_to" - "date_from" == "2 weeks".
Something like this:
User.where('date_to - date_from == 2.weeks').
Define this on user:
def self.two_weeks
where(date_to: date_from + 2.weeks)
end
Then you can just call User.two_weeks
Can also be done as a scope, if you prefer:
scope :two_weeks, -> { where(date_to: date_from + 2.weeks) }
It's good to keep these kinds of things on the model anyway, so that you can change them in one place if they need to be tweaked
This should work, for mysql at least
User.where('1 + DATEDIFF(date_to, date_from) >= 14')
Apparently, I could not compare the date in the peewee SQL.
START_DATE = datetime.datetime(2015, 7, 20, 0, 0, 0)
customer_records = Customers.select().\
join(Current_Insurers, on=(Customers.current_insurer == Current_Insurers.id)).\
switch(Current_Insurers).\
join(Insurers, on=(Current_Insurers.insurer == Insurers.id)).\
where(Customers.pol_type == "PC" & \
Current_Insurers.effective_date == START_DATE )
Where Customers, Current_Insurers, Insurers are three class. The result is always 0 records. But if I removed the datetime condition from the sql and compare as follows
customer_records = Customers.select().\
join(Current_Insurers, on=(Customers.current_insurer == Current_Insurers.id)).\
switch(Current_Insurers).\
join(Insurers, on=(Current_Insurers.insurer == Insurers.id)).\
where(Customers.pol_type == "PC"
for r in customer_records:
if(r.current_insurer.effective_date == START_DATE):
print(r.policy_id)
Surprisingly we can compare now and print out customers.
What do I need to do to add the datetime condition in the peewee sql?
Many thanks,
Apparently, I could not compare the date in the peewee SQL.
That's completely incorrect. Do you honestly think that the library would be that broken??
The problem is Python operator precedence. You need to wrap the equality expressions with parentheses. So you where clause should look like this instead:
where((Customers.pol_type == "PC") & \
(Current_Insurers.effective_date == START_DATE))
Additionally, it's typically only necessary to call switch() when you have multiple joins to a single model.
Put together, your query should be:
query = (Customers
.select()
.join(Current_Insurers, on=(Customer.current_insurer == Current_Insurers.id))
.join(Insurers, on=(Current_Insurers.insurer == Insurer.id))
.where(
(Customers.pol_type == "PC") &
(Current_Insurers.effective_date == START_DATE)))
I came here because I had the same problem abd subsequently the same question.
The cause of my issue was that mariaDB was stripping the milliseconds when the original insert was done and python/peewee was passing in the milliseconds in the predicate on the later update. Very frustrating.
So I am trying to handle a fairly complex query within active record. I am using rails 4.0 and mysql. The problem is, my query requires single quotes in the WHERE statement.
This is what the sql statement (WHERE portion) needs to end up looking like:
WHERE
`location_hours`.`open_at` <=
hour(CONVERT_TZ(now(), 'US/Mountain',`locations`.`time_zone`)) * 60 * 60
AND
`location_hours`.`close_at` >=
hour(CONVERT_TZ(now(), 'US/Mountain', `locations`.`time_zone`)) * 60 * 60
The issue is with 'US/Mountain'. I have tried the following two ways to make this work:
1.
string = "hour(CONVERT_TZ(now(), 'US/Mountain', `locations`.`time_zone`)) * 60 * 60";
filtered = filtered.joins(:hours)
.where("`location_hours`.`closed` = ?", false)
.where("`location_hours`.`open_at` <= %s AND `location_hours`.`close_at` >= %s",
string, string)
2.
filtered = filtered.joins(:hours)
.where("`location_hours`.`day` = weekday(CONVERT_TZ(now(), \"US/Mountain\", `locations`.`time_zone`))")
.where("`location_hours`.`closed` = ?", false)
.where("`location_hours`.`open_at` <= hour(CONVERT_TZ(now(), \"US/Mountain\", `locations`.`time_zone`)) * 60 * 60 AND `location_hours`.`close_at` >= hour(CONVERT_TZ(now(), \"US/Mountain\", `locations`.`time_zone`)) * 60 * 60")
The sql the second option generates (copy,pasted from console) I can literally drop as a raw query into phpmyadmin, and I get results back.
For some reason Rails is doing something funny with my query. Any pointers or suggestions would be greatly appreciated!
Thanks!!
The easiest way to get single quotes into your SQL is to just put them in there:
.where("location_hours.open_at <= hour(CONVERT_TZ(now(), 'US/Mountain', ...
If the timezone is in a variable then use a placeholder:
.where("location_hours.open_at <= hour(CONVERT_TZ(now(), :tz, ...", :tz => time_zone_string)
Keep in mind that an SQL snippet such as
string = "hour(CONVERT_TZ(now(), 'US/Mountain', `locations`.`time_zone`)) * 60 * 60";
is not an SQL string and should not be escaped as such. That means that sending it through a %s won't do the right thing, the %s will just make a mess of the quotes in the snippet.
BTW, you don't need to backtick-quote everything, only identifiers that are keywords, case sensitive, contain whitespace, etc. Backticking everything just makes an ugly mess.
I write query to find out how much the user has reduced calories in a week, but I have this error.
How to avoid mistakes?
def self.calories_burned(current_user)
week = ((created_at - current_user.first_program_started_at.utc.beginning_of_day) / 86400 / 7).ceil.to_i || 1
find_by_sql("
SELECT
count(*) as cnt,
WEEK(#{week}) as week_number
FROM
user_daily_updates
WHERE
user_id=#{current_user.id}
GROUP BY
week_number
")
end
When you write this:
def self.calories_burned(current_user)
etc...
end
it's a method that can only be called like this:
MyUserModel.calories_burned(some_user)
In this case you are running the code on the class before it is instantiated, this means that the model hasn't even attached itself to a connection to the database and because of that you will not be able to access attributes that pertain to your model.
On the other hand, if you write this:
def calories_burned
etc...
end
You don't need to pass the user to the method, you simply call it on the controller after instantiating your model, like this:
id = 123
current_user = MyUserModel.find(id)
current_user.calories_burned
where current_user.calories_burned will return the value you are looking for based on the current user.
After taking a closer look at your method
it should look more like this:
def calories_burned
week = ((created_at - first_program_started_at.utc.beginning_of_day) / 86400 / 7).ceil.to_i || 1
UserDailyUpdate.where(["user_id = ?", id]).where(["week = ?",week]]).count(:all, :group => 'week_number')
end
If I understood correctly what you were trying to do with your query, this should give you the same result. Now, I should mention I am assuming that when you created the table user_daily_updates, you also created a model UserDailyUpdate.
You should take a look at ActiveRecord's documentation for searching using conditions (all of section 2) in order to have a better understanding of what I just did.
Apneadiving is right, but I would also look to refactor your code:
def self.calories_burned(current_user)
week = ((##created_at## - current_user.first_program_started_at.utc.beginning_of_day) / 86400 / 7).ceil.to_i || 1
select("#{self.table_name}.*, count(*) as cnt, WEEK(#{week}) as week_number").where(user_id: current_user.id).group(:week_number)
end
You'll need to pull created_at from the db, as it won't be present unless you have an instance of a class already loaded