I have been trying to retrieve data from my database. I was successful, however, this time inside an if statement. The code looks like:
cur_msql = conn_mysql.cursor(cursor=pymysql.cursors.DictCursor)
select_query = """select x,y,z from table where type='sample' and code=%s"""
cur_msql.execute(select_query, code)
result2 = cur_msql.fetchone()
if(result2==None):
insert_func(code)
select_query = f"""select x,y,z from table where type='sample' and code='{code}'"""
mycur = conn_mysql.cursor(cursor=pymysql.cursors.DictCursor)
print(select_query)
mycur.execute(select_query)
result3 = mycur.fetchone()
if(result2==None):
result2=result3
Now I see that insert_func does successfully insert into the 'table'. However, on trying to fetch that row, immediately after the insertion, it returns None as if the row is absent. On debugging I find that result3 is also None. Nothing looks wrong to me but it's not working.
you don´t execute it in the right way, in the cur_msql.execute, you the to send the query and a tuple of values, and you are sending just a value:
cur_msql = conn_mysql.cursor(cursor=pymysql.cursors.DictCursor)
select_query = "select learnpath_code,learnpath_id,learnpath_name from contentgrail.knowledge_vectors_test where Type='chapters' and code=%s"
cur_msql.execute(select_query, (meta['chapter_code'],))
result2 = cur_msql.fetchone()
Related
I am querying my SQL table using the code below and converted the result to a list. Why is the list having unwanted commas and parenthesis?
The query result
[(34830,), (34650,), (35050,), (34500,), (35050,), (34500,), (34725,), (34550,), (34725,), (34760,), (34760,)]
It should just return a list with just numbers on it. Right?
The schema is simple (link text, price int);
What is the problem here? Is there something wrong with my code?
import pymysql
connection = pymysql.connect(host='localhost',
user='root',
password='passme',
db='hpsize') # connection obhect to pass the database details
sql = "SELECT price FROM dummy WHERE link ='https://www.flipkart.com/bose-noise-cancelling-700-anc-enabled-bluetooth-headset/p/itma57a01d3bd591?pid=ACCFGYZEVVGYM8FP'"
my_cursor = connection.cursor()
my_cursor.execute(sql)
result = list(my_cursor.fetchall())
print(result)
connection.close()
The query result
[(34830,), (34650,), (35050,), (34500,), (35050,), (34500,), (34725,), (34550,), (34725,), (34760,), (34760,)]
try
connection.row_factory = lambda cursor, row: row[0]
instead of list(my_cursor.fetchall())
then
result = connection.execute("""SELECT * FROM dummy""").fetchall()
or you can also use strip() to cut the unwanted part
I need to
1. run a select query on MYSQL DB and fetch the records.
2. Records are processed by python script.
I am unsure about the way I should proceed. Is xcom the way to go here? Also, MYSQLOperator only executes the query, doesn't fetch the records. Is there any inbuilt transfer operator I can use? How can I use a MYSQL hook here?
you may want to use a PythonOperator that uses the hook to get the data,
apply transformation and ship the (now scored) rows back some other place.
Can someone explain how to proceed regarding the same.
Refer - http://markmail.org/message/x6nfeo6zhjfeakfe
def do_work():
mysqlserver = MySqlHook(connection_id)
sql = "SELECT * from table where col > 100 "
row_count = mysqlserver.get_records(sql, schema='testdb')
print row_count[0][0]
callMYSQLHook = PythonOperator(
task_id='fetch_from_testdb',
python_callable=mysqlHook,
dag=dag
)
Is this the correct way to proceed?
Also how do we use xcoms to store the records for the following MySqlOperator?'
t = MySqlOperator(
conn_id='mysql_default',
task_id='basic_mysql',
sql="SELECT count(*) from table1 where id > 10",
dag=dag)
I was really struggling with this for the past 90 minutes, here is a more declarative way to follow for newcomers:
from airflow.hooks.mysql_hook import MySqlHook
def fetch_records():
request = "SELECT * FROM your_table"
mysql_hook = MySqlHook(mysql_conn_id = 'the_connection_name_sourced_from_the_ui', schema = 'specific_db')
connection = mysql_hook.get_conn()
cursor = connection.cursor()
cursor.execute(request)
sources = cursor.fetchall()
print(sources)
...your DAG() as dag: code
task = PythonOperator(
task_id = 'fetch_records',
python_callable = fetch_records
)
This returns to the logs the contents of your DB query.
I hope this is of use to someone else.
Sure, just create a hook or operator and call the get_records() method: https://airflow.apache.org/docs/apache-airflow/stable/_modules/airflow/hooks/dbapi.html
I'm having a problem with running a select query, using mysql-python, on an established database. The issue is that a number, what Python refers to as a long, is returned instead of the data queried- it should be noted that this number corresponds to the number of records which should be returned (I logged into the database and ran the query from MySQL to make sure).
Here is the code:
db = MySQLdb.connect(db = 'testdb', user='testuser', passwd='test', host='localhost', charset='utf8', use_unicode=True)
dbc = db.cursor()
result = dbc.execute("""SELECT %s FROM example_movie""", ('title',))
urls = [row[0] for row in result]
The last bit of code, urls = [row[0] for row in result] is to put everything into a list.
The error looks like this:
TypeError: 'long' object is not iterable
When I have python print result it returns:
('RESULT:', 52L)
When I enclose result like str(result) it just returns the number 52 (not long)
Any help and suggestions are greatly appreciated!
The return value from dbc.execute is not the results of the select; I believe it is the number of rows in the results. In order to get the actual results you need to call one of the fetch methods. See documentation here.
You should update your code to read:
db = MySQLdb.connect(db = 'testdb', user='testuser', passwd='test', host='localhost', charset='utf8', use_unicode=True)
dbc = db.cursor()
row_count = dbc.execute("""SELECT title FROM example_movie""")
results = dbc.fetchall()
urls = [row[0] for row in result]
I have 2 tables that I need to load together all the time, the both must exist together in the database. However I am wondering why Linq to Sql demands that I have to load in a collection and then do a join, I only want to join 2 single tables where a record where paramid say = 5, example...
var data = _repo.All<TheData>(); //why do I need a collection/IQueryable like this?
var _workflow = _repo.All<WorkFlow>()
.Where(x => x.WFID== paramid)
.Join(data, x => x.ID, y => y.WFID, (x, y) => new
{
data = x,
workflow = y
});
I gues then I need to do a SingleOrDefault()? If the record is not null pass it back?
I Understand the Sql query comes out correctly, is there a better way to write this?
NOTE: I need to search a table called Participants to see if the loggedonuser can actually view this record, so I guess I should leave it as this? (this is main requirement)
var participant = _repo.All<Participants>();
.Any(x=> x.ParticipantID == loggedonuser.ID); //add this to above query...
The line var data = _repo.All<TheData>(); is something like saying 'start building query against the TheData table'.
This function returns you an IQueryable which will contain a definition of the query against your database.
So this doesn't mean you load the whole TheData table data with this line!
The query will be executed the moment you do something like .Count(), .Any(), First(), Single(), or ToList(). This is called deferred execution.
If you would end your query with SingleOrDefault() this will create a sql query that joins the two tables, add the filter and select the top most record or null(or throw an error if there are more!).
You could also use Linq instead of query extension methods.
It would look like:
var data = _repo.All<TheData>();
var _workflow = from w in _repo.All<WorkFlow>()
join t in _repo.All<TheData> on w.Id equals t.WFID
where x.WIFD = paramid
select new
{
data = t,
workflow = x
});
I am trying to get the records from the 'many' table of a one-to-many relationship and add them as a list to the relevant record from the 'one' table.
I am also trying to do this in a single database request.
Code derived from Linq to Sql - Populate JOIN result into a List almost achieves the intended result, but makes one database request per entry in the 'one' table which is unacceptable. That failing code is here:
var res = from variable in _dc.GetTable<VARIABLE>()
select new { x = variable, y = variable.VARIABLE_VALUEs };
However if I do a similar query but loop through all the results, then only a single database request is made. This code achieves all goals:
var res = from variable in _dc.GetTable<VARIABLE>()
select variable;
List<GDO.Variable> output = new List<GDO.Variable>();
foreach (var v2 in res)
{
List<GDO.VariableValue> values = new List<GDO.VariableValue>();
foreach (var vv in v2.VARIABLE_VALUEs)
{
values.Add(VariableValue.EntityToGDO(vv));
}
output.Add(EntityToGDO(v2));
output[output.Count - 1].VariableValues = values;
}
However the latter code is ugly as hell, and it really feels like something that should be do-able in a single linq query.
So, how can this be done in a single linq query that makes only a single database query?
In both cases the table is set to preload using the following code:
_dc = _db.CreateLinqDataContext();
var loadOptions = new DataLoadOptions();
loadOptions.LoadWith<VARIABLE>(v => v.VARIABLE_VALUEs);
_dc.LoadOptions = loadOptions;
I am using .NET 3.5, and the database back-end was generated using SqlMetal.
This link may help
http://msdn.microsoft.com/en-us/vcsharp/aa336746.aspx
Look under join operators. You'll probably have to change from using extension syntax other syntax too. Like this,
var = from obj in dc.Table
from obj2 in dc.Table2
where condition
select