I run this code in python 3.6 (pythonanywhere.com) using MySQL
import MySQLdb
conn = MySQLdb.connect('username.mysql.pythonanywhere-services.com', 'username', 'password', 'username$to_do')
c = conn.cursor()
c.execute('SELECT task FROM list')
rows = c.fetchall()
list_em = []
number = 0
for eatchRow in rows:
list_em.append(eatchRow)
print("nr %s: %s" % (number, list_em[number]))
number += 1
del1 = int(input("Chose the number of the list item you want to delete: "))
delstatmt = "DELETE FROM list WHERE task = ?"
print (list_em[del1])
#c.execute(delstatmt, (list_em[del1],))
I am creating a list called "list_em" and fill with the content from the column "task" in the table "list". I would like "print (list_em[del1])" to return 'Gå med hunden' -A clean string that I can use to run the last script that is commented out. Instead get something that looks like this with brackets and a comma(from console):
nr 0: ('Skura Trappan',)
nr 1: ('Gå med hunden.',)
Chose the number of the list item you want to delete: 1
('Gå med hunden.',)
OBS! The table does not have any ID, just two columns:
mysql> select * from list;
+-----------------+-------------+
| task | status |
+-----------------+-------------+
| Skura Trappan | Not started |
| Gå med hunden. | Not started |
+-----------------+-------------+
2 rows in set (0.00 sec)
What you're getting back from the MySQL library is a list of tuples, one for each row returned by the query. The elements of each tuple are the values of the columns you're querying for.
In this case, you're querying for one column, so each element of the list is a one-element tuple, which Python will represent as (value,) -- which is what you're seeing.
So in order to print out just the value from the tuple, you need to extract it, just like you would with a list:
for eatchRow in rows:
list_em.append(eatchRow[0])
print("nr %s: %s" % (number, list_em[number]))
number += 1
Related
Goal: run checks.yml on all Tables in Database, implicitly / dynamically (not naming 100s of Tables).
Following Soda's quick start, I've completed sections:
Install Soda Core
Connect Soda Core to a data source - configuration.yml
Now I'm following Write a check and run a scan - checks.yml.
Problem
However, the documentation only gives examples for checking one Table each.
4 Checks
Sum of Tables (in Database)
Sum of Columns (across all Tables, in Database)
Sum of Tables' descriptions exist
Sum of Columns' descriptions exist
Queries return a COUNT().
So far, checks.yml:
# checks for MY_DATABASE:
sql_metrics:
name: num_tables, num_columns
sum_tables query: |
SELECT COUNT(*)
FROM information_schema.tables
WHERE table_schema = '*';
sum_columns query: |
SELECT COUNT(*)
FROM information_schema.columns
WHERE table_name = '*';
sum_tables_descriptions query: |
-- SQL
sum_columns_descriptions query: |
-- SQL
Your checks file should look like
checks for MY_DATABASE:
- sum_tables > 0:
sum_tables query: |
SELECT COUNT(*)
FROM information_schema.tables
WHERE table_schema ~~ '%'
- sum_columns > 0:
sum_columns query: |
SELECT COUNT(*)
FROM information_schema.columns
Your checks must be based on conditions(to be checked or verified) or filters that are not available yet on documentation.
~~ means like and % is the wildcard. Although condition all and no condition gives same result, therefore where clause is not neccessary.
Soda Core 3.0.5
Scan summary:
2/2 checks PASSED:
MY_DATABASE in postgres
sum_tables > 0 [PASSED]
sum_columns > 0 [PASSED]
All is good. No failures. No warnings. No errors.
Or you could dynamically create checks files with this script, using a list of datasets(tables) with 'for each dataset T: ' clause, like:
import psycopg2
import time
#File name with timestamp
filePath = '.'
timestr = time.strftime("%Y-%m-%d-%-H%M%S")
fileName = 'checks-' + timestr + '.yml'
try:
conn = psycopg2.connect("dbname=postgres user=postgres password=yourpassword host=localhost")
cur = conn.cursor()
cur.execute("SELECT ' - '||table_name||'\n' FROM information_schema.tables WHERE table_schema = 'public' order by 1 limit 5;")
row = cur.fetchone()
with open(fileName,'w') as f:
line="for each dataset T:\n datasets:\n"
f.write(line)
while row is not None:
f.write(row[0])
row = cur.fetchone()
cur.close()
line=" checks:\n - row_count > 0"
f.write(line)
except (Exception, psycopg2.DatabaseError) as error:
print(error)
finally:
if conn is not None:
conn.close()
For a list query I want to show how many prices there are for a specific product on a specific site.
I have found a way to do this with RawSQL, something I had never ever had to do before. Am I overlooking something?
queryset = Price.objects.all()
site_annotations = {}
for site in Site.objects.all():
site_annotations['num_prices_{}'.format(site.name)] = RawSQL(
'SELECT COUNT(id) '
'FROM `product_siteprice` '
'WHERE `product_siteprice`.`price_id` = `product_price`.`id` '
'AND site_id=%s', [site.pk]
)
queryset = queryset.annotate(**site_annotations)
edit
My models look simplified like this:
class Site(Model):
name = models.CharField(_('display name'), max_length=50)
class Price(Model):
name = models.CharField()
class SitePrice(Model):
price = models.ForeignKey(Price)
site = models.ForeignKey(Site)
Now the table I want is:
Product | Nr of prices on Site A | Nr of prices on Site B |
--------|------------------------|------------------------|
Iphone | 6 | 3 |
Xperia | 42 | 66 |
And I want to sort and filter on the number of prices, so thats why I need the annotation.
NB The name Price is a bit misleading if you don't know the rest of context, in this context it can be seen more like a Product
From what I understand, you just need to filter the prices on the related site, then get the count of the number of objects.
Price.objects.filter(site_price__site_id=site.pk).count()
Although you may just want to remove the site price model and replace it with a many to many field
I think the query that you want would look something like this:
qs = SitePrice.objects.values('price_id', 'site_id').annotate(d_count=Count('id'))
And to get the table that you want, you’d format it into a dict with something like this:
from collections import defaultdict
values = defaultdict(dict)
for x in qs:
values[ x['price_id'] ][ x['site_id'] ] = x['d_count']
To print out this table, you would use:
price_ids = list(values.keys()) # you don’t need `list` for python 2
site_ids = set()
for x in values.values():
site_ids |= set(x.keys())
site_ids = list(site_ids)
site_ids.sort()
prices = dict(Price.objects.filter(id__in=price_ids).values_list('id', 'name'))
sites = dict(Site.objects.filter(id__in=site_ids).values_list('id', 'name'))
print(' ', end='') # for python 3
# print ' ', # for python 2
for x in site_ids:
print('%-8s ' % sites[x], end='')
print('')
for x, counts in values.items():
print('%-8s ' % prices[x], end='')
for site_id in site_ids:
d_count = counts.get(site_id, 0)
print('%-8s ' % d_count, end='')
print('')
Suppose I have a simple database table that doesn't have an ID_KEY but has a name column. I want to display the output like this
+----+---------+
| | name |
+----+---------+
| 1 | dog |
| 2 | cat |
| 3 | penguin |
| 4 | lax |
| 5 | whale |
| 6 | ostrich |
+----+---------+
Then have a <STDIN> for like, say, 3 to select penguin. 3 is just the line number that appears when you do the select call.
Is there any way to do this, or is it only possible with an id key associated and then a subsequent select statement matching that id key?
I misunderstood you at first but I've caught on. But it doesn't make much sense, as when you're entering a number into a Perl program you won't be working with the MySQL command-line tool, and won't be able to see what numbers to enter..
What you need to do is to write your Perl program so that it prints all the name fields from the table together with a line number. Then your program can translate from an input animal number to its name because it knows what it printed.
Something like this would work. Of course you will have to set the name, IP address and credentials correctly so that DBI can connect to the database.
use strict;
use warnings;
use DBI;
my $dbh = DBI->connect(
'DBI:mysql:database=animal_test',
'username',
'password',
{ RaiseError => 1 },
);
my $names = map #$_, $dbh->selectall_arrayref('SELECT name FROM animals');
for my $i ( 0 .. $#$names ) {
printf "%3d: %s\n", $i+1, $names->[$i];
}
print "\n";
print "Enter animal number: ";
my $animal = <>;
chomp $animal;
my $name = $names->[$animal-1];
printf "Animal chosen is %s\n", $name;
Option 1 - You would have put a id field in the DB if you want to find by integer 3 because row 3 will not always be penguin from an SQL query.
Option 2 - Dump the data into and array or hash and use the index of that to find the item from with in the variable and not the DB when 3 is captured from STIN.
Just use query:
my $select = $dbh->prepare('
SET #id:=0;
SELECT name,
#id = #id+1
FROM table
');
I need to make a query where I will be looking for a specific string through several columns and I need to know which column (name) contains the value that I need.
In the example below I need a query where I can ask which column contains the value 1000000000002101214 and that it returns f1. DB is MySQL and I need the programming to be done in Perl.
+---------------------+---------------------+---------------------+
| f1 | f2 | f3 |
+---------------------+---------------------+---------------------+
| 1000000000002101214 | 1000000000001989129 | 1000000000001881637 |
| 1000000000002080453 | 1000000000001968481 | 1000000000001862284 |
| 1000000000002085919 | 1000000000001973677 | 1000000000001866854 |
| 1000000000002075076 | 1000000000001963189 | 1000000000001857288 |
+---------------------+---------------------+---------------------+
I was able to find an almost-answer to my question from another site where I could get the column names of the fields in the table with the following:
my #cols = #{$sth->{NAME}}; # or NAME_lc if needed
while (#row = $sth->fetchrow_array) {
print "#row\n";
}
$sth->finish;
foreach ( #cols ) {
printf( "Note: col : %s\n", $_ );
}
The problem is partially resolved. In the example table I provided in the original question I needed to know on which column my answer resides, the query contains several OR statemens:
SELECT * FROM table WHERE (f1='1000000000002101214' OR f2='1000000000002101214' OR f3='1000000000002101214')
And I need the result to show that the column name where the number is located is f1. So....
Any thoughts?
I don't even know where to start
Check out Perl's DBI module. Read the documentation. You'll have to do something like below:
#!/usr/bin/perl
use strict;
use warnings;
use DBI;
#Connect to your database, provide id, password
my $dbh = DBI->connect('dbi:mysql:perltest','root','password') or die "Connection Error: $DBI::errstr\n";
#Write your query
my $sql = "select * from database_schema";
my $sth = $dbh->prepare($sql);
#Execute it
$sth->execute or die "SQL Error: $DBI::errstr\n";
#Fetch the value
while (my #row = $sth->fetchrow_array) {
#Do something with your result
print "#row\n";
}
If you are new to Perl then see: http://learn.perl.org/
Edit: Query to find out column name based on the value found in column.
Select 'f1'
from database_schema
where database_schema.f1 = 1000000000002101214
union
Select 'f2'
from database_schema
where database_schema.f2 = 1000000000002101214
union
Select 'f3'
from database_schema
where database_schema.f3 = 1000000000002101214
I have a table with millions of rows and a single column of text that is exactly 11,159 characters long. It looks like this:
1202012101...(to 11,159 characters)
1202020120...
0121210212...
...
(to millions of rows)
I realize that I can use
SELECT SUBSTR(column,2,4) FROM table;
...if I wanted to pull out characters 2, 3, 4, and 5:
1202012101...
1202020120...
0121210212...
^^^^
But I need to extract noncontiguous characters, e.g. characters 1,5,7:
1202012101...
1202020120...
0121210212...
^ ^ ^
I realize this can be done with a query like:
SELECT CONCAT(SUBSTR(colm,1,1),SUBSTR(colm,5,1),SUBSTR(colm,7,1)) FROM table;
But this query gets very unwieldy to build for thousands of characters that I need to select. So for the first part of the question - how do I build a query that does something like this:
SELECT CHARACTERS(string,1,5,7) FROM table;
Furthermore, the indices of the characters I want to select are from a different table that looks something like this:
char_index keep_or_discard
1 keep
2 discard
3 discard
4 discard
5 keep
7 discard
8 keep
9 discard
10 discard
So for the second part of the question, how could I build a query to select specific characters from the first table based on whether keep_or_discard="keep" for that character's index in the second table?
this function does what you want:
CREATE DEFINER = `root`#`localhost` FUNCTION `test`.`getsubset`(selection mediumtext, longstring mediumtext)
RETURNS varchar(200)
LANGUAGE SQL
NOT DETERMINISTIC
CONTAINS SQL
SQL SECURITY DEFINER
COMMENT 'This function returns a subset of characters.'
BEGIN
SET #res:='';
SET #selection:=selection;
WHILE #selection<>'' DO
set #pos:=CONVERT(#selection, signed);
set #res := concat_ws('',#res,SUBSTRING(longstring,#pos,1));
IF LOCATE(',',#selection)=0 THEN
SET #selection:='';
END IF;
set #selection:=SUBSTRING(#selection,LOCATE(',',#selection)+1);
END WHILE;
RETURN #res;
END
Note: the CONVERT('1,2,3,4',signed) will yield 1, but it will give a warning.
I have it defined to be available in the database test.
The function takes two parameters; a string(!) with a list of positions, and a long string from where you want the characters taken.
An example of using this:
mysql> select * from keepdiscard;
+---------+------------+
| charind | keepordisc |
+---------+------------+
| 1 | keep |
| 2 | discard |
| 3 | keep |
| 4 | discard |
| 5 | keep |
| 6 | keep |
+---------+------------+
6 rows in set (0.00 sec)
mysql> select * from test;
+-------------------+
| longstring |
+-------------------+
| abcdefghijklmnopq |
| 123456789 |
+-------------------+
2 rows in set (0.00 sec)
mysql> select getsubset(group_concat(charind ORDER BY charind),longstring) as result from keepdiscard, test where keepordisc='keep' group by longstring;
+--------+
| result |
+--------+
| 1356 |
| acef |
+--------+
2 rows in set, 6 warnings (0.00 sec)
The warnings stem from the fast conversion to integer that is done in the function. (See comment above)
How about dynamic sql? (You will need to build the select part of the query)
CREATE PROCEDURE example_procedure()
BEGIN
--
--build the concat values here
--
SET #ids := '';
SET #S = 'SELECT #ids := built_concat_of_values FROM table';
PREPARE n_StrSQL FROM #S;
EXECUTE n_StrSQL;
DEALLOCATE PREPARE n_StrSQL;
END
You can write a php script to do this for you:
<?php
//mysql connect
$conn = mysql_connect('localhost', 'mysql_user', 'mysql_password');
if (!$conn) {
echo 'Unable to connect to DB: ' . mysql_error();
exit;
}
//database connect
$db = mysql_select_db('mydb');
if (!$db) {
echo 'Unable to select mydb: ' . mysql_error();
exit;
}
//get the keep numbers you’re going to use.
//and change the number into string so, for example, instead of 5 you get 'SUBSTR(colm,5,1)'
$result = mysql_query("SELECT number FROM number_table WHERE keep_or_discard='keep'");
$numbers = array();
while ($row = mysql_fetch_assoc($result)) {
$row = 'SUBSTR(colm,' . $row . ',1)';
$numbers = $row;
}
//implode the array so you get one long string with all the substrings
//eg. 'SUBSTR(colm,1,1),SUBSTR(colm,5,1),SUBSTR(colm,12,1)'
$numbers = implode(",", $numbers);
//pull the numbers you need and save them to an array.
$result = mysql_query("SELECT " . $numbers . " FROM table");
$concat = array();
while ($row = mysql_fetch_assoc($result)) {
$concat= $row;
}
And there you have an array with the correct numbers.
I'm sorry if you can't/don't want to use PHP for this, I just don't really know how to do this without PHP, Perl, Python or some other similar language. Hopefully this solution will help somehow...
The source of your difficulty is that your schema does not represent the true relationships between the data elements. If you wanted to achieve this with "pure" SQL, you would need a schema more like:
table
ID Index Char
1 0 1
1 1 2
1 2 0
charsToKeep
ID Index Keep
1 0 false
1 1 true
1 2 true
Then, you could perform a query like:
SELECT Char FROM table t JOIN charsToKeep c ON t.ID = c.ID WHERE c.Keep = true
However, you probably have good reasons for structuring your data the way you have (my schema requires much more storage space per character and the processing time is also probably much longer from what I am about to suggest).
Since SQL does not have the tools to understand the schema you have embedded into your table, you will need to add them with a user-defined function. Kevin's example of dynamic SQL may also work, but in my experience this is not as fast as a user-defined function.
I have done this in MS SQL many times, but never in MySql. You basically need a function, written in C or C++, that takes a comma-delimited list of the indexes you want to extract, and the string from which you want to extract them from. Then, the function will return a comma-delimited list of those extracted values. See these links for a good starting point:
http://dev.mysql.com/doc/refman/5.1/en/adding-functions.html
http://dev.mysql.com/doc/refman/5.1/en/adding-udf.html
To build the concatenated list of indexes you want to extract from the char_index table, try the group_concat function:
http://dev.mysql.com/doc/refman/5.0/en/group-by-functions.html#function_group-concat
Hope this helps!