How to do pivot table without knowledge of columns - mysql

I have read most of the posts on stackoverflow on how to do a pivot table but all of the posts show examples with prior knowledge of the columns. How do you construct a query if you have no knowledge of what the columns will be. here is some sample data:
id column value Row
1 age 13 1
2 height 55 1
3 fav number NULL 1
4 siblings 4 1
5 age 55 2
6 height 54 2
7 fav number 12 2
I am looking for this output:
row age height fav number siblings
1 13 55 NULL 4
2 55 54 12 NULL
As you can see there is no row 2 is missing an entry for siblings. The column names are unknown at the time of the query. How would you make this query.

I don't see any way you can just write some fancy SELECT query to get what you want. You're going to have to do some pre-processing.
You have to be executing this MySQL query from some sort of program, application, script, etc. Not sure what the language is, but here's what I would do in PHP:
/* $data is where our data is going to be stored in our desired format */
$data = array();
/* $columns is a list of all column names */
$columns = array();
/* $rows is a list of all row names (probably '1', '2', etc) */
$rows = array();
$result = mysql_query('SELECT column, value, row FROM TableName');
while ($row = mysql_fetch_array($result, MYSQL_ASSOC)) {
/* if this row isn't in $data yet, add it */
if (!array_key_exists($row['row'], $data) {
$data[$row['row']] = array();
}
/* if this column isn't in $columns yet, add it */
if (!in_array($row['column'], $columns)) {
array_push($columns, $row['column']);
}
/* if this row isn't in $rows yet, add it */
if (!in_array($row['row'], $rows)) {
array_push($rows, $row['row']);
}
/* set the actual value in our multi-dimensional array $data */
$data[$row['row']][$row['column']] = $row['value'];
}
/* free the result (php specific thing) */
mysql_free_result($result);
/* if we didn't set anything (row, column) pairs, set it to null in $data */
foreach ($rows as $r) {
foreach ($columns as $c) {
if (!array_key_exists($c, $data[$r])) {
$data[$r][$c] = null;
}
}
}
This will put all the data into a format you want in an array in PHP.
For example, after running this algorithm on the sample data you provided above, you would be able to do:
echo $data['2']['age']; // $data['row']['column']
Which would output 55.
OR if your database isn't being updated in real time (you have a bunch of data that you'd like to reformat once, rather than continuously), you could extend the script above to also have some "CREATE TABLE", "INSERT INTO" queries that basically recreate the table in the format you're looking for.
Furthermore, if you ARE receiving data in realtime, you can still write the script described above, but you'd just want to remove the rows from the original table as you processed them, and then just run the script whenever data is being put into the original table.

I doubt that you can do this in MySQL or PostgreSQL as you expect to, however there is an alternative which I have used where data is very free-form. Our use case is "attributes that the menu items pass back to the application" and of course we have no knowledge of these in the query. But you can't create a simple pivot table.
The reason you can't is that PostgreSQL requires that the tuple structure returned to be defined in advance. Not all db's do this (Informix for example, allows different rows to have different structures!) but most do.
Our approach is PostgreSQL-only. However, maybe with some tweaking you can find a MySQL equivalent somewhere. What we did would basically be in your version:
select row, array_agg("column" || '=' || "value") from sample_data group by row;
This produces output like:
1 {"age=3","height=55",null,"siblings=4"}
2 {"age=55","height=54","favorite_number=12"}
You can even get rid of the NULLS by:
select row, array_agg("column" || '=' || "value")
WHERE value is not null
GROUP BY row;
Then you get something like:
1 {"age=3","height=55","siblings=4"}
2 {"age=55","height=54","favorite_number=12"}
I don't know how to do the same in MySQL though.

Related

fast execution analitycs database with thousand rows to displaying in php

i've table with a thousand rows and i want to creating analitycs with chart display my front end php
my table structure is
and how i display this data :
by user_agent column i display operating system, browsers, and devices.
for now i still using the old algorithm with looping using for () method and parsing each rows. And it takes a long time respond and displaying the data.
anyone knows how i can display this data without take long respond in my website? any idea? with the database structure or my php script?
Thankyou before.
Assuming you're loading all your data in a PHP script and postprocessing it in a for-loop in PHP, you should alter your database query. A GROUP BY statement might help. Of course, you need to alter your script to work with the new data. Revisiting your database structure is a good idea, too. A better approach might be not to save the whole user-agent string in one column but to use several columns.
Example before:
$data = $db->query('SELECT * FROM table');
for ($i = 0; $i <= $data->max(); i++) {
$row = $data->getRow($i);
postprocessRow($row); /* $sum += 1; */
}
Example after:
$data = $db->query('SELECT count(*) as weight, * FROM table GROUP BY user_agent');
for ($i = 0; $i <= $data->max(); i++) {
$row = $data->getRow($i);
postprocessRowWeighted($row); /* $sum += $row['weight']; */
}

How optimize the research of next free "slot" in mysql?

i've a problem and i can't find an easy solution.
I have self expanding stucture made in this way.
database1 | table1
| table2
....
| table n
.
.
.
databaseN | table 1
table 2
table n
each table has a structire like this:
id|value
each time a number is generated is put into the right database/table/structure (is divided in this way for scalability... would be impossible to manage table of billions of records in a fas way).
the problem that N is not fixed.... but is like a base for calculating numbers (to be precise N is known....62 but I can onlyuse a subset of "digits" that could be different in time).
for exemple I can work only with 0 1 and 2 and after a while (when I've done all the possibilities) I want to add 4 and so on (up to base 62).
I would like to find a simple way to find the 1st free slot to put the next randomly generated id but that could be reverted.
Exemple:
I have 0 1 2 3 as numbers I want use....
the element 2313 is put on dabase 2 table 3 and there will be 13|value into table.
the element 1301 is put on dabase 1 table 3 and there will be 01|value into table.
I would like to generate another number based on the next free slot.
I could test every slot starting from 0 to the biggest number but when there will be milions of records for every database and table this will be impossible.
the next element of the 1st exemple would be 2323(and not 2314 since I'm using only the 0 1 2 3 digits).
I would like som sort of invers code in mysql to give me the 23 slot on table 3 database 2 to transform it into the number. I could randomly generate a number and try to find the nearest free up and down but since the set is variable could not be a good choice.
I hope it will be clear enought to tell me any suggestion ;-)
Use
show databases like 'database%' and a loop to find non-existent databases
show tables like 'table%' and a loop for tables
select count(*) from tableN to see if a table is "full" or not.
To find a free slot, walk the database with count in chunks.
This untested PHP/MySQL implementation will first fill up all existing databases and tables to base N+1 before creating new tables or databases.
The if(!$base) part should be altered if another behaviour is wanted.
The findFreeChunk can also be solved with iteration; but I leave that effort to You.
define (DB_PREFIX, 'database');
define (TABLE_PREFIX, 'table');
define (ID_LENGTH, 2)
function findFreeChunk($base, $db, $table, $prefix='')
{
$maxRecordCount=base**(ID_LENGTH-strlen($prefix));
for($i=-1; ++$i<$base;)
{
list($n) = mysql_fetch_row(mysql_query(
"select count(*) from `$db`.`$table` where `id` like '"
. ($tmp = $prefix. base_convert($i, 10, 62))
. "%'"));
if($n<$maxRecordCount)
{
// incomplete chunk found: recursion
for($k=-1;++$k<$base;)
if($ret = findFreeChunk($base, $db, $table, $tmp)
{ return $ret; }
}
}
}
function findFreeSlot($base=NULL)
{
// find current base if not given
if (!$base)
{
for($base=1; !$ret = findFreeSlot(++$base););
return $ret;
}
$maxRecordCount=$base**ID_LENGTH;
// walk existing DBs
$res = mysql_query("show databases like '". DB_PREFIX. "%'");
$dbs = array ();
while (list($db)=mysql_fetch_row($res))
{
// walk existing tables
$res2 = mysql_query("show tables in `$db` like '". TABLE_PREFIX. "%'");
$tables = array ();
while (list($table)=mysql_fetch_row($res2))
{
list($n) = mysql_fetch_row(mysql_query("select count(*) from `$db`.`$table`"));
if($n<$maxRecordCount) { return findFreeChunk($base, $db, $table); }
$tables[] = $table;
}
// no table with empty slot found: all available table names used?
if(count($tables)<$base)
{
for($i=-1;in_array($tmp=TABLE_PREFIX. base_convert(++$i,10,62),$tables););
if($i<$base) return [$db, $tmp, 0];
}
$dbs[] = $db;
}
// no database with empty slot found: all available database names used?
if(count($dbs)<$base)
{
for($i=-1;in_array($tmp=DB_PREFIX.base_convert(++$i,10,62),$dbs););
if($i<$base) return [$tmp, TABLE_PREFIX. 0, 0];
}
// none: return false
return false;
}
If you are not reusing your slots or not deleting anything, you can of course dump all this and simply remember the last ID to calculate the next one.

Display selected amount of records from database

I need a page that displays records from a database, sorted by their jobs. So the database holds different kind of persons with different jobs. For example, on the page "teacher" I just want to display all the teachers, not the other persons. I want to have 3 persons on a page and a button "previous" and "next" beneath it. If an user clicks the next-button I want the next 3 records to be shown. When the user reaches the last records, I need the next-button to disappear. Same goes for "previous".
What I have so far:
This piece of code creates a value named startrow.
if (!isset($_GET['startrow']) or !is_numeric($_GET['startrow'])) {
//give the value of the starting row 0 because nothing was found in URL
$startrow = 0;
//otherwise take the value from the URL
} else {
$startrow = (int)$_GET['startrow'];
}
The query I have:
$sql = mysql_query("SELECT * FROM $tbl_name WHERE jobs='teacher' LIMIT $startrow, 3")or
die(mysql_error());
$sql2 = "SELECT COUNT(*) AS TotalJobs FROM $tbl_name WHERE jobs='teacher'";
$result_count = mysql_query($sql2);
$count = mysql_fetch_array($result_count);
I created the $count for the if / else function in the next part (the part that doesn't seem to work the way I want it to):
if ($startrow < $count )
echo 'Next';
$prev = $startrow - 3;
//only print a "Previous" link if a "Next" was clicked
if ($prev >= 0)
echo 'Previous';
So the previous-button seems to work the way it should. The next doesn't. I've created the
if ($startrow < $count)
so basicly, if the value of startrow is smaller than the total number of records, it puts a next-button. But if I test this, it displays the next-button anyhow, no matter the value of startrow.
What am I missing here?
mysql_fetch_array returns an array containing the count, not the value itself. Use $count['TotalJobs'] instead.
Also, you shouldn't use the mysql extension anymore, instead use PDO or MySQLi.
Edit
You can change:
$count = mysql_fetch_array($result_count);
to
$countArr = mysql_fetch_array($result_count);
$count = $countArr['TotalJobs'];

MySql: Best way to run high number of search queries on a table

I have two tables, one is static database that i need to search in, the other is dynamic that i will be using to search the first database. Right now i have two separate queries. First on page load, values from second table are passed to first one as search term, and i am "capturing" the search result using cURL. This is very inefficient and probably really wrong way to do it, so i need help in fixing this issue. Currently page (html, front-end) takes 40 seconds to load.
Possible solutions: Turn it into function, but still makes so many calls out. Load table into memory and then run queries and unload cache once done. Use regexp to help speed up query? Possible join? But i am a noob so i can only imagine...
Search script:
require 'mysqlconnect.php';
$id = NULL;
if(isset($_GET['n'])) { $id = mysql_real_escape_string($_GET['n']); }
if(isset($_POST['n'])) { $id = mysql_real_escape_string($_POST['n']); }
if(!empty($id)){
$getdata = "SELECT id, first_name, last_name, published_name,
department, telephone FROM $table WHERE id = '$id' LIMIT 1";
$result = mysql_query($getdata) or die(mysql_error());
$num_rows = mysql_num_rows($result);
while($row = mysql_fetch_array($result, MYSQL_ASSOC))
{
echo <<<PRINTALL
{$row[id]}~~::~~{$row[first_name]}~~::~~{$row[last_name]}~~::~~{$row[p_name]}~~::~~{$row[dept]}~~::~~{$row[ph]}
PRINTALL;
}
}
HTML Page Script:
require 'mysqlconnect.php';
function get_data($url)
{
$ch = curl_init();
$timeout = 5;
curl_setopt($ch,CURLOPT_URL,$url);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch,CURLOPT_CONNECTTIMEOUT,$timeout);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
$getdata = "SELECT * FROM $table WHERE $table.mid != '1'ORDER BY $table.$sortbyme $o LIMIT $offset, $rowsPerPage";
$result = mysql_query($getdata) or die(mysql_error());
while($row = mysql_fetch_array($result, MYSQL_ASSOC))
{
$idurl = 'http://mydomain.com/dir/file.php?n='.$row['id'].'';
$p_arr = explode('~~::~~',get_data($idurl));
$p_str = implode(' ',$p_arr);
//Use p_srt and p_arr if exists, otherwise just output rest of the
//html code with second table values
}
As you can see, second table may or may not have valid id, hence no results but second table is quiet large, and all in all, i am reading and outputting 15k+ table cells. And as you can probably see from the code, i have tried paging but that solution doesn't fit my needs. I have to have all of the data on client side in single html page. So please advice.
Thanks!
EDIT
First table:
id_row id first_name last_name dept telephone
1 aaa12345 joe smith ANS 800 555 5555
2 bbb67890 sarah brown ITL 800 848 8848
Second_table:
id_row type model har status id date
1 ATX Hybrion 88-85-5d-id-ss y aaa12345 2011/08/12
2 BTX Savin none n aaa12345 2010/04/05
3 Full Hp 44-55-sd-qw-54 y ashley a 2011/07/25
4 ATX Delin none _ smith bon 2011/04/05
So the second table is the one that gets read and displayed, first is read and info displayed if ID is positive match. ID is only unique in the first one, second one has multi format input so it could or could not be ID as well as could be duplicate ID. Hope this gives better understanding of what i need. Thanks again!
A few things:
Curl is completely unnecessary here.
Order by will slow down your queries considerably.
I'd throw in an if is_numeric check on the ID.
Why are you using while and mysql_num_rows when you're limiting to 1 in the query?
Where are $table and these other things being set?
There is code missing.
If you give us the data structure for the two tables in question we can help you with the queries, but the way you have this set up now, I'm surprised its even working at all.
What you're doing is, for each row in $table where mid!=1 you're executing a curl call to a 2nd page which takes the ID and queries again. This is really really bad, and much more convoluted than it needs to be. Lets see your table structures.
Basically you can do:
select first_name, last_name, published_name, department, telephone FROM $table1, $table2 WHERE $table1.id = $table2.id and $table2.mid != 1;
Get rid of the curl, get rid of the exploding/imploding.

MySQL - Perl: How to get array of zip codes within submitted "x" miles of submitted "zipcode" in Perl example

I have found many calculations here and some php examples and most are just over my head.
I found this example:
SELECT b.zip_code, b.state,
(3956 * (2 * ASIN(SQRT(
POWER(SIN(((a.lat-b.lat)*0.017453293)/2),2) +
COS(a.lat*0.017453293) *
COS(b.lat*0.017453293) *
POWER(SIN(((a.lng-b.lng)*0.017453293)/2),2))))) AS distance
FROM zips a, zips b
WHERE
a.zip_code = '90210' ## I would use the users submitted value
GROUP BY distance
having distance <= 5; ## I would use the users submitted value
But, I am having trouble understanding how to implement the query with my database.
It looks like that query has all I need.
However, I cannot even find/understand what b.zip_code actually is! (whats the b. and zips a, zips b?)
I also do not need the state in the query.
My mySQL db structure is like this:
ZIP | LAT | LONG
33416 | 26.6654 | -80.0929
I wrote this in attempt to return some kind of results (not based on above query) but, it only kicks out one zip code.
## Just for a test BUT, in reality I desire to SELECT a zip code WHERE ZIP = the users submitted zip code
## not by a submitted lat lon. I left off the $connect var, assume it's there.
my $set1 = (26.6654 - 0.20);
my $set2 = (26.6654 + 0.20);
my $set3 = (-80.0929 - 0.143);
my $set4 = (-80.0929 + 0.143);
my $test123 = $connect->prepare(qq{SELECT `ZIP` FROM `POSTAL`
WHERE `LAT` >= ? AND `LAT` <= ?
AND `LONG` >= ? AND `LONG` <= ?}) or die "$DBI::errstr";
$test123->execute("$set1","$set2","$set3","$set4") or die "$DBI::errstr";
my $cntr;
while(#zip = $test123->fetchrow_array()) {
print qq~$zip[$cntr]~;
push(#zips,$zip[$cntr]);
$cntr++;
}
As you can see, I am quite the novice so, I need some hand holding here with verbose explanation.
So, in Perl, how can I push zip codes into an array from a USER SUBMITTED ZIP CODE and user submitted DISTANCE in miles. Can be a square instead of a circle, not really that critical of a feature. Faster is better.
I'll tackle the small but crucial part of the question:
However, I cannot even find/understand what b.zip_code actually is! (whats the "b." and "zips a, zips b"?)
Basically, the query joins two tables. BUT, both tables being joined are in fact the same table - "zips" (in other words, it joins "zips" table to itself"). Of course, since the rest of the query needs to understand when you are referring to the first copy of the "zips" table and when to the second copy of the "zips" table, you are giving a table alias to each copy - to wit, "a" and "b"'.
So, "b.xxx" means "column xxx from table zips, from the SECOND instance of that table being joined".
I don't see what's wrong with your first query. You have latitude and longitude in your database (if I'm understanding, you're comparing a single entry to all others). You don't need to submit or return the state that's just part of the example. Make the first query work like this:
my $query = "SELECT b.zip_code,
(3956 * (2 * ASIN(SQRT(
POWER(SIN(((a.lat-b.lat)*0.017453293)/2),2) +
COS(a.lat*0.017453293) *
COS(b.lat*0.017453293) *
POWER(SIN(((a.lng-b.lng)*0.017453293)/2),2))))) AS distance
FROM zips a, zips b WHERE
a.zip_code = ?
GROUP BY distance having distance <= ?";
my $sth = $dbh->prepare($query);
$sth->execute( $user_submitted_zip, $user_submitted_distance );
while( my ($zip, $distance) = $sth->fetchrow() ) ) {
# do something
}
This won't be that fast, but if you have a small record set ( less than 30k rows ) it should be fine. If you really want to go faster you should look into a search engine such as Sphinx which will do this for you.
fetchrow_array returns a list of list references, essentially a two-dimensional array, where each row represents a different result from the database query and each column represents a field from the query (in your case, there is only one field, or column, per row).
Calling while ($test123->fetchrow_array()) will cause an infinite loop as your program executes the query over and over again. If the query returns results, then the while condition will be satisfied and the loop will repeat. The usual idiom would be to say something more like for my $row ($test123->fetchrow_array()) { ..., which will only execute the query once and then iterate over the results.
Each result is a list reference, and the zip code you are interested in is in the first (and only) column, so you could accumulate the results in an array like this:
my #zips = (); # for final results
for my $row ($test123->fetchrow_array()) {
push #zips, $row->[0];
}
or even more concisely with Perl's map statement:
my #zips = map { $_->[0] } $test123->fetchrow_array()
which does the same thing.