Does SQLite support seeding the RANDOM() function the same way MySQL does with RAND()?
$query = "SELECT * FROM table ORDER BY RAND(" . date('Ymd') . ") LIMIT 1;";
From the MySQL Manual about RAND(N):
If a constant integer argument N is
specified, it is used as the seed
value, which produces a repeatable
sequence of column values. In the
following example, note that the
sequences of values produced by
RAND(3) is the same both places where
it occurs.
If not, is there any way to archive the same effect using only one query?
Have a look at the sqlite3_randomness() function:
SQLite contains a high-quality pseudo-random number generator (PRNG) used to select random ROWIDs when inserting new records into a table that already uses the largest possible ROWID. The PRNG is also used for the build-in random() and randomblob() SQL functions.
...
The first time this routine is invoked (either internally or by the application) the PRNG is seeded using randomness obtained from the xRandomness method of the default sqlite3_vfs object. On all subsequent invocations, the pseudo-randomness is generated internally and without recourse to the sqlite3_vfs xRandomness method.
Looking at the source of this xRandomness method, you can see that it reads from /dev/urandom on Unix. On Windows, it just returns the return values of some time functions. So it seems that your only option is to start hacking on the SQLite source code.
If you need a pseudo-random order, you can do something like this (PHP):
$seed = md5(mt_rand());
$prng = ('0.' . str_replace(['0', 'a', 'b', 'c', 'd', 'e', 'f'], ['7', '3', '1', '5', '9', '8', '4'], $seed )) * 1;
$query = 'SELECT id, name FROM table ORDER BY (substr(id * ' . $prng . ', length(id) + 2))';
Plus, you can set $seed to the predefined value and always get same results.
I've learned this trick from my colleague http://steamcooker.blogspot.com/
Related
I want to perform a query like the following one:
SELECT id, name
FROM mytable
ORDER BY FIELD(name, 'B', 'A', 'D', 'E', 'C')
FIELD is a MySQL specific function, and 'B', 'A', 'D', 'E', 'C' are values coming from a List.
I tried using fragment, but it doesn't seem to allow dynamic arity known only in the runtime.
Except going full-raw using Ecto.Adapters.SQL.query, is there a way to handle this using Ecto's query DSL?
Edit: Here's the first, naive approach, which of course does not work:
ids = [2, 1, 3] # this list is of course created dynamically and does not always have three items
query = MyModel
|> where([a], a.id in ^ids)
|> order_by(fragment("FIELD(id, ?)", ^ids))
ORM are wonderful, until they leak. All do, eventually. Ecto is young (f.e., it only gained ability to OR where clauses together 30 days ago), so it's simply not mature enough to have developed an API that considers advanced SQL gyrations.
Surveying possible options, you're not alone in the request. The inability to comprehend lists in fragments (whether as part of order_by or where or any where else) has been mentioned in Ecto issue #1485, on StackOverflow, on the Elixir Forum and this blog post. The later is particulary instructive. More on that in a bit. First, let's try some experiments.
Experiment #1: One might first try using Kernel.apply/3 to pass the list to fragment, but that won't work:
|> order_by(Kernel.apply(Ecto.Query.Builder, :fragment, ^ids))
Experiment #2: Then perhaps we can build it with string manipulation. How about giving fragment a string built-at-runtime with enough placeholders for it to pull from the list:
|> order_by(fragment(Enum.join(["FIELD(id,", Enum.join(Enum.map(ids, fn _ -> "?" end), ","), ")"], ""), ^ids))
Which would produce FIELD(id,?,?,?) given ids = [1, 2, 3]. Nope, this doesn't work either.
Experiment #3: Creating the entire, final SQL built from the ids, placing the raw ID values directly in the composed string. Besides being horrible, it doesn't work, either:
|> order_by(fragment(Enum.join(["FIELD(id,", Enum.join(^ids, ","), ")"], "")))
Experiment #4: This brings me around to that blog post I mentioned. In it, the author hacks around the lack of or_where using a set of pre-defined macros based on the number of conditions to pull together:
defp orderby_fragment(query, [v1]) do
from u in query, order_by: fragment("FIELD(id,?)", ^v1)
end
defp orderby_fragment(query, [v1,v2]) do
from u in query, order_by: fragment("FIELD(id,?,?)", ^v1, ^v2)
end
defp orderby_fragment(query, [v1,v2,v3]) do
from u in query, order_by: fragment("FIELD(id,?,?,?)", ^v1, ^v2, ^v3)
end
defp orderby_fragment(query, [v1,v2,v3,v4]) do
from u in query, order_by: fragment("FIELD(id,?,?,?)", ^v1, ^v2, ^v3, ^v4)
end
While this works and uses the ORM "with the grain" so to speak, it requires that you have a finite, manageable number of available fields. This may or may not be a game changer.
My recommendation: don't try to juggle around an ORM's leaks. You know the best query. If the ORM won't accept it, write it directly with raw SQL, and document why the ORM does not work. Shield it behind a function or module so you can reserve the future right to change its implementation. One day, when the ORM catches up, you can then just rewrite it nicely with no effects on the rest of the system.
Create a table with 2 columns:
B 1
A 2
D 3
E 4
C 5
Then JOIN LEFT(name, 1) to it and get the ordinal. Then sort by that.
(Sorry, I can't help with Elixir/Ecto/Arity.)
I would try to resolve this using the following SQL SELECT statement:
[Note: Don't have access right now to a system to check the correctness of the syntax, but I think it is OK]
SELECT A.MyID , A.MyName
FROM (
SELECT id AS MyID ,
name AS MyName ,
FIELD(name, 'B', 'A', 'D', 'E', 'C') AS Order_By_Field
FROM mytable
) A
ORDER BY A.Order_By_Field
;
Please note that the list 'B','A',... can be passed as either an array or any other method and replace what is written in the above code sample.
This was actually driving me crazy until I found that (at least in MySQL), there is a FIND_IN_SET function. The syntax is a bit weird, but it doesn't take variable arguments, so you should be able to do this:
ids = [2, 1, 3] # this list is of course created dynamically and does not always have three items
ids_string = Enum.join(ids, ",")
query = MyModel
|> where([a], a.id in ^ids)
|> order_by(fragment("FIND_IN_SET(id, ?)", ^ids_string))
I've looked around and have only seen this being used when using SELECT, not INSERT and it has gotten me quite confused.
What I've been trying is:
$statement = $pdo->prepare("INSERT INTO shipments (crate_type_id, username, arrival, quantity, type, status) VALUES ('$crate_type_id', '$username', 'DATE_ADD(NOW() + INTERVAL 1 HOUR)', '$quantity', 'purchase', 'active')");
I don't get any errors, it only inserts "0000-00-00 00:00:00". I've tried troubleshooting in phpmyadmin etc but haven't made any progress.
You should remove the quotes arount the date_add expression and use comma not +
statement = $pdo->prepare("INSERT INTO shipments
(crate_type_id, username, arrival, quantity, type, status)
VALUES ('$crate_type_id', '$username',
DATE_ADD(NOW() , INTERVAL 1 HOUR) ,
'$quantity', 'purchase', 'active')");
First of all, you're using PDO::prepare() incorrectly. The whole purpose of the tool is to separate code and data but you're injecting the latter into the former as if prepared statements had not been invented yet. You have several examples in the manual so I won't repeat that here but in short you have to:
Insert placeholders in your SQL code (? or :foo)
Provide actual values on execution
Secondly, I suggest you configure MySQL in strict mode so you don't end up with junk like 0000-00-00 00:00:00. You can set a proper SQL mode on connection, e.g.:
SET ##SESSION.sql_mode='TRADITIONAL'
Finally, your final SQL code is possibly not even valid, but apparently you aren't being notified about it. I recommend you add something like this to your connection code:
$pdo = new PDO($dsn, $user, $pass, [PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION]);
... and ensure you've configured your PHP development box to display all errors.
If I have a database having 2 fields, Roll no and name and I have a list (of n values) of roll numbers for which I have to search the corresponding names.
Can this be done using just one query in SQL or HQL?
SELECT name FROM [table] WHERE id IN ([list of ids])
where [list of ids] is for example 2,3,5,7.
Use the IN operator and separate your Roll no's by a comma.
SELECT name
FROM yourtable
WHERE [Roll no] IN (1, 2, 3, 4, etc)
You can use the IN statement as shown above.
There are a couple of minor issues with this. It can perform poorly if the number of values in the clause gets too large.
The second issue is that in many development environments you land up needing to dynamically create the query with a variable number of items (or a variable number of placeholders if using parameterised queries). While not difficult if does make your code look messy and mean you haven't got a nice neat piece of SQL that you can copy out and use to test.
But examples (using php).
Here the IN is just dynamically created with the SQL. Assuming the roll numbers can only be integers it is applying intval() to each member of the array to avoid any non integer values being used in the SQL.
<?php
$list_of_roll_no = array(1,2,3,4,5,6,7,8,9);
$sql = "SELECT FROM some_table WHERE `Roll no` IN (".implode(", ", array_map ('intval', $list_of_roll_no)).")";
?>
Using mysqli bound parameters is a bit messy. This is because the bind parameter statement expects a variable number of parameters. The 2nd parameter onwards are the values to be bound, and it expects them to be passed by reference. So the foreach here is used to generate an array of references:-
<?php
$list_of_roll_no = array(1,2,3,4,5,6,7,8,9);
if ($stmt = $mysqli->prepare("SELECT FROM some_table WHERE `Roll no` IN (".implode(",", array_fill(0, count($list_of_roll_no), '?')).")"))
{
$bind_arguments = [];
$bind_arguments[] = str_repeat("i", count($list_of_roll_no));
foreach ($list_of_roll_no as $list_of_roll_no_key => $list_of_roll_no_value)
{
$bind_arguments[] = & $list_of_roll_no[$list_of_roll_no_key]; # bind to array ref, not to the temporary $recordvalue
}
call_user_func_array(array($statement, 'bind_param'), $bind_arguments);
$statement->execute();
}
?>
Another solution is to push all the values into another table. Can be a temp table. Then you use an INNER JOIN between your table and your temp table to find the matching values. Depending on what you already have in place then this is quite easy to do (eg, I have a php class to insert multiple records easily - I just keep passing them across and the class batches them up and inserts them occasionally to avoid repeatedly hitting the database).
I have to insert more than 200000 records at one go, into mysql db table, Insert query is resulting in performance issue, what could be the substitute to this.
Below is the code I am using
$xml = simplexml_load_file("247electrical.xml");
foreach($xml->merchant as $merchant){
define('API', 'PS');
require_once('constants.inc.php');
require_once('classes/class.ClientFactory.php');
$oClient = ClientFactory::getClient(API_USERNAME, API_PASSWORD, API_USER_TYPE); $merchattrs=$merchant->attributes();
$aParams100 = array('iMerchantId' => array($merchattrs->id)); $merchantinfo= $oClient->call('getMerchant', $aParams100);
//Get Products
foreach($xml->merchant->prod as $product){
$attrs=$product->attributes();
//Insert Products into DB
mysql_query('INSERT INTO productstemp (merchant_id, merchant_name, aw_product_id, merchant_product_id, product_name, description, category_id, merchant_category, aw_deep_link, aw_image_url, search_price, delivery_cost, merchant_image_url, aw_thumb_url, brand_name, delivery_time, display_price, in_stock, merchant_thumb_url, model_number, pre_order, stock_quantity, store_price, valid_from, valid_to, web_offer, merchantimage, cleancompany) VALUES("'.$merchattrs->id.'","'.$merchattrs->name.'","'.$attrs->id.'"," ","'.$product->text->name.'","'.$product->text->desc.'","'.$product->cat->awCatId.'","'.$product->cat->mCat.'","'.$product->uri->awTrack.'","'.$product->uri->awImage.'","'.$product->price->buynow.'","'.$product->price->delivery.'","'.$product->uri->mImage.'","'.$product->uri->awThumb.'","'.$product->brand->brandName.'","'.$product->delTime.'","'.$product->price->buynow.'","'.$attrs->in_stock.'","'.$product->uri->mThumb.'","'.$product->modelNumber.'","'.$attrs->pre_order.'","'.$attrs->stock_quantity.'","'.$product->price->store.'","'.$product->valFrom.'","'.$product->valTo.'","'.$attrs->web_offer.'","'.$merchantinfo->oMerchant->sLogoUrl.'","247electrical" ) ')
or die(mysql_error());
}
}
Thanks
I dont think that the INSERT queries per se are the problem. 200.000 inserts arent that much for mysql after all.
First I guess reading the file is slow. SimpleXML is convenient but for large files it results in a huge memory overhead. Think about a streaming XML reader like PHP´s XMLReader.
You are sending individual statements to the mysql server which is way slower then sending one huge statement. Also, your single insert statement should be wrapped in a transaction. What happens if you processed 10.000 records and inserted them and then your script dies/mysql server dies etc.? How do you safely start the script again without manual work (clearing table, lookup which were already processed etc.).
Apart from that, one single INSERT statement with many VALUES should be way faster. I would make your PHP script output the query so it looks in the end like this:
INSERT INTO table(field_1, field_2, field 3)
VALUES('foo 1', 'bar 1', 'baz 1'),
VALUES('foo 2', 'bar 2', 'baz 2'),
...
And then import that file via:
$ mysql ... credentials options etc ... < output.sql
If thats still too slow… buying more hardware might help, too.
Has anyone seen a DBI-type module for Perl which capitalizes, easily, on MySQL's multi-insert syntax
insert into TBL (col1, col2, col3) values (1,2,3),(4,5,6),...?
I've not yet found an interface which allows me to do that. The only thing I HAVE found is looping through my array. This method seems a lot less optimal vs throwing everything into a single line and letting MySQL handle it. I've not found any documentation out there IE google which sheds light on this short of rolling my own code to do it.
TIA
There are two approaches. You can insert (?, ?, ?) a number of times based on the size of the array. The text manipulation would be something like:
my $sql_values = join( ' ', ('(?, ?, ?)') x scalar(#array) );
Then flatten the array for calling execute(). I would avoid this way because of the thorny string and array manipulation that needs to be done.
The other way is to begin a transaction, then run a single insert statement multiple times.
my $sql = 'INSERT INTO tbl (col1, col2, col3)';
$dbh->{AutoCommit} = 0;
my $sth = $dbh->prepare_cached( $sql );
$sth->execute( #$_ ) for #array;
$sth->finish;
$dbh->{AutoCommit} = 1;
This is a bit slower than the first method, but it still avoids reparsing the statement. It also avoids the subtle manipulations of the first solution, while still being atomic and allowing disk I/O to be optimized.
If DBD::mysql supported DBI's execute_for_fetch (see DBI's execute_array and execute_for_fetch) this is the typical usage scenario i.e., you have multiple rows of inserts/updates/deletes available now and want to send them in one go (or in batches). I've no idea if the mysql client libs support sending multiple rows of bound parameters in one go but most other database client libs do and can take advantage of DBI's execute_array/execute_for_fetch. Unfortunately few DBDs actually implement execute_array/execute_for_fetch and rely on DBI implementing it one row at a time.
Jim,
Frezik has it. That is probably the most optimal:
my $sth = $dbh->prepare( 'INSERT INTO tbl (?, ?, ?)' );
foreach(#array) { $sth->execute( #{$_} ); }
$sth->finish;