Something just came to mind and I'd like to bounce it off:
Say you have a user profile, with 10 fields that the user can edit, and not all of them are required. When issuing update commands, is it more efficient to either:
A) Collect all of the fields, filled in or not, and issue one all encompassing update statement to the server's DB
or
B) Use client side validation to check to see which fields have been filled out or changed, and have a selection of SQL methods that only send and update these fields
or
C) Create groupings, like "updateRequiredFields(...) and updateExtraFields(...)", which would issue one smaller transfer if the changes only belong in one group, however two transfers if both are edited
General consensus? Clearly option B is the far more verbose approach, I'm just wondering if it's worth coding it all out or if it'll actually make a noticeable impact on the server (think "scaled for big data").
You could do something like this on your DB update function:
public function updateFields(array $fields) {
$updateQuery = array();
foreach($fields as $fieldKey => $fieldValue) {
//if $fieldValue is false, leave it unchanged
if ($fieldValue !== false) {
//NOTE: make sure you escape this or use PDO
$updateQuery[] = $fieldKey . '=' . $fieldValue;
}
}
$query = 'UPDATE UserInfo SET ' . implode(",", $updateQuery) . ' WHERE ...';
}
You just need to build $fields array based on what was modified on client side and then pass in with either new value or with false if no change.
Related
I have a page that inserts records into a database file called ports that holds two fields, called id and port.
The data is checked by an include, checkform.php, that strips out any bad data and blank entries.
It works fine, and as I have more data files of a similar construction it seems logical to use the same page for inserting records by passing the file and field names to the page as parameters.
The SQL that is used for the stand alone page is:
$sql='INSERT IGNORE INTO ports(port) VALUES(?)';
I want to do some thing like:
$sql='INSERT IGNORE INTO $filename ($fieldname) VALUES(?)';
I have looked on the forum and found many solutions that do not appear to work
Like :
$sql='INSERT IGNORE INTO '$filename' ('$fieldname') VALUES(?)';
$sql='INSERT IGNORE INTO "'$filename'" ("'$fieldname'") VALUES(?)';
$sql='INSERT IGNORE INTO `$filename` (`$fieldname`) VALUES(?)';
as well as :
$sql="INSERT IGNORE INTO `$filename` (`$fieldname`) VALUES (`$fieldname`);";
and many others. The combination seems endless, and so far I would have been better just copying the pages and changing the variables by hand. The code for the insert is below:
// check if form submitted and has a value
If (isset($_POST['insert']))
{ require('../includes/checkform.inc.php');
// continue if the field is OK
if (empty($missing)) // ** missing is empty if the data is clean and exists
{ // process the input.
require_once('../includes/connection.inc.php');
// initialize a flag
$OK = false;
//create database connection
$conn = mysqli_connect( $DatabaseServer,$DatabaseUser, $DatabasePassword, $DatabaseName);
// Initialize prepared statement
$stmt = $conn->stmt_init();
//create SQL
$sql='INSERT IGNORE INTO ports(port) VALUES(?)'; //#
//bind parameters and execute statement
if($stmt->prepare($sql)) {
$stmt->bind_param('s',$_POST['port']);//#
$stmt->execute();
if ($stmt->affected_rows > 0)
$OK = true;
}//if $tmt
}// if empty
// redirect if successful or display an error - on page below
if ($OK) {
header('Location:insertok.php');
exit;
} else {
$error = htmlspecialchars($stmt->error);
The lines with //# against them are the ones that I need help with.
Most of the code is modified from a book by David Powers.
Howard Walker
To interpolate variables in a string, you have to use double quotes "$var". Note that you shouldn't surround $var with single quotes. And your table and column names might be one of the reserved words. It complains when that happens. You use backticks to escape the reserved words.
$sql="INSERT IGNORE INTO `$filename` (`$fieldname`) VALUES (?);";
This should work just fine.
EDIT
Your file/field might also include the characters that mySQL doesn't like. In that case, escape the query string before executing it. Refer: http://us3.php.net/manual/en/mysqli.real-escape-string.php
$sql = $stmt->real_escape_string($sql);
I have constructed a function where two queries are performed. Both of these queries insert data into two separate tables, data that is related to the registration of a user.
In one table things like username,password are held and in the other table stuff like address, phone etc...
Here is the function:
function register_biz_user($post,$connection)
{
$name=$connection-> real_escape_string($_POST['name']);
$lastname= $connection->real_escape_string($_POST['lastname']);
$pass_hashed = password::hash($_POST['password']);
$passwd= $connection->real_escape_string($pass_hashed);
$buztype= $connection->real_escape_string($_POST['buztype']);
$usertype= $connection->real_escape_string($_POST['usertype']);
$address= $connection->real_escape_string($_POST['address']);
$city= $connection->real_escape_string($_POST['city']);
$municipality= $connection->real_escape_string($_POST['municipality']);
$url= $connection->real_escape_string($_POST['wwwaddress']);
$email= $connection->real_escape_string($_POST['e-mail']);
$phone= $connection->real_escape_string($_POST['phone']);
$hash =$connection->real_escape_string(md5( rand(0,1000) )) ;
$connection->set_charset("utf8");
$result1 = $connection->query("insert into users values
(NULL,'" .$name. "','" .$lastname . "','".$email."','". $passwd."','".
$hash."','". $usertype."')");
if (!$result1) {
throw new Exception('error');
return false;
}
else{$result2=$connection->query("insert into business_users values
('".$connection->insert_id."','" .$address."','".$url ."','".$phone.
"','".$city. "','".$municipality. "','".$buztype. "')");
}
if(!$result2)
{ throw new Exception('error');
return false;}
return true;
}
And here is my problem:
If you look at the code you might notice that there is the problem that the 1st query runs without problem and the second throws an exception or vice verca.
My point is that there is the danger that the db WILL have ONLY partial data of the registered user. The goal is that either both queries run successfully or none runs.
How I must write the above code such that I can achieve the above statement?
I hope I was clear enough.
Use transactions: http://dev.mysql.com/doc/refman/5.0/en/commit.html
BEGIN
... queries ...
COMMIT or ROLLBACK
Note: "or vice verca" - that's not possible. In that case the 2nd query never gets executed.
Note2:
what's $post? seems to be unused.
why don't you use prepared statements? escaping everyhing is very error prone.
why do you have a procedural interface, passing $connection? you should have objects which know about the database connections... you have mixed code for at least 3 different layers... not necessary bad if you plan to create write-once-get-rid-of-code but probably not a good idea for a project which you have to maintain for months/years.
The normal result() method described in the documentation appears to load all records immediately. My application needs to load about 30,000 rows, and one at a time, submit them to a third-party search index API. Obviously loading everything into memory at once doesn't work well (errors out because of too much memory).
So my question is, how can I achieve the effect of the conventional MySQLi API method, in which you load one row at a time in a loop?
Here is something you can do.
while ($row = $result->_fetch_object()) {
$data = array(
'id' => $row->id
'some_value' => $row->some_field_name
);
// send row data to whatever api
$this->send_data_to_api($data);
}
This will get one row at the time. Check the CodeIgniter source code, and you will see that they will do this when you execute the result() method.
For those who want to save memory on large result-set:
Since CodeIgniter 3.0.0,
There is a unbuffered_row function,
All the methods above will load the whole result into memory (prefetching). Use unbuffered_row() for processing large result sets.
This method returns a single result row without prefetching the whole result in memory as row() does. If your query has more than one row, it returns the current row and moves the internal data pointer ahead.
$query = $this->db->query("YOUR QUERY");
while ($row = $query->unbuffered_row())
{
echo $row->title;
echo $row->name;
echo $row->body;
}
You can optionally pass ‘object’ (default) or ‘array’ in order to specify the returned value’s type:
$query->unbuffered_row(); // object
$query->unbuffered_row('object'); // object
$query->unbuffered_row('array'); // associative array
Official Document: https://www.codeigniter.com/userguide3/database/results.html#id2
Well, the thing is that result() gives away the entire reply of the query. row() simply fetches the first case and dumps the rest. However the query can still fetched 30 000 rows regardles of which function you use.
One design that would fit your cause would be:
$offset = (int)#$_GET['offset'];
$query = $this-db->query("SELECT * FROM table LIMIT ?, 1", array($offset));
$row = $query->row();
if ($row) {
/* Run api with values */
redirect(current_url().'?offset'.($offset + 1));
}
This would take one row, send it to api, update the page and use the next row. It will alos prevent the page from having a timeout. However it would most likely take a while with 30 000 records and refreshes, so you may wanna adjust your LIMIT ?, 1 to a higher number than 1 and go result() and foreach() multiple apis per pageload.
Well, there'se the row() method, which returns just one row as an object, or the row_array() method, which does the same but returns an array (of course).
So you could do something like
$sql = "SELECT * FROM yourtable";
$resultSet = $this->db->query($sql);
$total = $resultSet->num_rows();
for($i=0;$i<$total;$i++) {
$row = $resultSet->row_array($i);
}
This fetches in a loop each row from the whole result set.
Which is about the same as fetching everyting and looping over the $this->db->query($sql)->result() method calls I believe.
If you want a row at a time either you make 30.000 calls, or you select all the results and fetch them one at a time or you fetch all and walk over the array. I can't see any way out now.
I am updating rows of a MySQL database with groovy and with the method I am using things are very slow. I was hoping you could improve on the performance of my example:
sql.resultSetConcurrency = ResultSet.CONCUR_UPDATABLE
sql.eachRow("SELECT * FROM email) { bt ->
bt.extendedDesc = update(bt.name, bt.direction)
}
sql.resultSetConcurrency = ResultSet.CONCUR_READ_ONLY
Then there is the update method:
def update(name, direction) {
if (direction == 'Outgoing') {
result = 'FROM: '+name
} else {
result = 'TO: '+name
}
if(result.size() > 75) {
result = result.substring(0, 72) + "..."
}
return result
}
So it updates one field of each entry in email (extendedDesc in this example) using a method that needs 2 other fields passed to it as parameters.
It is very slow, around 600 entries updated per minute, and email has 200000+ entries =/
Is there a better method to accomplish this? Should use Groovy if possible, as it needs to run with all my other Groovy scripts.
You are doing your update as a cursor based, updatable query, which has to read every record and conditionally write something back. You're doing all the heavy lifting in the code, rather than letting the database do it.
Try using an update query to only update the records matching your criteria. You won't need eachRow to do this.
Follow up to this question. I have the following code:
string[] names = new[] { "Bob", "bob", "BoB" };
using (MyDataContext dataContext = new MyDataContext())
{
foreach (var name in names)
{
string s = name;
if (dataContext.Users.SingleOrDefault(u => u.Name.ToUpper() == s.ToUpper()) == null)
dataContext.Users.InsertOnSubmit(new User { Name = name });
}
dataContext.SubmitChanges();
}
...and it inserts all three names ("Bob", "bob" and "BoB"). If this was Linq-to-Objects, it wouldn't.
Can I make it look at the pending changes as well as what's already in the table?
I don't think that would be possible in general. Imagine you made a query like this:
dataContext.Users.InsertOnSubmit(new User { GroupId = 1 });
var groups = dataContext.Groups.Where(grp => grp.Users.Any());
The database knows nothing about the new user (yet) because the insert wasn't commited yet, so the generated SQL query might not return the Group with Id = 1. The only way the DataContext could take into account the not-yet-submitted insert in cases like this would be to get the whole Groups-Table (and possibly more tables, if they are affected by the query) and perform the query on the client, which is of course undesirable. I guess the L2S designers decided that it would be counterintuitive if some queries took not-yet-committed inserts into account while others wouldn't, so they chose to never take them into account.
Why don't you use something like
foreach (var name in names.Distinct(StringComparer.InvariantCultureIgnoreCase))
to filter out duplicate names before hitting the database?
Why dont you try something like this
foreach (var name in names)
{
string s = name;
if (dataContext.Users.SingleOrDefault(u => u.Name.ToUpper() == s.ToUpper()) == null)
{
dataContext.Users.InsertOnSubmit(new User { Name = name });
break;
}
}
I am sorry, I don't understand LINQ to SQL as much.
But, when I look at the code, it seems you are telling it to insert all the records at once (similar to a transaction) using SubmitChanges and you are trying to check the existence of it from the DB, when the records are not inserted at all.
EDIT: Try putting the SubmitChanges inside the loop and see that the code will run as per your expectation.
You can query the appropriate ChangeSet collection, such as
if(
dataContext.Users.
Union(dataContext.GetChangeSet().Inserts).
Except(dataContext.GetChangeSet().Deletes).
SingleOrDefault(u => u.Name.ToUpper() == s.ToUpper()) == null)
This will create a union of the values in the Users table and the pending Inserts, and will exclude pending deletes.
Of course, you might want to create a changeSet variable to prevent multiple calls to the GetChangeSet function, and you may need to appropriately cast the object in the collection to the appropriate type. In the Inserts and Deletes collections, you may want to filter it with something like
...GetChangeSet().Inserts.Where(o => o.GetType() == typeof(User)).OfType<User>()...