I'm trying to pass data from a Webscraper to a MySQL database. I have a lot of variables that need to be entered at a time into the database and below is a snippet of the code I'm using. (where the etc. is there are a bunch more variables.
con.query(INSERT INTO Paper_2 (referenceCodeSubject,referenceCode,subject, etc.) values ('"+referenceCodeSubject+"','"+referenceCode+"','"+subject+"', etc.))
The columns in the database have types INT, VARCHAR and CHAR.
My issue is that when I scrape not all of the variables will be assigned values and will remain as 'null' and I cannot pass this null as NULL to MySQL. It would also be quite complicated to sort the different cases for when to pass what due to the large amount of variables.
I'm hoping theres a simple way of doing this as the only solutions I've seen so far are omit the value in the query (which is hard because I would then need to decide which values to omit) or pass a string of "NULL" or just a value of 0. Is there any other way of doing this?
Better use the built in escaping feature to avoid sql injection attacks!
conn.query(
'INSERT INTO Paper_2 (referenceCodeSubject,referenceCode,subject) VALUES ?'
[
['refCodeSubject1', 'refCode1', 'subject1'],
['refCodeSubject2', 'refCode2', null]
],
(error, results, fields) => {
...
}
)
If you have the case, that the bind values can sometime be a valid string and sometimes undefined, use an or operator in sqlValues to handle both cases with shorthand code:
let nameValue;
let sql="insert into user (name) values (?)"
let sqlValues[] = [nameValue || null ]
Related
Let's suppose I have a set of integers of a variable length. I apply a function on this set of integers and I obtain a result.
myFunction(setOfIntegers) => myResult
Let's suppose a call to myFunction is very expensive and I would like to somehow store the results of this function calls.
In my application I am already using MySQL and what I was thinking was to somehow create a table with the setOfIntegers as a PK and myResult as an additional field.
I was thinking that I could do this by transforming the setOfIntegers to a string before storing it in the DB.
Can this be done in any other way? Or would there be a better way to store results of such function calls in order to avoid calling them a 2nd time with the same set of integers?
I don't know about Java, but Perl has my $str = join(',', $array) and PHP has $str = implode(',', $array). Then the string $str could be used as the PRIMARY KEY (assuming it is not too long). And the result would go in the other column.
Your app code (in Java) would need to first do an implode and SELECT to see if the function has already been evaluated for the given array. If not, then perform the function and end by INSERTing a new row.
If this will be multi-threaded, you could use INSERT IGNORE to deal with dups. (There are other solutions, too.)
Another note: If your set-of-integers is ordered, then what I describe is 'complete'. If it is unordered, then sort it before imploding. This will provide a canonical representation.
If the function can be implemented in MySQL directly, I would suggest using Views.
https://www.mysqltutorial.org/mysql-views-tutorial.aspx/
I'm using Nodejs and Express for my project. Which way do you think is better to store data. I'm using two methods to store data but I'm not sure which one is better and secure.
First Method:
var insertQuery = "INSERT INTO mytable (value1,value2) values (?, ?)";
con.query(insertQuery, [value1,value2],(err, rows) => { //Some Code here })
but I heard that I must use SQL variables to prevent from SQL injection and actually I'm not sure is it right or not?
Second Method:
var values = " #v1 :='"+value1"' , #v2 :='"+value2"'";
var insertQuery = "INSERT INTO mytable (value1,value2) values (" + values +")";
con.query(insertQuery, (err,rows)=>{ //Some Code here })
So what do you think?
For your question - first approach is better because it uses prepared statement, which means that at least your input will be escaped, but there are a lot of cases you need to prevent, some of it:
Validate user input
Properly encode all symbols, it's the thing why injection is possible
If you don't want to use any ORM's I advice you to use knex , which can work either as sql builder/builder + runner/ builder+runner+mapper. If you are really interested how to build own sql builder, you can read source code of it.
So i do a simple query like so:
connection.query("UPDATE workers SET timestamp='"+thedate+"' WHERE id = " + id,function(err,upres){
connection.release();
if(!err) {
console.log('updated record')
console.log(upres);
}
})
console.log reveals the data format as: 2015-04-02 19:29:14
And if i debug the SQL statement, that turns out to be:
UPDATE workers SET timestamp='2015-04-02 21:31:16' WHERE id = 3;
However, when i list the data, the output is:
[{"id":3,"worker":"John Doe","timestamp":"2015-04-01T22:00:00.000Z","duration":30}]
This is way off compared to the time that is being reported?
What is causing this?
You do not know how MySQL is turning your VARCHAR into a date. There are a lot of configuration options. It would be better to use the STR_TO_DATE function to circumvent all of the assumptions. Here is a link to the docs for STR_TO_DATE().
As a side note, I would strongly recommend using prepared statements as a way to safeguard your application against errors and sql injection.
EDITS:
In regards to your questions, the column could be DATETIME, but your value you are assigning is a VARCHAR
'UPDATE workers SET timestamp = ? WHERE id = ?', ['4/2/2015 3:00:00 PM', 3'], [callBackFunction]
Based on what you said about the conversion not working, I am suspicious about the data type for the timestamp column.
SELECT * FROM INFORMATION_SCHEMA.COLUMNS WHERE NAME = 'workers'
A statement like that should give you all of the information about that column. You could also find this in a GUI, if you have access. There are three different date types in MySQL date, datetime, or timestamp. This is most likely a DATE column, that will not be able to hold the time.
Imagine you have the following SQL query:
SELECT *
FROM foo
WHERE type = ?
AND subtype IN (?)
And you have the following possible data (we imagine that a user interface can set these data):
var Type int
var SubTypes []int
In the case of SubTypes, we are talking about a multiple choice selection.
Now, the following code won't work:
rows, err := sqldb.Query(`SELECT *
FROM foo
WHERE type = ?
AND subtype IN (?)`, Type, SubTypes)
Because the driver (at least the mysql driver used in this example) doesn't recognise a []slice. Typing to explode it (SubTypes...) doesn't work either, because A) you cannot have more than one exploded parameter and B) even if you could, your SQL only supports a single item ((?)).
However, there is a solution. First of all, since we can only have a single exploding parameter and no others, we should first put together our parameters in a single []slice:
var params []interface{}
params = append(params, Type)
for _, subtype := range SubTypes {
params = append(params, SubTypes)
}
Since the SQL will not expand on its own, let's expand that loop:
var params []interface{}
params = append(params, Type)
inCondition := ""
for _, subtype := range SubTypes {
params = append(params, SubTypes)
if inCondition != "" {
inCondition += ", "
}
inCondition += "?"
}
Assuming SubTypes contains []int{1,2,3}, inCondition should now contain ?, ?, ?.
We then combine that to our SQL statement and explode the argument:
sqlstr := fmt.Sprintf(`SELECT *
FROM foo
WHERE type = ?
AND subtype IN (%s)`, inCodition)
rows, err := sqldb.Query(sqlstr, params...)
Of course, it would be pretty cool, if you could simply pass []slices to your prepared statements, and the automatically expanded. But that might give some unexpected results if you are dealing with more 'unknown' data.
Prepared statements do not work that way, at least not in major DBMS I know. I mean, in Go, the support for prepared statements implemented by database/sql drivers is supposed to use the corresponding facility provided by the underlying DBMS (a driver might opt to simulate such support if it's not provided by the DB engine it interfaces with).
Now in all the DBMS-s I'm familiar with, the whole idea of prepared statement is that it's processed once by the DB engine and cached; "processed" here means syntax checking, compiling into some DB-specific internal representation and its execution plan figured out. As follows from the term "compiled", the statement's text is processed exactly once, and then each call to the prepared statement just essentially tells the server "here is the ID of that prepared statement I supplied you earlier, and here's the list of actual parameters to use for placeholders it contained". It's like compiling a Go program and then calling it several times in a row with different command-line flags.
So the solution you have come up with is correct: if you want to mess with the statement text between invocation then by all means use client-side text manipulations1 but do not attempt to use the result of it as a prepared statement unless you really intend to execute the resulting text more than once.
And to be may be more clear: your initial attempt to prepare something like
SELECT a, b FROM foo WHERE a IN (?)
supposedly fails at your attempt to supply a set of values for that IN (?) placeholder because commas which would be required there to specify several values are syntax, not parts of the value.
I think it should still be fine to prepare something like
SELECT a, b FROM foo WHERE a IN (?, ?, ?)
because it does not break that rule. Not that it's a solution for you…
See also this and this — studying the latter would allow you to play with prepared statements directly in the MySQL client.
1 Some engines provide for server-side SQL generation with subsequent execution of the generated text.
I am using DBIx::Class and I would like to select rows based on what day of the year they were inserted on. Below is my query:
$rows = $c->model("DB::Test")->search(
{
"DAYOFYEAR(entry_time)"=>$day_of_year,
});
However this doesn't work because DBIx::Class treats DAYOFYEAR(entry_time) as a column. Is there anyway I could have it use that value litteraly? I know sometimes making it a scalar such as \'DAYOFYEAR(entry_time)' will work for some situations, but I've tried that and it doesn't work. Does anyone know of a way that I could do this? Thanks!
Using \ and 'DAYOFYEAR(entry_time)' is the right approach, and part of the FAQ.
[How do I] .. search with an SQL function on the left hand side?
To use an SQL function on the left hand side of a comparison you currently need to resort to literal SQL:
->search( \[ 'YEAR(date_of_birth) = ?', [ plain_value => 1979 ] ] );
Note: the plain_value string in the [ plain_value => 1979 ] part should be either the same as the name of the column (do this if the type of the return value of the function is the same as the type of the column) or in the case of a function it's currently treated as a dummy string (it is a good idea to use plain_value or something similar to convey intent). The value is currently only significant when handling special column types (BLOBs, arrays, etc.), but this may change in the future.