I want to use the "?" char for a search in a MySQL request in rails.
For example, classical way is:
Model.where("name = ?", '#{#search}')
My question is about long queries and multilines condition.
If I want to build a condition manually:
where = ""
where << " status = 1 "
where << " AND MATCH (name) AGAINST (? IN BOOLEAN MODE), #search " if #search
#records = Model.where(where)
Of course, it won't work.
So how to use the "?" (for security and simplicity) with a multilines conditions ?
A simple way is to do that:
where << " MATCH (name) AGAINST ('#{#search}' IN BOOLEAN MODE) "
But I will lose security (SQL injection) and can have problems with quotes if #search contains quotes.
Thanks,
You're getting a bit confused about the contents of where: you're putting variable names inside the string, which won't work: "#search" inside a string becomes literally the word "#search" and not a variable.
The best way to think of the arguments to where is as an array of objects, and you can build it like this. The first object is the query string (with ? symbols) and the other elements are the values for the ? symbols, which will be sanitized and translated by rails.
eg
User.where(["first_name = ? and last_name = ?", "John", "Smith"])
you can pass other things to where, like a hash of values, or a single string, but the array is the most flexible, especially in your case.
Bearing that in mind, you can do something like this to build a dynamically-created, complex query: i use this pattern a lot as it's very flexible and also very readable.
condition_strings = []
condition_values = []
condition_strings << "status = ?"
condition_values << 1 #or some dynamic data
condition_strings << "MATCH (name) AGAINST (? IN BOOLEAN MODE)"
condition_values << #search
conditions = [condition_strings.join(" AND ")] + condition_values
# => ["status = ? AND MATCH (name) AGAINST (? IN BOOLEAN MODE)", 1, "foo"]
#now we can use the array as an argument to `.where`:
#records = Model.where(conditions)
Related
I am new to python and currently learning to use SQL with python. I have the following code:
word = input("Enter a word: ")
query = cursor.execute("SELECT * FROM Dictionary WHERE Expression LIKE '%s%' " % word)
results = cursor.fetchall()
The second line throws an error since I don't think I can use '%s%' like that? How would I change this so as to be able to make this work? I want to be able to return all related entries to the users input. So if the user inputs "rain", then I want the query to return all possible results e.g. "raining", "rainy" etc. Thank you.
You can try
query = cursor.execute(f"SELECT * FROM Dictionary WHERE Expression LIKE '%{word}%' ")
You should use cursor.execute() parameter substitution rather than string formatting, to prevent SQL injection.
Then use CONCAT() to surround the search string with %.
query = cursor.execute("SELECT * FROM Dictionary WHERE Expression LIKE CONCAT('%', %s, '%' "), (word,))
I am new in python as well as mysql. I am having trouble in populating proper query statement for mysql.
sql = "SELECT * FROM Persons WHERE %s"
cur = db.cursor()
cur.execute(sql,(where,))
where is a string variable which creates a string for WHERE clause; this is the point of question. When I print this variable it give the following result:
Gender = True And IsLate = False
(without any quotes) but when I add this variable to the query to execute it, it adds single quotes around the string.
I used the command
print(cur.statement)
and it prints:
SELECT * FROM Persons WHERE 'Gender = True And IsLate = False'
After supplying parameter, it puts it within single quotes and query returns 0 rows.
I have worked around by concatenating the query statement and variable together and execute the string as query, that worked,
sql = sql + where
cur.execute(sql)
But I know that is not the professional way, as I have searched and found the professional way is to use parameterized query and use variable to store the condition(s) and supplying it at the execution of query.
Looking for advice, am I thinking the right way or otherwise?
The whole point of using parameter substitution in cursor.execute() is that it protects you from SQL injection. Each parameter is treated as a literal value, not substituted into the query and re-interpreted.
If you really want it to be interprted, you need to use string formatting or concatenation, as you discovered. But then you will have to be very careful in validating the input, because the user can supply extra SQL code that you may not have expected, and cause the query to malfunction.
What you should do is build the where string and parameter list dynamically.
where = []
params = []
if gender_supplied:
where.append('gender = %s')
params.append(gender)
if islate_supplied:
where.append*('islate = %s')
params.append(islate)
sql = 'select * from persons'
if where:
query = sql + ' where ' + ' and '.join(where)
else:
query = sql
cur.execute(query, params)
Im trying to replicate the searching list style of crunchbase using ruby on rails.
I have an array of filters that looks something like this:
[
{
"id":"0",
"className":"Company",
"field":"name",
"operator":"starts with",
"val":"a"
},
{
"id":"1",
"className":"Company",
"field":"hq_city",
"operator":"equals",
"val":"Karachi"
},
{
"id":"2",
"className":"Category",
"field":"name",
"operator":"does not include",
"val":"ECommerce"
}
]
I send this json string to my ruby controller where I have implemented this logic:
filters = params[:q]
table_names = {}
filters.each do |filter|
filter = filters[filter]
className = filter["className"]
fieldName = filter["field"]
operator = filter["operator"]
val = filter["val"]
if table_names[className].blank?
table_names[className] = []
end
table_names[className].push({
fieldName: fieldName,
operator: operator,
val: val
})
end
table_names.each do |k, v|
i = 0
where_string = ''
val_hash = {}
v.each do |field|
if i > 0
where_string += ' AND '
end
where_string += "#{field[:fieldName]} = :#{field[:fieldName]}"
val_hash[field[:fieldName].to_sym] = field[:val]
i += 1
end
className = k.constantize
puts className.where(where_string, val_hash)
end
What I do is, I loop over the json array and create a hash with keys as table names and values are the array with the name of the column, the operator and the value to apply that operator on. So I would have something like this after the table_names hash is created:
{
'Company':[
{
fieldName:'name',
operator:'starts with',
val:'a'
},
{
fieldName:'hq_city',
operator:'equals',
val:'karachi'
}
],
'Category':[
{
fieldName:'name',
operator:'does not include',
val:'ECommerce'
}
]
}
Now I loop over the table_names hash and create a where query using the Model.where("column_name = :column_name", {column_name: 'abcd'}) syntax.
So I would be generating two queries:
SELECT "companies".* FROM "companies" WHERE (name = 'a' AND hq_city = 'b')
SELECT "categories".* FROM "categories" WHERE (name = 'c')
I have two problems now:
1. Operators:
I have many operators that can be applied on a column like 'starts with', 'ends with', 'equals', 'does not equals', 'includes', 'does not includes', 'greater than', 'less than'. I am guessing the best way would be to do a switch case on the operator and use the appropriate symbol while building the where string. So for example, if the operator is 'starts with', i'd do something like where_string += "#{field[:fieldName]} like %:#{field[:fieldName]}" and likewise for others.
So is this approach correct and is this type of wildcard syntax allowed in this kind of .where?
2. More than 1 table
As you saw, my approach builds 2 queries for more than 2 tables. I do not need 2 queries, I need the category name to be in the same query where the category belongs to the company.
Now what I want to do is I need to create a query like this:
Company.joins(:categories).where("name = :name and hq_city = :hq_city and categories.name = :categories[name]", {name: 'a', hq_city: 'Karachi', categories: {name: 'ECommerce'}})
But this is not it. The search can become very very complex. For example:
A Company has many FundingRound. FundingRound can have many Investment and Investment can have many IndividualInvestor. So I can select create a filter like:
{
"id":"0",
"className":"IndividualInvestor",
"field":"first_name",
"operator":"starts with",
"val":"za"
}
My approach would create a query like this:
SELECT "individual_investors".* FROM "individual_investors" WHERE (first_name like %za%)
This query is wrong. I want to query the individual investors of the investments of the funding round of the company. Which is a lot of joining tables.
The approach that I have used is applicable to a single model and cannot solve the problem that I stated above.
How would I solve this problem?
You can create a SQL query based on your hash. The most generic approach is raw SQL, which can be executed by ActiveRecord.
Here is some concept code that should give you the right idea:
query_select = "select * from "
query_where = ""
tables = [] # for selecting from all tables
hash.each do |table, values|
table_name = table.constantize.table_name
tables << table_name
values.each do |q|
query_where += " AND " unless query_string.empty?
query_where += "'#{ActiveRecord::Base.connection.quote(table_name)}'."
query_where += "'#{ActiveRecord::Base.connection.quote(q[fieldName)}'"
if q[:operator] == "starts with" # this should be done with an appropriate method
query_where += " LIKE '#{ActiveRecord::Base.connection.quote(q[val)}%'"
end
end
end
query_tables = tables.join(", ")
raw_query = query_select + query_tables + " where " + query_where
result = ActiveRecord::Base.connection.execute(raw_query)
result.to_h # not required, but raw results are probably easier to handle as a hash
What this does:
query_select specifies what information you want in the result
query_where builds all the search conditions and escapes input to prevent SQL injections
query_tables is a list of all the tables you need to search
table_name = table.constantize.table_name will give you the SQL table_name as used by the model
raw_query is the actual combined sql query from the parts above
ActiveRecord::Base.connection.execute(raw_query) executes the sql on the database
Make sure to put any user submitted input in quotes and escape it properly to prevent SQL injections.
For your example the created query will look like this:
select * from companies, categories where 'companies'.'name' LIKE 'a%' AND 'companies'.'hq_city' = 'karachi' AND 'categories'.'name' NOT LIKE '%ECommerce%'
This approach might need additional logic for joining tables that are related.
In your case, if company and category have an association, you have to add something like this to the query_where
"AND 'company'.'category_id' = 'categories'.'id'"
Easy approach: You can create a Hash for all pairs of models/tables that can be queried and store the appropriate join condition there. This Hash shouldn't be too complex even for a medium-sized project.
Hard approach: This can be done automatically, if you have has_many, has_one and belongs_to properly defined in your models. You can get the associations of a model using reflect_on_all_associations. Implement a Breath-First-Search or Depth-First Search algorithm and start with any model and search for matching associations to other models from your json input. Start new BFS/DFS runs until there are no unvisited models from the json input left. From the found information, you can derive all join conditions and then add them as expressions in the where clause of the raw sql approach as explained above. Even more complex, but also doable would be reading the database schema and using a similar approach as defined here by looking for foreign keys.
Using associations: If all of them are associated with has_many / has_one, you can handle the joins with ActiveRecord by using the joins method with inject on the "most significant" model like this:
base_model = "Company".constantize
assocations = [:categories] # and so on
result = assocations.inject(base_model) { |model, assoc| model.joins(assoc) }.where(query_where)
What this does:
it passes the base_model as starting input to Enumerable.inject, which will repeatedly call input.send(:joins, :assoc) (for my example this would do Company.send(:joins, :categories) which is equivalent to `Company.categories
on the combined join, it executes the where conditions (constructed as described above)
Disclaimer The exact syntax you need might vary based on the SQL implementation you use.
Full blown SQL string is a security issue, because it exposes your application to a SQL injection attack. If you can get your way around this, it is completely ok to make those query concatenations, as long as you make them compatible with your DB(yes, this solution is DB specific).
Other than that you can make some field that marks some querys as joined, as I have mentioned in the comment, you would have some variable to mark the desired table to be the output of the query, something like:
[
{
"id":"1",
"className":"Category",
"field":"name",
"operator":"does not include",
"val":"ECommerce",
"queryModel":"Company"
}
]
Which, when processing the query, you would use to output the result of this query as the queryModel instead of the className, in those cases the className would be used only to join the table conditions.
I would suggest altering your JSON data. Right now you only send name of the model, without the context, it would be easier if your model would have context.
In your example data would have to look like
data = [
{
id: '0',
className: 'Company',
relation: 'Company',
field: 'name',
operator: 'starts with',
val: 'a'
},
{
id: '1',
className: 'Category',
relation: 'Company.categories',
field: 'name',
operator: 'equals',
val: '12'
},
{
id: '3',
className: 'IndividualInvestor',
relation: 'Company.founding_rounds.investments.individual_investors',
field: 'name',
operator: 'equals',
val: '12'
}
]
And you send this data to QueryBuilder
query = QueryBuilder.new(data)
results = query.find_records
Note: find_records returns array of hashes per model on which you execute query.
For example it would return [{Company: [....]]
class QueryBuilder
def initialize(data)
#data = prepare_data(data)
end
def find_records
queries = #data.group_by {|e| e[:model]}
queries.map do |k, v|
q = v.map do |f|
{
field: "#{f[:table_name]}.#{f[:field]} #{read_operator(f[:operator])} ?",
value: value_based_on_operator(f[:val], f[:operator])
}
end
db_query = q.map {|e| e[:field]}.join(" AND ")
values = q.map {|e| e[:value]}
{"#{k}": k.constantize.joins(join_hash(v)).where(db_query, *values)}
end
end
private
def join_hash(array_of_relations)
hash = {}
array_of_relations.each do |f|
hash.merge!(array_to_hash(f[:joins]))
end
hash.map do |k, v|
if v.nil?
k
else
{"#{k}": v}
end
end
end
def read_operator(operator)
case operator
when 'equals'
'='
when 'starts with'
'LIKE'
end
end
def value_based_on_operator(value, operator)
case operator
when 'equals'
value
when 'starts with'
"%#{value}"
end
end
def prepare_data(data)
data.each do |record|
record.tap do |f|
f[:model] = f[:relation].split('.')[0]
f[:joins] = f[:relation].split('.').drop(1)
f[:table_name] = f[:className].constantize.table_name
end
end
end
def array_to_hash(array)
if array.length < 1
{}
elsif array.length == 1
{"#{array[0]}": nil}
elsif array.length == 2
{"#{array[0]}": array[1]}
else
{"#{array[0]}": array_to_hash(array.drop(1))}
end
end
end
I feel you are over complicating things by having one single controller for everything. I would create a controller for every model or entity that you would want to show and then implement the filters like you said.
Implementing a dynamic where and order by is not very hard but if, as you said, you need to have also the logic to implement some joins you are not only over complicating the solution (because you will have to keep this controller updated every time you add a new model, entity or change the basic logic) but you are also enabling people start playing with your data.
I am not very familiar with Rails so sadly I cannot give you any specific cde other than saying that your approach seems OK to me. I would explode it into multiple controllers.
I have this query:
$sql = "
INSERT INTO table SET
name = '$name',
sku = '$number',
description = '$desc'
";
But the rows containing some special characters (in my case this ') are not inserted.. How I can solve?
Thanks in advance.
When you construct your query, you need to escape the data you are inserting.
You need to at least use addslashes() function in PHP, like this:
$sql = "INSERT INTO table SET name = '".addslashes($name)."', sku = '".addslashes($number)."', description = '".addslashes($desc)."'";
However more correct way is to use a different function than addslashes, which would properly handle all characters in the data, not only apostrophes.
I am using my custom 'escape' function like this:
function escape($text)
{
return str_replace(array('\\', "\0", "\n", "\r", "'", '"', "\x1a"), array('\\\\', '\\0', '\\n', '\\r', "\\'", '\\"', '\\Z'), $text);
}
So using this function, you would write:
$sql = "INSERT INTO table SET name = '".escape($name)."', sku = '".escape($number)."', description = '".escape($desc)."'";
You must use parameterised queries instead of manually appending those values. Currently if name, number or description would contain any sql it would get executed.
A lot more detailed answer is in How can I prevent SQL injection in PHP?
Read about escaping characters in mysql. I think it is done with \
User Form Input - City
User Form Input - Venue
User Form Input - Cover
User Form Input - Time
User Form Input - Date
User Form Input - Number1
User Form Input - Number2
(if any are blank they are coverted to '*' on the way in. But could be whatever works.)
my $grabgig = $hookup->prepare(qq{SELECT `VenueNumber`,`Venue`,`CoverCharge`,`SetLength`,`City`,`Owner`,`Date`,`Time`,`Image1`,`Number`
FROM `gigs`
WHERE VenueNumber > ? AND `City` = ? AND `Venue` = ? AND `CoverCharge` = ?
AND Date = ? AND `Number` > ? AND `Number` < ?
AND `Time` LIKE ? LIMIT ?,?});
##########################################
$grabgig->execute('100',$city,$venue,$cover,'*',$number1,$number2,?,'0','6')
or die "Did not execute";
That is a basic example above.
I want to be able to return results based on the City Input.
If more input is present, then narrow down results accordingly.
But the query returns nothing if fields are empty (*).
I tried wildcards and so on then, I experimented with LIKE and NOT LIKE.
This seemingly simple search is driving me nuts.
Can someone help this newbie?
OK, I'm pretty unsure what you mean, BUT, my best undererstanding of what you're trying to do is to query like you do now BUT if a particular field is not populated in the form, to avoid adding that field to the where clause; as opposed to current query which instead does and myField="*".
Correct?
If that's so, you need to build your query, and replacement list, in pieces:
my $sql = qq{SELECT MY_FIELD_LIST_TOO_LAZY_TO_TYPE FROM `gigs` WHERE 2=2};
my #replacement_values = (); # These go into execute() instead of "?"s
if ($city ne "*") {
$sql .= qq[AND city = ?];
push #replacement_values, $city;
}
if ($number1 ne "*") {
$sql .= qq[AND number > ?];
push #replacement_values, $number1;
}
# ... more values processed the same way
my $grabgig = $hookup->prepare($sql);
$grabgig->execute(#replacement_values) or die "Did not execute";
If you want to do it more intelligently (i.e. to generalize), you will have the form fields in a hash; have a config hash mapping the form field name to the DB column name and the operator, and instead do the above as:
my %fields = (
city => ["city" , "="]
,number1 => ["number", ">"]
,number2 => ["number", "<"]
);
my $sql = qq{SELECT MY_FIELD_LIST_TOO_LAZY_TO_TYPE FROM `gigs` WHERE 2=2};
my #replacement_values = (); # These go into execute() instead of "?"s
foreach my $field (keys %form_data) {
next unless exists $fields{$field};
if ($form_data{$field} ne "*") {
$sql .= qq[ AND $fields{$field}->[0] $fields{$field}->[1] ?];
push #replacement_values, $form_data{$field};
}
}
my $grabgig = $hookup->prepare($sql);
$grabgig->execute(#replacement_values) or die "Did not execute";
I am assuming that you want to construct a query where only a few input parameters have valid values and the rest are undefined. If that is indeed what you want, here is what you could do: Construct the query dynamically. Here are the steps you could take assuming you are using CGI.pm and assuming that the where clause is just a series of "this = that" - In your case you have different operators - but the idea is the same.
First construct a "where" string from the CGI query parameter (Sorry untested code):
my $qrystr = '';
foreach ($query->param) {
if (my $val = $query->param($_)) {
$qrystr .= "where $_ = " . $dbh->quote($val) . ' and ';
}
}
$qrystr .= "where 1 = 1";
Now you can just prepare and execute the query : "select * from table $qrystr"
If you want automatic quoting you will have to use bind parameters which is an easy extension of the code above
Update There was a missing "where" in the last true clause "1 = 1" - Sorry, added it now
Sorry, the formatting bar was not appearing so, I rebooted. Now I cannot edit my question or comment.
What I am trying to do is provide a search for the users.
They select a city from a dropdown then some optional data can be entered / selected to narrow the results.
The optional data May or May Not be present in the table, could be a blank field.
I would like the results to show based on the selected criteria of the search in that City.
So, WHERE selected "input city" = "tables city column" look for the other options (ignore that particular criteria if field is empty) and return any matches that exist for that city.
I am then pushing into array in a While for output display.
I guess it would be like a car query. Select make where doors = 2 and color = red and engine = hamsterwheel but, the color field may be empty in the database..