Keeping code modular when accessing a database - mysql

I've been working on the database for a very basic social networking site: all you can do is register for an account and send/accept friend requests. The server is written in Javascript (NodeJS)
I have a method getUser(username) that accesses the database to get the JSON object representing a particular user given their username, and a method to list the friends of a particular user. From the listFriends(username) function, I want to return an array of the user objects rather than just their usernames and ideally I would like to utilize my own get() function rather than altering the SQL query that listFriends() uses
It might be easier to understand with an example. If I have three tables:
TABLE: UsersName
username (unique) | firstName | lastName |
------------------|-----------|------------|
pjfry | Phillip | Fry |
proff | Professor | Farnsworth |
bender | Bender | Rodriguez |
TABLE: UsersProfile (their profile description)
username (unique) | description |
------------------|------------------------|
pjfry | I went to the future |
proff | I am very old |
bender | Destroy all humans |
TABLE: Friendships (for simplicity assume that if (a,b) is an entry then so is (b,a))
user1 | user2
-----------|---------------
bender | pjfry
pjfry | bender
pjfry | proff
proff | pjfry
And a function to get the user object:
//the callback accepts the user object
function get (username, callback) {
db.query(
'select * from (UsersName n, UsersProfile p) where n.username=p.username and n.username=\'' + username + '\'',
function (rows) {
//for simplicity assume that this call always succeeds
//the user object is altered a bit:
callback({
username: rows[0].username,
name: {
first: rows[0].firstName,
last: rows[0].lastName,
full: rows[0].firstName + ' ' + rows[0].lastName
},
profile: {
description: rows[0].description
}
});
}
}
And here is the function to list the friends of a given user
//callback accepts the array of friends
function listFriends(username, callback) {
db.query(
'select user2 from Friendships where user1=\'' + username + '\'',
function(rows) {
//assume this call always succeeds
callback(rows)
}
)
}
The problem here is that listFriends() will just return the array of usernames rather than user objects. How could I modify the listFriends() function so it returns the user objects by utilizing the get() function?
It could be done by modifying the SQL statement in listFriends() but it would be much cleaner to use the get() method so that if the structure of the user object is ever changed, it only needs to be changed in one place.

Trying to keep things as DRY as possible, something like this should fit your requirements:
// Disclaimer: I have not run tested this code myself
//the callback accepts the user object
function get(username, callback) {
getFriendsWithConditions('n.username=\'' + username + '\'', function(rows){
// returns only a single object to the input callback
callback(rows[0]);
});
}
//callback accepts the array of friends
function listFriends(username, callback) {
getFriendsWithConditions('n.username in (select user2 from Friendships where user1=\'' + username + '\')', callback);
}
function getFriendsWithConditions(whereCondition, callback) {
db.query(
'select * from (UsersName n, UsersProfile p) where n.username=p.username and (' + whereCondition + ')',
function(rows) {
// Using ECMAScript 5 Array.map
callback(rows.map(rowToUserObject));
}
);
}
function rowToUserObject(row)
{
return {
username: row.username,
name: {
first: row.firstName,
last: row.lastName,
full: row.firstName + ' ' + row.lastName
},
profile: {
description: row.description
}
};
}
A few things that came to mind as I was writing this:
UserName and UserProfile feel like they should only be one table, but for all I know this is a simplified version of your actual database, so I left them as is.
I used Array.map as defined in ECMAScript 5 which should be available in Node.
The SQL alias names for each table are hardcoded throughout each function, meaning that you would also need to a) set some sort of constant or come of with a convention to keep all aliases consistent throughout all your functions (which can be a pain to maintain as the project grows) or b) look into using an ORM that can do these things for you (and more!) This will help you avoid sweating the details of how to construct queries, leaving you to worry about more important things, like what data you actually need. I'm not terribly familiar with what's available for NodeJS in terms of ORMs, but the official NodeJS wiki should be a good place start.

Might it be better not to destroy the existing functionality of listFriends()?
Instead, my thought would be to create a new function called getFriends() which calls listFriends() and for each returned username, calls get() for it. The concatenated series of results from each get() would serve as the return for getFriends().

This is a fairly simple solution that does what I want in 2 network calls (rather than 1, like I had originally hoped to do).
I added a function getUserArray(usernames) that takes an array of usernames and returns the objects from that. It does this by generating SQL that looks like: "SELECT ... FROM (UsersName n, UsersProfile p) WHERE n.username=p.username and n.username in (?, ?, ?, ...) order by field(username, ?, ?, ?, ...)" and passes the array of usernames as the values to be escaped and replaced with the ?'s.
The functions getUserArray() and getUser() share a lot in common, so common parts of the code for those can easily be put into common helper functions.
The functionality of listFriends() is augmented as follows: just before it would have returned the array of usernames, it calls getUserArray() to map the usernames to objects.
Simple, and only 2 network calls. And it keeps listFriends() from messing with the structure of the user metadata tables.

Related

Convert Dapper raw SQL Result to Nested JSON Array

I have developed a web api in .Net Core 5 which uses dapper to run a tabled valued function and return the SQL results. These results are then used to fill various select boxes on the front end in VueJS. However, when I began to build out my front end more I realized my JSON arrays could be nested to really help reduce the number of requests I make to the server as my select boxes are dependent. For instance, one select box includes states and then next select box relates to the cities in those states. Adjusting the tabled value function to return a single table was easy by adding a innerjoin between my state table in the database and the cities table. The joining key was a field called STATE_ID. Therefore I just have multiple rows due to multiple cities per state. So now what I am trying to figure out is how to take this result in my web api and my table valued function result without the use of models into a nested json array such that my results are as follows:
[{state: 'Maryland', cities :[{city: 'Baltimore'}, {city: 'Harford County'}]} ,
{state: 'Pennsylvania', cities :[{city: 'York'}, {city: 'Fawn Grove'}]}]
Table valued function result from A2Q00001_StateInfo(USERNUMBER):
| State_ID | State_Name | City_Name |
|---------------------|------------------|---------------------|
| 1 | Maryland | Baltimore |
| 1 | Maryland | Harford County |
| 2 | Pennsylvania | York |
| 2 | Pennsylvania | Fawn Grove |
My controller is as follows:
public ActionResult StateAndCities([FromQuery] String USERNUMBER)
{
//We have parameters here just in case we want to use them
IEnumerable queryResult;
String query = "select * from dbo.A2Q00001_StateInfo(#USERNUMBER);";
using (var connection = new SqlConnection(connectionString))
{
queryResult = connection.Query(query, new { USERNUMBER = USERNUMBER });
}
return Ok(queryResult);
}
All of the tutorials I have seen online use models to create the nested JSON object and return it however I am not sure how to create the nested object using the serialization in the Ok() function in asp.net core. Is this even posssible or do I need to perform operations on the queryResult from the dapper query? Any point in the right direction would be great.
My advice: split this into steps. I'm guessing your A2Q00001_StateInfo UDF here returns a State and City column (edit: I was close, it was State_Name, via the edit), among other things. So first step: let's just read that:
class SomeType
{
public string State_Name { get; set; }
public string City { get; set; }
}
//...
var queryResult = connection.Query<SomeType>(
"select State_Name, City from dbo.A2Q00001_StateInfo(#USERNUMBER);",
new { USERNUMBER }).AsList();
This gets our data from the database into local memory. Note that I filtered out irrelevant columns to reduce overheads.
Now, the next step is to structure that data - it looks like you want to aggregate by state, and create an array of the cities in each; so: let's do that:
var structured =
from grp in queryResult.GroupBy(x => x.State_Name)
select new
{
state = grp.Key,
cities = grp.Select(row => new { city = row.City }).ToArray()
};
This gives us a projection (using anonymous types) that does the restructuring we want. Finally, we need to convert it to JSON; this might be as simple as:
return Ok(structured);
Or you might need to use the Json/JsonResult APIs directly. However, now that the data is structured: any JSON serializer should know what we want to do here.
Note: you probably can rewrite all this into a single expression, but: don't do that; you're not trying to impress the compiler - it won't care either way. Make the code clear and obvious for the next person who is going to need to touch it (which might well be you).

Multi-parameter search with mysql and node.js

Let me preface by saying I'm very new to SQL (and back end design) in general. So for those annoyed with noob questions, please be gentle.
BACKGROUND:
I'm trying to build a product test database (storing test data for all our products) where I want a user to be able to refine a search to find test data they actually want. For example, they may start by searching for all products of a certain brand name, and then refine it with a product type, and/or refine it with a date range of when the test was done.
PROBLEM:
I'm having a hard time finding information on how to implement multi-parameter searches with mysql and node.js. I know you can do nested queries and joins and such within pure SQL syntax, but it's not abundantly clear to me how I would do this from node.js, especially when certain search criteria aren't guaranteed to be used.
Ex:
CREATE PROCEDURE `procedureName`(
IN brandname VARCHAR(20),
producttype VARCHAR(30))
BEGIN
SELECT * FROM products
WHERE brand = brandname
AND product_type = producttype;
END
I know how to pass data from node.js to this procedure, but what if the user didn't specify a product type? Is there a way to nullify this part of the query? Something like:
AND product_type = ALL;
WHAT I'VE TRIED:
I've also looked into nesting multiple SQL procedures, but passing in dynamic data to the "FROM" clause doesn't seem to be possible. Ex: if I had a brandname procedure, and a product type procedure, I don't know how/if I can pass the results from one procedure to the "FROM" clause of the other to actually refine the search.
One idea was to create tables with the results in each of these procedures, and pass those new table names to subsequent procedures, but that strikes me as an inefficient way to do this (Am I wrong? Is this a completely legit way to do this?).
I'm also looking into building a query string on the node side that would intelligently decide what search criteria have been specified by the front end, and figure out where to put SQL AND's and JOIN's and what-nots. The example below actually works, but this seems like it could get ugly quick as I add more search criteria, along with JOINS to other tables.
// Build a SQL query based on the parameters in a request URL
// Example request URL: http://localhost:3000/search?brand=brandName&type=productType
function qParams(req) {
let q = "SELECT * FROM products WHERE ";
let insert = [];
if(req.query.brand) {
brandname = req.query.brand; // get brandname from url request
q = q + `brand = ?`, // Build brandname part of WHERE clause
insert.push(brandname); // Add brandname to insert array to be used with query.
};
if(req.query.type) {
productType = req.query.type; // get product type from url request
insert.length > 0 ? q = q + ' AND ' : q = q; // Decide if this is the first search criteria, add AND if not.
q = q + 'product_type = ?'; // Add product_type to WHERE clause
insert.push(productType); // Add product_type variable to insert array.
}
// Return query string and variable insert array
return {
q: q,
insert: insert
};
};
// Send Query
async function qSend(req, res) {
const results = await qParams(req); // Call above function, wait for results
// Send query string and variables to MySQL, send response to browser.
con.query(results.q, results.insert, (err, rows) => {
if(err) throw err;
res.send(rows);
res.end;
})
};
// Handle GET request
router.use('/search', qSend);
CONCISE QUESTIONS:
Can I build 1 SQL procedure with all my search criteria as variables, and nullify those variables from node.js if certain criteria aren't used?
Is there way to nest multiple MySQL procedures so I can pick the procedures applicable to the search criteria?
Is creating tables of results in a procedure, and passing those new table names to other procedures a reasonable way to do that?
Building the query from scratch in node is working, but it seems bloated. Is there a better way to do this?
Googling "multi-parameter search mysql nodejs" is not producing useful results for my question, i.e. I'm not asking the right question. What is the right question? What do I need to be researching?
One option is to use coalesce():
SELECT p.*
FROM products p
WHERE
p.brand = COALESCE(:brandname, p.brand)
AND p.product_type = COALESCE(:producttype, p.producttype);
It may be more efficient do explicit null checks on the parameters:
SELECT p.*
FROM products p
WHERE
(:brandname IS NULL OR p.brand = :brandname)
AND (:producttype IS NULL OR p.product_type = :producttype);

why a variable defined in a node module is shared between two users

I run a node API server with pm2 cluster mode that will communicate with a mysql DB server.
In module x.js I have a code like this:
let insertMappingQuery = ``;
...
...
const constructInsertMappingQuery = () => {
insertMappingQuery += `
INSERT IGNORE INTO messages_mapping (message_id, contact_id)
VALUES (` + message_id + `, ` + contact_id + ` + `);`;
}
When a user sends a message a function will call module x and the code above is executed for his message (let's say message_id = 1)
INSERT IGNORE INTO messages_mapping (message_id, contact_id)
VALUES (1, some_id);
then another user sends a message and the code is executed for let's say message_id = 2 however the query will look like this:
INSERT IGNORE INTO messages_mapping (message_id, contact_id)
VALUES (1, some_id);
INSERT IGNORE INTO messages_mapping (message_id, contact_id)
VALUES (2, some_id);
So basically when user two sends a message, this query will contain what user one already executed. So user one will have his record inserted twice.
This doesn't happen all the time but it happens a lot (I would say 30% to 50%) and I couldn't find any pattern when this happens.
Users don't have to do it at the same time, there might be some time difference (minutes or even hours).
could this be related to the variable not being cleared in the memory? or a memory leakage of some kind?
I don't understand how two different users will share a variable.
Remember that require caches modules and all subsequent require calls are given the same things, so write something that exports a function, or class, so that you can safely call/instantiate things without variables getting shared.
For example:
const db = require(`your/db/connector`);
const Mapper = {
addToMessageMapping: async function(messageId, contactId) {
const query = `
INSERT IGNORE INTO messages_mapping (message_id, contact_id)
VALUES (${message_id}, ${contact_id});
`;
...
return db.run(query);
},
...
}
module.exports = Mapper;
And of course this could have been a class, too, or it could even have been that function directly - the only thing that changes is how you make it run that non-conflicting-with-any-other-call function.
Now, consumers of this code simply trust that the following is without side effects:
const mapper = require('mapper.js');
const express, app, etc, whatever = ...
....
app.post(`/api/v1/mappings/message/:msgid`, (req, res, next) => {
const post = getPOSTparamsTheUsualWay();
mapper.addToMessageMapping(req.params.msgId, post.contactId)
.then(() => next());
.catch(error => next(error));
}, ..., moreMiddleware, ... , (req,res) => {
res.render(`blah.html`, {...});
});
Also note that template strings exist specifically to prevent string composition by concatenating strings with +, the whole point is that they can take ${...} inside them and template in "whatever is in those curly braces" (variables, function calls, any JS really).
(The second power they have is that you can prefix tag them with a function name and that function will run as part of the templating action, but not a lot of folks need this on a daily basis. ${...} templating though? Every day, thousands of times).
And of course on a last note: it looks like you're creating raw SQL, which is always a bad idea. Use prepared statements for whatever database library you're using: it supports them, and means any user-input is made safe. Right now, someone could post to your API with a message id that's ); DROP TABLE messages_mapping; -- and done: your table's gone. Fun times.
Apparently I didn't know that requireing a module will cache it and reuse it. Thus global variables in that module will be cached too.
So the best solution here is to avoid using global variables and restructure the code. However if you need a quick solution you can use:
delete require.cache[require.resolve('replaceWithModulePathHere')]
Example:
let somefuncThatNeedsModuleX = () => {
delete require.cache[require.resolve('./x')];
const x = require('./x');
}

Query Google Admin Directory users for partial name match

I'm trying to do two things:
Query AdminDirectory.Users.list by name field to return a list of anyone whose name matches a given string (even partially)
If possible, prevent the email address field from being included in the search
For example, "donald" should return data for the users Donald Duck, Donald Trump, and Ronald McDonald. If someone searched for "onald", that should work too.
The below sort of works. In the "donald" scenario, it would return only Donald Duck's data. For some reason it won't return more than one user.
function processForm(formObject) { // formObject comes from form on front end
var textSearchObject = formObject.textSearch; // "donald"
var userList = AdminDirectory.Users.list({
domain: 'somedomain.com',
query: "name:'" + textSearchObject + "'",
viewType: 'domain_public',
projection: 'full'
}).users;
return userList;
}
I know, query: "name:'" + textSearchObject + "'" looks very strange and most people would just use query: textSearchObject. The problem is that this searches email addresses - I need to avoid this if possible.
You're limited to "starts with" searching e.g. Donald* but you can search just the givenName, familyName or the combined name (fullName). Full documentation is here: https://developers.google.com/admin-sdk/directory/v1/guides/search-users

Play + Slick: How to do partial model updates?

I am using Play 2.2.x with Slick 2.0 (with MYSQL backend) to write a REST API. I have a User model with bunch of fields like age, name, gender etc. I want to create a route PATCH /users/:id which takes in partial user object (i.e. a subset of the fields of a full user model) in the body and updates the user's info. I am confused how I can achieve this:
How do I use PATCH verb in Play 2.2.x?
What is a generic way to parse the partial user object into an update query to execute in Slick 2.0?I am expecting to execute a single SQL statement e.g. update users set age=?, dob=? where id=?
Disclaimer: I haven't used Slick, so am just going by their documentation about Plain SQL Queries for this.
To answer your first question:
PATCH is just-another HTTP verb in your routes file, so for your example:
PATCH /users/:id controllers.UserController.patchById(id)
Your UserController could then be something like this:
val possibleUserFields = Seq("firstName", "middleName", "lastName", "age")
def patchById(id:String) = Action(parse.json) { request =>
def addClause(fieldName:String) = {
(request.body \ fieldName).asOpt[String].map { fieldValue =>
s"$fieldName=$fieldValue"
}
}
val clauses = possibleUserFields.flatMap ( addClause )
val updateStatement = "update users set " + clauses.mkString(",") + s" where id = $id"
// TODO: Actually make the Slick call, possibly using the 'sqlu' interpolator (see docs)
Ok(s"$updateStatement")
}
What this does:
Defines the list of JSON field names that might be present in the PATCH JSON
Defines an Action that will parse the incoming body as JSON
Iterates over all of the possible field names, testing whether they exist in the incoming JSON
If so, adds a clause of the form fieldname=<newValue> to a list
Builds an SQL update statement, comma-separating each of these clauses as required
I don't know if this is generic enough for you, there's probably a way to get the field names (i.e. the Slick column names) out of Slick, but like I said, I'm not even a Slick user, let alone an expert :-)