Node.js redis pubsub - mysql

I'm really beginner of nodejs. I want to make a chatting service using nodejs. I use nodejs/jade/mysql to construct basic part of my system and now I want to provide pub/sub to users.
We receive users' interests from text field or using hash tags (anyway we received users' interests and stored in MySQL -> we did it). Then, we want to show users chatting room list according to their interests. For instance A's interests are 'game', 'car' and 'food', then we search chat rooms with 'game', 'car', 'food' and show A these chat rooms first.
I want to use redis to provide this service but i really have no idea!
1) I installed redis and can run redis-server.
2)
//redis
var redis = require('redis');
var publisher = redis.createClient();
var subscriber = redis.createClient();
subscriber.on('message', function(channel, message){
console.log('Message ' + message + ' on channel ' + channel + ' arrived!');
});
subscriber.on('subscribe', function(channel){
publisher.publish('test', 'the a team');
publisher.publish('test', 'the b team');
})
subscriber.subscribe('test');
This is short code that I tried to understand redis.
3) I don't know how can I read data stored in Mysql and show users chat room according to their interests using redis.

Redis is a advanced key-value cache and store.Its operations cannot be directly mapped to mysql.
In redis you can set either key value pair or a hash under a key.
That is :
If you want to store your name in redis it can be done by:
var client = redis.createClient();
client.set("name", "John")
Retrieve the values using client.get("name")
Similarly under a single key you can store multiple key value pairs, as hash.
That under a name if you want to store their details like age, place, company etc.Then hash should be used.
Redis has method "hmset" and "hmget" for hash opertaions.
In redis like in cache you can set expiry time.
There are different method available. You can explore those.
For reference http://redis.io/commands

Related

multi-tenancy Join or multiple Select queries postgres

I am implementing multi tenancy using single database and separating data for each tenant using a tenant_id. This id is passed in the jwt token as well. I have two tables right now genre and books. genre table has columns like
tenant_id, genre_id, ..... and books table has columns genre_id, book_id, book_name, ....
So 1 genre can have multiple books associated with it and 1 tenant can have multiple genres associated with it.
Now every time a book is fetched or updated I want to make sure the right person is making these calls.
I know of two ways to do it.
First way:
Make two queries. First fetch the book, get the associated genre_id in the object. Then fetch that genre and compare the jwt tenant_id with the tenant_id inside this genre object.
Something like this
const book= await ReadBook(req.query.book_id); // fetches my book from db
const genre = await ReadBook(book.genre_id); // fetches the genre from db
if (genre.tenant_id === jwtToken.tenant_id) {..} // compare if same or not
Second way:
Do this query in db
select b.*, g.tenant_id as tenant_id
from book_data b, genre_data g
where b.book_id = '0eokdpz0l' and g.tenant_id = 'M1MzgzMDM' and b.genre_id = g.genre_id
Which method is more efficient?
If theres a more efficient method then these then please let me know too
It's good practice to stick to ORM abstraction if possible, while minimising how much and how often data is transferred to/from db. Sequelize is able to construct an equivalent to that query for you, with the necessary joins and filters on the ids. Something among the lines of:
Books.findAll({
where: {book_id: '0eokdpz0l'},
include: [{
model: Genre,
where: {tenant_id : jwtToken.tenant_id}
}]
}).then(books => {
/* ... */
});
Running multiple queries in sequence not only adds latency due to additional round trips to/from db (and possibly connection setup if you're not pooling or holding them open) but it's also moving more bytes of data around, needlessly. tenant_id mismatch on db would send back a shorter message with an empty result. Checking it on client side requires downloading data even when you'll have to discard it.

Register custom audit events for billing

We are running a custom app on Invantive Data Access Point which adds business functionality to Exact Online. For billing purposes, we would like to somehow register actual use of the software as defined in business terms instead of memory used, CPU, SQL statements executed, etc.
We do not yet have custom tables and I would like to keep it that way, so the whole state is kept in memory and in Exact Online only. So "insert into mytable#sqlserver..." is not an option. Neither does Exact Online offer the possibility to create custom tables as with Salesforce.
How can we somehow register billable events, such as "Performed an upload of 8 bank transactions" under this condition?
For billing purposes, you can lift along on the Customer Service infrastructure, which is similar to functionality offered by AWS or Apple for this purpose in their eco system. The "table" which stores the billing events like a Call Detail Record of a PBX is managed by Customer Service infrastructure.
There are two options:
Your apps use the default audit and license event registrations like "User logged on", "First use of partition #xyz", etc. each with a specific message code like 'itgenlic125'.
Your apps define their own event types like "Performed an upload of bank transactions", with a message code 'mybillingmessagecode123' and the number '8' as quantity in the natural key.
The first option is automatically and always done. These data is also used to manage resource consumption and detect runaways.
The second option is best done using Invantive SQL with the data dictionary table "auditevents". All records inserted into auditevents are automatically asynchronously forwarded to Customer Service. To see the current register audit events since start of application:
select *
from auditevents#datadictionary
where:
occurrence_date: when it happened.
logging_level: always "Audit".
message_code: code identifying the type of event.
data_container_d: ID of the data container, used with distributed SQL transactions.
partition: partition within the data container for platforms such as Exact Online or Microsoft SQL Server which store multiple databases under one customer/instance.
session_id: ID of the session.
user_message: actual text.
last_nk: last used natural key
application_name: name of the appplication.
application_user: user as known to the application.
gui_action: action within the GUI.
And some auditing and licensing information fields.
To register a custom event:
insert into auditevents#datadictionary select * from auditevents#datadictionary
Only some fields can be provided; the rest are automatically determined:
message_code
user_message
last_natural_key
application_name
application_user
gui_action
gui_module
partition
provider_name
reference_key
reference_table_code
session_id
To receive the billing events yourself from the infrastructure, you will need to access the Customer Service APIs or have them automatically forwarded to mail, Slack, RocketChat or Mattermost channel.
A sample SQL:
insert into auditevents#datadictionary
( message_code
, user_message
, last_natural_key
, application_name
, gui_action
, gui_module
, reference_key
, reference_table_code
, partition
)
select 'xxmycode001' message_code
, 'Processed PayPal payments in Exact Online for ' || divisionlabel user_message
, 'today' last_natural_key
, 'PayPalProcessor' application_name
, 'xx-my-paypal-processor-step-2' gui_action
, 'xx-my-payal-processor' gui_module
, clr_id reference_key
, 'clr' reference_table_code
, division partition
from settings#inmemorystorage

Firebase web authenticate and add to database

I am stuck on this and the other posts I have read on here are not useful. So I've reached a point where i need to ask for help after many hours on not resolving what I feel should be a simple task. I program in Swift usually and really know little about html or javasript.
I am building a simple webpage to log-in to Firebase and a second linked page to upload data to a database. Both work fine. The problem is getting the uploaded data to link to the uid of the current user.
So I am logged into an existing user with it's own uid. How do I then upload the data to the current user did in the database? Should be simple but I am just not getting it :-(
Code for uploading data is as follows (note I have tried using both set and push):
// Generate a reference to a new location and add some data using push()
var postsRef = ref.child("users");
var newPostRef = postsRef.push({
// var newPostRef = postsRef.set({
name: _name,
property: _property,
email: _email,
phone: _phone,
Any help, or better still a working simple example would be useful. I have read the docs on Firebase, so please don't direct me there :-)
Many thanks in anticipation
It is a best practice to create a new database node using the UID generated by the account creation as the path after /users.
Right now, when you push data into /users, Firebase creates a uid for that particular array item that does not correspond to the UID of the user.
If you use set, you need to specify the path you will set which should include the long UI: /users/longGUIDhere
You can get the user id with something like this (from Firebase docs):
var user = firebase.auth().currentUser;
var name, email, photoUrl, uid;
if (user != null) {
name = user.displayName;
email = user.email;
photoUrl = user.photoURL;
uid = user.uid; // The user's ID, unique to the Firebase project. Do NOT use
// this value to authenticate with your backend server, if
// you have one. Use User.getToken() instead.
}
And then you shouuld use uid to populate the path like below to save their info:
function writeUserData(userId, name, email, imageUrl) {
firebase.database().ref('users/' + userId).set({
username: name,
email: email,
profile_picture : imageUrl
});
}
I know you asked not to be referred to the Firebase docs, but it also looks like you are using an older version of the SDK, so that could be part the issue as well. I recommend taking a look at these two page, since that is where I pulled these verbatim examples:
https://firebase.google.com/docs/database/web/read-and-write
https://firebase.google.com/docs/auth/web/manage-users
var postsRef = ref.child("users/current user id or json key");
this will help you to update the details of current user.

Anyway to get dkims records for verifying ses domain in boto?

Tinkering around with verifying a couple of domains and found the manual process rather tedius. My DNS controller offers API access so I figured why not script the whole thing.
Trick is I can't figure out how to access the required TXT & CNAME records for DKIMS verification from boto, when I punch in
dkims = conn.verify_domain_dkim('DOMAIN.COM')
it adds DOMAIN.COM to the list of domains pending verification but doesn't provide the needed records, the returned value of dkims is
{'VerifyDomainDkimResponse': {
'ResponseMetadata': {'RequestId': 'REQUEST_ID_STRING'},
'VerifyDomainDkimResult': {'DkimTokens': {
'member': 'DKIMS_TOKEN_STRING'}}}}
Is there some undocumented way to take the REQUEST_ID or TOKEN_STRING to pull up these records?
UPDATE
If you have an aws account you can see the records I'm after at
https://console.aws.amazon.com/ses/home?region=us-west-2#verified-senders:domain
tab: Details:: Record Type: TXT (Text)
tab: DKIM:: DNS Record 1, 2, 3
these are the records required to add to the DNS controller to validate & allow DKIM signatures to take place
This is how I do it with python.
DOMINIO = 'mydomain.com'
from boto3 import Session
session = Session(
aws_access_key_id=MY_AWS_ACCESS_KEY_ID,
aws_secret_access_key=MY_AWS_SECRET_ACCESS_KEY,
region_name=MY_AWS_REGION_NAME)
client = session.client('ses')
# gets VerificationToken for the domain, that will be used to add a TXT record to the DNS
result = client.verify_domain_identity(Domain=DOMINIO)
txt = result.get('VerificationToken')
# gets DKIM tokens that will be used to add 3 CNAME records
result = client.verify_domain_dkim(Domain=DOMINIO)
dkim_tokens = result.get('DkimTokens') # this is a list
At the end of the code, you will have "txt" and "dkim_tokens" variables, a string and a list respectively.
You will need to add a TXT record to your dns, where the host name is "_amazonses" and the value is the value of "txt" variable.
Also you will need to add 3 CNAME records to your dns, one for each token present in "dkim_tokens" list, where the host name of each record is of the form of [dkimtoken]._domainkey and the target is [dkimtoken].dkim.amazonses.com
After adding the dns records, after some minutes (maybe a couple of hours), Amazon will detect and verify the domain, and will send you an email notification. After that, you can enable Dkim signature by doing this call:
client.set_identity_dkim_enabled(Identity=DOMINIO, DkimEnabled=True)
The methods used here are verify_domain_identity, verify_domain_dkim and set_identity_dkim_enabled.
You may also want to take a look a get_identity_verification_attributes and get_identity_dkim_attributes.
I think the get_identity_dkim_attributes method will return the information you are looking for. You pass in the domain name(s) you are interested in and it returns the status for that identity as well as the DKIM tokens.

Want to store in Redis via Node.js

I want to store a hash/JSON data of users in Redis and want to add the user in users hash the user data like this.
For example, users = {};
When user rahul logs in then users will become.
users = {
rahul: {
username: 'rahul',
}
}
And when user namita login then
users = {
rahul: {
username: 'rahul',
},
namita: {
username: 'namita',
}
}
What will the code be to do this in Redis?
How will I initialise the key users and add rahul to it?
Out of this set, hset, etc., which function do I need to use?
And how will I retrieve the data of users['rahul'] via Redis?
Probably the most optimal solution to store single hash/json would be to use hashes commands. I also had this "dilemma" and there are several questions regarding data structure containing users with JSON-like objects in Redis.
EDIT
Use node_redis module. It's actively maintained by a pro a probably the most used node.js driver for Redis. First you should use SADD command to add your new JSON object name into the set in order to track which items "users" contain. Then use HMSET to store "user:rahul" object key-value hashes. Example:
// add first user
client.sadd("users", "user:rahul");
client.hmset("user:rahul", "username", "rahul", "foo", "bar");
// add second user
client.sadd("users", "user:namita");
client.hmset("user:namita", "username", "namita", "foo", "baz");
You can now access your users by various types of redis hash command depending if you want to retrieve only certain hash values or the entire single user object. To get all of the users you can use SMEMBERS command for example. Try to look at the node_redis readme and examples where you should find more information about its API.