In Express, it's very easy to block access to all routes starting, say, with the /admin prefix, simply by adding a middleware to that path before adding handlers for any specific endpoints under that path.
In Feathers, it looks like we have to create some common hook modules and add them to every service we created, individually. Same goes for event filters.
I find the thought of forgetting to add an authentication hook or event filter scary, because I wouldn't notice the mistake unless I reviewed all service initialization code or got hacked. In that sense, an Express middleware with some sort of white-listing I can easily implement for exceptional endpoints gives me much more peace of mind.
Is it possible to do something like that in Feathers?
(P.S.: I just noticed I had protected my app's REST API, but had forgotten to protect all real-time events).
As of v1.6.0 feathers-hooks supports application hooks that run on all services via app.hooks:
app.hooks({
before(hook) {
console.log('Global before hook');
},
after(hook) {
console.log('Global after hook');
},
error(hook) {
console.error(`Error in ${hook.path} method ${hook.method}`, hook.error.stack);
}
});
For more examples see this blog post about error and application hooks.
As for the real-time events, channels are used which provide a safe way to send events only to the clients that should see them.
Related
I would like to serve a Next.js app in europe using Firebase Hosting & Functions capabilities.
I do understand from the doc that:
If you are using HTTP functions to serve dynamic content for Firebase
Hosting, you must use us-central1
and that
Firebase Hosting supports Cloud Functions in us-central1 only
It's pretty clear: you must use us-central. But my main target is Europe..
I've read the following on the Cloud Functions locations guide:
For HTTP and callable functions, we recommend that you first set your
function to the destination region, or closest to where most expected
customers are located, and then alter your original function to
redirect its HTTP request to the new function (they can have the same
name). [Solution 1] If clients of your HTTP function support
redirects, you can simply change your original function to return an
HTTP redirect status (301) along with the URL of your new function.
[Solution 2] If your clients do not handle redirects well, you can
proxy the request from the original function to the new function by
initiating a new request from the original function to the new
function. The final step is to ensure that all clients are calling the
new function.
I've highlighted what seems to be two solutions to my initial problem:
Solution 1
Have a us-central1 function that send a 301 redirection to https://europe-west1-[myProject].cloudfunctions.net/[myEuropeanFunction]
Have a europe-west1 function that does the job (in my case, serve the Next.js app)
Happily using Firestore located in europe-west1
This would only work if clients of the HTTP function support redirects. In my case, it's fine: all browsers support redirection.
exports.nextServer = functions
.https
.onRequest((req, res) => {
res.set('location', 'https://europe-west1-<my-project>.cloudfunctions.net/nextServerEurope');
res.status(301).send()
});
exports.nextServerEurope = functions
.region('europe-west1')
.https
.onRequest((req, res) => {
return server.prepare().then(() => nextjsHandle(req, res));
});
The issue with that solution is that the URL changes in the browser to https://europe-west1-.cloudfunctions.net/nextServerEurope :-/
Solution 2
Have a us-central1 function that initiate a new/proxy request to the europe-west1 function
Have the same europe-west1 function that does the job (in my case, serve the Next.js app)
Still happily using Firestore located in europe-west1
By proxy request (as suggested in the guide), it would mean using a lib like axios I suppose. I know there are some libraries to perform proxy request available for node as well.
However, with that solution, the first issue I can think of is the unnecessary delay introcuded by passing by the us endpoint:
client -> us endpoint -> eu endpoint -> do stuff -> us endpoint -> client
Billing wise, I'm wondering what would be the impact..
I know that two services from different regions calling each others can increase the latency and the billing (egress).
With the first solution, there's no egress traffic as it's only a redirection to the european endpoint. But the redirection itself is not a valid solution in my case.
It's unclear for me what would be the additional billing cost with the second solution (beside the latency cost): is the traffic for the proxy request from us to eu going to be expensive?
To wrap-up:
The solution 1 is easy but leads to a non-transparent redirection
The solution 2 seems ok but it requires extra http request which leads to extra-latency (and potentially extra billing)
In the end, both solutions doesn't seem quite okay.
Therefore my question:
How do you serve in Europe dynamic content using Firebase Hosting and Functions?
Firebase Hosting only supports Cloud Functions in Us-Central as you mentioned and as stated in the Firebase Hosting Official Documentation.
I have created a Feature Request in Public Issue Tracker to support other regions when using Firebase Hosting with Cloud Functions. Please note, there is no ETA when this will be implemented.
So as #Doug Stevenson suggest, you can use Firebase Hosting with Cloud Run instead to serve your Dynamic Content.
Just to update. As of August 2022.
Finally, latency issue can be solved easily for now.
Firebase Hosting rewrites to CF3 are able to be done to any CF3
region, not just us-central1.
Reference: Feature Request Ticket
I want to restrict calls to a Feathers service method for externals calls with associateCurrentUser.
I also want to allow the server to call this service method without restricting it.
The use case is that through this service then clients use a lock table, all clients can see all locks, and occasionally the server should clear out abandoned rows in this table. Row abandonment can happen on network failures etc. When the server removes data then the normal Feathers remove events should be emitted to the clients.
I would imagine that this should be a mix of associateCurrentUser and disallow hooks but I can't even begin to experiment with this as I don't see how it would be put together.
How would one implement this, please?
Update:
I found this answer User's permissions in feathers.js API from Daff which implies that if the hook's context.params.provider is null then the call is internal, otherwise external. Can anyone confirm if this is really so in all cases, please?
It seems to be so from my own tests but I don't know if there are any special cases out there that might come and bite me down the line.
If the call is external params.provider will be set to the transport that has been used (currently either rest, socketio or primus, documented here, here and here).
If called internally on the server there is not really any magic. It will be whatever you pass as params. If you pass nothing it will be undefined if you pass (or merge with) hook.params in a hook it will be the same as what the original method was called with.
// `params` is an empty object so `params.provider` will be `undefined`
app.service('messages').find({})
// `params.provider` will be `server`
app.service('messages').find({ provider: 'server' })
// `params.provider` will be whatever the original hook was called with
function(hook) {
hook.app.service('otherservice').find(hook.params);
}
I'm looking for a way to create a simple HTML table that can be updated in real-time upon a database change event; specifically a new record added.
In other words, think of it like an executive dashboard. If a sale is made and a new line is added in a database (MySQL in my case) then the web page should "refresh" the table with the new line.
I have seen some information on the new using EVENT GATEWAY but all of the examples use Coldfusion as the "pusher" and not the "consumer". I would like to have Coldfusion both update / push an event to the gateway and also consume the response.
If this can be done using a combination of AJAX and CF please let me know!
I'm really just looking to understand where to get started with real-time updating.
Thank you in advance!!
EDIT / Explanation of selected answer:
I ended up going with #bpeterson76's answer because at the moment it was easiest to implement on a small scale. I really like his Datatables suggestion, and that's what I am using to update in close to real time.
As my site gets larger though (hopefully), I'm not sure if this will be a scalable solution as every user will be hitting a "listener" page and then subsequently querying my DB. My query is relatively simple, but I'm still worried about performance in the future.
In my opinion though, as HTML5 starts to become the web standard, the Web Sockets method suggested by #iKnowKungFoo is most likely the best approach. Comet with long polling is also a great idea, but it's a little cumbersome to implement / also seems to have some scaling issues.
So, let's hope web users start to adopt more modern browsers that support HTML5, because Web Sockets is a relatively easy and scalable way to get close to real time.
If you feel that I made the wrong decision please leave a comment.
Finally, here is some source code for it all:
Javascript:
note, this is a very simple implementation. It's only looking to see if the number of records in the current datatable has changed and if so update the table and throw an alert. The production code is much longer and more involved. This is just showing a simple way of getting a close to real-time update.
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.6.1/jquery.js"></script>
<script type="text/javascript" charset="utf-8">
var originalNumberOfRecsInDatatable = 0;
var oTable;
var setChecker = setInterval(checkIfNewRecordHasBeenAdded,5000); //5 second intervals
function checkIfNewRecordHasBeenAdded() {
//json object to post to CFM page
var postData = {
numberOfRecords: originalNumberOfRecsInDatatable
};
var ajaxResponse = $.ajax({
type: "post",
url: "./tabs/checkIfNewItemIsAvailable.cfm",
contentType: "application/json",
data: JSON.stringify( postData )
})
// When the response comes back, if update is available
//then re-draw the datatable and throw an alert to the user
ajaxResponse.then(
function( apiResponse ){
var obj = jQuery.parseJSON(apiResponse);
if (obj.isUpdateAvail == "Yes")
{
oTable = $('#MY_DATATABLE_ID').dataTable();
oTable.fnDraw(false);
originalNumberOfRecsInDatatable = obj.recordcount;
alert('A new line has been added!');
}
}
);
}
</script>
Coldfusion:
<cfset requestBody = toString( getHttpRequestData().content ) />
<!--- Double-check to make sure it's a JSON value. --->
<cfif isJSON( requestBody )>
<cfset deserializedResult = deserializeJSON( requestBody )>
<cfset numberOFRecords = #deserializedResult.originalNumberOfRecsInDatatable#>
<cfquery name="qCount" datasource="#Application.DBdsn#" username="#Application.DBusername#" password="#Application.DBpw#">
SELECT COUNT(ID) as total
FROM myTable
</cfquery>
<cfif #qCount.total# neq #variables.originalNumberOfRecsInDatatable#>
{"isUpdateAvail": "Yes", "recordcount": <cfoutput>#qCount.total#</cfoutput>}
<cfelse>
{"isUpdateAvail": "No"}
</cfif>
</cfif>
This isn't too difficult. The simple way would be to add via .append:
$( '#table > tbody:last').append('<tr id="id"><td>stuff</td></tr>');
Adding elements real-time isn't entirely possible. You'd have to run an Ajax query that updates in a loop to "catch" the change. So, not totally real-time, but very, very close to it. Your user really wouldn't notice the difference, though your server's load might.
But if you're going to get more involved, I'd suggest looking at DataTables. It gives you quite a few new features, including sorting, paging, filtering, limiting, searching, and ajax loading. From there, you could either add an element via ajax and refresh the table view, or simply append on via its API. I've been using DataTables in my app for some time now and they've been consistently cited as the number 1 feature that makes the immense amount of data usable.
--Edit --
Because it isn't obvious, to update the DataTable you call set your Datatables call to a variable:
var oTable = $('#selector').dataTable();
Then run this to do the update:
oTable.fnDraw(false);
UPDATE -- 5 years later, Feb 2016:
This is much more possible today than it was in 2011. New Javascript frameworks such as Backbone.js can connect directly to the database and trigger changes on UI elements including tables on change, update, or delete of data....it's one of these framework's primary benefits. Additionally, UI's can be fed real-time updates via socket connections to a web service, which can also then be caught and acted upon. While the technique described here still works, there are far more "live" ways of doing things today.
You can use SSE (Server Sent Events) a feature in HTML5.
Server-Sent Events (SSE) is a standard describing how servers can initiate data transmission towards clients once an initial client connection has been established. They are commonly used to send message updates or continuous data streams to a browser client and designed to enhance native, cross-browser streaming through a JavaScript API called EventSource, through which a client requests a particular URL in order to receive an event stream.
heres a simple example
http://www.w3schools.com/html/html5_serversentevents.asp
In MS SQL, you can attach a trigger to a table insert/delete/update event that can fire a stored proc to invoke a web service. If the web service is CF-based, you can, in turn, invoke a messaging service using event gateways. Anything listening to the gateway can be notified to refresh its contents. That said, you'd have to see if MySQL supports triggers and accessing web services via stored procedures. You'd also have to have some sort of component in your web app that's listening to the messaging gateway. It's easy to do in Adobe Flex applications, but I'm not sure if there are comparable components accessible in JavaScript.
While this answer does not come close to directly addressing your question, perhaps it will give you some ideas as to how to solve the problem using db triggers and CF messaging gateways.
M. McConnell
With "current" technologies, I think long polling with Ajax is your only choice. However, if you can use HTML5, you should take a look at WebSockets which gives you the functionality you want.
http://net.tutsplus.com/tutorials/javascript-ajax/start-using-html5-websockets-today/
WebSockets is a technique for two-way communication over one (TCP) socket, a type of PUSH technology. At the moment, it’s still being standardized by the W3C; however, the latest versions of Chrome and Safari have support for WebSockets.
http://html5demos.com/web-socket
Check out AJAX long polling.
Place to start Comet
No, you can't have any db code execute server side code. But you could write a service to poll the db periodically to see if a new record has been added then notify the code you have that needs pseudo real-time updates.
The browser can receive real-time updates via BOSH connection to Jabber/XMPP server. All bits and pieces can be found in this book http://professionalxmpp.com/ which I highly recommend. If you can anyhow send XMPP message upon record addition in your DB, then it is relatively easy to build the dashboard you want. You need strophe.js, Jabber/XMPP server (e.g. ejabberd), http server for proxying http-bind requests. All the details can be found in the book. A must read which I strongly believe will solve your problem.
The way I would achieve the notification is after the database update has been successfully committed I would publish an event that would tell any listening systems or even web pages that the change has occurred. I've detailed one way of doing this using an e-commerce solution in a recent blog post. The blog post shows how to trigger the event in ASP.NET but the same thing can easily be done in any other language since ultimately the trigger is performed via a REST API call.
The solution in this blog post uses Pusher but there's not reason why you couldn't install your own real-time server or use a Message Queue to communication between your app and the realtime server, which would then push the notification to the web page or client application.
I want to create some kind of AJAX script or call that continuously will check a MySQL database if any new messages has arrived. When there is a new message in the database, the AJAX script should invoke a kind of alert box or message box.
I’m not quite a AJAX expert (yet anyway) and have Googled around to find a solution but I’m having a hard time to figure out where to begin. I imagine that it is kind of the same method that an AJAX chat is using to see if any new chat-message has been send.
I’ve also tried to search for AJAX (httpxmlrequest) call in a continuously and infinity loop but still haven’t got a solution yet.
I hope there is someone, which can help me with such a AJAX script or maybe nudge me in the right direction.
Thanks
Sincerely
Mestika
Step 1 - You need a server-side page that you can call that checks to see if something new has arrived.
Step 2 - You could adapt the sequential AJAX request script from here (it uses jQuery to simplify the AJAX requests):
http://www.stevefenton.co.uk/Content/Blog/Date/201004/Blog/AJAX-and-Race-Conditions/
Currently, this script is for queuing a list of sequential AJAX requests, but you could use it to continually check by changing it like this...
var InfiniteAjaxRequest = function (uri) {
$.ajax({
url: uri,
success: function(data) {
// do something with "data"
if (data.length > 0) {
alert(data);// Do something sensible with it!
}
InfiniteAjaxRequest (uri);
},
error: function(xhr, ajaxOptions, thrownError) {
alert(thrownError);
}
});
};
InfiniteAjaxRequest ("CheckForUpdate.php");
What are the benefits of using this script?
Well, rather than checking every "x" seconds, it will only check once the previous request has been received, so it chains the requests. You could add in a delay to throttle this constant request, which I would highly recommend you do - otherwise you will be hitting your site with way too much traffic. You would add that delay in AFTER you've done something with the response, but BEFORE you call back into "InfiniteAjaxRequest".
Here's your nudge:
Get one of the available JavaScript frameworks (jQuery seems to be the most common, but there are others)
flip though the documentation on the AJAX methods it provides, choose a method for your task that seems appropriate
build a request to your site that fetches the info and reacts on the response (shows a message box or updates some part of your page), wrap that in a function
make sure request errors do not go unnoticed by implementing an error handler
check out setInterval() to call that function you've just made repeatedly
final step: make sure that the interval will be stopped in case of an error condition (or provide a on/off button for the user, even) so the server is not hammered needlessly
There is a technique called Comet where-by your client-side script would instantiate a HTTP request which remains open for a long time. The server can then push data into the response as they happen. It's a technique to deliver a push notification.
The Wikipedia link has more information on real-world implementations.
Instead of polling the server with AJAX calls you could also use push technology (COMET).
This way you can push the results to the client(s) as soon as the server is done with it's work.
There are many frameworks available like:
JQuery plugin
Cometd
Atmosphere (if your on java)
Is there any way to "subscribe" from GWT to JSON objects stream and listen to incoming events on keep-alive connection, without trying to fetch them all at once? I believe that the buzzword-du-jour for this technology is "Comet".
Let's assume that I have HTTP service which opens keep-alive connection and put JSON objects with incoming stock quotes there in real time:
{"symbol": "AAPL", "bid": "88.84", "ask":"88.86"}
{"symbol": "AAPL", "bid": "88.85", "ask":"88.87"}
{"symbol": "IBM", "bid": "87.48", "ask":"87.49"}
{"symbol": "GOOG", "bid": "305.64", "ask":"305.67"}
...
I need to listen to this events and update GWT components (tables, labels) in realtime. Any ideas how to do it?
There is a GWT Comet Module for StreamHub:
http://code.google.com/p/gwt-comet-streamhub/
StreamHub is a Comet server with a free community edition. There is an example of it in action here.
You'll need to download the StreamHub Comet server and create a new SubscriptionListener, use the StockDemo example as a starting point, then create a new JsonPayload to stream the data:
Payload payload = new JsonPayload("AAPL");
payload.addField("bid", "88.84");
payload.addField("ask", "88.86");
server.publish("AAPL", payload);
...
Download the JAR from the google code site, add it to your GWT projects classpath and add the include to your GWT module:
<inherits name="com.google.gwt.json.JSON" />
<inherits name="com.streamhub.StreamHubGWTAdapter" />
Connect and subscribe from your GWT code:
StreamHubGWTAdapter streamhub = new StreamHubGWTAdapter();
streamhub.connect("http://localhost:7979/");
StreamHubGWTUpdateListener listener = new StockListener();
streamhub.subscribe("AAPL", listener);
streamhub.subscribe("IBM", listener);
streamhub.subscribe("GOOG", listener);
...
Then process the updates how you like in the update listener (also in the GWT code):
public class StockListener implements StreamHubGWTUpdateListener {
public void onUpdate(String topic, JSONObject update) {
String bid = ((JSONString)update.get("bid")).stringValue();
String ask = ((JSONString)update.get("ask")).stringValue();
String symbol = topic;
...
}
}
Don't forget to include streamhub-min.js in your GWT projects main HTML page.
I have used this technique in a couple of projects, though it does have it's problems. I should note that I have only done this specifically through GWT-RPC, but the principle is the same for whatever mechanism you are using to handle data. Depending on what exactly you are doing, there might not be much need to over complicate things.
First off, on the client side, I do not believe that GWT can properly support any sort of streaming data. The connection has to close before the client can actually process the data. What this means from a server-push standpoint is that your client will connect to the server and block until data is available at which point it will return. Whatever code executes on the completed connection should immediately re-open a new connection with the server to wait for more data.
From the server side of things, you simply drop into a wait cycle (the java concurrent package is particularly handy for this with blocks and timeouts), until new data is available. At that point in time, the server can return a package of data down to the client which will update accordingly. There are a bunch of considerations depending on what your data flow is like, but here are a few to think about:
Is a client getting every single update important? If so, then the server needs to cache any potential events between the time the client gets some data and then reconnects.
Are there going to be gobs of updates? If this is the case, it might be wiser to package up a number of updates and push down chunks at a time every several seconds rather than having the client get one update at a time.
The server will likely need a way to detect if a client has gone away to avoid piling up huge amounts of cached packages for that client.
I found there were two problems with the server push approach. With lots of clients, this means lots of open connections on the web server. Depending on the web server in question, this could mean lots of threads being created and held open. The second has to do with the typical browser's limit of 2 requests per domain. If you are able to serve your images, css and other static content fro second level domains, this problem can be mitigated.
there is indeed a cometd-like library for gwt - http://code.google.com/p/gwteventservice/
But i ve not personally used it, so cant really vouch for whether its good or not, but the doco seems quite good. worth a try.
Theres a few other ones i ve seen, like gwt-rocket's cometd library.
Some preliminary ideas for Comet implementation for GWT can be found here... though I wonder whether there is something more mature.
Also, some insight on GWT/Comet integration is available there, using even more cutting-and-bleeding edge technology: "Jetty Continuations". Worth taking a look.
Here you can find a description (with some source samples) of how to do this for IBM WebSphere Application Server. Shouldn't be too different with Jetty or any other Comet-enabled J2EE server. Briefly, the idea is: encode your Java object to JSON string via GWT RPC, then using cometd send it to the client, where it is received by Dojo, which triggers your JSNI code, which calls your widget methods, where you deserialize the object again using GWT RPC. Voila! :)
My experience with this setup is positive, there were no problems with it except for the security questions. It is not really clear how to implement security for comet in this case... Seems that Comet update servlets should have different URLs and then J2EE security can be applied.
The JBoss Errai project has a message bus that provides bi-directional messaging that provides a good alternative to cometd.
We are using Atmosphere Framewrok(http://async-io.org/) for ServerPush/Comet in GWT aplication.
On a client side Framework has GWT integration that is pretty straightforward. On a server side it uses plain Servlet.
We are currently using it in production with 1000+ concurent users in clustered environment. We had some problems on the way that had to be solved by modifying Atmosphere source. Also the documentation is really thin.
Framework is free to use.