As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
At work, we need to build a jobs server for things like sending emails, building PDFs, crunching some data, etc. Obviously, we'd like to build on some sort of generic queueing system. I'm familiar with Gearman, and that this is exact problem that it tries to solve: putting jobs on a queue where workers come to pick them up. However, I'm seeing lots of mentions of Rabbitmq and am unclear how it's used in this scenario.
Is Rabbitmq a good framework to build a distributed jobs system on top of?
I would say that Gearman is better for queuing "jobs" and RabbitMQ is better for queuing "data". Of course, they are both really the same thing, but the way it works out for me is that if you are trying to "fan out" work to be done, and the workers can work independently, Gearman is the better way to do it. But if you are trying to feed data from a lot of sources down into fewer data consumers, RabbitMQ is the better solution.
The history of RabbitMQ, as something that allowed Twitter to take bursty loads of messages, and feed them into crusty old SMS gateways that could keep only one connection open, were rate limited, and didnt have retries, is illustrative of the kind of problems that RabbitMQ is good at solving.
It all depends what semantics you want to expose. It's really easy to do what Gearman does on top of RabbitMQ, which can certainly 'fan out' messages to independent workers.
But Gearman is built for purpose. IIUC, Gearman is a framework for processing jobs and not a messaging system as such. There are other such frameworks such as Celery that use RabbitMQ under the hood for that. Here is an article about Celery that is worth reading.
Related
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
I know the basic concept of a WebSocket . I know that it allows simultaneous bidirectional(Full duplex), persistent communication support built into it . So it is very useful for a server push kind of scenarios . What other scenarios are WebSockets best suited for ? What are some of the common situation where we as programmers should actively look at WebSockets as the solution instead of reinventing the wheel ?
Well, server push is the main component of the bidirectional support that the single direction of HTTP is lacking. It also supports cross-domain requests. Because the server can now contact the client asynchronously, it enables a whole bunch of techniques and applications:
Built-in heartbeats. I use this in one of my apps, and I no longer have to check myself if someone disconnects.
Have a client app that is served in real-time by different backend applications
Real-time updates of streaming data, news feeds, etc
Multi-user games that run in the browser
All of these you could somewhat-do before with long-polling, but it was inefficient (tons of overhead), complex (hard to implement) and ugly (not natural to use at all). WebSocket simpilfies much of that. According to this article, WebSocket can typically reduce overhead by 500 times and latency by 3 times. http://www.websocket.org/quantum.html
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
Which database should I use with node.js? I'm condering MySQL and MONGO DB.
The correct answer is: it depends.
Like many other programming languages node has various adapters for databases. Ture, some are better than others, but they are all changing and evolving so fast that any statement here would be outdated soon. Even if you select one database, you will find at least two or three adapters with different qualities.
In the end it does not really matter what is a good fit for Node, but what is a good fit for your project. Trying to squeeze a relational problem into a NoSQL storage will do you no good.
The node community provides you with lots of possibilities. Take a look at the Node Toolbox to find some solutions for CouchDB, MongoDB, Redis, Database abstraction layers, MySQL, … As always you need a bit of common sense to choose the right module. Look at commit history, how many maintainers, activity, last commit timestamp, issue queue, etc. You will surely find a solution for a database that you are comfortable with.
I personally find it rather overwhelming when I have to learn more than one technology at once. So if you are new to node, I would rather stick to a database you know for your first project.
Every one has different requirements for completion so it really depends on what you are trying to do. Like Joseph said MongoDB and Node.js works really well together but for certain use case, MySQL could probably be a better choice. Also, you might want to considered the database you are the most comfortable with. Personnaly, I would recommand MongoDB but I think it's a matter of preference.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
in the last weeks I have built a small notification system using HTML5 Websockets and node.js with socket.io.
There are still some things that I haven´t solved, yet.
I have read that realtime notification using node.js should be the best performance solution vs long polling etc because of non blocking i/o operations.
Now assume we have to notify friends of this user. Something like "I have bought a new iphone". How do I teach node.js to realtime notify specific user 1:n? Till know I have only got sending and receiving message 1-1 communication.
This means I have to go check against facebook api friendlist first and find out which facebook_id is connected with which socket_id somehow. Then check the users privacy settings, if he would like to receive notifications and then send and save it into the database.
So to me this somehow seems not to be the most effective solution...
At the moment I would use mongo-db native driver for node.js and have db operations done by serversided javascript.
I still need to implement facebooks javascript sdk to get the friendslist, but I would have trouble with the users privacy settings. Doesn´t this destroy all the advantages of non-blocking I/O?
How do you actually solve this. To me, it seems very complex.
Thank you very much.
You shouldn't go to Facebook to check friends list and such..
you should cache them, i assume the users are logged in if so the solution
seems to be very easy.
Get the user.
check if the sender is his friend
check if the recipient want to receive the message.
btw, redis is a lot faster.
Edit:
if you are worried that the user data won't be updated
and that's why you do not cache you should check Real-time Updates.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 12 years ago.
I'm starting a web application and I'm hesitating if it is worth to use Oracle in order to have a better performance / scalability. Thanks
Oracle requires significantly more infrastructure and specialized staff to be maintained than MySql for example. It offers, arguably, more in return. But since you are talking about scalability / performance, bear in mind that Wikipedia, Flicker, Facebook (and many of the most visited sites in the world today) run under MySQL...
So, put your focus on designing well your database instead of trying to find out which platform to use. Every single one of them has many strengths today, and are ready for production and demanding environments.
I, myself, would never ever even consider using Oracle from the start in a new Web Application. I would only jump to that more expensive RDBMS if really needed.
It can also pay off to use a ORM tool that will give you database independence, so changing the RDBMS will not be a big issue at all.
Oracle can be very fast, but you have to spend millions of dollars for licences, hardware and dba-consulting hours. PostgreSQL is fast, has no licence costs and offers (almost) the same rich functionality as Oracle. PostgreSQL speed depends on your hardware and configuration as well, but you don't have to invest all your money right now to get started.
MySQL is populair, but has limited functionality and isn't as strict as you would expect from a DBMS.
Write you application to be database generic. That way you can move to whatever database works the best for a given installation. That's relatively easy to do in most web environments these days.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 11 years ago.
In 5 days I'm going to ETL interview. It's my first interview on this subject. What question would I be asked? Most likely they will be about MS SQL Server Integration Service.
If possible, provide the answers. =)
If possible, provide the answers. =)
Keep it high-level if you have to, but don't ask a question that couldn't answer yourself.
I agree with Brad that syntax is not important, it's the thought process.
Another idea is to ask them about how they would pack up and move an office. It gives you insight into the same kinds of decisions needed in ETL (prep, actual moving of stuff, and validation), and you might be more comfortable talking about that than the details of SSIS
Think practically. Hand them a printout of a sample file that might need to be imported (possibly simplified to save time). Have them talk about database design, considerations, concerns, possible ways to improve the data. Then bring out a second printout of somehow related and see if they can figure how to validate the one from the other.
Make sure you talk about how much time is available to perform the ETL processes based on business rules and environment.
Require as much pseudo-code as you like, but I personally subscribe to the idea that syntax can be taught cheaply, but learning how to think is a very expensive thing to teach someone; and sometimes it's not even successful.
Also, ask them what standards they would implement if they were to design the optimum layout of the source data. Make sure you consider data distribution beyond your company (if applicable).