We have a requirement to implement Azure Service Bus as Integration point to various Applications (including apps hosted in AWS). Each application will have its own SQS. So the idea is to have Azure Service Bus with Topics and Subscription filters to route messages to each SQS accordingly. However I am not sure as to how we can pick messages from a subscription filter and push the message to AWS SQS. I am not able to see any solution for this.
These two are inherently two different messages services and you will either need to find a third party connector/bridge between the two or create your own. This would be a process that would be retrieving messages from one broker and forwarding it to another.
When it comes to a third party, there's an example that you could have a look at. NServiceBus has a community extension called Router. The router allows achieving exactly what you're looking for.
Disclaimer: I contribute and work on NServiceBus
Related
in my setup an IoT device is connected to an MQTT broker and publishes measurements. We duplicate this traffic to another PC where we want to perform analytics on the MQTT data. We cannot create a new client to this broker and subscribe to the topics, we just want to implement a sort of sniffer for these messages and extract the measurements as a JSON.
I have experimented with scapy and various python scripts but haven't succeeded. For example, it seems that the mqtt-paho library for python requires a connection to the actual broker, but as I said this is not an option. Any idea how to approach the problem?
This article describes how a websocket server for a chat application can look. We are planning to implement something similar; when a message is sent to the server it is sent to the correct recipient based on an authentication token and the message gets saved in a mysql database.
We will eventually host the server on Google App Engine, and I suspect that that will cause some issues with the above described approach, since that depends on all clients being connected to the same server, and that probably won't be the case since multiple instances will be created as needed. Is there a way to connect all instances so that this won't be a problem (Pub/Sub maybe? (That will cause additional costs though)), or should we find a different solution?
One idea I had was to use mysql-events to monitor the binlog from the websocket server for the creation of new rows in the messages table, but I read somewhere that that wasn't recommend. But I can't find where I read that, and maybe that is the best solution.
Since you asked about other solutions, I would recommend looking at Firebase and specifically the Realtime Database. Out of the box it provides all of the functionality that you need for realtime communication between connected clients and Cloud Messaging for clients who aren't.
Here's a tutorial that uses Firestore to create a realtime chat web app, but it can all be applied to the Realtime Database with minor modification. I say that because Firestore has expensive writes, which in my opinion make it unsuitable for a chat backend.
Do you know how does any queue services work behind the scenes? Do they store the messages in some datastore? Eg. Amazon SQS keeps the messages in its queue until any workers pulls those messages from SQS.
Where does it store behind the scenes? I think in some data store like Dynamodb or RDS. Is that correct to say? Are there any resources which can give me more insight on how the design of general Queue service looks like?
I designed the SQS replicated datastore. It was a completely custom implementation tailored for messaging use cases. Relational databases are not a good fit for high throughput queues.
Unfortunately, I cannot share the details of the datastore or replication algorithm as it was never made public by Amazon.
Kafka is open source, so you can get information about its store design.
I am doing some research for a mobile app I want to develop, and was wondering whether I could get feedback on the following architecture. Within my future app users should be able to authenticate and register themselves via the mobile app and retrieve and use their settings after a successful authentication.
What I am looking for is an architecture in which user accounts are managed by AWS Cognito, but all application related information is stored in a MySQL database hosted somewhere else.
Why host the database outside of AWS? Because of high costs / vendor lock-in / for the sake of learning about architecture rather than going all-in on AWS or Azure
Why not build the identity management myself? Because in the end I want to focus on the App and don't spent a lot of energy on something that AWS can already provide me with (yeah I know, not quite in line with my last argument above, but otherwise all my time goes into database AND IAM)
One of my assumptions in this design (please correct me if I am wrong) is that it is only possible to retrieve data from a MySQL database with 'fixed credentials'. Therefore, I don't want the app (the user's device) to make these queries (but do this on the server instead) as the credentials to the database would otherwise be stored on the device.
Also, to make it (nearly) impossible for users to run queries on the database with a fake identity, I want the server to retrieve the User ID from AWS Cognito (rather than using the ID token from the device) and use this in the SQL query. This, should protect the service from a fake user ID injection from the device/user.
Are there functionalities I have missed in any of these components that could make my design less complicated or which could improve the flow?
Is that API (the one in the step 3) managed by the AWS API Gateway? If so, your cognito user pool can be set as Authorizer in your AWS API Gateway, then the gateway will take care automatically of the token verification (Authorizers enable you to control access to your APIs using Amazon Cognito User Pools or a Lambda function).
You can also do the token verification in a Lambda if you need to verify something else in the token.
Regarding to the connection between NodeJS (assuming that is an AWS lambda) that will work fine, but keep in mind the security as your customers data will travel outside AWS, and try to use tools like AWS Secret Manager to keep your database passwords safe and rotate them from time to time in your lambda.
I have a PHP/MySQL application test-deployed on a server, with a domain name that I own. In order for this to be a real world scalable product, I decided to use Amazon Web Services. However, I'm new to using cloud services (this is my first), and since the past 2 days, after going through tutorials and "how to start" guides given on Amazon, I've still been unable to grasp "what exactly should I do, so that I can use my present domain name and use Amazon's services?" My users should be able to access my product using, let's say www.xyz.com which is the name I own. My PHP code gets some data from client, which it then stores in a SQL DB. This is the existing, working set up.
Now, how do I get my PHP code, to use Amazon Web Servics and store it in a database that Amazon provides? My product's DB will be continuously growing, and I will pay for whatever is used. Also, if I decide to use the PHP services from Amazon too, does Amazon host my code? In that case, what will be the domain name?
To summarize, my biggest concern is the domain name I've bought, and I've seen no documentation on how to go forward in such a case.
This is the only part I have been unable to figure out, rest was clear from the documentation..
Thanks for your help!
Amazon Web Services (AWS) is a cloud platform composed of multiple services that jointly enable you to host infrastructure and applications on it. It's not a single offering that magically does everything for you. In order to achieve your goal you will want to do the following:
Use Amazon Elastic Compute Cloud (EC2) to spin up servers that host your PHP application. They will handle the incoming traffic for you. Have a look at this link to get started.
In order to store data you will want to use some sort of database. AWS offers various database types. Since you are looking for a SQL-type database, you will want to use RDS. This service allows you to provision a functional database and relieves you of certain administrative tasks.
In order to use your current domain, you will have to transfer its registration to AWS Route53. Just Google 'Route53 domain transfer' and the documentation will show you how to do it.
There are many whitepapers available that show architectural patterns across the AWS cloud. I suggest you read them so you can get a better understanding of the platform.
To get started quickly I recommend using Amazon Elastic Beanstalk for your purposes:
Amazon Web Services (AWS) comprises dozens of services, each of which
exposes an area of functionality. While the variety of services offers
flexibility for how you want to manage your AWS infrastructure, it can
be challenging to figure out which services to use and how to
provision them.
With Elastic Beanstalk, you can quickly deploy and manage applications
in the AWS Cloud without worrying about the infrastructure that runs
those applications. AWS Elastic Beanstalk reduces management
complexity without restricting choice or control. You simply upload
your application, and Elastic Beanstalk automatically handles the
details of capacity provisioning, load balancing, scaling, and
application health monitoring.
Learn more about it here
regarding the domain, you could transfer it to route 53
OR
route your domain traffic by using route53 name servers