I need to provide every user with data from different sites. Each of the site provides data in JSON format, but some of them have restriction to maximal number of request.
My idea for solution is to download the data to firebase periodically, than users will access just the firebase database.
From docs it seems to me that firebase can somehow use http requests.
Can I use firebase to periodically update itself by http request?
Or should I establish server which will do the task?
I am pretty new to those topics so any tip where to look for information will be appreciated.
Related
I need to retrieve data from an API source that has a massive amount of entries. (1800+) The problem is that the source has no pagination or way to group the entries. We are then saving the entries as post on the site and will run through a Cronjob daily.
Using curl_init() to retrieve the data from the API source. But the we keep getting a 503 error, timing out. When it works it retrieves the data as json saving important info with as metadata and the rest as json.
Is there a more reliable way to retrieve the data. On other sites I have worked on we have been able to programmatically run through an API per page in the backend.
You might try saving the JSON to a file first, then running the post creation on the JSON in the file vs. the direct cURL connection. I ran into similar issues in the past, even with an API that had pagination.
I've read a lot of questions and look after a lot of sources, but I could not get the idea. That's because there is lot redirects happening with my application, let's see a quick look over the steps.
Step1. Login with [MyApp1]
1.1 get tokens via POST method
Step2. [Save tokens temporarily] < Stuck here.
Step3. Login with [MyApp2]
3.1 get tokens via POST method
Step4. Encrypt all tokens
4.1 Save it to MySQL Database
I do not know what is best to save the data temporarily, I looked over sessions and Redis, I am working on Node.js Web Application, HTML with a Server and MySQL Database.
Perhaps store them in the query? OAuth2 provides a state query parameter that is applied on redirect so your application can pass some information to itself after redirecting.
I'm currently implementing an IoT solution that has a bunch of sensors sending information in JSON format through a gateway.
I was reading about doing this on azure but couldn't quite figure out how the JSON scheme and the Event Hubs work to display the info on PowerBI?
Can I create a schema and upload it to PowerBI then connect it to my device?
there's multiple sides to this. To start with, the IoT ingestion in Azure is done tru Event Hubs as you've mentioned. If your gateway is able to do a RESTful call to the Event Hubs entry point, Event Hubs will get this data and store it temporarily for the retention period specified. Then stream analytics, will consume the data from Event Hubs and will enable you to do further processing and divert the data to different outputs. In your case, you can set one of the outputs to be a PowerBI dashboard which you can authorize with an organizational account (more on that later) and the output will automatically tied to PowerBI. The data schema part is interesting, the JSON itself defines the data table schema to be used on PowerBI side and will propagate from EventHubs to Stream Analytics to PowerBI with the first JSON package sent. Once the schema is there it is fixed and the rest of the data being streamed in should be in the same format.
If you don't have an organizational account at hand to use with PowerBI, you can register your domain under Azure Active Directory and use that account since it is considered within your org.
There may be a way of altering the schema afterwards using PowerBI rest api. Kindly find the links below..Haven't tried it myself tho.
https://msdn.microsoft.com/en-us/library/mt203557.aspx
Stream analytics with powerbi
Hope this helps, let me know if you need further info.
One way to achieve this is to send your data to Azure Events Hub, read it and send it to PowerBI with Stream Analytics. Listing all the steps here would be too long. I suggest that you take a look at a series of blog posts I wrote describing how I built a demo similar to what you try to achieve. That should give you enough info to get you started.
http://guyb.ca/IoTAzureDemo
I have a web app with a MySQL database we maintain in the cloud that we are trying to integrate with our QuickBooks Online account. We want to sync data between or web app's database and QuickBooks online, such as customer names and addresses. If they update their address in or web app, it's easy to then update it in QuickBooks online using the QuickBooks Online API. However, if they tell us their new address over the phone and we change it directly in QuickBooks online, we have no idea how to have that trigger something so that it automatically updates our MySQL web app. How do we go about doing this or learning about this process?
Intuit/QuickBooks has an API that's specifically geared towards this use-case. From the docs:
The change data capture (CDC) operation returns a list of entities that have changed since a specified time. This operation is for an app that periodically polls Data Services and then refreshes its local copy of entity data.
Docs are here:
https://developer.intuit.com/docs/0100_accounting/0300_developer_guides/change_data_capture
Basically you make an OAuth signed HTTP GET request like this:
https://quickbooks.api.intuit.com/v3/company/1234/cdc?entities=Class,Item,Invoice&changedSince=2012-07-20T22:25:51-07:00
And you get back a list of objects that have changed since the given date/time.
Your application can remember the last time you called this, and periodically call this API to get things that have changed since the last time you called it.
You get back something like this:
<IntuitResponse xmlns="http://schema.intuit.com/finance/v3" time="2013-04-03T10:36:19.393Z">
<CDCResponse>
<QueryResponse>
<Customer>...
</Customer>
<Customer>...
</Customer>
</QueryResponse>
<QueryResponse>
<Invoice>...
</Invoice>
<Invoice>...
</Invoice>
</QueryResponse>
</CDCResponse>
</IntuitResponse>
I'm trying to figure out how to insert/update data into offsite databases that don't have an API available. Since they don't have an API, I thought of an approach I can take to insert/update data into their database.
They would first need to build a script and place it in an accessible location on their webserver that I can access via a URL. They would be required to supply the URL to me. I then can do a cURL POST request to that URL and pass a JSON array of the data that needs to be inserted. The script on their server would handle the parsing of the JSON array and the insert/update into the database.
I think this should work, but what security issues would I be opening them up to?
What you described is them creating an API. Just because the url invokes a script and isn't written in something like Java or PhP doesn't mean its not an api.
You need to make sure your url is secure so only authorized people can invoke it, and they would probably want to do data validation.
You should let them decide whether that is easier than standing up a more robust/non-script based solution