What I'm trying to do is host a JSON document that will then, essentially, serve as a hosted version of json-server. I'm aware I can do something similar with My JSON Server, but I plan to move my entire architecture to GCP so want to get more familiar with it.
At first I looked into the Storage JSON API, but it seems like that's just for getting data about buckets rather than the items in the buckets itself. I created a bucket called test-json-api and added a test-data.json, but there's seemingly no way to access the data in the json file via this API.
I'm trying to keep it as simple as possible for testing purposes. In time, I'll probably use a firestore allocation, but for now I'd like to avoid all that complexity, and instead have a simple GET and a PUT/PATCH to a json file.
The Storage JSON API you are talking about are only for getting and updating the metadata and not for getting and updating the data inside the object. Objects inside the Google Cloud Storage bucket are immutable and one way to update them may be to get the object data from Google Cloud Storage bucket within the code, updating it, then uploading it again into the Google Cloud Storage bucket.
As you want to deal with JSON files you may explore using Cloud Datastore or Cloud Firestore. Also if you wish to use Firebase then you may explore Firebase Realtime Database.
A very quick and dirty way to make it easy to read some of the information in the json doc is to use that information in the blob name. For example the key information in
doc = {'id':3, 'name':'my name', ... }
could be stored in an object called "doc_3_my name", so that it can be read while browsing the bucket. You can then download the right doc if you need to see all the original data. An object name can be up to 1024 bytes of UTF-8 (with some exclusions), which is normally sufficient for surfacing basic information.
Note that you should never store PII like this.
https://cloud.google.com/storage/docs/objects
Related
Is there a mechanism within Azure API Management to disable the generation of the sample data that is injected into the api specification by the APIM processes?
I have noticed in APIs that have significant large/complex models and a large number of operations that the sampleGenerated attribute is creating an extremely large overhead to the configuration of the api. For example we have an api that is ~260k on original import of the swagger file and when it ends up in the APIM repository the configuration file has expanded out to over 13 megs of data. This sample data doesn't appear to be used in the admin or developer portal so not sure of its value stored in the primary configuration file. I have attempted to update via the repository to clear these values however it appears to be recreated after the repository update.
The only way to do so is to provide your own samples.
I have run a Natural Language AutoML Text Classification model on google cloud platform. The data I used to train the model can be exported to a csv file in a bucket. The file has 3 columns (1. train/validation/test, 2. gs:// reference to the actual content, 3. prediction result/label) as in the below example.
TRAIN gs://[bucket_name]/[folder]/uploads/content/RrpGCDwgse0.txt Website
My question is how to get the actual content out of the gs:// reference so I can look at each row to determine if the predicted labels are correct or not. The output should be in a csv file with the string content, not the gs:// reference.
This bucket would have been created by someone before using AutoML to store the documents used to train your model. So if you want access to the objects within the bucket, someone with the correct permissions will need to give you access to the bucket or individual objects within the bucket. The gs:// reference is the object itself.
There's quite a few options available for this:
Cloud Identity and Access Management (Cloud IAM) permissions to grant access to buckets and bulk access to objects in the bucket.
Access Control Lists (ACLs) to grant read or write access to users for individual buckets or objects
Signed URLs (query string authentication) to give time-limited read or write access to an object through a URL you generate
I need to provide every user with data from different sites. Each of the site provides data in JSON format, but some of them have restriction to maximal number of request.
My idea for solution is to download the data to firebase periodically, than users will access just the firebase database.
From docs it seems to me that firebase can somehow use http requests.
Can I use firebase to periodically update itself by http request?
Or should I establish server which will do the task?
I am pretty new to those topics so any tip where to look for information will be appreciated.
I'm currently implementing an IoT solution that has a bunch of sensors sending information in JSON format through a gateway.
I was reading about doing this on azure but couldn't quite figure out how the JSON scheme and the Event Hubs work to display the info on PowerBI?
Can I create a schema and upload it to PowerBI then connect it to my device?
there's multiple sides to this. To start with, the IoT ingestion in Azure is done tru Event Hubs as you've mentioned. If your gateway is able to do a RESTful call to the Event Hubs entry point, Event Hubs will get this data and store it temporarily for the retention period specified. Then stream analytics, will consume the data from Event Hubs and will enable you to do further processing and divert the data to different outputs. In your case, you can set one of the outputs to be a PowerBI dashboard which you can authorize with an organizational account (more on that later) and the output will automatically tied to PowerBI. The data schema part is interesting, the JSON itself defines the data table schema to be used on PowerBI side and will propagate from EventHubs to Stream Analytics to PowerBI with the first JSON package sent. Once the schema is there it is fixed and the rest of the data being streamed in should be in the same format.
If you don't have an organizational account at hand to use with PowerBI, you can register your domain under Azure Active Directory and use that account since it is considered within your org.
There may be a way of altering the schema afterwards using PowerBI rest api. Kindly find the links below..Haven't tried it myself tho.
https://msdn.microsoft.com/en-us/library/mt203557.aspx
Stream analytics with powerbi
Hope this helps, let me know if you need further info.
One way to achieve this is to send your data to Azure Events Hub, read it and send it to PowerBI with Stream Analytics. Listing all the steps here would be too long. I suggest that you take a look at a series of blog posts I wrote describing how I built a demo similar to what you try to achieve. That should give you enough info to get you started.
http://guyb.ca/IoTAzureDemo
I'm trying to figure out how to insert/update data into offsite databases that don't have an API available. Since they don't have an API, I thought of an approach I can take to insert/update data into their database.
They would first need to build a script and place it in an accessible location on their webserver that I can access via a URL. They would be required to supply the URL to me. I then can do a cURL POST request to that URL and pass a JSON array of the data that needs to be inserted. The script on their server would handle the parsing of the JSON array and the insert/update into the database.
I think this should work, but what security issues would I be opening them up to?
What you described is them creating an API. Just because the url invokes a script and isn't written in something like Java or PhP doesn't mean its not an api.
You need to make sure your url is secure so only authorized people can invoke it, and they would probably want to do data validation.
You should let them decide whether that is easier than standing up a more robust/non-script based solution