I am routing messages from an Azure IoT Hub to a blob container (Azure Storage as a routing endpoint). The messages sent to the IoT Hub are of Content Type: 'application/json' and Content Encoding: 'UTF-8'. However, when they arrive in blob storage several of these messages are batched together into one file with Content Type 'application/octet-stream'. Thus, for instance Power BI is not able to read these files in JSON format when reading directly from the blob.
Is there any way to route these messages so that each single message is saved as a json file in the blob container?
Tl;dr : Please make use of the Encoding option to specify AVRO or JSON format & Batch Frequency/Size to control the batch.
"With an Azure Storage container as a custom endpoint, IoT Hub will write messages to a blob based on the batch frequency and block size specified by the customer. After either the batch size or the batch frequency is hit, whichever happens first, IoT Hub will then write the enqueued messages to the storage container as a blob. You can also specify the naming convention you want to use for your blobs, as shown below."
The below image shows how we navigate to the IoTHub's message routing section to add a custom endpoint of a blob storage account.
-The below image shows how we configure the settings of the batch count and the size. Also please make use of the Encoding section to specify the message format such as AVRO or JSON
Please leave a comment below to let us know if you need further help in this matter.
The message encoding needs to be done by the device stream or as part of a module to translate the protocol. Each protocol (AMQP, MQTT, and HTTP) uses a different method to encode the message from base64 to UTF-8.
To route messages based on message body, you must first add property 'contentType' (ct) to the end of the MQTT topic and set its value to be application/json;charset=utf-8. An example is shown below.
devices/{device-id}/messages/events/$.ct=application%2Fjson%3Bcharset%3Dutf-8
https://learn.microsoft.com/en-us/azure/iot-hub/iot-hub-mqtt-support
Related
We are using a data acquisition system as a device and send some signals values via MQTT protocol into a container which is assigned to an iot-hub. The connection works well between device and iot-hub, and we receive some JSON data. When we open a JSON data, We cannot read the temperature values in "Body" inside the JSON data, since they are encoded. I would be thankful if you tell us, how we should automatically convert the JSON data to a proper format so that we could read the values in numbers?
Please find below three of our code's lines in JSON Data. The rest of the lines are the same, but they are encoded differently.
{"EnqueuedTimeUtc":"2022-02-09T10:00:30.8600000Z","Properties":{"Sensor":""},"SystemProperties":{"connectionDeviceId":"Iba","connectionAuthMethod":"{"scope":"device","type":"sas","issuer":"iothub","acceptingIpFilterRule":null}","connectionDeviceGenerationId":"637799949903534194","enqueuedTime":"2022-02-09T10:00:30.8600000Z"},"Body":"My42MjI3NTQ="}
{"EnqueuedTimeUtc":"2022-02-09T10:00:30.8750000Z","Properties":{"Sensor":""},"SystemProperties":{"connectionDeviceId":"Iba","connectionAuthMethod":"{"scope":"device","type":"sas","issuer":"iothub","acceptingIpFilterRule":null}","connectionDeviceGenerationId":"637799949903534194","enqueuedTime":"2022-02-09T10:00:30.8750000Z"},"Body":"My42ODEyNDY="}
{"EnqueuedTimeUtc":"2022-02-09T10:00:30.9070000Z","Properties":{"Sensor":""},"SystemProperties":{"connectionDeviceId":"Iba","connectionAuthMethod":"{"scope":"device","type":"sas","issuer":"iothub","acceptingIpFilterRule":null}","connectionDeviceGenerationId":"637799949903534194","enqueuedTime":"2022-02-09T10:00:30.9070000Z"},"Body":"My43Mzk1OTI="}
Thanks in advance!
Br
Masoud
you should add to the message topic two parameters such as the content-type (ct) and content-encoding (ce) like is shown in the following example:
devices/device1/messages/events/$.ct=application%2Fjson&$.ce=utf-8
I am using Python SDK for OCI. I tried the Upload manager example and its working perfectly fine when i try to upload files from file system. But i have to expose this python code as REST service (using flask) and files to be uploaded to object storage will come as payload for REST. Does it have to multipart/mixed content type in this case or can it be multipart/form-data as well.
#user1945183, are you asking if Upload Manager can support multipart/form-data? Yes, Upload Manager can take in multipart/form-data.
The Upload Manager uses Object Storage's CreateMultipartUpload API. You can learn more about the CreateMultipartUpload API doc.
From CreateMultipartUploadDetails Reference you will find that content-type is optional and have no effect on Object Storage behavior.
The optional Content-Type header that defines the standard MIME type format of
the object to upload. Specifying values for this header has no effect on
Object Storage behavior. Programs that read the object determine what to do
based on the value provided. For example, you could use this header to
identify and perform special operations on text only objects.
Using
NotificationHubClient hub = NotificationHubClient.CreateClientFromConnectionString(notificationHubConnection, notificationHubName, enableTestSend);
NotificationOutcome outcome = await hub.SendDirectNotificationAsync(fcmNotification, deviceUri);
I am able to send and receive notifications using FCM via the Azure hub to a Xamarin Android app, finally. However the payload is not present in the received RemoteMessage even though the sent fcmNotification json payload looks good and passes validation. I am basically looking at the RemoteMessage.Data property, but not finding the expected payload array. Looking at RemoteMessage structure, I haven't found any part of the payload array either.
I know that the Azure hub manipulates the notification by adding the necessary headers like content type, e.g. "application/json". Are there any other settings that are needed to be passed to enable the "data" only payload?
Additional settings are not necessary, but the format of the entire notification contents has to have this type of structure:
"{ \"data\":{ \"A\": \"aaa\",\"B\": \"bbb\",\"C\": \"ccc\",\"D\": \"ddd\",\"E\": \"eee\",\"F\": \"fff\"}}"
The number of data elements is up to you. The data element name can be anything as well as the associated content except for the need for backslash usage for special characters. Both the element name and content can be inserted using variables creating a traditional composite string.
What is especially important is the inserted spaces as shown. Also note that traditional Json formatting is not acceptable because of the need for those spaces.
Is there any limit on sending JSON data to a rest API?I am building an app where I have used a rest API to send an email. Here I want to send app logs to API,but before going through that I want to know about the limitation on send data to rest API.
Secondly which one is better option file or send JSON data to rest API?
In ASP.NET there's by default a 4MB limit of the size of a request. This can be adjusted using the maxRequestLength attribute on the httpRuntime element:
<httpRuntime targetFramework="4.5.2" maxRequestLength="1048576" />
In this example we set the maximum request size to be 1GB.
Secondly which one is better option file or send JSON data to rest API?
If you are planning to send large request payloads I would recommend you using the multipart/form-data content type for the request instead of JSON. This would allow you to directly send the raw bytes in the request payload. If you use JSON then you would need to encode those raw bytes to something like base64 which would make the request even larger. To even further optimize network traffic the client could gzip the raw bytes before sending them over the wire and then unzip the stream on the server.
You may also find the following article useful in setting up this file upload on the server side.
I am using json service(.net RIA service) to push data to server from Extjs. Currently I have a requirement to upload document to server. I saw some examples using form submit to php files.
Is this possible through json service? or Is it necessery to create some server logic where i can accept form submits?
Is it possible to read some binary data from client side and push as json data to server?
In Extjs, File uploads are not performed using normal 'Ajax' techniques, that is they are not performed using XMLHttpRequests. Instead a hidden element containing all the fields is created temporarily and submitted with its target set to refer to a dynamically generated, hidden which is inserted into the document but removed after the return data has been gathered.
Be aware that file upload packets are sent with the content type multipart/form and some server technologies
check : http://docs.sencha.com/ext-js/4-1/#!/api/Ext.form.Basic-method-hasUpload
Yes it's possible. You should use FileReader to get binary base64 encoded content (FileReader.readAsDataURL(Blob|File)), which you can transfer with JSON.
More insights here: http://www.html5rocks.com/en/tutorials/file/dndfiles/