I am using Python SDK for OCI. I tried the Upload manager example and its working perfectly fine when i try to upload files from file system. But i have to expose this python code as REST service (using flask) and files to be uploaded to object storage will come as payload for REST. Does it have to multipart/mixed content type in this case or can it be multipart/form-data as well.
#user1945183, are you asking if Upload Manager can support multipart/form-data? Yes, Upload Manager can take in multipart/form-data.
The Upload Manager uses Object Storage's CreateMultipartUpload API. You can learn more about the CreateMultipartUpload API doc.
From CreateMultipartUploadDetails Reference you will find that content-type is optional and have no effect on Object Storage behavior.
The optional Content-Type header that defines the standard MIME type format of
the object to upload. Specifying values for this header has no effect on
Object Storage behavior. Programs that read the object determine what to do
based on the value provided. For example, you could use this header to
identify and perform special operations on text only objects.
Related
I am routing messages from an Azure IoT Hub to a blob container (Azure Storage as a routing endpoint). The messages sent to the IoT Hub are of Content Type: 'application/json' and Content Encoding: 'UTF-8'. However, when they arrive in blob storage several of these messages are batched together into one file with Content Type 'application/octet-stream'. Thus, for instance Power BI is not able to read these files in JSON format when reading directly from the blob.
Is there any way to route these messages so that each single message is saved as a json file in the blob container?
Tl;dr : Please make use of the Encoding option to specify AVRO or JSON format & Batch Frequency/Size to control the batch.
"With an Azure Storage container as a custom endpoint, IoT Hub will write messages to a blob based on the batch frequency and block size specified by the customer. After either the batch size or the batch frequency is hit, whichever happens first, IoT Hub will then write the enqueued messages to the storage container as a blob. You can also specify the naming convention you want to use for your blobs, as shown below."
The below image shows how we navigate to the IoTHub's message routing section to add a custom endpoint of a blob storage account.
-The below image shows how we configure the settings of the batch count and the size. Also please make use of the Encoding section to specify the message format such as AVRO or JSON
Please leave a comment below to let us know if you need further help in this matter.
The message encoding needs to be done by the device stream or as part of a module to translate the protocol. Each protocol (AMQP, MQTT, and HTTP) uses a different method to encode the message from base64 to UTF-8.
To route messages based on message body, you must first add property 'contentType' (ct) to the end of the MQTT topic and set its value to be application/json;charset=utf-8. An example is shown below.
devices/{device-id}/messages/events/$.ct=application%2Fjson%3Bcharset%3Dutf-8
https://learn.microsoft.com/en-us/azure/iot-hub/iot-hub-mqtt-support
Can anyone comment on if it should be possible to use rclone's swift support to access buckets in OCI object storage (new OCI, not classic).
I'm interested in it because S3 compatibility mode is limited to a single designated compartment and I'd like to be able to use rclone with any bucket in my tenancy.
I know that for public buckets there is still a swift style URL. The 3 functional URLs styles seem to be:
Native: https://objectstorage.{region}.oraclecloud.com/n/{object-storage-namespace}/b/{bucket}/o/{filename}
Swift: https://swiftobjectstorage.{region}.oraclecloud.com/v1/{object-storage-namespace}/{bucket}/{filename}
S3: https://{object-storage-namespace}.compat.objectstorage..oraclecloud.com/{bucket}/{filename}
https://docs.cloud.oracle.com/en-us/iaas/Content/Identity/Tasks/managingcredentials.htm talks a little bit about Swift password (Auth Tokens) and you can create one in the console.
But I can't find anything about what the auth URL would be for the non-classic version of object storage. And storage_url with a auth_token doesn't seem to work either.
Using -vvvv doesn't show anything more than 401 Unauthorized.
I'm interested in it because S3 compatibility mode is limited to a single designated compartment and I'd like to be able to use rclone with any bucket in my tenancy.
The designated compartment only controls where buckets created via that protocol (S3 or Swift) are placed. The designated compartment does not affect authorization. Authorization is controlled by the relevant IAM policies.
But I can't find anything about what the auth URL would be for the non-classic version of object storage. And storage_url with a auth_token doesn't seem to work either.
The new/current OCI Object Storage does not support auth URLs. You must use HTTP basic-style auth with Swift on OCI. It does not seem that rclone supports HTTP basic auth with swift directly (it is possible to create the basic auth header yourself and have rclone send it).
All that said, using rclone with s3 is the best approach for OCI Object Storage. Ensure you set the "region" option to the correct region name like "us-phoenix-1" and you should be good.
Thanks!
When I'm checking web requests in Chrome's DevTools on Mac (Network tab), I've got the Payload in the following format:
7|0|6|https://www.example.com/app/Basic/|00D1D071AC218DFE91521C012683E911|com.optionfair.client.common.services.nongenerated.RefreshService|getCometUpdates|I|J|1|2|3|4|3|5|6|6|173|VvAwAqy|o$UN|
which is basically separated by vertical bar character (|).
How I can copy or convert above payload from Chrome into some meaningful format such as JSON? Any ideas?
Btw. In this question it looks fine on the screenshot, but in my case, I don't have view parsed and it doesn't look like JSON format at all.
Using Google Chrome on Mac (Version 57.0.2987.133, 64-bit).
Reproducible steps:
Go to this page.
Open DevTools on Network/XHR tab and look for refresh requests.
My goal is to reuse/replicate the POST data in Request Payload in the command-line tool such as curl so it can be recognized (not necessary on the page mentioned above, but I'd like to know the general approach to deal with this blob format). I would expect JSON format, but it's not.
Here you need to look at the request header content-type to determine how this request was encoded before knowing what might parse it:
This is GWT RPC, so it can include serializations of built-in and custom Java Objects, where knowledge of the class is in both server-side Java and transpiled Java running on the client via Google Web Toolkit.
There is no reason for Chrome to understand this format directly, and it need not have a JSON or XML cannonicalization. Fully interpreting these calls to the extent it is possible on the client may require disassembly or introspection tricks against the transpiled client code, assuming the program wasn't transpiled with source maps.
Without digging into the client code, one can interpret the literal rpc while guessing at or probing the definition of classes and their methods by modifying the call.
I'm writing a RESTful web application where I need to provide the service of uploading images for a user. Currently, I have been able to upload an image from my current machine but I need to send it as JSON data over the web through the REST protocol.
In the server, there is a Java application running Jax-RS to manage the RESTful service. I was planning to save the JSON data that contains the image in the server and then provide a URL to the user for him to be able to locate it's image on the server.
Can someone provide some ideas on how can I do this?
If you want to send the image in a JSON object, then the image should be Base64 encoded it, or some other form of encoding. Then on the server side you will need to unmarshal the JSON and then decode back the image. You can get some ideas here on how that can be done.
Optionally, instead of doing all the converting inside the resource method (as in the example linked above), you could write a custom MessageBodyReader, where you can do the unmarshalling and decoding there.
If you decide you don't want to work with JSON, you can go the normal route and use Multipart. Depending on the implementation of JAX-RS you are using, multipart support will be different. You can see some examples (all examples have links to the official documentation)
Jersey example
Resteasy example
CXF example
There are other implementations, but I don't have examples for those. You will need to search for the documentation if you're using an implementation other than listed above.
I am using json service(.net RIA service) to push data to server from Extjs. Currently I have a requirement to upload document to server. I saw some examples using form submit to php files.
Is this possible through json service? or Is it necessery to create some server logic where i can accept form submits?
Is it possible to read some binary data from client side and push as json data to server?
In Extjs, File uploads are not performed using normal 'Ajax' techniques, that is they are not performed using XMLHttpRequests. Instead a hidden element containing all the fields is created temporarily and submitted with its target set to refer to a dynamically generated, hidden which is inserted into the document but removed after the return data has been gathered.
Be aware that file upload packets are sent with the content type multipart/form and some server technologies
check : http://docs.sencha.com/ext-js/4-1/#!/api/Ext.form.Basic-method-hasUpload
Yes it's possible. You should use FileReader to get binary base64 encoded content (FileReader.readAsDataURL(Blob|File)), which you can transfer with JSON.
More insights here: http://www.html5rocks.com/en/tutorials/file/dndfiles/