Be default, Dart-RPC uses JSON serialisation when transferring objects (class instances) between the server and the client.
How can I use Protobuf (Protocol Buffers) serialisation instead?
Is it possible to specify the serialisation method (like a content-type) using the Accept request header?
Here's what I tried,
I've used the following .proto definition file representing a Person entity:
message Person {
required string name = 1;
required int32 id = 2;
optional string email = 3;
}
Which generated person.pb.dart for me using the protoc-gen-dart plugin, by running the command:
protoc person.proto --dart_out=. --plugin ./protoc-gen-dart
And some boilerplate dart-rpc code:
import 'dart:io';
import 'package:rpc/rpc.dart';
import 'person.pb.dart';
const String _API_PREFIX = '/api';
final ApiServer _apiServer =
new ApiServer(apiPrefix: _API_PREFIX, prettyPrint: true);
main() async {
_apiServer.addApi(new Cloud());
_apiServer.enableDiscoveryApi();
HttpServer server = await HttpServer.bind(InternetAddress.ANY_IP_V4, 8080);
server.listen(_apiServer.httpRequestHandler);
}
#ApiClass(version: 'v1')
class Cloud {
#ApiMethod(method: 'GET', path: 'resource/{name}')
Person getResource(String name) {
Person p = new Person()
..id = 1
..name = name
..email = 'a#a.a';
return p; // ??? p.writeToBuffer(); ???
}
}
Update
Opened a feature request: https://github.com/dart-lang/rpc/issues/62
rpc only supports JSON. You can create a feature request in the GitHub repository.
Related
I'm trying to use Firebase and its callable Cloud Functions for my Unity project.
With the docs and different posts I found on the web I struggle to understand how returning data works. (I come from Azure Functions in C#)
I use TypeScript, and try to return a custom object CharactersResponse:
export class CharactersResponse //extends CustomResponse
{
Code!: CharactersCode;
CharacterID?: string;
}
export enum CharactersCode
{
Success = 0,
InvalidName = 2000,
CharacterNameAlreadyExists = 2009,
NoCharacterSlotAvailable = 3000,
InvalidCharacterClass = 4000,
EmptyResponse = 9000,
UnknownError = 9999,
}
(Custom Response is a parent class with only an UnknownErrorMessage string property, that I use for adding extra message when needed, but only in Unity. I don't need it in my functions.)
I have the same in my C# Unity Project:
public class CharactersResponse : CustomResponse
{
public CharactersCode Code;
public string CharacterID;
}
public enum CharactersCode
{
Success = 0,
InvalidName = 2000,
CharacterNameAlreadyExists = 2009,
NoCharacterSlotAvailable = 3000,
InvalidCharacterClass = 4000,
EmptyResponse = 9000,
UnknownError = 9999,
}
I'm still learning but I found it useful to do this way for displaying correct messages in Unity (and also regarding localization).
When the Code is 0 (Success), I will usually need to get some data at the same time like in this example CharacterID, or CharacterLevel, CharacterName etc.. CharacterResponse will be used for all functions regarding Characters like "GetAllCharacters", "CreateNewCharacter" etc..
My Function (CreateNewCharacter) looks like this:
import * as functions from "firebase-functions";
import { initializeApp } from "firebase-admin/app";
import { getFirestore } from "firebase-admin/firestore";
import { CharactersResponse } from "./CharactersResponse";
import { CharactersCode } from "./CharactersResponse";
import { StringUtils } from "../Utils/StringUtils";
// DATABASE INITIALIZATION
initializeApp();
const db = getFirestore();
// CREATE NEW CHARACTER
export const CreateNewCharacter =
functions.https.onCall((data, context) =>
{
// Checking that the user is authenticated.
if (!context.auth)
{
// Throwing an HttpsError so that the client gets the error details.
throw new functions.https.HttpsError('failed-precondition', 'The function must be called ' +
'while authenticated.');
}
// TEST
data.text = '';
// Authentication / user information is automatically added to the request.
const uid: string = context?.auth?.uid;
const characterName: string = data.text;
// Check if UserID is present
if (StringUtils.isNullOrEmpty(uid))
{
// Throwing an HttpsError so that the client gets the error details.
throw new functions.https.HttpsError('failed-precondition', 'Missing UserID in Auth Context.');
}
const response = new CharactersResponse();
if (StringUtils.isNullOrEmpty(characterName))
{
response.Code = CharactersCode.InvalidName;
console.log("character name null or empty return");
return response; // PROBLEM IS HERE *****************
}
console.log("end return");
return "Character created is named : " + characterName + ". UID = " + uid;
});
In Unity, the function call looks like this:
private static FirebaseFunctions functions = FirebaseManager.Instance.Func;
public static void CreateNewCharacter(string text, Action<CharactersResponse> successCallback, Action<CharactersResponse> failureCallback)
{
Debug.Log("Preparing Function");
// Create the arguments to the callable function.
var data = new Dictionary<string, object>();
data["text"] = text;
// Call the function and extract the operation from the result.
HttpsCallableReference function = functions.GetHttpsCallable("CreateNewCharacter");
function.CallAsync(data).ContinueWithOnMainThread((task) =>
{
if (task.IsFaulted)
{
foreach(var inner in task.Exception.InnerExceptions)
{
if (inner is FunctionsException)
{
var e = (FunctionsException)inner;
// Function error code, will be INTERNAL if the failure
// was not handled properly in the function call.
var code = e.ErrorCode;
var message = e.Message;
Debug.LogError($"Code: {code} // Message: {message}");
if (failureCallback != null)
{
failureCallback.Invoke(new CharactersResponse()
{
Code = CharacterCode.UnknownError,
UnknownErrorMessage = $"ERROR: {code} : {message?.ToString()}"
});
}
}
}
}
else
{
Debug.Log("About to Deserialize response");
// PROBLEM IS HERE *********************
CharactersResponse response = JsonConvert.DeserializeObject<CharactersResponse>(task.Result.Data.ToString());
Debug.Log("Deserialized response");
if (response == null)
{
Debug.LogError("Response is NULL");
}
else
{
Debug.Log("ELSE");
Debug.Log($"Response: {response}");
Debug.Log(response.Code.ToString());
}
}
});
}
The problem :
In my Unity C# code, task.Result.Data contains the CharactersCode I've set in my function, but I can't find a way to convert it to CharactersResponse. (It worked in Azure Functions). Moreover, the line just after Deserialization Debug.Log("Deserialized response"); is not executed. The code seems stuck in the deserialization process.
I tried with and without extending my TypeScript class with CustomResponse(because I don't need it in my Function so I didn't extended it at first).
I also tried setting a CharacterID because I thought maybe it didn't like the fact that this property was missing but the result is the same.
I don't understand what is the problem here? If any of you can help.
Thanks.
HttpsCallableResult.Data is of type object!
=> Your ToString will simply return the type name something like
System.Object
or in your case the result is a dictionary so it prints out that type.
=> This is of course no valid JSON content and not what you expected.
Simply construct the result yourself from the data:
var result = (Dictionary<string, object>)task.Result.Data;
CharactersResponse response = new CharactersResponse
{
Code = (CharactersCode)(int)result["Code"],
CharacterID = (string)result["CharacterID"];
};
I wanted to implement derHugo's solution but couldn't find a way to convert task.Result.Data to Dictionary<string, object>.
The code was stuck at var result = (Dictionary<string, object>)task.Result.Data; even in step by step debugging and no error popped up.
OLD SOLUTION:
So I did a little research and stumbled upon this post and ended up using this instead :
var json = JsonConvert.SerializeObject(task.Result.Data);
CharactersResponse response = JsonConvert.DeserializeObject<CharactersResponse>(json);
I basically convert the task.Result.Data to JSON and convert it back to CharactersResponse and it works. I have what I wanted.
However, I seem to understand that it is not the best solution performance-wise, but for now it is okay and I can now move forward in the project, I'll try to find a better solution later.
NEW SOLUTION:
I wanted to try one last thing, out of curiosity. I wondered what if I convert to JSON at the beginning (in my function) instead of at the end (in my Unity app). So I did this in my function's TypeScript code:
response.Code = CharactersCode.InvalidName;
var r = JSON.stringify(response); // Added this line
return r; // return 'r' instead of 'response'
In my C# code, I retried this line of code:
CharactersResponse response = JsonConvert.DeserializeObject<CharactersResponse>(task.Result.Data.ToString());
And it works ! I just needed to convert my object to JSON in my function before returning it. It allows me to "save" one line of code to process on the client side compared to the old solution.
Thanks derHugo for your answer as it helped me finding what I want.
I've been in the process of migrating a code base from Angular 4.x to 5.x, and I'm running into a strange issue. I have a service function, that is intended to return a list of objects to the front end, which I then massage into a specific data format. I know I'll need to keep the mapping, but I'm a little miffed that it's returning just plain string data.
The original function is this: (using Http from #angular/http just renamed to HttpClient)
public GetVendors(showAll = true, screenArea: number = 0): Observable<Array<SelectModel>> {
let link = AppSettings.API_COMMON_VENDORS;
let params: URLSearchParams = new URLSearchParams();
params.set('showAll', showAll.toString());
params.set('screenArea', screenArea.toString());
let requestOptions = new RequestOptions();
requestOptions.search = params;
return this.httpClient.get(link, requestOptions).map(response => {
let result = JSON.parse(response.json());
let list = new Array<SelectModel>();
let vendors: Array<any> = result;
vendors.forEach(vendor => {
list.push(this.CreateSelectModel(vendor));
});
return list;
});
}
and after ripping out ALL of the Http code, here's the function again using HttpClient from #angular/common/http
public GetVendors(showAll = true, screenArea: number = 0): Observable<Array<SelectModel>> {
let link = AppSettings.API_COMMON_VENDORS;
let params: HttpParams = new HttpParams()
.set('showAll', showAll.toString())
.set('screenArea', screenArea.toString());
return this.httpClient.get<Array<any>>(link, {params}).map(response => {
let list = new Array<SelectModel>();
response.forEach(vendor => {
list.push(this.CreateSelectModel(vendor));
});
return list;
});
}
The issue with this is it kind of defeats the purpose of the new client parsing json for me. The response object is a string representing the JSON of the data I requested, but it's still in a string form, and not the type defined in the get<>() call.
What am I doing wrong here? shouldn't it be parsed already?
Sample Response Data A'la Network Tools in Chrome Dev Tools:
Sample Response Body:
Dev Tools Screenshot with Value of response
The backend (C#) responds with this:
[HttpGet]
public JsonResult Vendors(bool showAll = false, int screenArea = 0)
{
var vendors = _commonBL.GetVendorsSlimForUser(UserModel, UserModel.CustomerId, showAll, screenArea);
return GetJson(vendors);
}
this is how it worked before the Http => HttpClient migration, and it worked with ONE JSON.parse() The data in the return line is simply a standard List<T>
This is what the raw response for your data should look like:
[{"Id":1234,"Name":"Chris Rutherford"}]
But this is what it actually looks like:
"[{\"Id\":1234,\"Name\":\"Chris Rutherford\"}]"
So somewhere in your server code, you have applied JSON encoding twice. Once you correct that, HttpClient will do the right thing.
I'd quote an answer from this thread. Hope it will shed some light on how things work, read it thoroughly it enlightened me tough its not easy to find.
TypeScript only verifies the object interface at compile time. Any object that the code fetches at runtime cannot be verified by
TypeScript.
If this is the case, then things like HttpClient.Get should not
return Observable of type T. It should return Observable of type Object because
that's what is actually returned. Trying to state that it returns T
when it returns Object is misleading.
In the documentation the client's return section says this:
#return an Observable of the body as
type T.
In reality, the documentation should say:
#return an Observable of the body which
might be T. You do not get T back. If you got T back, it would
actually be T, but it's not.
This is my put request in Unity:
UnityWebRequest request = UnityWebRequest.Put(baseUrl + "miniGame1s/", JsonUtility.ToJson(cardsToSend));
And the "cardsToSend" is an object form class below:
[Serializable]
class Cards
{
public string uid;
public List<Card> cards = new List<Card>();
}
[Serializable]
class Card
{
public int index;
public int theCard;
public bool theAnswer;
public int theState;
public bool playerAnswer;
public float answerMoment;
public float timeAfterAnswer;
public float scoreAfterAnswer;
public int validOrNot;
}
And this is the code in the server that written in Node js and Express:
app.put('/api/miniGame1s',(req,res)=>{
console.log(req.body);
const objectSchema = Joi.object().keys({
index: Joi.number().required(),
theCard: Joi.number().required(),
theAnswer: Joi.boolean().required(),
theState: Joi.number().required(),
playerAnswer: Joi.boolean().required(),
answerMoment: Joi.number().required(),
timeAfterAnswer: Joi.number().required(),
scoreAfterAnswer: Joi.number().required(),
validOrNot: Joi.number().integer().required()
});
const arraySchema = {
uid: Joi.string().required(),
cards: Joi.array().items(objectSchema)
};
const result = Joi.validate(req.body, arraySchema);
if(result.error) return res.status(400).send(result.error.details[0].message);
else{
thatGame1InProgressIndex = game1sInProgress.findIndex(g => g.uid == req.body.uid);
console.log('Index: ' + thatGame1InProgressIndex);
for(let i = 0; i < req.body.cards.length;i++)
{
game1sInProgress[thatGame1InProgressIndex].cards[req.body.cards[i].index] = req.body.cards[i];
}
// TODO: validating incomming info and set the validation parameter
// TODO: generate new cards and send them to client.
let newCards = {
uid: req.body.uid,
cards: []
};
for(let i = 0; i < 10 ;i++)
{
let card = CreateACard(game1sInProgress[thatGame1InProgressIndex].cards[game1sInProgress[thatGame1InProgressIndex].cards.length-1].theCard, game1sInProgress[thatGame1InProgressIndex].cards.length);
game1sInProgress[thatGame1InProgressIndex].cards.push(card);
newCards.cards.push(card);
}
res.send(newCards);
}
});
Now, the server code is incomplete yet. ( that's not the issue )
the problem is that I can't retrieve the JSON object in the server.
I tried different things, like use ToString() in unity instead of JSON.ToJson(), or Parsing the "req.body" in the server with JSON.parse(), or things like this.
In the best case, I will give the error: "uid" is required
that is the Joi Validation error. that means it can't reveal the parameters of Object.
I tried to Send the same request with the same data to the server with POSTMAN app. and it works.
I think the problem is that the structure of the JSON that Produced by JSONUtility, is the problem.
Any Help Will Be Appreciated.
First, I should say that this link was so useful:
UnityWebRequest.Post(url, jsonData) sending broken Json
And the solution is that you need to set the "content-type" to "application/json". this way the Express can realize the JSON object and read it perfectly. to do so, add this line after creating your request and before sending it to server:
request.SetRequestHeader("Content-Type", "application/json");
I want to send file and json data from HttpClient to web api server.
I cant seem to access the json in the server via the payload, only as a json var.
public class RegulationFilesController : BaseApiController
{
public void PostFile(RegulationFileDto dto)
{
//the dto is null here
}
}
here is the client:
using (var client = new HttpClient())
{
using (var content = new MultipartFormDataContent())
{
client.BaseAddress = new Uri(ConfigurationManager.AppSettings["ApiHost"]);
content.Add(new StreamContent(File.OpenRead(#"C:\\Chair.png")), "Chair", "Chair.png");
var parameters = new RegulationFileDto
{
ExternalAccountId = "1234",
};
JavaScriptSerializer serializer = new JavaScriptSerializer();
content.Add(new StringContent(serializer.Serialize(parameters), Encoding.UTF8, "application/json"));
var resTask = client.PostAsync("api/RegulationFiles", content); //?ApiKey=24Option_key
resTask.Wait();
resTask.ContinueWith(async responseTask =>
{
var res = await responseTask.Result.Content.ReadAsStringAsync();
}
);
}
}
this example will work:HttpClient Multipart Form Post in C#
but only via the form-data and not payload.
Can you please suggest how to access the file and the submitted json And the file at the same request?
Thanks
I have tried many different ways to submit both file data and metadata and this is the best approach I have found:
Don't use MultipartFormDataContent, use only StreamContent for the file data. This way you can stream the file upload so you don't take up too much RAM on the server. MultipartFormDataContent requires you to load the entire request into memory and then save the files to a local storage somewhere. By streaming, you also have the benefit of copying the stream into other locations such as an Azure storage container.
This solves the issue of the binary data, and now for the metadata. For this, use a custom header and serialize your JSON into that. Your controller can read the custom header and deserialize it as your metadata dto. There is a size limit to headers, see here (8-16KB), which is a large amount of data. If you need more space, you could do two separate requests, one to POST the minimum need, and then a PATCH to update any properties that needed more than a header could fit.
Sample code:
public class RegulationFilesController : BaseApiController
{
public async Task<IHttpActionResult> Post()
{
var isMultipart = this.Request.Content.IsMimeMultipartContent();
if (isMultipart)
{
return this.BadRequest("Only binary uploads are accepted.");
}
var headerDto = this.GetJsonDataHeader<RegulationFileDto>();
if(headerDto == null)
{
return this.BadRequest("Missing X-JsonData header.");
}
using (var stream = await this.Request.Content.ReadAsStreamAsync())
{
if (stream == null || stream.Length == 0)
{
return this.BadRequest("Invalid binary data.");
}
//save stream to disk or copy to another stream
var model = new RegulationFile(headerDto);
//save your model to the database
var dto = new RegulationFileDto(model);
var uri = new Uri("NEW URI HERE");
return this.Created(uri, dto);
}
}
private T GetJsonDataHeader<T>()
{
IEnumerable<string> headerCollection;
if (!this.Request.Headers.TryGetValues("X-JsonData", out headerCollection))
{
return default(T);
}
var headerItems = headerCollection.ToList();
if (headerItems.Count() != 1)
{
return default(T);
}
var meta = headerItems.FirstOrDefault();
return !string.IsNullOrWhiteSpace(meta) ? JsonConvert.DeserializeObject<T>(meta) : default(T);
}
}
is it possible to get metadata of an OData service in JSON format?
When I try to use format=json , it doesn't work. Here is what I tried:
http://odata.informea.org/services/odata.svc/$metadata/?format=json
The $metadata document is in the CSDL format, which currently only has an XML representation. (As a side note, if you do want to request the json format for a different kind of OData payload, make sure the format query token has a $ in front of it: $format=json.)
So, no it is not possible. You can, however, get the service document in JSON, which is a subset of the $metadata document:
http://odata.informea.org/services/odata.svc?$format=json
This won't have type information, but it will list the available entry points of the service (i.e., the entity sets).
As an alternative to ?$format=json, you could also just set the following two headers :
Accept: application/json
Content-Type: application/json; charset=utf-8
I'm not sure which is the minimum Odata version required, but this works perfectly for me on Microsoft Dynamics NAV 2016, which uses Odata v4.
I agreed with the previous answer. This isn't supported by the specification but some OData frameworks / libraries are about to implement this feature.
I think about Olingo. This is could be helpful for you if you also implement the server side. See this issue in the Olingo JIRA for more details:
OLINGO-570 - https://issues.apache.org/jira/browse/OLINGO-570
Hope it helps you,
Thierry
You can use jQuery to get the relevant information from an OData service $metadata.
Take for example:
You write a unit test to check the OData entities property names matches with your application entities. Then you have to retrieve the properties of the OData entity.
$.ajax({
type: "GET",
url: "/destinations/odata-service/$metadata",
beforeSend: function() {
console.log("before send check");
},
dataType: "xml",
contentType: "application/atom+xml",
context: document.body,
success: function(xml) {
console.log("Success ResourceTypes");
var ODataTypeINeed = $(xml).find('EntityType').filter(function(){
return $(this).attr('Name') == 'ODataTypeINeed'
});
$(ODataTypeINeed).find('Property').each(function() {
console.log($(this).attr('Name')); //List of OData Entity properties
});
},
error: function(err) {
console.log(err);
}
});
I wrote a simple provider to parse out some of the needed information from the metadata, Feel free to expand on it. First you'll need some simple models, to expresss the data, we'll want to convert from there ugly XML names
export class ODataEntityType
{
name: string;
properties: ODataProperty[];
}
export class ODataProperty
{
name: string;
type: ODataTypes;
isNullable: boolean;
}
//Hack Until Ionic supports TS 2.4
export class ODataTypeMap
{
"Edm.Int32" = ODataTypes.Int;
"Edm.Int64" = ODataTypes.Long;
"Edm.Decimal" = ODataTypes.Decimal;
"Edm.Double" = ODataTypes.Double;
"Edm.Guid" = ODataTypes.Guid;
"Edm.String" = ODataTypes.String;
"Edm.Boolean" = ODataTypes.Bool;
"Edm.DateTime" = ODataTypes.DateTime;
"Edm.DateTimeOffset" = ODataTypes.DateTimeOffset;
}
export enum ODataTypes
{
Int,
Long,
Decimal,
Double,
Guid,
String,
Bool,
DateTime,
DateTimeOffset
}
This is the provider:
import { Injectable } from "#angular/core";
import { Http } from "#angular/http";
import * as X2JS from 'x2js';
import * as _ from 'underscore';
import { ODataEntityType, ODataProperty, ODataTypes, ODataTypeMap } from "../models/ODataEntityType";
#Injectable()
export class ODataMetadataToJsonProvider {
x2js = new X2JS();
public entityTypeMap: Dictionary = new Dictionary();
public entityTypes : ODataEntityType[];
constructor(public http: Http) {
}
parseODataMetadata(metadataUrl: string) {
this.http.get(metadataUrl).subscribe(data => {
let metadata: any = this.x2js.xml2js(data.text());
let rawEntityTypes = _.filter(metadata.Edmx.DataServices.Schema, x => x["EntityType"] != null);
if(rawEntityTypes.length == 0)
{
return;
}
this.entityTypes = _.map(rawEntityTypes[0]["EntityType"], t => {
let oDataEntityType = new ODataEntityType();
oDataEntityType.name = t["_Name"];
oDataEntityType.properties = _.map(t["Property"], p => {
let property = new ODataProperty();
property.name = p["_Name"];
let typeStr: string = p["_Type"];
property.type = ODataTypeMap[typeStr];
property.isNullable = !!p["_Nullable"];
return property;
});
return oDataEntityType;
});
});
}
}
In 2021, possibly...
The OData 4.01 specification includes support for JSON: https://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part1-protocol.html#sec_MetadataDocumentRequest
Microsoft's OData implementation added support from the following versions:
ODataLib 7.7.3 (released 24 Sep 2020)
WebAPI 7.5.4 (released 29 Dec 2020)
Additionally, JSON Metadata is only supported at platform implementing .NETStardard 2.0. [sic].
If the service you're calling supports it, you can do it in the query string...
$format=application/json
$format=json
...or using the http headers:
Accept=application/json
It it doesn't... ask the provider of the service to upgrade?