flutter: FormatException: Unterminated string while decoding long stream response - json

I was working on a flutter package that I made https://github.com/pratikbaid3/flutter_client_sse. This is used to consume server-sent events. This works fine when the response returned is comparatively small. But as soon as the response grows large, I start getting this error flutter: FormatException: Unterminated string
This is the code snippet used to get the sse
while (true) {
try {
_client = http.Client();
var request = new http.Request("GET", Uri.parse(url));
request.headers["Cache-Control"] = "no-cache";
request.headers["Accept"] = "text/event-stream";
request.headers["Cookie"] = token;
Future<http.StreamedResponse> response = _client.send(request);
await for (final data in response.asStream()) {
await for (final d in data.stream) {
final rawData = utf8.decode(d);
final event = rawData.split("\n")[1];
if (event != null && event != '') {
yield SSEModel.fromData(rawData);
}
}
}
} catch (e) {
print('---ERROR---');
print(e);
yield SSEModel(data: '', id: '', event: '');
}
await Future.delayed(Duration(seconds: 1), () {});
}
My guess is that a part of the response is lost due to the large size.

So after hours of debugging, I found that in my current approach when the response is very large, part of it is truncated (Not sure why). I changed the code to read one line at a time and got it to work.
data.stream
..transform(Utf8Decoder())
.transform(LineSplitter())
.listen((dataLine) {
//Do what you want with the data here
}
This seemed to solve the issue.
The complete code can be found at https://github.com/pratikbaid3/flutter_client_sse

Related

How can I send custom object from my callable Firebase Cloud Function in TypeScript to my Unity app?

I'm trying to use Firebase and its callable Cloud Functions for my Unity project.
With the docs and different posts I found on the web I struggle to understand how returning data works. (I come from Azure Functions in C#)
I use TypeScript, and try to return a custom object CharactersResponse:
export class CharactersResponse //extends CustomResponse
{
Code!: CharactersCode;
CharacterID?: string;
}
export enum CharactersCode
{
Success = 0,
InvalidName = 2000,
CharacterNameAlreadyExists = 2009,
NoCharacterSlotAvailable = 3000,
InvalidCharacterClass = 4000,
EmptyResponse = 9000,
UnknownError = 9999,
}
(Custom Response is a parent class with only an UnknownErrorMessage string property, that I use for adding extra message when needed, but only in Unity. I don't need it in my functions.)
I have the same in my C# Unity Project:
public class CharactersResponse : CustomResponse
{
public CharactersCode Code;
public string CharacterID;
}
public enum CharactersCode
{
Success = 0,
InvalidName = 2000,
CharacterNameAlreadyExists = 2009,
NoCharacterSlotAvailable = 3000,
InvalidCharacterClass = 4000,
EmptyResponse = 9000,
UnknownError = 9999,
}
I'm still learning but I found it useful to do this way for displaying correct messages in Unity (and also regarding localization).
When the Code is 0 (Success), I will usually need to get some data at the same time like in this example CharacterID, or CharacterLevel, CharacterName etc.. CharacterResponse will be used for all functions regarding Characters like "GetAllCharacters", "CreateNewCharacter" etc..
My Function (CreateNewCharacter) looks like this:
import * as functions from "firebase-functions";
import { initializeApp } from "firebase-admin/app";
import { getFirestore } from "firebase-admin/firestore";
import { CharactersResponse } from "./CharactersResponse";
import { CharactersCode } from "./CharactersResponse";
import { StringUtils } from "../Utils/StringUtils";
// DATABASE INITIALIZATION
initializeApp();
const db = getFirestore();
// CREATE NEW CHARACTER
export const CreateNewCharacter =
functions.https.onCall((data, context) =>
{
// Checking that the user is authenticated.
if (!context.auth)
{
// Throwing an HttpsError so that the client gets the error details.
throw new functions.https.HttpsError('failed-precondition', 'The function must be called ' +
'while authenticated.');
}
// TEST
data.text = '';
// Authentication / user information is automatically added to the request.
const uid: string = context?.auth?.uid;
const characterName: string = data.text;
// Check if UserID is present
if (StringUtils.isNullOrEmpty(uid))
{
// Throwing an HttpsError so that the client gets the error details.
throw new functions.https.HttpsError('failed-precondition', 'Missing UserID in Auth Context.');
}
const response = new CharactersResponse();
if (StringUtils.isNullOrEmpty(characterName))
{
response.Code = CharactersCode.InvalidName;
console.log("character name null or empty return");
return response; // PROBLEM IS HERE *****************
}
console.log("end return");
return "Character created is named : " + characterName + ". UID = " + uid;
});
In Unity, the function call looks like this:
private static FirebaseFunctions functions = FirebaseManager.Instance.Func;
public static void CreateNewCharacter(string text, Action<CharactersResponse> successCallback, Action<CharactersResponse> failureCallback)
{
Debug.Log("Preparing Function");
// Create the arguments to the callable function.
var data = new Dictionary<string, object>();
data["text"] = text;
// Call the function and extract the operation from the result.
HttpsCallableReference function = functions.GetHttpsCallable("CreateNewCharacter");
function.CallAsync(data).ContinueWithOnMainThread((task) =>
{
if (task.IsFaulted)
{
foreach(var inner in task.Exception.InnerExceptions)
{
if (inner is FunctionsException)
{
var e = (FunctionsException)inner;
// Function error code, will be INTERNAL if the failure
// was not handled properly in the function call.
var code = e.ErrorCode;
var message = e.Message;
Debug.LogError($"Code: {code} // Message: {message}");
if (failureCallback != null)
{
failureCallback.Invoke(new CharactersResponse()
{
Code = CharacterCode.UnknownError,
UnknownErrorMessage = $"ERROR: {code} : {message?.ToString()}"
});
}
}
}
}
else
{
Debug.Log("About to Deserialize response");
// PROBLEM IS HERE *********************
CharactersResponse response = JsonConvert.DeserializeObject<CharactersResponse>(task.Result.Data.ToString());
Debug.Log("Deserialized response");
if (response == null)
{
Debug.LogError("Response is NULL");
}
else
{
Debug.Log("ELSE");
Debug.Log($"Response: {response}");
Debug.Log(response.Code.ToString());
}
}
});
}
The problem :
In my Unity C# code, task.Result.Data contains the CharactersCode I've set in my function, but I can't find a way to convert it to CharactersResponse. (It worked in Azure Functions). Moreover, the line just after Deserialization Debug.Log("Deserialized response"); is not executed. The code seems stuck in the deserialization process.
I tried with and without extending my TypeScript class with CustomResponse(because I don't need it in my Function so I didn't extended it at first).
I also tried setting a CharacterID because I thought maybe it didn't like the fact that this property was missing but the result is the same.
I don't understand what is the problem here? If any of you can help.
Thanks.
HttpsCallableResult.Data is of type object!
=> Your ToString will simply return the type name something like
System.Object
or in your case the result is a dictionary so it prints out that type.
=> This is of course no valid JSON content and not what you expected.
Simply construct the result yourself from the data:
var result = (Dictionary<string, object>)task.Result.Data;
CharactersResponse response = new CharactersResponse
{
Code = (CharactersCode)(int)result["Code"],
CharacterID = (string)result["CharacterID"];
};
I wanted to implement derHugo's solution but couldn't find a way to convert task.Result.Data to Dictionary<string, object>.
The code was stuck at var result = (Dictionary<string, object>)task.Result.Data; even in step by step debugging and no error popped up.
OLD SOLUTION:
So I did a little research and stumbled upon this post and ended up using this instead :
var json = JsonConvert.SerializeObject(task.Result.Data);
CharactersResponse response = JsonConvert.DeserializeObject<CharactersResponse>(json);
I basically convert the task.Result.Data to JSON and convert it back to CharactersResponse and it works. I have what I wanted.
However, I seem to understand that it is not the best solution performance-wise, but for now it is okay and I can now move forward in the project, I'll try to find a better solution later.
NEW SOLUTION:
I wanted to try one last thing, out of curiosity. I wondered what if I convert to JSON at the beginning (in my function) instead of at the end (in my Unity app). So I did this in my function's TypeScript code:
response.Code = CharactersCode.InvalidName;
var r = JSON.stringify(response); // Added this line
return r; // return 'r' instead of 'response'
In my C# code, I retried this line of code:
CharactersResponse response = JsonConvert.DeserializeObject<CharactersResponse>(task.Result.Data.ToString());
And it works ! I just needed to convert my object to JSON in my function before returning it. It allows me to "save" one line of code to process on the client side compared to the old solution.
Thanks derHugo for your answer as it helped me finding what I want.

Why HttpClient.GetFromJsonAsync doesn't throw an exception when the response is HTML instead of JSON?

I'm learning Blazor.
I have created a Blazor WASM App with the "ASP.NET Core Hosted" option.
So I have 3 projects in the solution: Client, Server and Shared.
The following code is in the Client project and works perfectly when the endpoint is correct (obviously). But at some point I made a mistake and messed up the request URI, and then I noticed that the API returned an HTML page with code 200 OK (as you can see in the Postman screenshot below the code).
I expected one of my try-catches to get this, but the debugger jumps to the last line (return null) without throwing an exception.
My first question is why?
My second question is how can I catch this?
I know fixing the endpoint fixes everything, but would be nice to have a catch that alerts me when I have mistyped an URI.
Thanks.
private readonly HttpClient _httpClient;
public async Task<List<Collaborator>> GetCollaborators()
{
string requestUri = "api/non-existent-endpoint";
try
{
var response = await _httpClient.GetFromJsonAsync<CollaboratorsResponse>(requestUri);
if (response == null)
{
// It never enters here. Jumps to the last line of code.
}
return response.Collaborators;
}
catch (HttpRequestException)
{
Console.WriteLine("An error occurred.");
}
catch (NotSupportedException)
{
Console.WriteLine("The content type is not supported.");
}
catch (JsonException)
{
Console.WriteLine("Invalid JSON.");
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
return null;
}
it is a never good idea to use GetFromJsonAsync, You are not the first who are asking about the strange behavior. Try to use GetAsync. at least you will now what is going on.
var response = await client.GetAsync(requestUri);
if (response.IsSuccessStatusCode)
{
var stringData = await response.Content.ReadAsStringAsync();
var result = JsonConvert.DeserializeObject<CollaboratorsResponse>(stringData);
... your code
}
else
{
var statusCode = response.StatusCode.ToString(); // HERE is your error status code, when you have an error
}

How can I invoke and use Google Cloud Functions in a Flutter app?

I have created a url scraper function, working and tested on Google Cloud, but I am really drawing a blank on how to invoke it. I have tried two methods, one using the cloud_functions package, and the other using a standard HTTPS get. I've tried looking online, but none of the solutions/guides involve functions with an input from the Flutter app, and an output back to the app.
Here's the structure of the function (which is working alright). I've named this function Parse in Google Cloud Platform.
<PYTHON PACKAGE IMPORTS>
def Parser(url):
<URL PARSE FUNCTIONS>
return source, datetime, imageurl, keyword
def invoke_parse(request):
request_json = request.get_json(silent=True)
file = Parser(request_json['url'])
return jsonify({
"source": file[0],
"datetime": file[1],
"imageurl": file[2],
"keyword": file[3],
})
The first method I tried was using an HTTP CALL to get the function. But that isn't working, even though there are no errors - I suspect it's just returning nothing.
parser(String url) async{ // Here I honestly don't know where to use the url input within the function
var uri = Uri.parse(<Function URL String>);
HttpClient client;
try {
var request = await client.getUrl(uri);
var response = await request.close();
if (response.statusCode == HttpStatus.ok) {
var json = await response.transform(utf8.decoder).join();
Map data = jsonDecode(json) as Map;
source = data['source']; // These are the variables used in the main Flutter app
postedAt = data['datetime'];
_imageUrl = data['image'];
keyword = data['keyword'];
} else {
print('Error running parse:\nHttp status ${response.statusCode}');
}
} catch (exception) {
print('Failed invoking the parse function.');
}
}
That didn't work, so I thought I might alternatively use the cloud_functions package as follows (in lieu of the previous):
parser(String url) async {
var functionUrl = <FUNCTION URL>;
HttpsCallable callable = CloudFunctions.instance.getHttpsCallable(functionName: 'Parse')
..timeout = const Duration(seconds: 30);
try {
final HttpsCallableResult result = await callable.call(
<String, dynamic>{
'url': url,
}
);
setState(() {
source = result.data['source']; //THESE ARE VARIABLES USED IN THE FLUTTER APP
postedAt = result.data['datetime'];
_imageUrl = result.data['image'];
keyword = result.data['keyword'];
});
}
on CloudFunctionsException catch (e) {
print('caught firebase functions exception');
print(e.code);
print(e.message);
print(e.details);
} catch (e) {
print('caught generic exception');
print(e);
}
}
In the latter case, the code ran without errors but doesn't work. My flutter log states the following error:
I/flutter ( 2821): caught generic exception
I/flutter ( 2821): PlatformException(functionsError, Cloud function failed with exception., {code: NOT_FOUND, details: null, message: NOT_FOUND})
which I'm assuming is also an error at not being able to read the function.
Any help on how I should go about processing my function would be appreciated. Apologies if something is a really obvious solution, but I am not familiar as much with HTTP requests and cloud platforms.
Thanks and cheers.
Node Js Backend Function
const functions = require("firebase-functions");
const admin = require("firebase-admin");
admin.initializeApp();
exports.test = functions.https.onCall(async (data, context) => {
functions.logger.info("Hello logs: ", {structuredData: true});
functions.logger.info( data.token, {structuredData: true});
}
Flutter frontend
1- pubspec.yaml
cloud_functions: ^1.1.2
2 - Code
HttpsCallable callable = FirebaseFunctions.instance.httpsCallable('test');
final HttpsCallableResult results = await callable.call<Map>( {
'token': token,
});

Why is this data parsing failing?

I'm writing a Node.js server and I'm trying to fetch data from an API and return it to my user. I'm taking the insightlyResponse and trying to convert to JSON. Here's my code:
insightlyResponse.setEncoding('utf8');
let rawData = '';
insightlyResponse.on('data', (chunk) => rawData += chunk);
insightlyResponse.on('end', () => {
try {
const parsedData = JSON.parse(rawData);
responseData = "PARSED";
} catch (e) {
responseData = `Got error: ${e.message}`
}
response.end(responseData);
});
The error is Got error: Unexpected token \u001f in JSON at position 0. What does this mean and what am I doing wrong?
Figured it out! I double-checked the API and realized that it can return compressed responses. Turned off that setting and works like a charm!

Post a list and iterate it using Dart

I'm trying to post some data from a dart project to another and store them in a mongoDB
Post code:
import 'dart:io';
void main() {
List example = [
{"source": "today", "target": "tomorrow"},
{"source": "yesterday", "target": "tomorrow"},
{"source": "today", "target": "yesterday"}
];
new HttpClient().post('localhost', 4040, '')
.then((HttpClientRequest request) {
request.headers.contentType = ContentType.JSON;
request.write(example);
return request.close();
});
}
Code that receives it, inside another file
void start() {
HttpServer.bind(address, port)
.then((HttpServer server) {
// Log in console to show that server is listening
print('Server listening on ${address}:${server.port}');
server.listen((HttpRequest request) {
request.transform(UTF8.decoder).listen(sendToDatastore);
});
});
}
void sendToDatastore(String contents) {
var dbproxy = new dbProxy("myDb");
dbproxy.write("rawdata", contents);
index++;
// non related to the problem code
}
bool write(collectionName, document)
{
Db connection = connect();
DbCollection collection = connection.collection(collectionName);
connection.open().then((_){
print('writing $document to db');
collection.insert(document);
}).then((_) {
print('closing db');
connection.close();
});
return true;
}
What I'm struggling with is that I'm using
request.transform(UTF8.decoder).listen(sendToDatastore);
so I'm converting the request stream to a string as I couldn't find the way to send it as Json.
And then in sendToDatastore I'm not able to parse it properly in order to store it. As far as I understand I'd need to get every Json object as a Map to store it as I'm getting this error
Uncaught Error: type 'String' is not a subtype of type 'Map' of 'document'.
Thanks,
UPDATE
If I try to do something like this in sendToDatastore
void sendToDatastore(String contents) {
var dbproxy = new dbProxy("myDb");
var contentToPass = JSON.decode(contents);
contentToPass.forEach((element) => dbproxy.write("rawdata", element));
index++;
// non related to the problem code
}
It raises this error
Uncaught Error: FormatException: Unexpected character (at character 3)
[{source: today, target: tomorrow}, {source: yesterday, target: tomorrow}, ...
^
In the use of JSON.decode
UPDATE2
The error was that I wasn't sending actual Json from the "post code". I used
// ...
request.write(JSON.encode(example));
// ...
and everything worked fine
Thanks
You should be able to use the dart:convert package.
You can then use:
String str = JSON.encode(obj)
and
var obj = JSON.decode(str)
to convert string/json.