Post a list and iterate it using Dart - json

I'm trying to post some data from a dart project to another and store them in a mongoDB
Post code:
import 'dart:io';
void main() {
List example = [
{"source": "today", "target": "tomorrow"},
{"source": "yesterday", "target": "tomorrow"},
{"source": "today", "target": "yesterday"}
];
new HttpClient().post('localhost', 4040, '')
.then((HttpClientRequest request) {
request.headers.contentType = ContentType.JSON;
request.write(example);
return request.close();
});
}
Code that receives it, inside another file
void start() {
HttpServer.bind(address, port)
.then((HttpServer server) {
// Log in console to show that server is listening
print('Server listening on ${address}:${server.port}');
server.listen((HttpRequest request) {
request.transform(UTF8.decoder).listen(sendToDatastore);
});
});
}
void sendToDatastore(String contents) {
var dbproxy = new dbProxy("myDb");
dbproxy.write("rawdata", contents);
index++;
// non related to the problem code
}
bool write(collectionName, document)
{
Db connection = connect();
DbCollection collection = connection.collection(collectionName);
connection.open().then((_){
print('writing $document to db');
collection.insert(document);
}).then((_) {
print('closing db');
connection.close();
});
return true;
}
What I'm struggling with is that I'm using
request.transform(UTF8.decoder).listen(sendToDatastore);
so I'm converting the request stream to a string as I couldn't find the way to send it as Json.
And then in sendToDatastore I'm not able to parse it properly in order to store it. As far as I understand I'd need to get every Json object as a Map to store it as I'm getting this error
Uncaught Error: type 'String' is not a subtype of type 'Map' of 'document'.
Thanks,
UPDATE
If I try to do something like this in sendToDatastore
void sendToDatastore(String contents) {
var dbproxy = new dbProxy("myDb");
var contentToPass = JSON.decode(contents);
contentToPass.forEach((element) => dbproxy.write("rawdata", element));
index++;
// non related to the problem code
}
It raises this error
Uncaught Error: FormatException: Unexpected character (at character 3)
[{source: today, target: tomorrow}, {source: yesterday, target: tomorrow}, ...
^
In the use of JSON.decode
UPDATE2
The error was that I wasn't sending actual Json from the "post code". I used
// ...
request.write(JSON.encode(example));
// ...
and everything worked fine
Thanks

You should be able to use the dart:convert package.
You can then use:
String str = JSON.encode(obj)
and
var obj = JSON.decode(str)
to convert string/json.

Related

How can I send custom object from my callable Firebase Cloud Function in TypeScript to my Unity app?

I'm trying to use Firebase and its callable Cloud Functions for my Unity project.
With the docs and different posts I found on the web I struggle to understand how returning data works. (I come from Azure Functions in C#)
I use TypeScript, and try to return a custom object CharactersResponse:
export class CharactersResponse //extends CustomResponse
{
Code!: CharactersCode;
CharacterID?: string;
}
export enum CharactersCode
{
Success = 0,
InvalidName = 2000,
CharacterNameAlreadyExists = 2009,
NoCharacterSlotAvailable = 3000,
InvalidCharacterClass = 4000,
EmptyResponse = 9000,
UnknownError = 9999,
}
(Custom Response is a parent class with only an UnknownErrorMessage string property, that I use for adding extra message when needed, but only in Unity. I don't need it in my functions.)
I have the same in my C# Unity Project:
public class CharactersResponse : CustomResponse
{
public CharactersCode Code;
public string CharacterID;
}
public enum CharactersCode
{
Success = 0,
InvalidName = 2000,
CharacterNameAlreadyExists = 2009,
NoCharacterSlotAvailable = 3000,
InvalidCharacterClass = 4000,
EmptyResponse = 9000,
UnknownError = 9999,
}
I'm still learning but I found it useful to do this way for displaying correct messages in Unity (and also regarding localization).
When the Code is 0 (Success), I will usually need to get some data at the same time like in this example CharacterID, or CharacterLevel, CharacterName etc.. CharacterResponse will be used for all functions regarding Characters like "GetAllCharacters", "CreateNewCharacter" etc..
My Function (CreateNewCharacter) looks like this:
import * as functions from "firebase-functions";
import { initializeApp } from "firebase-admin/app";
import { getFirestore } from "firebase-admin/firestore";
import { CharactersResponse } from "./CharactersResponse";
import { CharactersCode } from "./CharactersResponse";
import { StringUtils } from "../Utils/StringUtils";
// DATABASE INITIALIZATION
initializeApp();
const db = getFirestore();
// CREATE NEW CHARACTER
export const CreateNewCharacter =
functions.https.onCall((data, context) =>
{
// Checking that the user is authenticated.
if (!context.auth)
{
// Throwing an HttpsError so that the client gets the error details.
throw new functions.https.HttpsError('failed-precondition', 'The function must be called ' +
'while authenticated.');
}
// TEST
data.text = '';
// Authentication / user information is automatically added to the request.
const uid: string = context?.auth?.uid;
const characterName: string = data.text;
// Check if UserID is present
if (StringUtils.isNullOrEmpty(uid))
{
// Throwing an HttpsError so that the client gets the error details.
throw new functions.https.HttpsError('failed-precondition', 'Missing UserID in Auth Context.');
}
const response = new CharactersResponse();
if (StringUtils.isNullOrEmpty(characterName))
{
response.Code = CharactersCode.InvalidName;
console.log("character name null or empty return");
return response; // PROBLEM IS HERE *****************
}
console.log("end return");
return "Character created is named : " + characterName + ". UID = " + uid;
});
In Unity, the function call looks like this:
private static FirebaseFunctions functions = FirebaseManager.Instance.Func;
public static void CreateNewCharacter(string text, Action<CharactersResponse> successCallback, Action<CharactersResponse> failureCallback)
{
Debug.Log("Preparing Function");
// Create the arguments to the callable function.
var data = new Dictionary<string, object>();
data["text"] = text;
// Call the function and extract the operation from the result.
HttpsCallableReference function = functions.GetHttpsCallable("CreateNewCharacter");
function.CallAsync(data).ContinueWithOnMainThread((task) =>
{
if (task.IsFaulted)
{
foreach(var inner in task.Exception.InnerExceptions)
{
if (inner is FunctionsException)
{
var e = (FunctionsException)inner;
// Function error code, will be INTERNAL if the failure
// was not handled properly in the function call.
var code = e.ErrorCode;
var message = e.Message;
Debug.LogError($"Code: {code} // Message: {message}");
if (failureCallback != null)
{
failureCallback.Invoke(new CharactersResponse()
{
Code = CharacterCode.UnknownError,
UnknownErrorMessage = $"ERROR: {code} : {message?.ToString()}"
});
}
}
}
}
else
{
Debug.Log("About to Deserialize response");
// PROBLEM IS HERE *********************
CharactersResponse response = JsonConvert.DeserializeObject<CharactersResponse>(task.Result.Data.ToString());
Debug.Log("Deserialized response");
if (response == null)
{
Debug.LogError("Response is NULL");
}
else
{
Debug.Log("ELSE");
Debug.Log($"Response: {response}");
Debug.Log(response.Code.ToString());
}
}
});
}
The problem :
In my Unity C# code, task.Result.Data contains the CharactersCode I've set in my function, but I can't find a way to convert it to CharactersResponse. (It worked in Azure Functions). Moreover, the line just after Deserialization Debug.Log("Deserialized response"); is not executed. The code seems stuck in the deserialization process.
I tried with and without extending my TypeScript class with CustomResponse(because I don't need it in my Function so I didn't extended it at first).
I also tried setting a CharacterID because I thought maybe it didn't like the fact that this property was missing but the result is the same.
I don't understand what is the problem here? If any of you can help.
Thanks.
HttpsCallableResult.Data is of type object!
=> Your ToString will simply return the type name something like
System.Object
or in your case the result is a dictionary so it prints out that type.
=> This is of course no valid JSON content and not what you expected.
Simply construct the result yourself from the data:
var result = (Dictionary<string, object>)task.Result.Data;
CharactersResponse response = new CharactersResponse
{
Code = (CharactersCode)(int)result["Code"],
CharacterID = (string)result["CharacterID"];
};
I wanted to implement derHugo's solution but couldn't find a way to convert task.Result.Data to Dictionary<string, object>.
The code was stuck at var result = (Dictionary<string, object>)task.Result.Data; even in step by step debugging and no error popped up.
OLD SOLUTION:
So I did a little research and stumbled upon this post and ended up using this instead :
var json = JsonConvert.SerializeObject(task.Result.Data);
CharactersResponse response = JsonConvert.DeserializeObject<CharactersResponse>(json);
I basically convert the task.Result.Data to JSON and convert it back to CharactersResponse and it works. I have what I wanted.
However, I seem to understand that it is not the best solution performance-wise, but for now it is okay and I can now move forward in the project, I'll try to find a better solution later.
NEW SOLUTION:
I wanted to try one last thing, out of curiosity. I wondered what if I convert to JSON at the beginning (in my function) instead of at the end (in my Unity app). So I did this in my function's TypeScript code:
response.Code = CharactersCode.InvalidName;
var r = JSON.stringify(response); // Added this line
return r; // return 'r' instead of 'response'
In my C# code, I retried this line of code:
CharactersResponse response = JsonConvert.DeserializeObject<CharactersResponse>(task.Result.Data.ToString());
And it works ! I just needed to convert my object to JSON in my function before returning it. It allows me to "save" one line of code to process on the client side compared to the old solution.
Thanks derHugo for your answer as it helped me finding what I want.

flutter: FormatException: Unterminated string while decoding long stream response

I was working on a flutter package that I made https://github.com/pratikbaid3/flutter_client_sse. This is used to consume server-sent events. This works fine when the response returned is comparatively small. But as soon as the response grows large, I start getting this error flutter: FormatException: Unterminated string
This is the code snippet used to get the sse
while (true) {
try {
_client = http.Client();
var request = new http.Request("GET", Uri.parse(url));
request.headers["Cache-Control"] = "no-cache";
request.headers["Accept"] = "text/event-stream";
request.headers["Cookie"] = token;
Future<http.StreamedResponse> response = _client.send(request);
await for (final data in response.asStream()) {
await for (final d in data.stream) {
final rawData = utf8.decode(d);
final event = rawData.split("\n")[1];
if (event != null && event != '') {
yield SSEModel.fromData(rawData);
}
}
}
} catch (e) {
print('---ERROR---');
print(e);
yield SSEModel(data: '', id: '', event: '');
}
await Future.delayed(Duration(seconds: 1), () {});
}
My guess is that a part of the response is lost due to the large size.
So after hours of debugging, I found that in my current approach when the response is very large, part of it is truncated (Not sure why). I changed the code to read one line at a time and got it to work.
data.stream
..transform(Utf8Decoder())
.transform(LineSplitter())
.listen((dataLine) {
//Do what you want with the data here
}
This seemed to solve the issue.
The complete code can be found at https://github.com/pratikbaid3/flutter_client_sse

How can I invoke and use Google Cloud Functions in a Flutter app?

I have created a url scraper function, working and tested on Google Cloud, but I am really drawing a blank on how to invoke it. I have tried two methods, one using the cloud_functions package, and the other using a standard HTTPS get. I've tried looking online, but none of the solutions/guides involve functions with an input from the Flutter app, and an output back to the app.
Here's the structure of the function (which is working alright). I've named this function Parse in Google Cloud Platform.
<PYTHON PACKAGE IMPORTS>
def Parser(url):
<URL PARSE FUNCTIONS>
return source, datetime, imageurl, keyword
def invoke_parse(request):
request_json = request.get_json(silent=True)
file = Parser(request_json['url'])
return jsonify({
"source": file[0],
"datetime": file[1],
"imageurl": file[2],
"keyword": file[3],
})
The first method I tried was using an HTTP CALL to get the function. But that isn't working, even though there are no errors - I suspect it's just returning nothing.
parser(String url) async{ // Here I honestly don't know where to use the url input within the function
var uri = Uri.parse(<Function URL String>);
HttpClient client;
try {
var request = await client.getUrl(uri);
var response = await request.close();
if (response.statusCode == HttpStatus.ok) {
var json = await response.transform(utf8.decoder).join();
Map data = jsonDecode(json) as Map;
source = data['source']; // These are the variables used in the main Flutter app
postedAt = data['datetime'];
_imageUrl = data['image'];
keyword = data['keyword'];
} else {
print('Error running parse:\nHttp status ${response.statusCode}');
}
} catch (exception) {
print('Failed invoking the parse function.');
}
}
That didn't work, so I thought I might alternatively use the cloud_functions package as follows (in lieu of the previous):
parser(String url) async {
var functionUrl = <FUNCTION URL>;
HttpsCallable callable = CloudFunctions.instance.getHttpsCallable(functionName: 'Parse')
..timeout = const Duration(seconds: 30);
try {
final HttpsCallableResult result = await callable.call(
<String, dynamic>{
'url': url,
}
);
setState(() {
source = result.data['source']; //THESE ARE VARIABLES USED IN THE FLUTTER APP
postedAt = result.data['datetime'];
_imageUrl = result.data['image'];
keyword = result.data['keyword'];
});
}
on CloudFunctionsException catch (e) {
print('caught firebase functions exception');
print(e.code);
print(e.message);
print(e.details);
} catch (e) {
print('caught generic exception');
print(e);
}
}
In the latter case, the code ran without errors but doesn't work. My flutter log states the following error:
I/flutter ( 2821): caught generic exception
I/flutter ( 2821): PlatformException(functionsError, Cloud function failed with exception., {code: NOT_FOUND, details: null, message: NOT_FOUND})
which I'm assuming is also an error at not being able to read the function.
Any help on how I should go about processing my function would be appreciated. Apologies if something is a really obvious solution, but I am not familiar as much with HTTP requests and cloud platforms.
Thanks and cheers.
Node Js Backend Function
const functions = require("firebase-functions");
const admin = require("firebase-admin");
admin.initializeApp();
exports.test = functions.https.onCall(async (data, context) => {
functions.logger.info("Hello logs: ", {structuredData: true});
functions.logger.info( data.token, {structuredData: true});
}
Flutter frontend
1- pubspec.yaml
cloud_functions: ^1.1.2
2 - Code
HttpsCallable callable = FirebaseFunctions.instance.httpsCallable('test');
final HttpsCallableResult results = await callable.call<Map>( {
'token': token,
});

Elasticsearch js Bulk Index TypeError

Background
Recently, I have been working with the Elasticsearch Node.js API to bulk-index a large JSON file. I have successfully parsed the JSON file. Now, I should be able to pass the index-ready array into the Elasticsearch bulk command. However, using console log, it appears as though the array shouldn't be causing any problems.
Bugged Code
The following code is supposed to take an API URL (with a JSON response) and parse it using the Node.js HTTP library. Then using the Elasticsearch Node.js API, it should bulk-index every entry in the JSON array into my Elasticsearch index.
var APIUrl = /* The url to the JSON file on the API providers server. */
var bulk = [];
/*
Used to ready JSON file for indexing
*/
var makebulk = function(ParsedJSONFile, callback) {
var JSONArray = path.to.array; /* The array was nested... */
var action = { index: { _index: 'my_index', _type: 'my_type' } };
for(const item of items) {
var doc = { "id": `${item.id}`, "name": `${item.name}` };
bulk.push(action, doc);
}
callback(bulk);
}
/*
Used to index the output of the makebulk function
*/
var indexall = function(madebulk, callback) {
client.bulk({
maxRetries: 5,
index: "my_index",
type: "my_type",
body: makebulk
}, function(err, resp) {
if (err) {
console.log(err);
} else {
callback(resp.items);
}
});
}
/*
Gets the specified URL, parses the JSON object,
extracts the needed data and indexes into the
specified Elasticsearch index
*/
http.get(APIUrl, function(res) {
var body = '';
res.on('data', function(chunk) {
body += chunk;
});
res.on('end', function() {
var APIURLResponse = JSON.parse(body);
makebulk(APIURLResponse, function(resp) {
console.log("Bulk content prepared");
indexall(resp, function(res) {
console.log(res);
});
console.log("Response: ", resp);
});
});
}).on('error', function(err) {
console.log("Got an error: ", err);
});
When I run node bulk_index.js on my web server, I receive the following error: TypeError: Bulk body should either be an Array of commands/string, or a String. However, this doesn't make any sense because the console.log(res) command (From the indexall function under http.get client request) outputs the following:
Bulk content prepared
Response: [ { index: { _index: 'my_index', _type: 'my_type', _id: '1' } },
{ id: '5', name: 'The Name' }, ... },
... 120690 more items ]
The above console output appears to show the array in the correct format.
Question
What does TypeError: Bulk body should either be an Array of commands/string, or a String indicate is wrong with the array I am passing into the client.bulk function?
Notes
My server is currently running Elasticsearch 6.2.4 and Java Development Kit version 10.0.1. Everything works as far as the Elaticsearch server and even my Elaticsearch mappings (I didn't provide the client.indices.putMapping code, however I can if it is needed). I have spent multiple hours reading over every scrap of documentation I could find regarding this TypeError. I couldn't find much in regards to the error being thrown, so I am not sure where else to look for information regarding this error.
Seems a typo in your code.
var indexall = function(**madebulk**, callback) {
client.bulk({
maxRetries: 5,
index: "my_index",
type: "my_type",
body: **makebulk**
Check the madebulk & makebulk.

MVC controller action not returning JSON

I am unable to get a json response from my controller action. The network shows as a post which is correct as I am posting a file to the server, however, needs a JSON response sent back to my view.
public JsonResult Upload(HttpPostedFileBase file, int id)
{
Homes thishomes= _db.Homes.FirstOrDefault(t => t.Id == id);
FileUploader fileupload = new FileUploader();
fileupload.PostIt(file.InputStream);
return Json(new { success = true, response = "File uploaded.", JsonRequestBehavior.AllowGet });
}
JQUERY using Dropzonejs:
Dropzone.options.DropzoneForm = {
paramName: "file",
maxFilesize: 2000,
maxFiles: 28,
dictMaxFilesExceeded: "Custom max files msg",
init: function () {
this.on("success", function () {
alert("Added file");
})
}
};
Can anyone see an this issue?
Try to write [HttpPost] attribute over your action. Also "The network shows as a post which is correct" if its post then you don't need JsonRequestBehavior.AllowGet
when you are returning Json to your request