I am working with an Arduino project that uses Google AppEngine to post data that it collects from various sensors. To give you an idea, here is a link to the project: http://www.iowa-aquaponics.com
I am finding that occasionally the Arduino will post, or goggle app engine receives, data that isn't valid.
This would be a good entry:
http://www.mysite.appspot.com/adacs/arduino?&Temp=80.6&Humidity=82.2&AmbientLDR=16&WaterTemp=79.3&pHValue=5.03&doValue=2.5
Occasionally, a character will drop and i will either get a decimal point that is missing or it will drop an & and what should be a number value like 80.6 turns into a sting like 80.6umidity.
Since the Google AppEngine is seeing this as a string, it goes in the datastore no problem. When I query this data into a JSON table, it will fail because it is looking for a number and it is getting a string.
I was thinking of writing a CRON job that would run each time a new data set was submitted and it would validate the data and delete the record if any of the elements were not valid. I am collecting data every 10 minutes or so, so I am fine with occasionally dropping some records. I would rather have this over not being able to see any data because one element of the JSON table is not valid.
I am curious of some other ideas to handle this situation. If there is a best practice for this, please let me know. Thanks everyone.
Related
First off, let me apologise for any mistakes, as this is my first post.
Now, for the actual issue:
I've coded a PHP script that handles MySQL Queries, including one that fetches all data from a certain table.
I made it so the final result is a JSON object, containing all rows from said table.
I also coded a Flutter class (Dart) that handles the request for the JSON object via the URL, and I got it to successfully log said JSON to the console.
However, when I use that data to show on an actual List of Widgets, this is what appears on screen:
Error Shown
Any ideas on how to fix this?
(For NDA reasons, I am not allowed to show the console logs.).
Any help is appreciated!
I send JSON's to my app via Postman in a list with a type of mapping(CRUD) to send it to my database.
I want my controller to put all this data, from multiple senders, in a list that will send the information to my DB. The problem is that i don't know how to store in the same list the Json and the Mapping, so when my threads do their work to know if that json must be inserted, updated, deleted and so on.
Do you guys have any ideea ?
PS: It is a spring-boot app that need to be able to send 12000 objects ( made from that jsons ) to the db.
I don't see a reason for putting all data in one list and sharing it later, each HTTP request receives own thread.
On decent server you can handle couple thousands of requests/sec which perform simple CRUD operations.
I have an API that produces JSON like this:
)]}',
{
//JSON DATA
}
The //JSON DATA is valid JSON, but the )]}', up top is not.
When I try to GET this data via a Logic App, I get:
BadRequest. Http request failed: the content was not a valid JSON.
So, a few related questions:
1) Can I tell the logic app to return the invalid JSON anyway?
2) How can debug the issue better? I happen to know that the response is invalid, but what if I didn't? Can I see the raw data somewhere?
3) This is all done via the Azure web portal. Are there better tools? Visual Studio?
I should also mention that if I call a route on the same API that returns XML instead of JSON, then the Logic App works fine. So it definitely doesn't like the JSON response in particular.
Thanks!
First of all, please do not post three questions as a single question.
Question 1). The best thing you can do is make the API return a valid JSON object. This is good for million reasons. Here're a few:
it's pretty much a standard (either valid JSON or XML -- yeah, old school way);
therefore, no users of this API (including you) will need to struggle and guess what's going on and why;
your Logic App's step will just work without adding extra complexity;
you will make this world and your karma better.
If API-side changes are not within your reach, I don't think you can do much. If you're lucky and the HTTP action is successful (Status Code 2xx), you can try to use a Query Action with a function that truncates the first characters. It will look something like this (I don't know the exact syntax): #Substring(body('myHttpGet'), 4, length(body('myHttpGet')) - 4) where myHttpGet is the id of the Http Get action.
However, once again, if possible, I strongly recommend fixing up the API which is the root cause of the problem, instead of dealing with garbage response after that.
UPDATE Another thing you can do is wrap the dirty API. For example, you could create a trivial Azure Function that invokes the API you don't directly control, and sanitizes the response for you consumption requirements. This Azure Function function should be easy to call from the Logic App. It costs almost nothing (unless we're talking millions of requests/month). The only drawback here is the increasing latency, which may be not an issue at all -- test it and see whether it adds less than 100ms or so... Oh, and don't forget to file a ticket with the API owner, they make our world a bad place!
Question 2) In Azure Logic App web UI you can Look into the execution details and the error will definitely be there.
Question 3) You're asking for a tool recommendation which is by definition a highly subjective thing and is off-topic on StackOverflow.
TL/DR: The other app is not producing valid JSON.
Meaning, this is not a problem for you to solve. The other app has to return valid JSON if the owner claims it should.
If they cannot or will not produce valid JSON, then the first thing you need to do is inform your management that you will have to spend a lot of extra time accommodating their non-standard format.
I have a following architectural question - my application back-end is written at Java and client side at AngularJS. Right now I need to store the user input on the page in order to be able to share and bookmark my application urls and restore the state by this url.
I'm going to implement the following approach - every time the user interacts with my application via selecting the data and conditions at the page I'll collect all of his input at a complex JSON document and store this document at Elasticsearch. Key of this document from ES I'll send back to client application(AngularJS) and based on this key I'll update the page url. For example the original url looks like:
http://example.com/some-page
based on a key from server I'll update this url to following:
http://example.com/some-page/analysis/234532453455
where 234532453455 is a key of a document in ES.
Every time the users will try to access the following url - http://example.com/some-page/analysis/234532453455 AngularJS application will try to get a saved state by key (234532453455) via Java backend REST endpoint.
Will it work ?
Also, I'm in doubt right now how to prevent the duplication of the documents in ES. Right now I have no experience with ES so don't know what approach from ES can be used out of the box for this purpose.
For example is it a good idea to calculate some hash code of each JSON document and store this hash code as a key of a document.. so before storing the new document I can check the old document by hash code. Also performance is very important to me so please take this into account also.
For me it sounds you try to implement cache.
Yes you can do this but if you will use ES only for this solution then I think you should better look to redis or memcached.
I can't say that ES is bad solution for this but ES has some tricks which you must remember for instance its near realtime search. After you index data they are not immediately available to search it takes few seconds depends on configuration(But you can also call _refresh but I am not sure about performance if you index data very often).
Hash: I dont see reasons to use has I'd better create right id. So if you have types of reports per user than id could be "reporttype_{userid}", because if you will use hash as an ID then each new object will have new id and instead of rewriting you will end up with having many copies of old data for that user. if you go with pattern reporttype_{userid} then each time when user regenerate report with new data you will just override it.
As an option you could add to that option fields userid and expireat for future cleanup, for instance you could have job which will cleanup expired reports, but this is valid only if you go with ES, since in redis and memcached there is option to set expiration when you save data
There is a web service that allows me to go to a URL, with my API-key, and request a page of data. The data is returned as JSON. The JSON is well-formed, I ran it through JSONLint and confirmed its OK.
What I would like to do is retrieve the JSON data from within MS Access (2003 or 2007), if possible, and build a table from that data (first time thru), then append/update the table on subsequent calls to that URL. I would settle for "pre-step" where I retrieve this information via another means. Since I have an API key in the URL, I do not want to do this server-side. I would like to keep it all within Access, run it on my PC at home (its for personal use anyway).
If I have to use another step before the database load then Javascript? But I dont know that very well. I dont even really know what JSON is other than what I have read in Wikipedia. The URL looks similar to:
http://www.SomeWebService.com/MyAPIKey?p=1&s=50
where: p = page number
s = records per page
Access DB is a JavaScript Lib for MS Access, quick page search says they play nicely with JSON, and you can input/output with. boo-ya.
http://www.accessdb.org/
EDIT:
dead url; wayback machine ftw:
http://web.archive.org/web/20131007143335/http://www.accessdb.org/
also sourceforge
http://sourceforge.net/projects/accessdb/