I am using spring data to fetch couchbase documents. I assume the document expiry cannot be set as a configurable value externalized in a property file (#Document(expiry = )) based on answer in this link
However can I define a view in cochbase and have query to return only the documents which have been created in the past 15 minutes? I don't want to save additional information regarding date in my document and I am looking to see if there is any meta data that can be used for this purpose
function (doc, meta) {
if(doc._class=="com.customer.types.CustomerInfo" ){
emit(meta.id, doc);
}
}
If you go into the webconsole and create a development view, you can have a look at what metadata are available using the preview:
emit(meta.id, meta)
This gives us
{
"id": "someKey",
"rev": "3-00007098b90700000000000002000000",
"seq": "3",
"vb": "56",
"expiration": 0,
"flags": 33554432,
"type": "json"
}
So as you can see, there is no explicit metadata that you can use to tell if a doc has been created/updated in the last 15 minutes.
Side note: the id of the doc will always be part of the view's response so emitting it is a bit redundant. More importantly, don't emit the whole doc as a value (if in doubt on what to emit as the second field, just emit null). This value field is stored in the index, so that means that you are basically storing your document's content twice (once in the primary store, once in the view index)!
Related
I have a requirement to convert the json into csv(or a SQL table) or any other flatten structure using Data Flow in Azure Data Factory. I need to take the property names at some hierarchy and values of the child properties at lower of hierrarchy from the source json and add them both as column/row values in csv or any other flatten structure.
Source Data Rules/Constraints :
Parent level data property names will change dynamically (e.g. ABCDataPoints,CementUse, CoalUse, ABCUseIndicators names are dynamic)
The hierarchy always remains same as in below sample json.
I need some help in defining Json path/expression to get the names ABCDataPoints,CementUse, CoalUse, ABCUseIndicators etc. I am able to figure out how to retrieve the values for the properties Value,ValueDate,ValueScore,AsReported.
Source Data Structure :
{
"ABCDataPoints": {
"CementUse": {
"Value": null,
"ValueDate": null,
"ValueScore": null,
"AsReported": [],
"Sources": []
},
"CoalUse": {
"Value": null,
"ValueDate": null,
"AsReported": [],
"Sources": []
}
},
"ABCUseIndicators": {
"EnvironmentalControversies": {
"Value": false,
"ValueDate": "2021-03-06T23:22:49.870Z"
},
"RenewableEnergyUseRatio": {
"Value": null,
"ValueDate": null,
"ValueScore": null
}
},
"XYZDataPoints": {
"AccountingControversiesCount": {
"Value": null,
"ValueDate": null,
"AsReported": [],
"Sources": []
},
"AdvanceNotices": {
"Value": null,
"ValueDate": null,
"Sources": []
}
},
"XYXIndicators": {
"AccountingControversies": {
"Value": false,
"ValueDate": "2021-03-06T23:22:49.870Z"
},
"AntiTakeoverDevicesAboveTwo": {
"Value": 4,
"ValueDate": "2021-03-06T23:22:49.870Z",
"ValueScore": "0.8351945854483925"
}
}
}
Expected Flatten structure
Background:
After having multiple calls with ADF experts at Microsoft(Our workplace have Microsoft/Azure partnership), they concluded this is not possible with out of the box activities provided by ADF as is, neither by Dataflow(need not to use data flow though) nor Flatten feature. Reasons are Dataflow/Flatten only unroll the Array objects and there are no mapping functions available to pick the property names - Custom expression are in internal beta testing and will in PA in near future.
Conclusion/Solution:
We concluded with an agreement based on calls with Microsoft emps ended up to go multiple approaches but both needs the custom code - with out custom code this is not possible by using out of box activities.
Solution-1 : Use some code to flatten as per requirement using a ADF Custom Activity. The downside of this you need to use an external compute(VM/Batch), the options supported are not on-demand. So it is little bit expensive but works best if have continuous stream workloads. This approach also continuously monitor if input sources are of different sizes because the compute needs to be elastic in this case or else you will get out of memory exceptions.
Solution-2 : Still needs to write the custom code - but in a function app.
Create a Copy Activity with source as the files with Json content(preferably storage account).
Use target as Rest Endpoint of function(Not as a function activity because it has 90sec timeout when called from an ADF activity)
The function app will takes Json lines as input and parse and flatten.
If you use the above way so you can scale the number of lines cane be send in each request to function and also scale the parallel requests.
The function will do the flatten as required to one file or multiple files and store in blob storage.
The pipeline will continue from there as needed from there.
One problem with this approach is if any of the range is failed the copy activity will retry but it will run the whole process again.
Trying something very similar, is there any other / native solution to address this?
As mentioned in the response above, has this been GA yet? If yes, any reference documentation / samples would be of great help!
Custom expression are in internal beta testing and will in PA in near future.
I have a document like -
{
"fullUserName": "xxyz",
"userFirstName": "xx",
"userLastName": "xx",
"primaryRole": "xy",
"actualRole": "rrr",
"userId": "abcd1234",
"password":"c28f5c7cb675d41c7763ab0c42d",
"type":"login",
"channels":"*"
}
and view -
function (doc, meta) {
if(doc.userId,doc.password,doc.type){
emit([doc.userId,doc.password,doc.type],doc);
}
}
When the key matches with the docment's property it return the document otherwise it return empty JSON like -
{"total_rows":2,"rows":[
]
}
Now I want to response the error message in JSON format when the key does not match for example-
{
"Error-Code":"400",
"Error-Msg":"user id and password does not match"
}
Is there any way to do so,Please correct if I am moving in the wrong direction.
Thanks in advance.
You shouldn't directly expose the view query result to your users but interpret it instead.
So make a view request, look at the response and do the business logic of checking there. For example:
"if result is empty it can only be because the user is unknown or the password hash didn't match the user, so return a business-specific error message, otherwise carry on with login"
There's no way you can change the behavior and response format of the server, and that doesn't make much sense to do so anyway. This is the API and contract of how you interact with the server. You should add your own business logic in a layer in between.
I am trying to create a view to group on a particular attribute inside an array. However, the below map function is not returning any result.
JSON Document Structure :
{
"jsontype": "survey_response",
"jsoninstance": "xyz",
"jsonlanguage": "en_us",
"jsonuser": "test#test.com",
"jsoncontact": "test#mayoclinic.com",
"pages": [
{
q-placeholder": "q1-p1",
q:localized": "q1-localized-p1",
q-answer-placeholder": "jawaabu121",
q-answer-localized": "localized jawaabu1"
},
{
q-placeholder": "q2-p2",
q:localized": "q2-localized-p2",
q-answer-placeholder": "jawaabu221",
q-answer-localized": "localized jawaabu2"
},
{
"q-placeholder": "q3-p3",
"q:localized": "q3-localized-p3",
"q-answer-placeholder": "jawaabu313",
"q-answer-localized": "localized jawaabu3"
}
]
}
Map Function :
function(doc, meta){
emit(doc.jsoninstance,[doc.pages[0].q-placeholder, doc.pages[0].q-localized,doc.pages[0].q-answer-placeholder,q-answer-localized]);
}
It looks like you made a typo at the end of your emit statement:
doc.pages[0].q-answer-placeholder,q-answer-localized.
Instead q-answer-localized should be changed to doc.pages[0].q-answer-localized.
Further to this it seems that you have defined a field as q-localized in your emit statement, but actually according to the sample document that you posted this should actually be q:localized, I assume that this was a mistake in the snippet of the document and not the emit statement, but if not then will also need amending.
I would imagine errors like this would be flagged up in the view engine's map-reduce error log, in future you should check this log so that you will be able to debug errors like this yourself.
The location of the mapreduce_errors log can be found in the Couchbase documentation
I am getting JSON returned in this format:
{
"status": "success",
"data": {
"debtor": {
"debtor_id": 1301,
"key": value,
"key": value,
"key": value
}
}
}
Somehow, my RESTAdapter needs to provide my debtor model properties from "debtor" section of the JSON.
Currently, I am getting a successful call back from the server, but a console error saying that Ember cannot find a model for "status". I can't find in the Ember Model Guide how to deal with JSON that is nested like this?
So far, I have been able to do a few simple things like extending the RESTSerializer to accept "debtor_id" as the primaryKey, and also remove the pluralization of the GET URL request... but I can't find any clear guide to reach a deeply nested JSON property.
Extending the problem detail for clarity:
I need to somehow alter the default behavior of the Adapter/Serializer, because this JSON convention is being used for many purposes other than my Ember app.
My solution thus far:
With a friend we were able to dissect the "extract API" (thanks #lame_coder for pointing me to it)
we came up with a way to extend the serializer on a case-by-case basis, but not sure if it really an "Ember Approved" solution...
// app/serializers/debtor.js
export default DS.RESTSerializer.extend({
primaryKey: "debtor_id",
extract: function(store, type, payload, id, requestType) {
payload.data.debtor.id = payload.data.debtor.debtor_id;
return payload.data.debtor;
}
});
It seems that even though I was able to change my primaryKey for requesting data, Ember was still trying to use a hard coded ID to identify the correct record (rather than the debtor_id that I had set). So we just overwrote the extract method to force Ember to look for the correct primary key that I wanted.
Again, this works for me currently, but I have yet to see if this change will cause any problems moving forward....
I would still be looking for a different solution that might be more stable/reusable/future-proof/etc, if anyone has any insights?
From description of the problem it looks like that your model definition and JSON structure is not matching. You need to make it exactly same in order to get it mapped correctly by Serializer.
If you decide to change your REST API return statement would be something like, (I am using mock data)
//your Get method on service
public object Get()
{
return new {debtor= new { debtor_id=1301,key1=value1,key2=value2}};
}
The json that ember is expecting needs to look like this:
"debtor": {
"id": 1301,
"key": value,
"key": value,
"key": value
}
It sees the status as a model that it needs to load data for. The next problem is it needs to have "id" in there and not "debtor_id".
If you need to return several objects you would do this:
"debtors": [{
"id": 1301,
"key": value,
"key": value,
"key": value
},{
"id": 1302,
"key": value,
"key": value,
"key": value
}]
Make sense?
I'm building my first API which outputs in JSON, and was wondering: If one of the parameters is empty, is it best to still include that parameter name with an empty value, or not include it at all? For example, if a certain product has batteries it would normally output
"batteries": [
{
"device": "Vehicle",
"number": "4",
"type": "AA",
"included": "Not Included"
},
{
"device": "Remote",
"number": "2",
"type": "AAA",
"included": "Not Included"
}
],
If there are no remote batteries, should I just not include that second section? What if there aren't batteries at all, should I remove the whole battery node?
From the perspective of the json interpreter it won't matter. You should send the JSON however you want the consumer to reconstruct your objects...
Do you want the consumer to have a "Remote" object indicating there are no batteries?
Your example doesn't look like an empty node to me, it looks like meaningful data!
For actually empty nodes it may only matter if you need to keep the serialized object as small as possible (for whatever reason) or if you need to have something else besides JSON look at the serialized object later.
In my personal opinion from an API I like to see all meaningful nodes populated because it gives me an idea of the possibilities of the API.... "Oh, I see, so some of them have remotes and include batteries and this API can tell me that!"
In Javascript, you can treat an absent property in almost the same way you would trean a property set to null:
> a_unset = {}
> a_null = {a: null}
> a_null.a == a_unset.a
true
> a_null.a ? 1 : 0
0
> a_unset.a ? 1 : 0
0
Therefore in JSON, which is based on Javascript and most often consumed by Javascript code, it is customary to omit empty values.
But this is not a hard rule. JSON does provide the null value, so if you think your client code or target users would need to know that a property is there but unset, null might be a good choice. Otherwise just omit it, you will save space.