I searched but didn't find anything specific to the issue I was having. My apologies if I overlooked it.
I have a scenario where I'm pulling data from an oData source and everything works fine. I have the oData source in a loop where I loop over each of our different companies and pull the data for that company. I'm doing this to reduce the volume of data that is being returned in each oData call.
Everything works fine as long as there is data being returned from the oData call. In the attached photo you can see that the call is being made and data is being returned.
But when the oData Service is called with parameters/filters that return no data I get the following and that is when the VS_NEEDSNEWMETADATA error is thrown. The image below shows what I get when no data is returned.
So the issue isn't that I have invalid metadata due to changes made to the service (adding/removing fields). It's that nothing is being returned so there is no Metadata. Now it's possible that this is an issue with the system that I'm pulling the oData from (SAP S4) and they way that system surfaces the oData call when there is no data??
Either way, trying to figure out a way to handle this within SSIS. I tried to set validate external metadata = false but the package still fails. I could also fix this by excluding those companies in the script but then once data does exists I'd have to remember to update the scripts and redeploy.
Ideas suggestions?
You've hit the nail on the head except your metadata is changing in the no data case because the shape is different - at least from the SSIS engine perspective.
You basically need to double invoke the OData source. First invocation will simply identify is data returned. If that evaluation is true, only then does the Data Flow start running.
How can you evaluate whether your OData will return data? You're likely looking at a Script Task and C#/VB.NET and hopefully the SAP data returns don't vary between this call and next call.
If that's the case, then you need to define an OnError event handler for the Data Flow, mark the System scoped variable Propagate to False and maybe even force a success status to be returned to get the loop to continue.
Related
I'm making a website that gets its info from a RESTapi I've written and hosted myself, have had no data problems thus far.
Now I'm trying a simple retrieve of a json object and I get all the info correctly as shown here in the API. (Visualized & tested in Swagger)
As you can clearly see, it retrieves it the complete object and underlying objects correctly (blurred sensitive info).
Pay attention to the property "AmountOfEggs" though.
Now when i call this api endpoint (exactly the same way) in my site and console.log the result.data, this is the feedback.
Now for some reason I can't recieve AmountOfEggs in my frontend.
I've recreated the entire object, wrote different names, passed different props (Id, NumberBus, etc are passed in this situation with no problem, which are also int (number) types).
I have no idea why this property gets "dropped" in the transfer and filled with/defaults to an empty string.
Any help would be greatly appreciated.
I found where it went wrong and it's due to different factors.
To start, I am making this site using the Vue framework, which has reactive components, which means, data gets adjusted and you get live updated components on your views/pages.
In the page that contained the call, I also wanted to add dynamic styling. The code I used for this is the following:
v-for="seller in retrievedSellers"
:key="seller.Id"
:class="[
'sellerStyle'
, (seller.AmountOfEggs = 0 ? 'grey-out-seller' : ''),
]"
Now two things to note, first of all this might look like backend coding but this is Vue logic on the .vue file which handles the dynamic options that Vue provides so I never thought to look here for the error.
The error of couse is the 'seller.AmountOfEggs = 0' part, using one equal sign (assignment) instead of two (comparison).
So I learned,
Vue doesn't throw errors on assignments in code that is used to generate frontend (where you should NEVER alter your data).
My console.log was the moment the response of the API came in, where apparently it was already assigned a new value (due to my code above) before hitting the console.log.
It still greatly surprises me that Vue handles ^this^ first to make sure your reactive components are up to date before finishing the api call itself.
However, I still don't understand how it said "" instead of 0.
I'm using typescript (javascript strongly-typed) so it definitely wants to contain a number and not an empty string. Even in the declaration of my DTO the property is defined as (and expects) a number.
I have a basic NIFI flow with one GenerateFlowFile processor and one LookupRecord processor with RestLookupService.
I need to do a rest lookup and enrich the incoming flow file with rest response.
I am getting errors that the lookup service is unable to lookup coordinates with the value I am extracting from the incoming flow file.
GenerateFlowFile is configured with simple JSON
LookupRecord is configured to extract the key from the JSON and populate it to the RestLookupService. Also, JsonReader and JsonSetWriter is configured to read the incoming flow file and to write the response back to the flow file
The RestLookupService itself exits with JsonParseException about unexpected character '<'
RestLookupService is configured with my API running in the cloud in which I am trying to use the extracted KEY from the incoming flow file.
The most interesting bug is that when I configure the URL to point for example to mocky.io everything works correctly so it means that the issue itself is tight with the API URL I am using (http://iotosk.ddns.net:3006/devices/${key}/getParsingData). I have tried also removing the $key, using the exact URL, using different URLs..
Of course the API is working OK over postman/curl anything else. I have checked the logs on the container that the API is running on and there is no requests in the logs what means that nifi is failing even before reaching the API. At least on application level...
I am absolutely out of options without any clue how to solve this. And with nifi also google is not helpful.
Does anybody see any issue in the configuration or can point me in some direction what can cause this issue?
After some additional digging. The issue was connected with authentication logic even
before the API receives it and that criples the request and and returned XML as Bryan Bende suggested in the comment.
But definitely better logging in nifi will help to solve this way faster...
I am retrieving some JSON data from an HTTP service and using JsonConverter to parse it.
When I first run the macro everthing is fine, but when I run the macro again I get same the same data as the first time (when I should be getting updated data, which includes a ticker changing every 100 ms at the client side).
If I close Excel and reopen again I get the latest data.
What is the problem here?
Any help appreciated.
The issue is due to the caching mechanism.
To avoid it, use Msxml2.ServerXMLHTTP.6.0 instead of Msxml2.XMLHTTP.6.0.
You could also keep using Msxml2.XMLHTTP.6.0, but you'll have to set the following header:
xhr.SetRequestHeader "If-None-Match", "-"
In this https://developer.gooddata.com/article/data-modeling-api there is a logical data model and its corresponding JSON. However, I can't seem to find out how to extract JSON from a logical data model via the REST API. Is there a way to do this other than using the single load interface (which would be very inefficient)?
For the record, my end goal is to make a tool that extracts that JSON (which would be in dev), then post that to the ldm manager2, and then apply the suggested changes through the returned MaQL to production. Any help is greatly appreciated
Currently this works only for Getting or Updating the entire Project. Anyway you can GET all model definition by simple API call. See the documentation:
http://docs.gooddatadrafts.apiary.io/
There is a GET request which is asynchronous. You can build some logic on the top of that on your end. You can get all models, store per datasets information, but at the end you need to POST the "final version" and all updates will be applied.
Let me know if I can help you with anything!
Regards,
JT
Is there common design pattern for dispatching deleted objects to the requestor (client of the API)?
Challenges we are having:
When object is deleted on the API completely, client will not know
that object is gone and will keep it locally (as API only shows objects changed after the certain date)
If we enable object's property to show that is deleted (ex. "deleted = TRUE") then
eventually number of objects in the API grows and slows down the transfer rate.
Another option we looking into is to have separate Endpoints on the API to show list of deleted objects only (is this the pattern that anyone uses?).
I'm looking for most "RESTful way" to delete local objects.
The way I handle it is a variation on your #1: each item has a last updated field in the database, and if something is deleted, I make an entry in another table of deleted items, and it's updated value is when it was deleted.
The client makes a request asking for "changes since X" which is their own locally stored last updated value...it returns new data, and an array of deleted items. Then on the client I purge those values
Stale data is always a problem with client/server applications. If clients loads some data, then some object is deleted on the server, and then client sends a DELETE request, the RESTFul thing to do would be to return a 404, which indicated "not found". If the client knows that if it sends a DELETE, and gets a 404, the resource was deleting from underneath...
What if you think of your resource not as a list, but rather as a changeset?
Eg. changesets what you have in git or SVN.
This way, there's always a "head" version, and the client always has some version, and the resource is the change between client's last and head.
That way you can apply whatever you've learned by examining/using version control systems.
If you need anything more complex, the science behind is called Operational Transformation (OT) - http://en.wikipedia.org/wiki/Operational_transformation