How do I save a historical (past) dataset in Foundry - palantir-foundry

I want to save a version of my dataset called (order_clean). But not the current version. The version I want to save is from the past. I understand this is a historical transaction that still exists within my retention policy window. How would I be able to do this?

This can be done via an API call and hitting a specific endpoint.
This answer assumes a working knowledge around authorization tokens and curl requests:
Obtain the desired dataset’s RID (Example in screenshot below), as well as the transaction ID of the transaction corresponding to the version of the dataset you want.
[
Create a new branch in your specific dataset by running a curl request populated as follows:
curl -X POST -H "Content-type: application/json" -H "Authorization: Bearer YOUR_AUTH_TOKEN" "STACK_URL/foundry-catalog/api/catalog/datasets/YOUR_DATASET_RID/branchesUnrestricted2/NEW_BRANCH_NAME" -d '{"parentRef": "YOUR_TRANSACTION_ID", "parentBranchId": "master"}'
Replace YOUR_AUTH_TOKEN, STACK_URL, YOUR_DATASET_RID, NEW_BRANCH_NAME and YOUR_TRANSACTION_ID with appropriate values. Short lived authentication token can be generated following these instructions.
Save the branch into a new dataset with a transform in a Code Repo, specifying the Input with the path to the dataset as well as the new branch’s name from #2.

Related

etherscan or ethplorer doesn't see test transaction

I want to get address transaction list and as I can see that is possible only if dancing with a tambourine. So I decided to use some third party service(which I think shame for ethereum developers but we have that what we have). But those service can't see transactions or balance on the address even balance showing via command
curl -X POST http://localhost:8545 -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"eth_getBalance","params":"0xcd88f3430a2e01e61554e655719e0f92d3e1ba37", "pending"],"id":1}'
{"jsonrpc":"2.0","id":1,"result":"0xde0b6b3a7640000"}
(sent via goerli)
but when I perform request like
https://api.etherscan.io/api?module=account&action=txlist&address=0xcd88f3430a2e01e61554e655719e0f92d3e1ba37&sort=asc&apikey=...
I get
message "No transactions found"
Or on
https://kovan-api.ethplorer.io/getAddressTransactions/0xcd88f3430a2e01e61554e655719e0f92d3e1ba37?apiKey=...
[]
Can you please explain to me what happen and how easily to extract transaction list for the address.
Found it, need to use different domain for goerli transactions
https://api-goerli.etherscan.io/

Is there a possibility to scrape an entire job offering database?

I am trying to automatically retrieve all active job listings from Deutsche Telekom from the following website: https://telekom.jobs/global-careers
At the moment, there are 1,813 job offerings.
There seems to be an API that returns JSON code (https://telekom.jobs/globaljobboard_api/v3/search/), however the results are limited to 10 (SearchResultCount).
Is there any possibility to parse some parameters to the API so it returns a JSON file with all 1,800 current job offerings?
Thanks
I have tried to add some parameters to the URL, however I was not succesfull with it
https://telekom.jobs/globaljobboard_api/v3/search/%7B%22JobadID%22:%22%22,%22LanguageCode%22:%222%22,%22SearchParameters%22:%7B%22FirstItem%22:1,%22CountItem%22:10000,%22Sort%22:[%7B%22Criterion%22:%22FavoriteJobIndicator%22,%22Direction%22:%22DESC%22%7D],%22MatchedObjectDescriptor%22:[%22ID%22,%22PositionTitle%22,%22ParentOrganization%22,%22ParentOrganizationName%22,%22PositionURI%22,%22PositionLocation.CountryName%22,%22PositionLocation.CountrySubDivisionName%22,%22PositionLocation.CityName%22,%22PositionLocation.Longitude%22,%22PositionLocation.Latitude%22,%22PositionBenefit.Code%22,%22PositionBenefit.Name%22,%22FavoriteJobIndicator%22,%22FavoriteJobIndicatorName%22]%7D,%22SearchCriteria%22:[%7B%22CriterionName%22:%22PositionLocation.Latitude%22,%22CriterionValue%22:[%2250.73743%22]%7D,%7B%22CriterionName%22:%22PositionLocation.Longitude%22,%22CriterionValue%22:[%227.098206800000071%22]%7D,%7B%22CriterionName%22:%22PositionLocation.Distance%22,%22CriterionValue%22:[%229.013064227023515%22]%7D,%7B%22CriterionName%22:%22PositionLocation.CountryCode%22,%22CriterionValue%22:[%22DE%22]%7D,%7B%22CriterionName%22:%22PositionLocation.AreaCode%22,%22CriterionValue%22:[%22DE%22]%7D]
My expected results are a JSON file of the whole database, not only the ten most recent entries.
You need to do a POST with the data {"SearchParameters":{"CountItem":1813}}.
Here an example using curl:
curl 'https://telekom.jobs/globaljobboard_api/v3/search/' -H 'Accept: application/json' --compressed -H 'Content-Type: application/json' --data '{"SearchParameters":{"CountItem":1813}}'
Notice this is particular to this API, this will not necessarily work for others.

salt state to run JSON API

I am quite new to SALT, I need to query a JSON API through slat state, may I know how best I should do. I am trying below is of no help though.
Salt ‘*’ cmd.run ‘curl --insecure -u "username:password" -X POST -H "Content-Type: application/json" -d '{"action":"RouterName","method":"methodname","data":[{"limit":100000, "uid" : "Query UID"}],"tid":1}' https:///zport/dmd/device_router’
Can I use any state file to query JSON API of my application through SALT.
There is a difference between module and state.
Modules: Module runs the command/function every time and does not care if the task needs to be run or not.
State: On the other hand, the state cares about the status of the task that needs to be executed. Based on the result, it decides if the task should be run or the system is in the desired state that the task wants it to be.
More about this issue this.
To somehow answer your question, you cannot be sure what is the state of a server that you are trying to curl from. That means, that you need to use a module to get the current output and decide upon that.

SVF POST job file conversion failure

A friend and I are following the Autodesk Viewer tutorial here:
https://developer.autodesk.com/en/docs/viewer/v2/tutorials/basic-viewer/
We've both completed all the previous steps except for the very last of using the APIs POST job endpoint to send a base64 encoded urn off to the Model Derivative API for conversion to a SVF file so that Autodesk's Viewer may use it.
This is what I'm using to send the POST job request:
curl -X 'POST' -H 'Authorization: Bearer cywr9WWu8kml3rQIVnPDohFDsoRl' -H
'Content-Type: application/json' -v 'https://developer.api.autodesk.com/modelderivative/v2/designdata/job'
-d '{"input": {"urn": "dXJuOmFkc2sub2JqZWN0czpvcy5vYmplY3Q6bXl0ZXN0YnVja2V0dGVzdDIvZ2lyaV9zaXJfY29weTEucnZ0"},
"output": {"formats": [{"type": "svf","views": ["2d","3d"]}]}}'
and this is the APIs response:
{"type":"manifest","hasThumbnail":"false","status":"failed","progress":"complete",
"region":"US","urn":"dXJuOmFkc2sub2JqZWN0czpvcy5vYmplY3Q6bXl0ZXN0YnVja2V0dGVzdDIvZ2lyaV9zaXJfY29weTEucnZ0",
"derivatives":[{"name":"giri_sir_copy1.rvt","hasThumbnail":"false","status":"failed",
"progress":"complete","messages":[{"type":"error","code":"Revit-InternalError",
"message":"<message>We have encountered some issues while preparing the file for viewing.
Please contact support for assistance.</message>"},{"type":"error",
"message":"Unrecoverable exit code from extractor: -1073741831",
"code":"TranslationWorker-InternalFailure"}],"outputType":"svf"}]}
Any insights is greatly appreciated.
Summarizing the comments on the question: here is the link to the sample running and source code (postJob function).
Please note this sample is storing the file directly under the developer account (OSS), not under the user account (Data Management), it's an important difference! The first uses 2-legged token, the second uses 3-legged token. Data Management contains some abstractions/metadata to organize the files, like hubs, projects and folders, but ultimately stores on OSS. Depending on the way it was stored, you need different tokens/permissions to read/write it.

Box.com API Get Information About a Folder - Stops working

Simple and quick question :-)
My application (https://www.box.com/services/rubibox) stops working. I use this request for retrieve items in folder http://developers.box.com/docs/#folders-get-information-about-a-folder and this request stops working yesterday. Is maintance on Box.com Apiv2 or request was removed from Apiv2?
PS: I know that is possible to use http://developers.box.com/docs/#folders-retrieve-a-folders-items but this is not good for me because not contain current folder informations.
Thanks
Due to a recent API change a call to /folders no longer includes the item_collection by default. You must now ask for the item_collection in the request:
curl https://api.box.com/2.0/folders/FOLDER_ID?fields=item_collection
-H "Authorization: Bearer ACCESS_TOKEN"