Related
I'm trying to update a Contentful entry using Postman.
What I did:
In Contentful space, I created a test post to play with.
Went to Settings - API Keys - Content management tokens and generated a Personal access token
Created a GET request in Postman, passing space ID, master environment, and ID of the test post:
https://cdn.contentful.com/spaces/{spaceID}i/environments/master/entries?sys.id={postID}
I also sent authorisation header with content delivery token.
The GET request goes successfully and I'm able to copy the JSON object-response.
{
"sys": {
"type": "Array"
},
"total": 1,
"skip": 0,
"limit": 100,
"items": [
{
"metadata": {
"tags": []
},
"sys": {
"space": {
"sys": {
"type": "Link",
"linkType": "Space",
"id": "d9r4mg123x4v"
}
},
"id": "2Fwow39hxxx1bvMkjpsyV9",
"type": "Entry",
"createdAt": "2021-11-10T14:00:11.935Z",
"updatedAt": "2021-11-10T14:06:51.393Z",
"environment": {
"sys": {
"id": "master",
"type": "Link",
"linkType": "Environment"
}
},
"revision": 3,
"contentType": {
"sys": {
"type": "Link",
"linkType": "ContentType",
"id": "hotelInfo"
}
},
"locale": "en-US"
},
"fields": {
"name": "Test entry",
"slug": "test-entry",
"address": "Lviv",
"cityName": "Lviv",
"phone": "+380931231212",
"coordinates": {
"lon": -115.302,
"lat": 36.18709
},
"dog": "100",
"cat": "100",
"delivery": "100",
"photo": [
{
"sys": {
"type": "Link",
"linkType": "Asset",
"id": "2hSnYhQDJzU99NvlsYdk3k"
}
}
],
"additionalInfo": {
"data": {},
"content": [
{
"data": {},
"content": [
{
"data": {},
"marks": [],
"value": "Test",
"nodeType": "text"
}
],
"nodeType": "paragraph"
}
],
"nodeType": "document"
},
"featuredHotel": true,
"phoneClicks": 1
}
}
],
"includes": {
"Asset": [
{
"metadata": {
"tags": []
},
"sys": {
"space": {
"sys": {
"type": "Link",
"linkType": "Space",
"id": "d9r4mg123x4v"
}
},
"id": "2hSnYhQDJzU99NvlsYdk3k",
"type": "Asset",
"createdAt": "2021-11-10T13:59:59.954Z",
"updatedAt": "2021-11-10T13:59:59.954Z",
"environment": {
"sys": {
"id": "master",
"type": "Link",
"linkType": "Environment"
}
},
"revision": 1,
"locale": "en-US"
},
"fields": {
"title": "JS",
"description": "Lorem Ipsum",
"file": {
"url": "//images.ctfassets.net/d9r4mg123x4v/2hSnYhQDJzU99NvlsYdk3k/6fbabc7be7f4b28dc8b7deadd9892205/JS.png",
"details": {
"size": 23078,
"image": {
"width": 1024,
"height": 1024
}
},
"fileName": "JS.png",
"contentType": "image/png"
}
}
}
]
}
}
Now I want to create PUT request to send the updated JSON to the Contentful.
I paste the JSON I got as a response from GET request.
I change one of the values:
"name": "Test entry"
to
"name": "Test entry 123"
I send PUT request to https://api.contentful.com/spaces/{spaceID}/environments/master/entries/{postID}
The Authorisation header contains the Personal access token I generated before.
The X-Contentful-Version header contains the version of the post, can be found in post details
When I send this request, I get JSON response with an empty "fields": {}
{
"metadata": {
"tags": []
},
"sys": {
"space": {
"sys": {
"type": "Link",
"linkType": "Space",
"id": "d9r4mg123x4v"
}
},
"id": "2Fwow39hxxx1bvMkjpsyV9",
"type": "Entry",
"createdAt": "2021-11-10T13:57:10.882Z",
"updatedAt": "2021-11-11T10:58:39.480Z",
"environment": {
"sys": {
"id": "master",
"type": "Link",
"linkType": "Environment"
}
},
"publishedVersion": 13,
"publishedAt": "2021-11-10T14:06:51.393Z",
"firstPublishedAt": "2021-11-10T14:00:11.935Z",
"createdBy": {
"sys": {
"type": "Link",
"linkType": "User",
"id": "4123123123zOn3MkhuVB"
}
},
"updatedBy": {
"sys": {
"type": "Link",
"linkType": "User",
"id": "4123123123zOn3MkhuVB"
}
},
"publishedCounter": 3,
"version": 23,
"publishedBy": {
"sys": {
"type": "Link",
"linkType": "User",
"id": "4123123123zOn3MkhuVB"
}
},
"contentType": {
"sys": {
"type": "Link",
"linkType": "ContentType",
"id": "hotelInfo"
}
}
},
"fields": {}
}
And in Contentful Admin area, all the fields of the post become empty.
Contentful documentation says:
Contentful doesn't merge changes made to content, so when updating
content, you need to send the entire body of an entry. If you update
content with a subset of properties, you will lose all existing
properties not included in that update.
You should always update resources in the following order:
Fetch current resource.
Make changes to the current resource.
Update the resource by passing the changed resource along with current version number.
This way no unseen changes are overridden and unexpected conflicts are
unlikely to occur.
Note: You can't update any of the sys property fields, including
sys.id.
...so, I guess, I'm doing everything right - taking the post, editing data and sending updated post back.
I tried editing my JSON data to send it without sys fields, but no luck.
I'm stuck, anyone has any ideas what should I proceed with?
Thanks to #whitep4nth3r I was able to solve the problem.
I needed to GET data from the same source I'm trying to PUT it to.
The Authorisation header needed to be replaced with the Personal access token used for PUT request.
I'm working on a sync server that keeps the Office 365 events and my calendering app in sync
I subscribed to a push notification for user's event calendar that has the following custom single extended property.
Sample subscription request payload
{
"resource": "/me/events?$expand=singleValueExtendedProperties($filter=id%20eq%20'\''String%20{66f5a359-4659-4830-9070-00047ec6ac6e}%20Name%20CUSTOM'\'')",
"notificationUrl": "https://serverurl.xxxx.com/dev/notification",
"changeType": "updated,created",
"clientState": "123456-fad3-4191-9a66-123456789",
"expirationDateTime": "2018-03-17T11:00:00.0000000Z"
}
Then I do a single update of calendar event that has the above custom single extended property. On update, the sync server receives multiple notifications (2-3 notifications) for just updating the title of the event.
Notification 1:
{
"value": [{
"subscriptionId": "9645fed8-dc53-4955-b3a1-2d2b9ac5728f",
"subscriptionExpirationDateTime": "2018-03-17T11:00:00+00:00",
"changeType": "updated",
"resource": "<masked>",
"resourceData": {
"#odata.type": "#Microsoft.Graph.Event",
"#odata.id": "<masked>",
"#odata.etag": "W/\"+JK7q7qG1U+aBgDQ2ypjIQAAVnP6yQ==\"",
"id": "AAMkAGJmZjA5NzI0LWM5NTgtNGRhYy04MDExLTJmZTY4ZTJkNmVlYQBGAAAAAAC-F40EZzuzSqgA9N8VQW0zBwD4kruruobVT5oGANDbKmMhAAAAAAENAAD4kruruobVT5oGANDbKmMhAABV-OaMAAA=",
"createdDateTime": "2018-03-13T13:30:54.740501Z",
"lastModifiedDateTime": "2018-03-13T13:42:33.7107864Z",
"changeKey": "+JK7q7qG1U+aBgDQ2ypjIQAAVnP6yQ==",
"categories": [],
"originalStartTimeZone": "Pacific Standard Time",
"originalEndTimeZone": "Pacific Standard Time",
"iCalUId": "040000008200E00074C5B7101A82E0080000000006474883CFBAD30100000000000000001000000021D57F3032480543BE96AE56D8816FDF",
"reminderMinutesBeforeStart": 15,
"isReminderOn": true,
"hasAttachments": false,
"subject": "Microsoft DEMO - 2 UPDATED",
"bodyPreview": "Let's get together!",
"importance": "Normal",
"sensitivity": "Normal",
"isAllDay": false,
"isCancelled": false,
"isOrganizer": true,
"responseRequested": true,
"seriesMasterId": null,
"showAs": "Busy",
"type": "SingleInstance",
"webLink": "<masked>",
"onlineMeetingUrl": null,
"responseStatus": {
"Response": "Organizer",
"Time": "0001-01-01T00:00:00Z"
},
"body": {
"ContentType": "HTML",
"Content": "<html>\r\n<head>\r\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=utf-8\">\r\n<meta content=\"text/html; charset=us-ascii\">\r\n</head>\r\n<body>\r\nLet's get together!\r\n</body>\r\n</html>\r\n"
},
"start": {
"DateTime": "2018-03-29T01:00:00Z",
"TimeZone": "tzone://Microsoft/Utc"
},
"end": {
"DateTime": "2018-03-29T06:00:00Z",
"TimeZone": "tzone://Microsoft/Utc"
},
"location": {
"DisplayName": "",
"LocationType": "Default",
"UniqueIdType": "Unknown",
"Address": {
"Type": "Unknown"
},
"Coordinates": {}
},
"locations": [],
"recurrence": null,
"attendees": [{
"Type": "Required",
"Status": {
"Response": "None",
"Time": "0001-01-01T00:00:00Z"
},
"EmailAddress": {
"Name": "<masked>",
"Address": "<masked>"
}
}],
"organizer": {
"EmailAddress": {
"Name": "<masked>",
"Address": "<masked>"
}
}
},
"clientState": "123456-fad3-4191-9a66-123456789"
}]
}
Notification 2:
{
"value": [{
"subscriptionId": "9645fed8-dc53-4955-b3a1-2d2b9ac5728f",
"subscriptionExpirationDateTime": "2018-03-17T11:00:00+00:00",
"changeType": "updated",
"resource": "<masked>",
"resourceData": {
"#odata.type": "#Microsoft.Graph.Event",
"#odata.id": "<masked>",
"#odata.etag": "W/\"+JK7q7qG1U+aBgDQ2ypjIQAAVnP6yw==\"",
"id": "AAMkAGJmZjA5NzI0LWM5NTgtNGRhYy04MDExLTJmZTY4ZTJkNmVlYQBGAAAAAAC-F40EZzuzSqgA9N8VQW0zBwD4kruruobVT5oGANDbKmMhAAAAAAENAAD4kruruobVT5oGANDbKmMhAABV-OaMAAA=",
"createdDateTime": "2018-03-13T13:30:54.740501Z",
"lastModifiedDateTime": "2018-03-13T13:42:33.869926Z",
"changeKey": "+JK7q7qG1U+aBgDQ2ypjIQAAVnP6yw==",
"categories": [],
"originalStartTimeZone": "Pacific Standard Time",
"originalEndTimeZone": "Pacific Standard Time",
"iCalUId": "<masked>",
"reminderMinutesBeforeStart": 15,
"isReminderOn": true,
"hasAttachments": false,
"subject": "Microsoft DEMO - 2 UPDATED",
"bodyPreview": "Let's get together!",
"importance": "Normal",
"sensitivity": "Normal",
"isAllDay": false,
"isCancelled": false,
"isOrganizer": true,
"responseRequested": true,
"seriesMasterId": null,
"showAs": "Busy",
"type": "SingleInstance",
"webLink": "<masked>",
"onlineMeetingUrl": null,
"responseStatus": {
"Response": "Organizer",
"Time": "0001-01-01T00:00:00Z"
},
"body": {
"ContentType": "HTML",
"Content": "<html>\r\n<head>\r\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=utf-8\">\r\n<meta content=\"text/html; charset=us-ascii\">\r\n</head>\r\n<body>\r\nLet's get together!\r\n</body>\r\n</html>\r\n"
},
"start": {
"DateTime": "2018-03-29T01:00:00Z",
"TimeZone": "tzone://Microsoft/Utc"
},
"end": {
"DateTime": "2018-03-29T06:00:00Z",
"TimeZone": "tzone://Microsoft/Utc"
},
"location": {
"DisplayName": "",
"LocationType": "Default",
"UniqueIdType": "Unknown",
"Address": {
"Type": "Unknown"
},
"Coordinates": {}
},
"locations": [],
"recurrence": null,
"attendees": [{
"Type": "Required",
"Status": {
"Response": "None",
"Time": "0001-01-01T00:00:00Z"
},
"EmailAddress": {
"Name": "<masked>",
"Address": "<masked>"
}
}],
"organizer": {
"EmailAddress": {
"Name": "<masked>",
"Address": "<masked>"
}
}
},
"clientState": "123456-fad3-4191-9a66-123456789"
}]
}
Notification 3:
{
"value": [{
"subscriptionId": "9645fed8-dc53-4955-b3a1-2d2b9ac5728f",
"subscriptionExpirationDateTime": "2018-03-17T11:00:00+00:00",
"changeType": "updated",
"resource": "<masked>",
"resourceData": {
"#odata.type": "#Microsoft.Graph.Event",
"#odata.id": "<masked>",
"#odata.etag": "W/\"+JK7q7qG1U+aBgDQ2ypjIQAAVnP6yw==\"",
"id": "AAMkAGJmZjA5NzI0LWM5NTgtNGRhYy04MDExLTJmZTY4ZTJkNmVlYQBGAAAAAAC-F40EZzuzSqgA9N8VQW0zBwD4kruruobVT5oGANDbKmMhAAAAAAENAAD4kruruobVT5oGANDbKmMhAABV-OaMAAA=",
"createdDateTime": "2018-03-13T13:30:54.740501Z",
"lastModifiedDateTime": "2018-03-13T13:42:33.869926Z",
"changeKey": "+JK7q7qG1U+aBgDQ2ypjIQAAVnP6yw==",
"categories": [],
"originalStartTimeZone": "Pacific Standard Time",
"originalEndTimeZone": "Pacific Standard Time",
"iCalUId": "<masked>",
"reminderMinutesBeforeStart": 15,
"isReminderOn": true,
"hasAttachments": false,
"subject": "Microsoft DEMO - 2 UPDATED",
"bodyPreview": "Let's get together!",
"importance": "Normal",
"sensitivity": "Normal",
"isAllDay": false,
"isCancelled": false,
"isOrganizer": true,
"responseRequested": true,
"seriesMasterId": null,
"showAs": "Busy",
"type": "SingleInstance",
"webLink": "<masked>",
"onlineMeetingUrl": null,
"responseStatus": {
"Response": "Organizer",
"Time": "0001-01-01T00:00:00Z"
},
"body": {
"ContentType": "HTML",
"Content": "<html>\r\n<head>\r\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=utf-8\">\r\n<meta content=\"text/html; charset=us-ascii\">\r\n</head>\r\n<body>\r\nLet's get together!\r\n</body>\r\n</html>\r\n"
},
"start": {
"DateTime": "2018-03-29T01:00:00Z",
"TimeZone": "tzone://Microsoft/Utc"
},
"end": {
"DateTime": "2018-03-29T06:00:00Z",
"TimeZone": "tzone://Microsoft/Utc"
},
"location": {
"DisplayName": "",
"LocationType": "Default",
"UniqueIdType": "Unknown",
"Address": {
"Type": "Unknown"
},
"Coordinates": {}
},
"locations": [],
"recurrence": null,
"attendees": [{
"Type": "Required",
"Status": {
"Response": "None",
"Time": "0001-01-01T00:00:00Z"
},
"EmailAddress": {
"Name": "<masked>",
"Address": "<masked>"
}
}],
"organizer": {
"EmailAddress": {
"Name": "<masked>",
"Address": "<masked>"
}
}
},
"clientState": "123456-fad3-4191-9a66-123456789"
}]
}
I expected just one notification for every update (In my case I just did an update on event title), but the server is receiving multiple notifications with just #odata.etag different in the notifications. Is this expected?
When making a change to any item, the server may need to perform several modifications on the item, as well as other related items behind the scenes. Each of these modifications would result in a notification. There also could be other entities that are listening to changes, and reacting to them, potentially causing further changes and hence more notifications on the item. So in short, there is no guarantee regarding the number of notifications that you may receive as a result of a single API call. That is specially true for high-level calendaring operations, which often involve complex workflows on the server.
In this case you can see that in fact you have received three different change keys, indicating that the item has been modified multiple times.
I created a task that talks to a REST API to retreive the values for 2 picklists.
Filling the first dropdown box works fine, when using just the jsonpath.
Based on the first picklist I'd like to retreive values of the second list.
I've tried some variations and I'm trying something like this:
The json which I receive in the first rest call is similar to:
{
"id": "45",
"href": "https://selfservice/api/configurations/45/",
"name": "Type",
"description": "",
"version": "0.0.4",
"attributes": [
{
"value": {
"sdk-object": {
"type": "ConfigurationElement",
"id": "56"
}
},
"type": "ConfigurationElement",
"name": "win"
},
{
"value": {
"sdk-object": {
"type": "ConfigurationElement",
"id": "57"
}
},
"type": "ConfigurationElement",
"name": "lin"
}
]
}
I try to show the attributes name in the list and need the id of the attribute in the second picklist.
I created the following datasourcebindings in the task.json. Of course, the targets exist in the task.
task.json:
{
"id": "GUID",
"name": "Spinup",
"friendlyName": "Create environment",
"description": "Starts Workflow to create an environment. ___ The following placeholders are created deplending on the template and can be used in the rest of the release:**XLDEnvironment** and **testAgentHostname**",
"category": "Deploy",
"author": "***",
"version": {
"Major": 0,
"Minor": 0,
"Patch": 33
},
"minimumAgentVersion": "1.95.3",
"inputs": [
{
"label": "Endpoint",
"name": "connectedServiceName",
"required": true,
"type": "connectedService:server",
"helpMarkDown": "endpoint to connect to."
},
{
"name": "stage",
"type": "string",
"label": "Stage",
"defaultValue": "$(Release.EnvironmentName)",
"required": true,
"helpMarkDown": "Stage of the release, default value is based on the pipeline."
},
{
"name": "releaseName",
"type": "string",
"label": "Environment name",
"defaultValue": "$(Release.ReleaseName)",
"required": true,
"helpMarkDown": "Name of the environment that will be created."
},
{
"name": "owner",
"type": "string",
"label": "Owner of the environment",
"defaultValue": "***",
"required": true,
"helpMarkDown": "It is common to use the initials of the product owner in this field. There has to be a valid AD user owner of each environment. This can also be set by using a variable through the pipeline."
},
{
"name": "group",
"type": "string",
"label": "Group of the owner",
"defaultValue": "",
"required": true,
"helpMarkDown": "group that owns the environment. groups can be requested by the infrastructure department. Example: ****"
},
{
"name": "platform",
"type": "pickList",
"label": "Platform (OS)",
"defaultValue": "",
"required": true,
"helpMarkDown": "Choose the type of the target platform."
},
{
"name": "size",
"type": "pickList",
"label": "Environment Template",
"defaultValue": "",
"required": true,
"helpMarkDown": "Size of the environment to create."
}
],
"dataSourceBindings": [
{
"dataSourceName": "GetCEPlatformType",
"endpointId": "$(connectedServiceName)",
"target": "platform",
"selector": "jsonpath:$.attributes[*].name",
"keySelector": "jsonpath:$.attributes[*].value.sdk-object.id"
},
{
"dataSourceName": "GetCEPlatformSize",
"endpointId": "$(connectedServiceName)",
"target": "size",
"parameters": {
"platformTypeId": "$(platform)"
}
}
],
"instanceNameFormat": "Spin up $(size) $(platform) environment",
"execution": {
"PowerShell3": {
"target": "$(currentDirectory)\\task.ps1",
"argumentFormat": "",
"workingDirectory": "$(currentDirectory)"
}
}
}
vss-extension.json:
{
"manifestVersion": 1,
"id": "*****",
"name": "Release Tasks",
"version": "1.0.1",
"publisher": "***",
"targets": [
{
"id": "Microsoft.VisualStudio.Services"
}
],
"description": "Tools to contact ***. Includes a task to spin up a platform and a task to remove the platform.",
"categories": [
"Build and release"
],
"icons": {
"default": "images/extension-icon.png"
},
"files": [
{
"path": "RemoveEnvironment"
},
{
"path": "SpinUpEnvironment"
}
],
"contributions": [
{
"id": "******",
"type": "ms.vss-distributed-task.task",
"targets": [
"ms.vss-distributed-task.tasks"
],
"properties": {
"name": "RemoveEnvironment"
}
},
{
"id": "*********",
"type": "ms.vss-distributed-task.task",
"targets": [
"ms.vss-distributed-task.tasks"
],
"properties": {
"name": "SpinUpEnvironment"
}
},
{
"description": "Service Endpoint type for all connections",
"id": "endpoint-type",
"properties": {
"authenticationSchemes": [
{
"inputDescriptors": [
{
"description": "Username",
"id": "username",
"inputMode": "textbox",
"name": "Username",
"validation": {
"dataType": "string",
"isRequired": true
}
},
{
"description": "Password",
"id": "password",
"inputMode": "passwordbox",
"isConfidential": true,
"name": "Password",
"validation": {
"dataType": "string",
"isRequired": false
}
}
],
"type": "ms.vss-endpoint.endpoint-auth-scheme-basic"
}
],
"dataSources": [
{
"endpointUrl": "api/configurations/*****/",
"name": "GetCEPlatformType",
"resultSelector": "jsonpath:$.attributes[*].name"
},
{
"endpointUrl": "api/configurations/$(platformTypeId)/",
"name": "GetCEPlatformSize",
"resultSelector": "jsonpath:$.attributes[*].name"
}
],
"displayName": "*****",
"helpMarkDown": "Enter the url and credentials to connect to the endpoint.",
"name": "server",
"url": {
"displayName": "Server URL",
"helpText": "Url for the server to connect to."
}
},
"targets": [
"ms.vss-endpoint.endpoint-types"
],
"type": "ms.vss-endpoint.service-endpoint-type"
}
]
}
Sadly I can't get this to work. The datasources exist in the endpoint, but I'm not sure what to do with the resultSelector.
Does anyone have an idea on how to get this to work? The documentation on this isn't too good.
I'm not familiar with mustache, but in the mustache test tool this seems to work.
Thoughts are appreciated!
Try with this in the first data source:
{
"dataSourceName": "dsList1",
"endpointId": "$(connectedServiceName)",
"target": "list1",
"selector": "jsonpath:$.attributes[*].name",
"keySelector": "jsonpath:$.attributes[*].value.sdk-object.id"
}
Update:
You were trying to use "platformTypeId" in vss-extension.json file while it was defined in task.json, this cause the error message you mentioned in the comment.
By the way, I just noticed that you are using "dataSourceBindings", then you cannot use "selector" and "KeySelector" to parse the result since they are used for "sourceDefinitions".
To achieve the feature you want with "dataSourceBindings", you can definition the endpoint url in the task.json directly instead of defining datasource in endpoint contribution.
So you can move the datasources section in the vss-extension.json file first, and then in the task.json file, change to following:
"dataSourceBindings": [
{
"endpointId": "$(connectedServiceName)",
"endpointURL": "api/configurations/*****/",
"target": "platform",
"resultselector": "jsonpath:$.attributes[*]",
"resultTemplate": "{ \"Value\" : \"{{{value.id}}}\", \"DisplayValue\" : \"{{{name}}}\" }"
},
{
"endpointId": "$(connectedServiceName)",
"endpointURL": "api/configurations/$(platformTypeId)/",
"target": "size"
}
],
I have been working on this for a couple days and cannot get past this error. I have 2 activities in this pipeline. The first activity copies data from an ODBC connection to an Azure database, which is successful. The 2nd activity transfers the data from Azure table to another Azure table and keeps failing.
The error message is:
Copy activity met invalid parameters: 'UnknownParameterName', Detailed message: An item with the same key has already been added..
I do not see any invalid parameters or unknown parameter names. I have rewritten this multiple times using their add activity code template and by myself, but do not receive any errors when deploying on when it is running. Below is the JSON pipeline code.
Only the 2nd activity is receiving an error.
Thanks.
Source Data set
{
"name": "AnalyticsDB-SHIPUPS_06shp-01src_AZ-915PM",
"properties": {
"structure": [
{
"name": "UPSD_BOL",
"type": "String"
},
{
"name": "UPSD_ORDN",
"type": "String"
}
],
"published": false,
"type": "AzureSqlTable",
"linkedServiceName": "Source-SQLAzure",
"typeProperties": {},
"availability": {
"frequency": "Day",
"interval": 1,
"offset": "04:15:00"
},
"external": true,
"policy": {}
}
}
Destination Data set
{
"name": "AnalyticsDB-SHIPUPS_06shp-02dst_AZ-915PM",
"properties": {
"structure": [
{
"name": "SHIP_SYS_TRACK_NUM",
"type": "String"
},
{
"name": "SHIP_TRACK_NUM",
"type": "String"
}
],
"published": false,
"type": "AzureSqlTable",
"linkedServiceName": "Destination-Azure-AnalyticsDB",
"typeProperties": {
"tableName": "[olcm].[SHIP_Tracking]"
},
"availability": {
"frequency": "Day",
"interval": 1,
"offset": "04:15:00"
},
"external": false,
"policy": {}
}
}
Pipeline
{
"name": "SHIPUPS_FC_COPY-915PM",
"properties": {
"description": "copy shipments ",
"activities": [
{
"type": "Copy",
"typeProperties": {
"source": {
"type": "RelationalSource",
"query": "$$Text.Format('SELECT COMPANY, UPSD_ORDN, UPSD_BOL FROM \"orupsd - UPS interface Dtl\" WHERE COMPANY = \\'01\\'', WindowStart, WindowEnd)"
},
"sink": {
"type": "SqlSink",
"sqlWriterCleanupScript": "$$Text.Format('delete imp_fc.SHIP_UPS_IntDtl_Tracking', WindowStart, WindowEnd)",
"writeBatchSize": 0,
"writeBatchTimeout": "00:00:00"
},
"translator": {
"type": "TabularTranslator",
"columnMappings": "COMPANY:COMPANY, UPSD_ORDN:UPSD_ORDN, UPSD_BOL:UPSD_BOL"
}
},
"inputs": [
{
"name": "AnalyticsDB-SHIPUPS_03shp-01src_FC-915PM"
}
],
"outputs": [
{
"name": "AnalyticsDB-SHIPUPS_03shp-02dst_AZ-915PM"
}
],
"policy": {
"timeout": "1.00:00:00",
"concurrency": 1,
"executionPriorityOrder": "NewestFirst",
"style": "StartOfInterval",
"retry": 3,
"longRetry": 0,
"longRetryInterval": "00:00:00"
},
"scheduler": {
"frequency": "Day",
"interval": 1,
"offset": "04:15:00"
},
"name": "915PM-SHIPUPS-fc-copy->[imp_fc]_[SHIP_UPS_IntDtl_Tracking]"
},
{
"type": "Copy",
"typeProperties": {
"source": {
"type": "SqlSource",
"sqlReaderQuery": "$$Text.Format('select distinct ups.UPSD_BOL, ups.UPSD_BOL from imp_fc.SHIP_UPS_IntDtl_Tracking ups LEFT JOIN olcm.SHIP_Tracking st ON ups.UPSD_BOL = st.SHIP_SYS_TRACK_NUM WHERE st.SHIP_SYS_TRACK_NUM IS NULL', WindowStart, WindowEnd)"
},
"sink": {
"type": "SqlSink",
"writeBatchSize": 0,
"writeBatchTimeout": "00:00:00"
},
"translator": {
"type": "TabularTranslator",
"columnMappings": "UPSD_BOL:SHIP_SYS_TRACK_NUM, UPSD_BOL:SHIP_TRACK_NUM"
}
},
"inputs": [
{
"name": "AnalyticsDB-SHIPUPS_06shp-01src_AZ-915PM"
}
],
"outputs": [
{
"name": "AnalyticsDB-SHIPUPS_06shp-02dst_AZ-915PM"
}
],
"policy": {
"timeout": "1.00:00:00",
"concurrency": 1,
"executionPriorityOrder": "NewestFirst",
"style": "StartOfInterval",
"retry": 3,
"longRetryInterval": "00:00:00"
},
"scheduler": {
"frequency": "Day",
"interval": 1,
"offset": "04:15:00"
},
"name": "915PM-SHIPUPS-AZ-update->[olcm]_[SHIP_Tracking]"
}
],
"start": "2017-08-22T03:00:00Z",
"end": "2099-12-31T08:00:00Z",
"isPaused": false,
"hubName": "adf-tm-prod-01_hub",
"pipelineMode": "Scheduled"
}
}
Have you seen this link?
They get the same error message and suggest using AzureTableSink instead of SqlSink
"sink": {
"type": "AzureTableSink",
"writeBatchSize": 0,
"writeBatchTimeout": "00:00:00"
}
It would make sense for you too since your 2nd copy activity is Azure to Azure
It could be a red herring but I'm pretty sure "tableName" is a require entry in the typeProperties for a sqlSource. Yours is missing this for the input dataset. Appreciate you have a join in the sqlReaderQuery so probably best to put a dummy (but real) table name in there.
Btw, not clear why you are using $$Text.Format and WindowStart/WindowEnd on your queries if you're not transposing these values into the query; you could just put the query between double quotes.
using CEP i'm trying to make a basic example of communication between Proton and Orion.
I simply want Proton to receive a notification of temperature from Orion and send back a payload changing one boolean attribute if the temperature value is superior or equal to 20.
Here is the definition i'm using.
{
"epn":
{
"events":
[
{
"name": "TemperatureContextUpdate",
"attributes":
[
{
"dimension": 0,
"description": "The certainty that this event happen (value between 0 to 1)",
"name": "Certainty",
"defaultValue": "1",
"type": "Double"
},
{
"dimension": 0,
"description": "No value means it equals the event detection time, other option is to use one of the defined distribution functions with parameters",
"name": "OccurrenceTime",
"type": "Date"
},
{
"dimension": 0,
"name": "ExpirationTime",
"type": "Date"
},
{
"dimension": 0,
"description": "The cost of this event occurrence. Negative if this is an opportunity",
"name": "Cost",
"type": "Double"
},
{
"dimension": 0,
"description": "Used in case the this event occur within an interval",
"name": "Duration",
"defaultValue": "0",
"type": "Double"
},
{
"dimension": "0",
"name": "entityId",
"type": "String"
},
{
"dimension": "0",
"name": "entityType",
"type": "String"
},
{
"dimension": "0",
"name": "temperature",
"type": "Integer"
},
{
"dimension": "0",
"name": "chaud",
"type": "Boolean"
}
],
"createdDate": "Wed Feb 22 2017"
},
{
"name": "Out_TemperatureContextUpdate",
"attributes":
[
{
"dimension": 0,
"description": "The certainty that this event happen (value between 0 to 1)",
"name": "Certainty",
"defaultValue": "1",
"type": "Double"
},
{
"dimension": 0,
"description": "No value means it equals the event detection time, other option is to use one of the defined distribution functions with parameters",
"name": "OccurrenceTime",
"type": "Date"
},
{
"dimension": 0,
"name": "ExpirationTime",
"type": "Date"
},
{
"dimension": 0,
"description": "The cost of this event occurrence. Negative if this is an opportunity",
"name": "Cost",
"type": "Double"
},
{
"dimension": 0,
"description": "Used in case the this event occur within an interval",
"name": "Duration",
"defaultValue": "0",
"type": "Double"
},
{
"dimension": "0",
"name": "entityId",
"type": "String"
},
{
"dimension": "0",
"name": "entityType",
"type": "String"
},
{
"dimension": "0",
"name": "temperature",
"type": "Integer"
},
{
"dimension": "0",
"name": "chaud",
"type": "Boolean"
}
],
"createdDate": "Wed Feb 22 2017"
}
],
"name": "TemperatureDemo",
"consumers":
[
{
"events":
[
{
"name": "Out_TemperatureContextUpdate"
}
],
"name": "Consumer_Rest",
"properties":
[
{
"name": "URL",
"value": "http://172.17.0.3:1026/v2/entities"
},
{
"name": "contentType",
"value": "application/json"
},
{
"name": "formatter",
"value": "json_ngsi"
},
{
"name": "delimiter",
"value": ";"
},
{
"name": "tagDataSeparator",
"value": "="
},
{
"name": "dateFormat",
"value": "dd/MM/yyyy-HH:mm:ss"
}
],
"type": "Rest",
"createdDate": "Wed Feb 22 2017"
},
{
"events":
[
{
"name": "TemperatureContextUpdate"
}
],
"name": "Consumer_File",
"properties":
[
{
"name": "filename",
"value": "/home/work/Documents/test.txt"
},
{
"name": "formatter",
"value": "json"
},
{
"name": "delimiter",
"value": ";"
},
{
"name": "tagDataSeparator",
"value": "="
}
],
"type": "File",
"createdDate": "Wed Feb 22 2017"
}
],
"epas":
[
{
"inputEvents":
[
{
"instanceSelectionPolicy": "First",
"alias": "TemperatureContextUpdate",
"filterExpression": "TemperatureContextUpdate.temperature>=20",
"name": "TemperatureContextUpdate",
"consumptionPolicy": "Consume"
}
],
"derivedEvents":
[
{
"name": "Out_TemperatureContextUpdate",
"reportParticipants": false,
"expressions":
{
"Cost": "TemperatureContextUpdate.Cost",
"Certainty": "TemperatureContextUpdate.Certainty",
"OccurrenceTime": "TemperatureContextUpdate.OccurrenceTime",
"Duration": "TemperatureContextUpdate.Duration",
"entityId": "TemperatureContextUpdate.entityId",
"ExpirationTime": "TemperatureContextUpdate.ExpirationTime",
"chaud": "true",
"entityType": "TemperatureContextUpdate.entityType",
"temperature": "TemperatureContextUpdate.temperature"
}
}
],
"evaluationPolicy": "Immediate",
"name": "EPA_Change",
"internalSegmentation":
[
],
"context": "always",
"epaType": "Basic",
"cardinalityPolicy": "Single",
"localPlacement": true,
"createdDate": "Wed Feb 22 2017",
"computedVariables":
[
]
}
],
"optimization":
{
},
"contexts":
{
"temporal":
[
{
"terminators":
[
],
"neverEnding": true,
"atStartup": true,
"name": "always",
"type": "TemporalInterval",
"createdDate": "Wed Feb 22 2017",
"initiators":
[
]
}
],
"segmentation":
[
],
"composite":
[
]
},
"producers":
[
]
}
}
I try to create a file while first update is receive but this file is always empty, btw it is correctly create in my directory and change everytime I change definition. The rest payload is never send like the condition is never triggered.
The payload I send :
{
"subscriptionId": "51c04a21d714fb3b37d7d5a7",
"data":
[ {
"id": "temp1",
"type": "Temperature",
"temperature": {
"type":"integer",
"value": 28,
"metadata":{}
},
"chaud": {
"type":"boolean",
"value": false,
"metadata":{}
},
"Certainty": {
"value": "1",
"type": "Float"
},
"Cost": {
"value": "0",
"type": "Float"
},
"Name": {
"value": "",
"type": "String"
},
"EventSource": {
"value": "",
"type": "String"
},
"Duration": {
"value": "0",
"type": "Float"
}
}
]
}
The log catalina.out :
INFOS: started event message body reader
févr. 23, 2017 11:23:35 AM com.ibm.hrl.proton.webapp.providers.EventJSONNgsiMessageReader parseVTwoFormat
INFOS: Event: TemperatureContextUpdate
févr. 23, 2017 11:23:35 AM com.ibm.hrl.proton.webapp.providers.EventJSONNgsiMessageReader readFrom
INFOS: finished event message body reader
févr. 23, 2017 11:23:35 AM com.ibm.hrl.proton.webapp.providers.EventJSONNgsiMessageReader readFrom
INFOS: EventJSONNgsiMessageReader: read event TemperatureContextUpdate; entityId=temp1; EventId=ba868559-7589-4720-ab75-658b920a3f14; chaud=false; entityType=Temperature; Chronon=null; temperature=28; DetectionTime=1487845415319; Name=; Certainty=1; Cost=0; EventSource=; OccurrenceTime=null; Duration=0; Annotation=; ExpirationTime=null; from broker...
févr. 23, 2017 11:23:35 AM com.ibm.hrl.proton.webapp.resources.EventResource submitNewEvent
INFOS: starting submitNewEvent
févr. 23, 2017 11:23:35 AM com.ibm.hrl.proton.webapp.resources.EventResource submitNewEvent
INFOS: events sent to proton runtime...
So any idea where is my problem ?
Thanks.
your mistake is that you send an event with an empty name.
I am sorry the same information is duplicated in the input, however this steams from the integration between Proton and Orion, each of which requires the information in different form and the difference was not patched smoothly.
The input you should send should contain the input event name, like this :
{
"subscriptionId": "51c04a21d714fb3b37d7d5a7",
"data":
[ {
"id": "temp1",
"type": "Temperature",
"temperature": {
"type":"integer",
"value": "28",
"metadata":{}
},
"chaud": {
"type":"boolean",
"value": false,
"metadata":{}
},
"Certainty": {
"value": "1",
"type": "Float"
},
"Cost": {
"value": "0",
"type": "Float"
},
"Name": {
"value": "TemperatureContextUpdate",
"type": "String"
},
"EventSource": {
"value": "",
"type": "String"
},
"Duration": {
"value": "0",
"type": "Float"
}
}
]
}
Note that the name attribute now has value "TemperatureContextUpdate"