Connection String error in Azure Data Factory Linked Service - ms-access

I am trying to create a Linked Service to a Access Database in Azure Data Factory. The issue I'm having is that I get this error with my connection string.
It seems to be the backslash in the file location, because it goes away when I erase that. But I need those to write the full file path. It even does this when I try an example connection string from Microsoft or ConnectionStrings.com
the example string = "Driver={Microsoft Access Driver (*.mdb)};Dbq=C:\mydatabase.mdb;"
I'm trying to use the JSON, but even trying to use a key vault secret doesn't work.
{
"name": "MicrosoftAccessLinkedService",
"properties": {
"type": "MicrosoftAccess",
"typeProperties": {
"connectionString": "Driver={Microsoft Access Driver (*.mdb)};Dbq=N:\Inventory Data\WhrsInvLK_22.mdb;",
"authenticationType": "Anonymous"
},
"connectVia": {
"referenceName": "IntegrationRuntime4",
"type": "IntegrationRuntimeReference"
}
}
}

Since the \ is part of the connection string and not used to escape some other character, you need to make sure that the \ character in the connection string itself is escaped.
For example, look at the following demonstration. If you are assigning the same exact string to a variable, it will store the value by placing an additional \ to escape the backslash present in the value.
The output would be as following:
So, you can use the following as the value for your connection string:
Driver={Microsoft Access Driver (*.mdb)};Dbq=N:\\Inventory Data\\WhrsInvLK_22.mdb;

Related

Azure Logic App - Parse JSON with dynamic key/name

just want to know if and how I can parse a HTTP response with a dynamic name in a JSON?
I used the Azure Management API to receive the managed identities (system- and user assigned managed identities) to receive all managed identities.
With a foreach I am iterating the results.
If a resource has a system assigned managed identity and user assigned managed identity, the response looks like this:
{
"principalId": "<principalId1>",
"tenantId": "<tenantId>",
"type": "SystemAssigned, UserAssigned",
"userAssignedIdentities": {
"/subscriptions/<subscriptionId>/resourcegroups/<resourceGroupName>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<userAssignedIdentitiesName>": {
"principalId": "<principalId2>",
"clientId": "<clientId>"
}
}
}
Now, I would like to get the <principalId2>.
Unfortunately, the Name of the object is dynamic related to the scope of the resource /subscriptions/<subscriptionId>/resourcegroups/<resourceGroupName>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<userAssignedIdentitiesName>.
How can I parse the JSON to receive the needed <principalId2>?
For all other responses I can easily use the Data operations Parse JSON with the payload I inserted from the HTTP response.
Is there a way to use a wildcard? Otherwise, could I somehow just select the first object of userAssignedIdentities to receive the needed value?
Ok, this should work for you. This is the flow I tested with ...
Initialise JSON
Your JSON as a string, how you do that in your solution may differ slightly.
Initialize XPath Result
Defined as an Array and the expression is ...
xpath(xml(json(concat('{ root: ', replace(variables('JSON'), 'PrincipalId', 'principalId'), '}'))), '(//principalId)[2]')
Initialize Result
A bit more work again but defined as a String and the expression is ...
array(xpath(xml(base64ToString(variables('XPath Result')[0]?['$content'])), '//text()'))[0]
The end result should be your value ...

Use Azure Data Factory to iterate over JSON and use values in parameters

I am attempting to automate downloading files from various sections of an FTP Server.
Thinking of having a JSON file that will have the relevant credentials, which would need to be iterated through to obtain the information and pass this as parameters to the FTP connection.
I've tried going through:
"Lookup" to the JSON file to obtain the array
"Set variable" for the JSON array
ForEach - to run through the JSON file
But I can't seem to get this to work.
I've followed the steps in here as a starting point but to no avail.
The main achievement of this exercise is to iterate over the JSON file and pass through the values as parameters in a ForEach Loop.
The JSON file is structured as follows (example):
{
"FilesToGet": [
{
"Description": "<Desc>",
"Username": "<Username>",
"Password": "<Password>",
"Subfolder": "<FTP_Subfolder>",
},
{
"Description": "<Desc>",
"Username": "<Username>",
"Password": "<Password>",
"Subfolder": "<FTP_Subfolder>",
}
]
}
This is the set up I have tried so far.
Another point to note is that I am thinking of storing the JSON file in Azure Key Vault (as it will contain sensitive info) - would iterating over this and passing the info to parameters still be viable?

AWS Step function string/json concatenation

I have orchestrated a data pipe line using AWS Step function.
In last state I want to send a custom notification. I'm using an Intrinsic function States.Format to format my message and subject. It works fine for Context object element. Here, I have tested that in Message parameter.
But it doesn't work with input JSON. This is my input JSON
{
"job-param":{
"pipe-line-name":"My pipe line name", "other-keys":"other values"
}
}
"Success State": {
"Type": "Task",
"Resource": "arn:aws:states:::sns:publish",
"Parameters": {
"Message.$": "States.Format('Execution Id:{}, completed successfully!', $$.Execution.Id)",
"Subject.$": "States.Format('[INFO] {} completed successfully!', $.job-param.pipe-line-name)",
"TopicArn": "arn:aws:sns:us-east-1:************:sns-topic"
},
"End": true
}
While saving this state machine, it gives me following error message:
The value for the field 'Subject.$' must be a valid JSON Path
I checked Input and Result path. They have this value. I can directly use this value as parameter. This is working fine. But I can't format with other string.
"Subject.$": "$.job-param.pipe-line-name"
Alternate approach would be to call lambda to customize and trigger SNS. But I want to avoid that.
Can I request some suggestions to fix this error?
Thanks in advance!
If you want to use any name with - in your JSON then you can write your JSON Path like this:
"Subject.$": "States.Format('[INFO] {} completed successfully!', $['job-param']['pipe-line-name'])",
But it would be easier if you change your input JSON and replace - with _:
"Subject.$": "States.Format('[INFO] {} completed successfully!', $.job_param.pipe_line_name)",

Linked Service to MySQL. Error converting invalid input with source encoding UTF-8 using ICU

In Azure Data Factory I created a MySQL linked service. But during data preview I get error:
ERROR [HY000] [Microsoft][Support] (50311) Error converting invalid input with source encoding UTF-8 using ICU. Rejected bytes began with: CFC2.. Activity ID:e3b7fc32-8fcd-4981-b0e3-f377c6b17150
Most likely a coding error. But I did not find how to configure it
Here is JSON of linked sevice
{
"name": "n_trans_connection",
"type": "Microsoft.DataFactory/factories/linkedservices",
"properties": {
"annotations": [],
"type": "MySql",
"typeProperties": {
"connectionString": "server=some-server;port=some-port;database=u5256_mti_spr;user=some-user;sslmode=1;usesystemtruststore=0",
"encryptedCredential": "some-hash"
}
}
}
How to set up encoding?
Since the error is when you preview data , I don't think that is coding issue . Can you please check if you select a different table and if you face the same issue .

How Can I parse out JSON in a Node-RED Function

I am working with Node-RED in Bluemix for IoT.
How can I parse out the individual pieces of information (like the cmdmsg and the tempr) in a function node so I can use it in other nodes in the flow?
I'm getting an error when I try (see below)
I am receiving the JSON complete message object (from an IoT in Node) that I see in my "debug" node when I set it to look at the complete message object. see the object below.
It appears to me that the JSON is formatted correctly.
I tried putting the following in the function node, but I'm getting an error that says "TypeError: Cannot read property 'tempr' of undefined"
Here is what the function parameter is:
return {payload:msg.payload.d.tempr};
and here is the message object
{
"topic": "iot-2/type/Arduino-tempsensor/id/FFFFFFFFFFFF/evt/status/fmt/json",
"payload": "{\n\"d\": {\n\"myName\": \"Arduino CF\",\n\"cmdmsg\": \"Weekly\",\n\"tempr\": -3,\n}\n}",
"deviceId": "FFFFFFFFFFFF",
"deviceType": "Arduino-tempsensor",
"eventType": "status",
"format": "json",
"_msgid": "ffffffff.55555"
}
note: I obfuscated the device ID (mac address) and msgid
Any ideas on how to parse the data out and why I'm getting an error?
Sorry, but your JSON Payload is completely messed up, it should look like this: {"d": {"myName": "Arduino CF","cmdmsg": "Weekly","tempr": -3}}
You shouldn't see any \or \nin the payload, they look like escape characters from the client side. I also believe that the last comma after the tempr value shouldn't be there for valid JSON.
I am not an Arduino expert but I have experimented with a Raspberry Pi and the Mosquitto client, this is how I can successfully send an event to IoTF:
mosquitto_pub -h <org>.messaging.internetofthings.ibmcloud.com -p 1883 -u "use-token-auth" -P "<token>" -i d:<org>:raspi:raspi2 -t iot-2/evt/message/fmt/json -m {"d":{"text":"Hello World"}}
If the paylod is correct JSON your statement return {payload:msg.payload.d.tempr}; will work.
Have you seen this: http://www.ibm.com/developerworks/cloud/library/cl-bluemix-arduino-iot2/
The JSON string you are showing should be converted to a Javascript object before you try to access its fields. To do that is as simple as wiring the arduino output to a "JSON" node, which does the conversion for you (or throws an error if the string is not valid JSON).
Wire the output of the JSON node to a debug node, if you want to see the structure of the msg object. You can also wire it to a "change" node, if you simply want to replace the msg.payload with the temperature value, for instance. You don't need any custom javascript code in a function node to do simple changes like that.
Here is a sample flow that you can import... the arduino output string is simulated by pasting your payload into a "template" node:
[{"id":"1a79abfe.b8abb4","type":"inject","z":"58c8eb7a.5496c4","name":"send output","topic":"iot-2/type/Arduino-tempsensor/id/FFFFFFFFFFFF/evt/status/fmt/json","payload":"true","payloadType":"bool","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":170,"y":2740,"wires":[["9fc678fb.ae18e8"]]},{"id":"69e91778.e0c6e8","type":"json","z":"58c8eb7a.5496c4","name":"","property":"payload","action":"","pretty":false,"x":390,"y":2800,"wires":[["d066800f.60a9b","cf991eb1.f2a1a"]]},{"id":"9d8d7da2.2a7da","type":"debug","z":"58c8eb7a.5496c4","name":"msg string","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","x":610,"y":2740,"wires":[]},{"id":"9fc678fb.ae18e8","type":"template","z":"58c8eb7a.5496c4","name":"arduino string","field":"payload","fieldType":"msg","format":"json","syntax":"plain","template":"{\n \"d\": {\n \"myName\": \"Arduino CF\",\n \"cmdmsg\": \"Weekly\",\n \"tempr\": -3\n }\n}","output":"str","x":360,"y":2740,"wires":[["69e91778.e0c6e8","9d8d7da2.2a7da"]]},{"id":"d066800f.60a9b","type":"change","z":"58c8eb7a.5496c4","name":"extract tempr","rules":[{"t":"set","p":"payload","pt":"msg","to":"payload.d.tempr","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":450,"y":2860,"wires":[["af730d72.2995a"]]},{"id":"af730d72.2995a","type":"debug","z":"58c8eb7a.5496c4","name":"tempr","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","x":630,"y":2860,"wires":[]},{"id":"cf991eb1.f2a1a","type":"debug","z":"58c8eb7a.5496c4","name":"msg object","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","x":610,"y":2800,"wires":[]}]
As was noted, your original JSON string was not valid, because of the trailing comma -- but it is ok to have double-quotes and newlines as long as they are escaped with a "\". When the string is parsed, they are stripped out anyway.