I have a potential client who wants to pull some data from a website via VBA. I am new to XML and JSON.
I found a link somewhere that provides the following code which uses MSXML to return data for a single item from that particular website.
Public Function GetItemSalePrice(item As String) As Double
Dim dblItem As Long
With CreateObject("msxml2.xmlhttp")
.Open "GET", "http://www.gw2spidy.com/api/v0.9/json/item/" & item, False
.send
dblItem = Split(Split(.responsetext, "min_sale_unit_price"":")(1), ",")(0)
GetItemSalePrice = dblItem / 100
End With
End Function
However, the data my client wants to return comes in pages of up to 500 records at a time. He indicates that the
he wants to pass in date ranges, and a page number, similar to the following.
https://api.appfigures.com/v2/reviews?client_key=xxxxxxxf&start=2015-01-01&end=2016-01-21&page=1
But because this is an https site, it wants a userid and password. Can I simply reformat that string to include userid and password? Or is there another method or property of the MSXML object that can be set for authentication?
The client indicates that the return value looks like:
{
"total": 140,
"pages": 28,
"this_page": 1,
"reviews": [{
"author": "DeveloperToDeveloper",
"title": "Just Spectacular",
"review": "Finally able to remove the ads! The description is hilarious!! Thanks!!!",
"original_title": null,
"original_review": null,
"stars": "5.00",
"iso": "US",
"version": "1.2",
"date": "2012-09-19T17:05:00",
"product": 6567539,
"weight": 0,
"id": "5561747L7xnbsMRu8UbPvy7A71Dv6A=="
}]
}
But with multiple records returned, as many as 500 at a time. Is there an efficient way of reading data in that format into a table? One or more records at a time? I can obviously write a text parser but I assume that someone has probably already done that leg work.
thanks Bob. If figured out the authentication issue, a popup dialog is used to enter userid and password and the client was satisfied with that.
And given the structure of the returned string, I was able to parse the data myself without too much difficulty.
Related
I'm trying to save data to my MySql db from a Node method. This includes a field called attachments.
console.log(JSON.stringify(post.acf.attachments[0])); returns:
{
"ID": 4776,
"id": 4776,
"title": "bla",
"filename": "bla.pdf",
"filesize": 1242207,
"url": "https://example.com/wp-content/uploads/bla.pdf",
"link": "https://example.com/bla/",
"alt": "",
"author": "1",
"description": "",
"caption": "",
"name": "bla",
"status": "inherit",
"uploaded_to": 0,
"date": "2020-10-23 18:05:13",
"modified": "2020-10-23 18:05:13",
"menu_order": 0,
"mime_type": "application/pdf",
"type": "application",
"subtype": "pdf",
"icon": "https://example.com/wp-includes/images/media/document.png"
}
This is indeed the data I want to save to the db:
await existing_post.save({
...
attachments: post.acf.attachments[0],
)};
However, the attachments field produces a 422 server error (if I comment out this field, the other fields save without a problem to the db). I'm not getting what is causing this error. Any ideas?
I've also tried
await existing_post.save({
...
attachments: post.acf.attachments,
)};
but then it seems to just save "[object Object]" to the database.
The field in the database is defined as text. I've also tried it by defining the field as json, but that made no difference.
exports.up = function (knex, Promise) {
return knex.schema.table("posts", function (table) {
table.longtext("attachments");
});
};
The 422 error code is about the server unable to process the data you are sending to it. In your case, your table field is longtext when post.acf.attachments seems like an object. That's why it saves [object Object] to your db (It is the return value of the toString() method).
Try using
await existing_post.save({
...
attachments: JSON.stringify(post.acf.attachments),
)};
MySQL and knex both support the JSON format, I'd suggest you change the field to json. (See knex docs and mysql 8 docs). You'll stiil need to stringify your objects tho.
EDIT: I just saw that Knex supports jsonInsert (and plenty other neat stuff) as a query builder that should be useful for you.
Mysql also support a large range of cool stuffs for handling jsons
In addition, when you fetch the results in the database, you'll need to parse the JSON result to get an actual JSON object:
const acf = await knex('posts').select('acf').first();
const attachment = JSON.parse(acf.attachment;
Knex also provide jsonExtract that should fill your needs (See also the mysql json_extract
I work a no-code app maker called Adalo. I am connected my app to a API that sends emails automatically. But in order for me to connect to the API, I must first "Run Test Request" which tries to send a request to the API. If I put an email into the "to" instead of "User Email" it will work perfectly. However, I currently have it set as a variable which is populated by data records. When I try to "Run Test Request", it does not work, because it does not no where to pull the data from during the test (it knows where to get the data outside the test). So what happens is the "To" email becomes "" which it says is an invalid email, so the test cannot pass. I am trying to think of a work around, because I know the code works, its just that the test wont let it work with a variable. So I wanted to know if I could some how create an if statement that said something like
if User Email == ""
then User Email = "g#gmail.com"
The syntax is wrong, but hopefully you understand what I am getting at.
Below is the current code I have. This has to be in JSON, my understand is that I cannot insert Java.
{
"sender": {
"name": "Peculiar Yogi Mail",
"email": "namaste#peculiaryogi.com"
},
"to": [{
"email": "USER EMAIL",
"name": "USER FIRST NAME"
}],
"subject": "Peculiar Yogi Class Booking Confirmation",
"htmlContent":"<html><head></head><body><p>Hello,</p><p>This is to confirm that you've successfully booked a class at Peculiar Yogi.</p><p>Class Name:</p><p>Class Instructor</p><p>Class Time:</p><p>Namaste.</p></body></html>"
}
I've setup Apache Drill, created http storage plugin and set its configuration as here:
{
"type": "http",
"cacheResults": false,
"connections": {
"accounts": {
"url": "https://my.datasource.url",
"method": "GET",
"headers": {
"Authorization": "Bearer access_token...",
"Accept": "application/json"
},
"authType": "none",
"userName": null,
"password": null,
"postBody": null,
"params": null,
"dataPath": "QueryResponse/Account",
"requireTail": false,
"inputType": "json"
}
},
"timeout": 0,
"proxyHost": null,
"proxyPort": 0,
"proxyType": "direct",
"proxyUsername": null,
"proxyPassword": null,
"enabled": true
}
I am able to run queries through rest call (as well as from web ui and odbc) as here:
{
"queryType": "SQL",
"query": "select * from myds.accounts"
}
The problem is, access token is short lived and multiple users need to access these data sources with their own access tokens, so saving token withing connection doesn't work for me.
Is there any way I could send access token from the client at the time of sending query? I have no preference of using either Rest API or ODBC, any of them would be good as far as it solves my problem. Thanks
It may be possible to specify some of the configuration at query time. The example below demonstrates, in the file system plugin, how to use the table() function to alter the configuration options at runtime. In this case, we're specifying which sheet to query in an excel file.
SELECT *
FROM table(dfs.`excel/test_data.xlsx` (type => 'excel', sheetName =>'secondSheet'))
I don't know if this will work for the REST plugin or not, but it's worth a try. (It is admittedly a bit of a hack)
Another option, which would require modification to the plugin, would be to create special variables that could be specified at query time. For instance, we could create a _headers variable so that you could insert items into the headers at query time. Thus, a query might look like:
SELECT...
FROM ...
WHERE _headers="Authorization=1234"
I'm really wondering what the best way to accomplish this is. I'm sure you're not the only one with this issue.
Very short version
How do I include an ADF Variable inside a JSON POST request, in a Web Activity within ADF?
I feel like this should be a very simple string concatenation, but i can't get it to work
Detail
We have a requirement to run a query / SProc from within ADF, which will return a string containing an error message. That string is to then be passed via the Web Activity in ADF to a Logic App, in order to fire off an email, containing the error.
The setup of the logic app is copied from here:
https://www.mssqltips.com/sqlservertip/5718/azure-data-factory-pipeline-email-notification--part-1/
and then here (part 2)
https://www.mssqltips.com/sqlservertip/5962/send-notifications-from-an-azure-data-factory-pipeline--part-2/
In ADF, I used the Lookup activity, to run a query, which brings back the error (appears to work, the preview returns the correct string)
Then I use the Set Variable activity, to take the output of the lookup and store it in a variable.
Last Step is to fire off the POST using the Web Activity.
With this code (tweaked slightly to remove personal details) in my Web Activity, everything works fine and I receive an email
{
"DataFactoryName": "#{pipeline().DataFactory}",
"PipelineName": "#{pipeline().Pipeline}",
"Subject": "Pipeline finished!",
"ErrorMessage": "Everything is okey-dokey!",
"EmailTo": "me#myEmail.com"
}
But any attempt to put the contents of the Variable into the Subject part has failed.
This (for example) sends me an email with the subject literally being #variables('EmailSubject')
{
"DataFactoryName": "#{pipeline().DataFactory}",
"PipelineName": "#{pipeline().Pipeline}",
"Subject": "#variables('EmailSubject')",
"ErrorMessage": "Everything is okey-dokey!",
"EmailTo": "me#myEmail.com"
}
But I've also attempted various other solutions that result in errors or the email subject just containing the literal thing that I put in there (e.g. + #variables('EmailSubject') +).
I also tried storing the entire JSON in the Variable, and then having the Web activity use only the variable, that returned no errors, but also did not send an email.
This attempt:
{
"DataFactoryName": "#{pipeline().DataFactory}",
"PipelineName": "#{pipeline().Pipeline}",
"Subject": "#{variables('EmailSubject')}",
"ErrorMessage": "Everything is okey-dokey!",
"EmailTo": "me#myEmail.com"
}
Resulted in this input into the web activity - which actually includes the text of the error, which is a bonus ... (text = Job Duration Warning):
{
"url": "https://azureLogicAppsSiteHere",
"method": "POST",
"headers": {
"Content-Type": "application/json"
},
"body": "{\n \"DataFactoryName\": \"DFNAMEHERE\",\n \"PipelineName\": \"pipeline1\",\n \"Subject\": \"{\"firstRow\":{\"\":\"Job Duration Warning\"},\"effectiveIntegrationRuntime\":\"DefaultIntegrationRuntime (West Europe)\",\"billingReference\":{\"activityType\":\"PipelineActivity\",\"billableDuration\":[{\"meterType\":\"AzureIR\",\"duration\":0.016666666666666666,\"unit\":\"DIUHours\"}]},\"durationInQueue\":{\"integrationRuntimeQueue\":0}}\",\n \"ErrorMessage\": \"Everything is okey-dokey!\",\n \"EmailTo\": \"me#myEmail.com\"\n}\t"
}
But then resulted in this error:
{
"errorCode": "2108",
"message": "{\"error\":{\"code\":\"InvalidRequestContent\",\"message\":\"The request content is not valid and could not be deserialized: 'After parsing a value an unexpected character was encountered: f. Path 'Subject', line 4, position 17.'.\"}}",
"failureType": "UserError",
"target": "Web1",
"details": []
}
[Edit] The PREVIEW from the Lookup Activity is the text: Job Duration Warning BUT when I debug the pipeline, it lets me see the actual Output, which is this:
{
"count": 1,
"value": [
{
"": "Job Duration Warning"
}
],
"effectiveIntegrationRuntime": "DefaultIntegrationRuntime (West Europe)",
"billingReference": {
"activityType": "PipelineActivity",
"billableDuration": [
{
"meterType": "AzureIR",
"duration": 0.016666666666666666,
"unit": "DIUHours"
}
]
},
"durationInQueue": {
"integrationRuntimeQueue": 0
}
}
So it appears that the problem is that the Lookup Output isn't what I thought it was, so the variable can't be used in the Web Activity, as it contains unsupported characters or something along those lines.
I just tested this and it worked ok:
Create a String Parameter with the value Job Duration Warning
Set the Variable value to be #pipeline().parameters.ParamSubject
Include the variable in the web activity with an # in front of it
I then receive my expected email with the right subject. I just don't know how to get the string output of my query, into a variable / parameter, so that i can use it in the web activity.
I don't know how well this applies to other people's issues, but I found a solution that has worked for me.
In the SELECT query within the Lookup Activity - name the output (in my case, I called that column 'Subject'- i.e. SELECT xyz AS Subject
In the Lookup Activity, turn on the setting 'First Row Only'
In the Set Variable Activity, use the code: #activity('Lookup1').output.firstRow.subject
(where 'Lookup1' is the name of your Lookup Activity and Subject is the name of the column you are outputting)
In the Web Activity, reference the variable as follows:
{
"DataFactoryName": "#{pipeline().DataFactory}",
"PipelineName": "#{pipeline().Pipeline}",
"Subject": "#{variables('EmailSubject')}",
"ErrorMessage": "Everything is okey-dokey!",
"EmailTo": "me#myEmail.com"
}
I am sending a raw Json requet using postman to an API service which feeds it to another web service and finally a database. I want to attach a file to the raw Json request.
I am attaching below the current request I am sending. Is it the right way? The first name and other information is going through but the attachment is not. Any suggestions?
{
"Prefix": "",
"FirstName": "test-resume-dlyon",
"LastName": "test-dlyon-resume",
"AddressLine1": "test2",
"AddressLine2": "",
"City": "Invalid Zipcode",
"State": "GA",
"Zip": "99999",
"Phone": "9999999999",
"Email": "testresumedlyon#gmail.com",
"Source": "V",
"WritingNumber": "",
"AgeVerified": true,
"AdditionalSource": "",
"EnableInternetSource": true,
"InternetSource": "",
"ExternalResult": "",
"PartnerID": "",
"SubscriberID": "15584",
"Languages": [
"English",
"Spanish"
],
"fileName": "resume",
"fileExtension": "docx",
"fileData": "UELDMxE76DDKlagmIF5caEVHmJYFv2qF6DpmMSkVPxVdtJxgRYV"
}
There is no "correct" format to attach a file to a JSON.
JSON is not multipart/form-data (which is designed to include files).
JSON is a text-based data format with a variety of data types (such as strings, arrays, and booleans) but nothing specific for files.
This means that to attach a file, you have to get creative.
For example, you could encode a file in text format (e.g. using base64), but it wouldn't be very efficient, and any Word document would result in you getting a much longer string than "UELDMxE76DDKlagmIF5caEVHmJYFv2qF6DpmMSkVPxVdtJxgRYV".
Of course, the method you use to encode the file has to be the method that whatever is reading the JSON expects you to use. Since there is no standard for this, and you have said nothing about the system which is consuming the JSON you are sending, we have no idea what that method is.
First of all, I'd recommend reading the postman API docs. They have some extremely useful information on there for using the API. Two particular articles that might of interest here are these:
Looking at it and running it through a validator like this one shows that there are no syntax errors so it must be to do with the JSON parameters the API is expecting.
Here's something you can try:
In postman, set method type to POST.
Then select Body -> form-data -> Enter your parameter name (file according to your code)
and on right side next to value column, there will be dropdown "text, file", select File. choose your image file and post it.
For rest of "text" based parameters, you can post it like normally you do with Postman. Just enter parameter name and select "text" from that right side dropdown menu and enter any value for it, hit send button. Your controller method should get called.