I have been having a strange error that I have been going in circles trying to figure out.
I decided to learn some JavaScript/google apps script to parse some information from an api of a game I play. I followed a bunch of tutorials and searched google and Stack Overflow for solutions but none have worked. I think I've narrowed it down to one particular Method: the "UrlFetchApp."
Right now I'm simply trying to grab some JSON data from the API and display it in the log window.
I keep getting this:
[21-09-13 09:01:40:770 EDT] Exception: Request failed for https://api.guildwars2.com returned code 429. Truncated server response: {"text":"too many requests"} (use muteHttpExceptions option to examine full response)
at getRequest(Untitled:4:30)
Now here's the strange part, I ran the script from my phone at work to test some theories out and got this:
[21-09-13 09:02:20:943 EDT] {
"name": "Abomination Hammer",
"type": "Weapon",
"level": 0,
"rarity": "Fine",
"vendor_value": 0,
"default_skin": 5014,
"game_types": [
"Activity",
"Wvw",
"Dungeon",
"Pve"
],
"flags": [
"NoSell",
"SoulbindOnAcquire",
"SoulBindOnUse"
],
"restrictions": [],
"id": 15,
"chat_link": "[&AgEPAAAA]",
"icon": "https://render.guildwars2.com/file/E8507FFB6CF3C9094A69956344CEDBD9B47D95B6/434872.png",
"details": {
"type": "Hammer",
"damage_type": "Physical",
"min_power": 146,
"max_power": 165,
"defense": 0,
"infusion_slots": [],
"attribute_adjustment": 20.736,
"infix_upgrade": {
"id": 112,
"attributes": []
},
"secondary_suffix_item_id": ""
}
}
I doubt it's the code now.
That is what was supposed to be returned. It's the same code unedited and ran multiple times on both my phone and computer and the results are always the same.
Here's the line of code I wrote:
function getRequest() {
var response = UrlFetchApp.fetch('https://api.guildwars2.com/v2/items/15');
Logger.log(response)
}
Can anyone help? I'm really frustrated now.
Response status code 429 shows when a user sent too many requests in a given amount of time. Based on the documentation of Rate Limiting in Guild Wars the rate limit is tracked per-IP and the maximum request per minute is 600.
In your script, you can confirm the rate limit by printing the header by using response.getAllHeaders() or response.getHeaders() and you will see x-rate-limit-limit=600.
X-Rate-Limit-Limit is the maximum number of requests that may be
issued within a minute. This is a static value doesn’t contain count
for requests you have already sent.
Usually, 429 status code includes Retry-After header. This indicates how long a user must wait before making a new request. You can also view this by printing the header.
If the Retry-After is not included in the headers, you can follow Elchanan shuky Shukrun comment above to use VPN since the rate limit is tracked per IP. You can also use the Guild Wars forum and create a post for a possible bug.
Reference:
429 Too Many Requests
Retry-Limit
Related
I want to send different JSONs to this endpoint:
{{URL_API}}/products/
I need to update several information related to different products so i need to specify the product within the endpoint, i mean, i.e:
If you access this particular endpoint: {{URL_API}}/products/ you will get all the products but i need to specify the product that i want to update:
{{URL_API}}/products/99RE345GT
So, i decided to create a CSV file, i will update all the different product passing that file in the COLLECTION RUNNER screen, do you get me?.
For that, i created a new collection, i put this request_url:
{{URL_API}}/ns/products/{{sku}}
I edited the request body:
{
"sku": "{{sku}}",
"price": "{{price}}",
"tax_percentage": "{{tax_percentage}}",
"store_code": "{{store_code}}",
"markup_top": "{{markup_top}}",
"status": "{{status}}",
"group_prices": [
{
"group": "{{class_a}}",
"price": "{{price_a}}",
"website": "{{website_a}}"
}
]
}
all those fields between the {{}} will be completed by the CSV but it shows an error message over the url_request, {{sku}} seems to be wrong... it throws this message error:
unresolved variable, this variable is not defined in the active
collection, environment or globals.
How can i solve this? I do not know what to do now.
What am i missing?
Very short version
How do I include an ADF Variable inside a JSON POST request, in a Web Activity within ADF?
I feel like this should be a very simple string concatenation, but i can't get it to work
Detail
We have a requirement to run a query / SProc from within ADF, which will return a string containing an error message. That string is to then be passed via the Web Activity in ADF to a Logic App, in order to fire off an email, containing the error.
The setup of the logic app is copied from here:
https://www.mssqltips.com/sqlservertip/5718/azure-data-factory-pipeline-email-notification--part-1/
and then here (part 2)
https://www.mssqltips.com/sqlservertip/5962/send-notifications-from-an-azure-data-factory-pipeline--part-2/
In ADF, I used the Lookup activity, to run a query, which brings back the error (appears to work, the preview returns the correct string)
Then I use the Set Variable activity, to take the output of the lookup and store it in a variable.
Last Step is to fire off the POST using the Web Activity.
With this code (tweaked slightly to remove personal details) in my Web Activity, everything works fine and I receive an email
{
"DataFactoryName": "#{pipeline().DataFactory}",
"PipelineName": "#{pipeline().Pipeline}",
"Subject": "Pipeline finished!",
"ErrorMessage": "Everything is okey-dokey!",
"EmailTo": "me#myEmail.com"
}
But any attempt to put the contents of the Variable into the Subject part has failed.
This (for example) sends me an email with the subject literally being #variables('EmailSubject')
{
"DataFactoryName": "#{pipeline().DataFactory}",
"PipelineName": "#{pipeline().Pipeline}",
"Subject": "#variables('EmailSubject')",
"ErrorMessage": "Everything is okey-dokey!",
"EmailTo": "me#myEmail.com"
}
But I've also attempted various other solutions that result in errors or the email subject just containing the literal thing that I put in there (e.g. + #variables('EmailSubject') +).
I also tried storing the entire JSON in the Variable, and then having the Web activity use only the variable, that returned no errors, but also did not send an email.
This attempt:
{
"DataFactoryName": "#{pipeline().DataFactory}",
"PipelineName": "#{pipeline().Pipeline}",
"Subject": "#{variables('EmailSubject')}",
"ErrorMessage": "Everything is okey-dokey!",
"EmailTo": "me#myEmail.com"
}
Resulted in this input into the web activity - which actually includes the text of the error, which is a bonus ... (text = Job Duration Warning):
{
"url": "https://azureLogicAppsSiteHere",
"method": "POST",
"headers": {
"Content-Type": "application/json"
},
"body": "{\n \"DataFactoryName\": \"DFNAMEHERE\",\n \"PipelineName\": \"pipeline1\",\n \"Subject\": \"{\"firstRow\":{\"\":\"Job Duration Warning\"},\"effectiveIntegrationRuntime\":\"DefaultIntegrationRuntime (West Europe)\",\"billingReference\":{\"activityType\":\"PipelineActivity\",\"billableDuration\":[{\"meterType\":\"AzureIR\",\"duration\":0.016666666666666666,\"unit\":\"DIUHours\"}]},\"durationInQueue\":{\"integrationRuntimeQueue\":0}}\",\n \"ErrorMessage\": \"Everything is okey-dokey!\",\n \"EmailTo\": \"me#myEmail.com\"\n}\t"
}
But then resulted in this error:
{
"errorCode": "2108",
"message": "{\"error\":{\"code\":\"InvalidRequestContent\",\"message\":\"The request content is not valid and could not be deserialized: 'After parsing a value an unexpected character was encountered: f. Path 'Subject', line 4, position 17.'.\"}}",
"failureType": "UserError",
"target": "Web1",
"details": []
}
[Edit] The PREVIEW from the Lookup Activity is the text: Job Duration Warning BUT when I debug the pipeline, it lets me see the actual Output, which is this:
{
"count": 1,
"value": [
{
"": "Job Duration Warning"
}
],
"effectiveIntegrationRuntime": "DefaultIntegrationRuntime (West Europe)",
"billingReference": {
"activityType": "PipelineActivity",
"billableDuration": [
{
"meterType": "AzureIR",
"duration": 0.016666666666666666,
"unit": "DIUHours"
}
]
},
"durationInQueue": {
"integrationRuntimeQueue": 0
}
}
So it appears that the problem is that the Lookup Output isn't what I thought it was, so the variable can't be used in the Web Activity, as it contains unsupported characters or something along those lines.
I just tested this and it worked ok:
Create a String Parameter with the value Job Duration Warning
Set the Variable value to be #pipeline().parameters.ParamSubject
Include the variable in the web activity with an # in front of it
I then receive my expected email with the right subject. I just don't know how to get the string output of my query, into a variable / parameter, so that i can use it in the web activity.
I don't know how well this applies to other people's issues, but I found a solution that has worked for me.
In the SELECT query within the Lookup Activity - name the output (in my case, I called that column 'Subject'- i.e. SELECT xyz AS Subject
In the Lookup Activity, turn on the setting 'First Row Only'
In the Set Variable Activity, use the code: #activity('Lookup1').output.firstRow.subject
(where 'Lookup1' is the name of your Lookup Activity and Subject is the name of the column you are outputting)
In the Web Activity, reference the variable as follows:
{
"DataFactoryName": "#{pipeline().DataFactory}",
"PipelineName": "#{pipeline().Pipeline}",
"Subject": "#{variables('EmailSubject')}",
"ErrorMessage": "Everything is okey-dokey!",
"EmailTo": "me#myEmail.com"
}
I am posting the results/logs of a CI/CD system to Microsoft Teams. While handling some failed builds with longer results, I stumbled upon the following error returned by the webhook URL https://outlook.office.com/webhook/bb6bfee7-1820-49fd-b9f9-f28f7cc679ff#<uuid1>/IncomingWebhook/<id>/<uuid2>:
Webhook message delivery failed with error: Microsoft Teams endpoint returned HTTP error 413 with ContextId tcid=3626521845697697778,server=DB3PEPF0000009A,cv=BmkbJ1NdTkv1EDoqr7n/rg.0..
As I observe, this is caused by too long payload posted to the Teams webhook URL.
The initial complex message (sections, titles, subtitles, formatted links, <pre> formatted text, etc.) was failing when the JSON payload was a above 18000 characters.
Testing a bit with the payload I observed that the more formatting I remove from the raw JSON payload, the longer can the Teams message be. The longest message I could post had (according cu cURL): Content-Length: 20711. The JSON payload for this message was:
{"themeColor":"ED4B35","text":"a....a"}
whitespaces in the JSON format seem not to count (i.e. adding spaces will not decrease the maximum message length that I can sent to the Teams webhook).
For reference, the initial message was looking similar to this:
{
"themeColor": "ED4B35",
"summary": "iris-shared-libs - shared-library-updates - failure",
"sections": [
{
"activityTitle": "Job: [iris-shared-libs](https://my.concourse.net/teams/hsm/pipelines/iris-shared-libs) - [shared-library-updates #89](https://my.concourse.sccloudinfra.net/teams/hsm/pipelines/iris-shared-libs/jobs/shared-library-updates/builds/89) (FAILURE)",
"activityImage": "https://via.placeholder.com/200.png/ED4B35/FFFFFF?text=F",
"facts": [
{
"name": "Failed step",
"value": "update-shared-libraries"
}
]
},
{
"text": "Trying a new strategy with gated versioned releases",
"facts": [
{
"name": "Repository",
"value": "[iris-concourse-resources](https://my.git.com/projects/IRIS/repos/iris-concourse-resources)"
},
{
"name": "Commit",
"value": "[2272145ddf9285c9933df398d63cbe680a62f2b7](https://my.git.com/projects/IRIS/repos/iris-concourse-resources/commits/2272145ddf9285c9933df398d63cbe680a62f2b7)"
},
{
"name": "Author",
"value": "me#company.com"
}
]
},
{
"activityTitle": "Job failed step logs part 1",
"text": "<pre>...very long log text goes here ...</pre>"
}
]
}
What is the actual maximum lengths of the Microsoft Teams connector webhook posted message?
The official page does not mention it. In the Feedback section at the bottom there is still an open question regarding "Messages size limits?" with the feedback: "We are currently investigating this."
From the tests I made so far, some limits observed (if this is independent of the server) are roughly, based on the JSON message payload (structure and formatting) between 18000 and 40000 (with length below 18000 never breaking and above 40000 always breaking).
Use case 18000: one long text for a section
Use case 40000: 600 facts with very short name and empty string as value
And removing a fragment of the JSON payload and adding an equal number of characters in another JSON value will not give you the same maximum.
I have observed a soft limit (message truncated, but no error) as well on the maximum number of sections: 10. The sections starting with the 11th are discarded.
I've got a API stage that's NOT using "Lambda Proxy integration" which has a Lambda function passing an error.
In the mapping template I have this:
$input.path("$.errorMessage")
Which results in the output of this:
{
"headers": {
"apiVersion": "20190218.1",
"isTesting": true
},
"body": {
"statusCode": 503,
"status": "Service Unavailable",
"title": "One or more of our data providers are currently offline for scheduled maintenance"
}
}
The header values are mapped to template headers and pull through correctly, however I need the body to transform to this:
{
"statusCode": 503,
"status": "Service Unavailable",
"title": "One or more of our data providers are currently offline for scheduled maintenance"
}
Whatever I have tried, body always returns as a blank string, an empty body, or an invalid JSON.
This is the closest I've got but it returns an invalid JSON:
$util.parseJson($input.path("$.errorMessage")).body
Result (comes back with no quotes):
{statusCode=503, status=Service Unavailable, title=One or more of our data providers are currently offline for scheduled maintenance}
Is it possible to do what I'm after? I can't find a reverse for $util.parseJson (i.e, stringify).
Thanks!
I think the original poster has probably moved on in the past 11 months, but in case anyone else stumbles across this question, $input.json('$.errorMessage.body') should work.
After creating a Group using POST /v1.0/groups with the body:
{
"description": "hello",
"displayName": "group_for_restore",
"groupTypes": [
"Unified"
],
"mailEnabled": true,
"mailNickname": "group_for_restore",
"securityEnabled": false,
"visibility": "Public"
}
A request to GET /v1.0/groups/{group-id}/planner/plans does not retrieve any plans.
As far as I know, after creating a Group, a Plan will be created too. On the web interface you can see that this plan is correctly created and shown, but it does not appear in JSON response:
{
"#odata.context": "https://graph.microsoft.com/v1.0/$metadata#Collection(microsoft.graph.plannerPlan)",
"#odata.count": 0,
"value": []
}
After clicking on it in the web interface you can easily get the plan using the request above.
Is it ok to do such magic steps to fetch plan id?
The Web App is provisioning the Group's Plan the first time you attempt to access it. When creating a Group through the API however, you'll need to create a new Plan yourself:
POST https://graph.microsoft.com/v1.0/planner/plans
Content-type: application/json
{
"owner": "group-id",
"title": "title-value"
}
Keep in mind this important note from the documentation:
When creating a new plan, make a group its owner by simply setting the owner property on a plan object.