I'm getting a 400 error when testing the examples provided by Bottleneck.
I double checked that I'm using the correct Associate Tag with the right region (do I include the '-20' in the AWS_ASSOCIATE_TAG?), I tried setting my Ubuntu's timezone (running on a VM) to GMT to follow the api_url builder (below) but that didn't help.
I checked the credentials using AWS-Cli and I was able to pull all EC2 instances without a permission error.
I tried reading the HTTPError using e.read, but I'm getting a class bytes object which I can't further investigate.
Is there a way to parse the error into XML or a simple string so that I could understand what the error actually is? Did anyone else encounter this problem and can think of a solution?
AWS Error codes
query = {
'Operation': self.Operation,
'Service': "AWSECommerceService",
'Timestamp': time.strftime(
"%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
'Version': self.Version,
}
I was skimming the docs and so I missed a crucial element - I was using AWS' API credentials, while the ones required belong to Amazon Product API.
To access those, please use this link.
Related
I have an node application deployed in GCP.
The application includes code to access ressources in AWS-cloud.
For this purpose it uses the aws-SDK with ChainableTemporaryCredentials.
The relevant code lines are...
const credentials = new ChainableTemporaryCredentials({
params: {
RoleArn: `arn:aws:iam::${this.accountId}:role/${this.targetRoleName}`,
RoleSessionName: this.targetRoleName,
},
masterCredentials: new WebIdentityCredentials({
RoleArn: `arn:aws:iam::${this.proxyAccountId}:role/${this.proxyRoleName}`,
RoleSessionName: this.proxyRoleName,
WebIdentityToken: token,
}),
})
await credentials.getPromise()
The WebIdentityToken was received from google and looks good.
At AWS-side I created an proxy-role (the line from masterCredentials RoleArn).
However at runtime I get the error:
Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1
I do not understand this error. Because my application runs in GCP and I use temporary credentials I do not understand why I should use aws-credentials in form of an credentials file or environment variables like AWS_ACCESS_KEY_ID or AWS_SECRET_ACCESS_KEY. I thought the idea to use ChainableTemporaryCredentials is NOT to have direct aws-credentials. Right?
You can see the public code at:
https://github.com/cloud-carbon-footprint/cloud-carbon-footprint/blob/trunk/packages/aws/src/application/GCPCredentials.ts
and documentation regarding env-variables at:
https://www.cloudcarbonfootprint.org/docs/configurations-glossary/
Any help which leads to understanding of this error message is welcome.
Thomas
Solved it. "Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1 was totally misleading." In reality it was a problem with the field-names in the GCP-JWT-token und the policy in aws. See https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_iam-condition-keys.html#ck_aud
I'm trying to make a community connector to connect my database in BigQuery to data studio with the service account that I hooked up as the Owner/DataViewer/JobUser of the BigQuery project. I know that the service account works when connecting to BigQuery because I've tested it elsewhere. I copied from the connector code from this tutorial (https://developers.google.com/datastudio/solution/blocks/using-service-accounts) almost exactly, replacing the SQL string with my query and adding some different query parameters. I also stored the service account's credentials in my script properties by pasting the json object and storing it like:
var service_account_creds_obj = {
"type": "service_account",
"project_id": ...
...
}
scriptProperties.setProperty('SERVICE_ACCOUNT_CREDS', JSON.stringify(service_account_creds_obj));
However, I always get stuck in the flow when my getData function calls getOauthService().getAccessToken(), which doesn't ever successfully return. When I create a report using the connector, I get this error: "Access not granted or expired." I can't find the documentation for getAccessToken and I'm having trouble understanding why it won't terminate. I can see that it doesn't return because a console.log immediately before that line displays but it never gets to the log on the next line. Then my try-catch block catches the error that I'm seeing. Note that my getOauthService function is exactly the same as the one from the documentation/tutorial example, except that I've played around with the input text in the call to createService. That input text shouldn't really matter though right?
Please, I've been trying to debug this for hours, but the documentation on this is pretty horrible, and it's really hard to debug since the flow of the code is handled in the background and stackdriver logging is really buggy.
I figured out my problem. The documentation posted above said to set the OAuth2 scope to https://www.googleapis.com/auth/bigquery.readonly. However, I naively included
"oauthScopes": ["https://www.googleapis.com/auth/bigquery.readonly"]
in my manifest file. Meanwhile, the code I copied over from the documentation already included this line:
.setScope(['https://www.googleapis.com/auth/bigquery.readonly']);
So I'm not sure exactly why this caused a problem. But it must have prevented the OAuth2.createService function from properly getting set.
im trying to connect my polymer element to my own elasticsearch-server.
My first problem was, that they are on two different ports, so it had to choose JSONP because of Cross-Domain problems.
So I found out, that I just have to add
http.jsonp.enable: true
in the elasticsearch.yml.
Im starting the server simply by executing the "elasticsearch.bat".
I've indexed data.
If I try to load the API via iron-jsonp-library, im always getting an unexpected token error.
<iron-jsonp-library id="libraryLoader"
library-url="http://127.0.0.1:9200/data/_search?pretty%%callback%%"
notify-event="api-load"
callbackName="jsonpCallback">
</iron-jsonp-library>
In Google Chrome, I'm getting following result from elasticsearch
{"took":2,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":5,"max_score":1.0,"hits":[{"_index":"data","_type":"data","_id":"5","_score":1.0,"_source":{"id":5,"name":"Meyr","manufacturer":"Meyr","weight":1.0,"price":1.0000,"popularity":1,"instock":true,"includes":"Meyr"}},{"_index":"data","_type":"data","_id":"2","_score":1.0,"_source":{"id":2,"name":"Meier","manufacturer":"Meier","weight":1.0,"price":1.0000,"popularity":1,"instock":true,"includes":"Meier"}},{"_index":"data","_type":"data","_id":"4","_score":1.0,"_source":{"id":4,"name":"Mair","manufacturer":"Mair","weight":1.0,"price":1.0000,"popularity":1,"instock":true,"includes":"Mair"}},{"_index":"data","_type":"data","_id":"1","_score":1.0,"_source":{"id":1,"name":"Maier","manufacturer":"Maier","weight":1.0,"price":1.0000,"popularity":1,"instock":true,"includes":"Maier"}},{"_index":"data","_type":"data","_id":"3","_score":1.0,"_source":{"id":3,"name":"Mayr","manufacturer":"Mayr","weight":1.0,"price":1.0000,"popularity":1,"instock":true,"includes":"Mayr"}}]}}
Due to some internet knowledge of JSONP, its not jsonp.
Why is my elasticsearch server, not formatting right?
Are you prior to v2.0? Looks like they removed jsonp in 2.0 (elastic.co/guide/en/elasticsearch/reference/2.2/…).
Alsopretty%%callback%% doesn't look right, the %%callback%% macro usually needs to be the value of name (like onload=%%callback%%). The element replaces %%callback%% with the name of a global function that is generated for you.
I was wondering if Wirecloud offers complete support for object storage with FI-WARE Testbed instead of Fi-lab. I have successfully integrated Wirecloud with Testbed and have developed a set of widgets that are able to upload/download files to specific containers in Fi-lab with success. However, the same widgets do not seem to work in Fi-lab, as i get an error 500 when trying to retrieve the auth tokens (also with the well known object-storage-test widget) containing the following response:
SyntaxError: Unexpected token
at Object.parse (native)
at create (/home/fiware/fi-ware-keystone-proxy/controllers/Token.js:343:25)
at callbacks (/home/fiware/fi-ware-keystone-proxy/node_modules/express/lib/router/index.js:164:37)
at param (/home/fiware/fi-ware-keystone-proxy/node_modules/express/lib/router/index.js:138:11)
at pass (/home/fiware/fi-ware-keystone-proxy/node_modules/express/lib/router/index.js:145:5)
at Router._dispatch (/home/fiware/fi-ware-keystone-proxy/node_modules/express/lib/router/index.js:173:5)
at Object.router (/home/fiware/fi-ware-keystone-proxy/node_modules/express/lib/router/index.js:33:10)
at next (/home/fiware/fi-ware-keystone-proxy/node_modules/express/node_modules/connect/lib/proto.js:195:15)
at Object.handle (/home/fiware/fi-ware-keystone-proxy/server.js:31:5)
at next (/home/fiware/fi-ware-keystone-proxy/node_modules/express/node_modules/connect/lib/proto.js:195:15)
I noticed that the token provided in the beggining (to start the transaction) is
token: Object
id: "%fiware_token%"
Any idea regarding what might have gone wrong?
The WireCloud instance available at FI-WARE's testbed is always the latest stable version while the FI-LAB instance is currently outdated, we're working on updating it as soon as possible. One of the things that changes between those versions is the Object Storage API, so sorry for the inconvenience as you will not be able to use widgets/operators using the Object Storage in both environments.
Anyway, the response you were obtaining seems to indicate the object storage instance you are accessing is not working properly, so you will need to send an email to one of the available mail lists for getting help (fiware-testbed-help or fiware-lab-help) telling what is happening to you (remember to include your account information as there are several object storage nodes and ones can be up and the others down).
Regarding the strange request body:
"token": {
id: "%fiware_token%"
}
This behaviour is normal, as the WireCloud client code has no direct access to the IdM token of the user. It's the WireCloud's proxy which replaces the %fiware_token% pattern with the correct value.
I am using R to write a program and perform some analyses. The data is being captured by an outside vendor with MongoDB in JSON format. They are providing it to me via a URI on port 443, which they want me to query using cURL. They have authentication in place and self signed SSL.
I can authenticate and dump the data via curl in Windows, however to create a long term sustainable solution it needs to all be done within R.
The vendor says that RCurl "should" work but they aren't providing any support and they basically just don't like the idea of using RMongo and have no comment on it (but if we could make it work that would be awesome, in my opinion).
I have the following packages loaded
- ggplot2
- DBI
- rjson
- RJSONIO (I sometimes don't load this one if I'm using rjson, or visa versa)
- RMongo
- rstudio
- RCurl
The self signed certificate caused issues even with curl, but those were resolved by editing settings in Ruby and then launching a cmd shell with Ruby and using curl that way. I'm not sure if the problems in R are related.
When trying to go the RCurl route I end up with commands/errors like this:
x <- getURL("https://xxx.xx.xxx.xxx:443/db/_authenticate", userpwd="xxxx:xxxxx") }{Error in function (type, msg, asError = TRUE) : couldn't connect to host
and when trying to use RMongo I'm even more clueless...
> mongo <- mongoDbConnect("xxx.xx.xxx.xxx")
username = "xxxx"
password="xxxxxxxxxxxxx"
authenticated <- dbAuthenticate(mongo, username, password)
Feb 25, 2013 4:00:09 PM com.mongodb.DBTCPConnector fetchMaxBsonObjectSize
WARNING: Exception determining maxBSON size using0
java.io.IOException: couldn't connect to [/127.0.0.1:27017] bc:java.net.ConnectException: Connection refused: connect
at com.mongodb.DBPort.open(DBPort.java:224)
at com.mongodb.DBPort.go(DBPort.java:101)
at com.mongodb.DBPort.go(DBPort.java:82)
at com.mongodb.DBPort.findOne(DBPort.java:142)
at com.mongodb.DBPort.runCommand(DBPort.java:151)
at com.mongodb.DBTCPConnector.fetchMaxBsonObjectSize(DBTCPConnector.java:429)
at com.mongodb.DBTCPConnector.checkMaster(DBTCPConnector.java:416)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:193)
at com.mongodb.DBApiLayer$MyCollection._find(DBApiLayer.java:303)
at com.mongodb.DB.command(DB.java:159)
at com.mongodb.DB.command(DB.java:144)
at com.mongodb.DB._doauth(DB.java:503)
at com.mongodb.DB.authenticate(DB.java:440)
at rmongo.RMongo.dbAuthenticate(RMongo.scala:24)
Error in .jcall(rmongo.object#javaMongo, "Z", "dbAuthenticate", username, :
com.mongodb.MongoException$Network: can't call something
Feb 25, 2013 4:00:10 PM com.mongodb.DBPortPool gotError
WARNING: emptying DBPortPool to 127.0.0.1:27017 b/c of error
java.io.IOException: couldn't connect to [/127.0.0.1:27017] bc:java.net.ConnectException: Connection refused: connect
at com.mongodb.DBPort._open(DBPort.java:224)
at com.mongodb.DBPort.go(DBPort.java:101)
at com.mongodb.DBPort.go(DBPort.java:82)
at com.mongodb.DBPort.call(DBPort.java:72)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:202)
at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:303)
at com.mongodb.DB.command(DB.java:159)
at com.mongodb.DB.command(DB.java:144)
at com.mongodb.DB._doauth(DB.java:503)
at com.mongodb.DB.authenticate(DB.java:440)
at rmongo.RMongo.dbAuthenticate(RMongo.scala:24)
any help would be greatly appreciated!
I had an issue in the past with RCurl where I needed to explicitly point it toward the security certificates to get it to work okay. I ended up needing something like this:
out <- postForm("https://url.org/api/",
token="IMATOKEN",
.opts=curlOptions(cainfo="C:/path/aaa.crt"))
I had manually exported the certificate I needed to get that working.
Also, it kind of looks like you should be doing a POST request given that URI, not a GET. Try the postForm() command, maybe?
EDITED TO ADD:
Okay, I think things might be a little more clear if we stepped back a second. Is your goal to get some file from a specific URL (basically, doing a wget but from within R)? Or is your goal to submit a form that subsequently returns the data you need?
IF you are just trying to get something that is behind basic (and also fairly INSECURE) HTTP authentication, you should do two things:
Tell your data provider to use a more secure option
Use the getURL() option as shown (using the www.omegahat.org example you posted about):
Code:
getURL("http://www.omegahat.org/RCurl/testPassword/",.opts=list(userpwd="bob:welcome"))
OR
getURL("http://bob:welcome#www.omegahat.org/RCurl/testPassword/")
Now, if you need to submit a form to get the data, you would generally pass authentication tokens, etc, as parameters (so, in the example above, `token='.