I am using R to write a program and perform some analyses. The data is being captured by an outside vendor with MongoDB in JSON format. They are providing it to me via a URI on port 443, which they want me to query using cURL. They have authentication in place and self signed SSL.
I can authenticate and dump the data via curl in Windows, however to create a long term sustainable solution it needs to all be done within R.
The vendor says that RCurl "should" work but they aren't providing any support and they basically just don't like the idea of using RMongo and have no comment on it (but if we could make it work that would be awesome, in my opinion).
I have the following packages loaded
- ggplot2
- DBI
- rjson
- RJSONIO (I sometimes don't load this one if I'm using rjson, or visa versa)
- RMongo
- rstudio
- RCurl
The self signed certificate caused issues even with curl, but those were resolved by editing settings in Ruby and then launching a cmd shell with Ruby and using curl that way. I'm not sure if the problems in R are related.
When trying to go the RCurl route I end up with commands/errors like this:
x <- getURL("https://xxx.xx.xxx.xxx:443/db/_authenticate", userpwd="xxxx:xxxxx") }{Error in function (type, msg, asError = TRUE) : couldn't connect to host
and when trying to use RMongo I'm even more clueless...
> mongo <- mongoDbConnect("xxx.xx.xxx.xxx")
username = "xxxx"
password="xxxxxxxxxxxxx"
authenticated <- dbAuthenticate(mongo, username, password)
Feb 25, 2013 4:00:09 PM com.mongodb.DBTCPConnector fetchMaxBsonObjectSize
WARNING: Exception determining maxBSON size using0
java.io.IOException: couldn't connect to [/127.0.0.1:27017] bc:java.net.ConnectException: Connection refused: connect
at com.mongodb.DBPort.open(DBPort.java:224)
at com.mongodb.DBPort.go(DBPort.java:101)
at com.mongodb.DBPort.go(DBPort.java:82)
at com.mongodb.DBPort.findOne(DBPort.java:142)
at com.mongodb.DBPort.runCommand(DBPort.java:151)
at com.mongodb.DBTCPConnector.fetchMaxBsonObjectSize(DBTCPConnector.java:429)
at com.mongodb.DBTCPConnector.checkMaster(DBTCPConnector.java:416)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:193)
at com.mongodb.DBApiLayer$MyCollection._find(DBApiLayer.java:303)
at com.mongodb.DB.command(DB.java:159)
at com.mongodb.DB.command(DB.java:144)
at com.mongodb.DB._doauth(DB.java:503)
at com.mongodb.DB.authenticate(DB.java:440)
at rmongo.RMongo.dbAuthenticate(RMongo.scala:24)
Error in .jcall(rmongo.object#javaMongo, "Z", "dbAuthenticate", username, :
com.mongodb.MongoException$Network: can't call something
Feb 25, 2013 4:00:10 PM com.mongodb.DBPortPool gotError
WARNING: emptying DBPortPool to 127.0.0.1:27017 b/c of error
java.io.IOException: couldn't connect to [/127.0.0.1:27017] bc:java.net.ConnectException: Connection refused: connect
at com.mongodb.DBPort._open(DBPort.java:224)
at com.mongodb.DBPort.go(DBPort.java:101)
at com.mongodb.DBPort.go(DBPort.java:82)
at com.mongodb.DBPort.call(DBPort.java:72)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:202)
at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:303)
at com.mongodb.DB.command(DB.java:159)
at com.mongodb.DB.command(DB.java:144)
at com.mongodb.DB._doauth(DB.java:503)
at com.mongodb.DB.authenticate(DB.java:440)
at rmongo.RMongo.dbAuthenticate(RMongo.scala:24)
any help would be greatly appreciated!
I had an issue in the past with RCurl where I needed to explicitly point it toward the security certificates to get it to work okay. I ended up needing something like this:
out <- postForm("https://url.org/api/",
token="IMATOKEN",
.opts=curlOptions(cainfo="C:/path/aaa.crt"))
I had manually exported the certificate I needed to get that working.
Also, it kind of looks like you should be doing a POST request given that URI, not a GET. Try the postForm() command, maybe?
EDITED TO ADD:
Okay, I think things might be a little more clear if we stepped back a second. Is your goal to get some file from a specific URL (basically, doing a wget but from within R)? Or is your goal to submit a form that subsequently returns the data you need?
IF you are just trying to get something that is behind basic (and also fairly INSECURE) HTTP authentication, you should do two things:
Tell your data provider to use a more secure option
Use the getURL() option as shown (using the www.omegahat.org example you posted about):
Code:
getURL("http://www.omegahat.org/RCurl/testPassword/",.opts=list(userpwd="bob:welcome"))
OR
getURL("http://bob:welcome#www.omegahat.org/RCurl/testPassword/")
Now, if you need to submit a form to get the data, you would generally pass authentication tokens, etc, as parameters (so, in the example above, `token='.
Related
I'm getting a 400 error when testing the examples provided by Bottleneck.
I double checked that I'm using the correct Associate Tag with the right region (do I include the '-20' in the AWS_ASSOCIATE_TAG?), I tried setting my Ubuntu's timezone (running on a VM) to GMT to follow the api_url builder (below) but that didn't help.
I checked the credentials using AWS-Cli and I was able to pull all EC2 instances without a permission error.
I tried reading the HTTPError using e.read, but I'm getting a class bytes object which I can't further investigate.
Is there a way to parse the error into XML or a simple string so that I could understand what the error actually is? Did anyone else encounter this problem and can think of a solution?
AWS Error codes
query = {
'Operation': self.Operation,
'Service': "AWSECommerceService",
'Timestamp': time.strftime(
"%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
'Version': self.Version,
}
I was skimming the docs and so I missed a crucial element - I was using AWS' API credentials, while the ones required belong to Amazon Product API.
To access those, please use this link.
I have been reading the documentation for last 2 days. I'm new to feathersjs.
First issue: any link related to feathersjs is not accessible. Such as this.
Giving the following error:
This page isn’t working
legacy.docs.feathersjs.com redirected you too many times.
Hence I'm unable to traceback to similar types or any types of previously asked threads.
Second issue: It's a great framework to start with Real-time applications. But not all real time application just require alone DB access, their might be access required to something like Amazon S3, Microsoft Azure etc. In my case it's the same and it's more like problem with setting up routes.
I have executed the following commands:
feathers generate app
feathers generate service (service name: upload, REST, DB: Mongoose)
feathers generate authentication (username and password)
I have the setup with me, ready but how do I add another custom service?
The granularity of the service starts in the following way (Use case only for upload):
Conventional way of doing it >> router.post('/upload', (req, res, next) =>{});
Assume, I'm sending a file using data form, and some extra param like { storage: "s3"} in the req.
Postman --> POST (Only) to /upload ---> Process request (isStorageExistsInRequest?) --> Then perform the actual upload respectively to the specific Storage in Req and log the details in local db as well --> Send Response (Success or Failure)
Another thread on stack overflow where you have answered with this:
app.use('/Category/ExclusiveContents/:categoryId', {
create(data, params) {
// do complex stuff here
params.categoryId // the id of the category
data // -> additional data from the POST request
}
});
The solution can viewed in this way as well, since featherjs supports micro service approach, It would be great to have sub-routes like:
/upload_s3 -- uploads to s3
/upload_azure -- uploads to azure and so on.
/upload -- main route which is exposed to users. User requests, process request, call the respective sub-route. (Authentication and Auth to be included as well)
How to solve these types of problems using existing setup of feathersjs?
1) This is a deployment issue, Netlify is looking into it. The current documentation is not on the legacy domain though, what you are looking for can be found at docs.feathersjs.com/api/databases/querying.html.
2) A custom service can be added by running feathers generate service and choosing the custom service option. The functionality can then be implemented in src/services/<service-name>/<service-name>.class.js according to the service interface. For file uploads, an example on how to customize the parameters for feathers-blob (which is used in the file uploading guide) can be found in this issue.
im trying to connect my polymer element to my own elasticsearch-server.
My first problem was, that they are on two different ports, so it had to choose JSONP because of Cross-Domain problems.
So I found out, that I just have to add
http.jsonp.enable: true
in the elasticsearch.yml.
Im starting the server simply by executing the "elasticsearch.bat".
I've indexed data.
If I try to load the API via iron-jsonp-library, im always getting an unexpected token error.
<iron-jsonp-library id="libraryLoader"
library-url="http://127.0.0.1:9200/data/_search?pretty%%callback%%"
notify-event="api-load"
callbackName="jsonpCallback">
</iron-jsonp-library>
In Google Chrome, I'm getting following result from elasticsearch
{"took":2,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":5,"max_score":1.0,"hits":[{"_index":"data","_type":"data","_id":"5","_score":1.0,"_source":{"id":5,"name":"Meyr","manufacturer":"Meyr","weight":1.0,"price":1.0000,"popularity":1,"instock":true,"includes":"Meyr"}},{"_index":"data","_type":"data","_id":"2","_score":1.0,"_source":{"id":2,"name":"Meier","manufacturer":"Meier","weight":1.0,"price":1.0000,"popularity":1,"instock":true,"includes":"Meier"}},{"_index":"data","_type":"data","_id":"4","_score":1.0,"_source":{"id":4,"name":"Mair","manufacturer":"Mair","weight":1.0,"price":1.0000,"popularity":1,"instock":true,"includes":"Mair"}},{"_index":"data","_type":"data","_id":"1","_score":1.0,"_source":{"id":1,"name":"Maier","manufacturer":"Maier","weight":1.0,"price":1.0000,"popularity":1,"instock":true,"includes":"Maier"}},{"_index":"data","_type":"data","_id":"3","_score":1.0,"_source":{"id":3,"name":"Mayr","manufacturer":"Mayr","weight":1.0,"price":1.0000,"popularity":1,"instock":true,"includes":"Mayr"}}]}}
Due to some internet knowledge of JSONP, its not jsonp.
Why is my elasticsearch server, not formatting right?
Are you prior to v2.0? Looks like they removed jsonp in 2.0 (elastic.co/guide/en/elasticsearch/reference/2.2/…).
Alsopretty%%callback%% doesn't look right, the %%callback%% macro usually needs to be the value of name (like onload=%%callback%%). The element replaces %%callback%% with the name of a global function that is generated for you.
import socket
def functions():
print ("hello")
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_address = ('192.168.137.1', 20000)
sock.bind(server_address)
sock.listen(1)
conn, addr = sock.accept()
print ('Connected by', addr)
sock.listen(1)
conn.sendall(b"Welcome to the server")
My question is how to send a function to the client,
I know that conn.sendall(b"Welcome to the server") will data to the client.
Which can be decoded.
I would like to know how to send a function to a client like
conn.sendall(function()) - this does not work
Also I would like to know the function that would allow the client to receive the function I am sending
I have looked on the python website for a function that could do this but I have not found one.
The functionality requested by you is principally impossible unless explicitly coded on client side. If this were possible, one could write a virus which easily spreads into any remote machine. Instead, this is client right responsibility to decode incoming data in any manner.
Considering a case client really wants to receive a code to execute, the issue is that code shall be represented in a form which, at the same time,
is detached from server context and its specifics, and can be serialized and executed at any place
allows secure execution in a kind of sandbox, because a very rare client will allow arbitrary server code to do anything at the client side.
The latter is extremely complex topic; you can read any WWW browser security history - most of closed vulnerabilities are of issues in such sandboxing.
(There are environments when such execution is allowed and desired; e.g. Erlang cookie-based peering cluster. But, in such cluster, side B is also allowed to execute anything at side A.)
You should start with searching an execution environment (high-level virtual machine) which conforms to your needs in functionality and security. For Python, you'd look at multiprocessing module: its implementation of worker pools doesn't pass the code itself, but simplifies passing data for execution requests. Also, passing of arbitrary Python data without functions is covered with marshal and pickle modules.
Hello I have got problem I have configured my Jmeter to test performance between play framework version 2.2 and 2.3 to check the speed of controllers but when I post something (here im creating account) in play console I see that account is creating but its not saving in my database, when I have done this same from another laptop by typing my IP Adress it saved it to database and worked good. I dont know where could be the problem cause configuration is the same.
In Jmaker in thread I have made
HTTP Head Manager and i added here
Name: Content-Type Value: application/json
I have made too HTTP Request
Server Adress: localhost Port:9000 Method: Post Content encoding: UTF-8
Path: api/accounts Implementation: HTTPClient 4 Portocol [http]: http
And in parameters I added
name:(nothing),
Value:{"username":"AccountTest1","password":"test6ccou49","email":"AccountTest1#dev.null"}
and I have turned only Include Equals
I added to Constant Timer and jp#gc Response Times over time from plugin which is making chart
I dont know why when I press Start: chart is building, in play console I see account creating but this is not saving to database.
I will be very thankful for every help.
Create a CSV file like below sample data:
AccountTest1,test6ccou49,AccountTest1#dev.null
AccountTest2,test6ccou50,AccountTest2#dev.null
AccountTest3,test6ccou51,AccountTest3#dev.null
..... so on
Add->Config Element->CSV Data Set Config
keep settings as mentioned below:
Filename: "Give complete file path with file name e.g. c:\folder1\folder2\file.csv"
File Encoding: "Leave Blank"
Variable Names: "user_name,pass_wd,e_mail"
Delimiter: ","
Allow Quoted Data: "false"
Recycle on EOF: "True"
Stop Thread on EOF: "false"
Sharing mode: "All threads"
Now Post body should look like
{"username":"${user_name}","password":"${pass_wd}","email":"${e_mail}"}
Hope this will help.
Try to use "Post Body" in the HTTP Sampler to post below values
{"username":"AccountTest1","password":"test6ccou49","email":"AccountTest1#dev.null"}
Hope this will help.
I have repaired it already, the problem was in Http Header :), I have got question, now I'm creating accounts by requesting
{"username":"AccountTest1","password":"test6ccou49","email":"AccountTest1#dev.null"}
But this create one account, I was testing some solutions on the Internet but them don't work also is here any way to make a lot of accounts in one time to test performance?
My create account function gets json object {xxx:yyy,bbb:ccc}