Octave urlread will not download data due to "Pee - octave

I am using Octave 4.0.0 for windows, and want to download stock prices from a web page that is open to all public. I use the following call:
data = urlread(https://www.netfonds.no/quotes/paperhistory.php?paper=API.A&csv_format=csv)
However, I get the following error message:
urlread: Peer certificate cannot be authenticated with given CA certificates
I have searched internet, including StackOverflow, for this error message, but do not understand the advices given there.
Q1: Is there something lacking on my pc? If so, what do I do?
Q2: Can I change the call somehow to adjust for something lacking on my pc?
Thanks in advance for any help : )

It appears that is a bug in urlread() for certain versions of Octave. For a course I'm doing, we changed this:
responseBody = urlread(submissionUrl, 'post', params);
to
[code, responseBody] = system(sprintf('echo jsonBody=%s | curl -k -X POST -d #- %s', body, submissionUrl));

Although the page is publicly available, the connection is encrypted. For an encrypted connection to make sense, it must use a key that you trust. The typical user does not thinks about whether to trust it, it leaves the job of deciding this to the OS or web browser (who then rely on certificate authorities). I am guessing this is your case.
The error you get is because the website you are accessing uses a key that was certified by something that urlread does not "trust". Ideally, you would have a single list of trusted certificates and all applications would use it. If your web browser trusts it, but the rest of your system does not, you have a configuration issue. Either your web browser is keeping its own list of trusted certificates, or libcurl (the library that urlread uses) is not finding the certificates installed on your system.
This "configuration" will be a directory with several .pem files. The specific certificate required for this website will most likely be named GlobalSign_Root_CA_-_R2.pem.
And it works here:
octave> data = urlread ("https://www.netfonds.no/quotes/paperhistory.php?paper=API.A&csv_format=csv")
data = quote_date,paper,exch,open,high,low,close,volume,value
20150508,API,Amex,0.39,0.40,0.39,0.40,85933,34194
20150507,API,Amex,0.40,0.41,0.38,0.39,163325,64062
...

For Windows a workaround is to use the curl command in the Windows console. This can be called by Octave via the system command. With the curl command you can chose the option '--insecure' that will also allow connection to websites without certificates. Only use this option if you're sure the website is safe.
sURLLink = 'https://www.netfonds.no/quotes/paperhistory.php?paper=API.A&csv_format=csv'
command=['curl --insecure ','"',sURLLink,'"'];
[status, output] =system(command);

Related

Puppeteer pass on access token to wget (from Dropbox link login)

Sometimes is hard to do just simple stuff with puppeteer. I was wondering if it is possible to use puppeteer to login to a dropbox link with password and then use someting like wget or curl to do the rest. I imagine that I would need to read and then pass on on some sort of access token after the login.
Would this be possible?
(yes, I know that using the dropbox API would perhaps be an easier and more correct solution)
I am not familiar how puppeteer stores cookies, but I am sure you can do that, see references below, and e.g. the Puppeteer API docu on cookies.
Here my 50 cents on wget and/or cURL with access control. In both cases it is possible to load cookies which provide you access to dropbox. Assume you have them stored in myAccessCookies.txt, you can reload them and use them e.g. with wget:
wget -qO- --load-cookies myAccessCookies.txt http://www.example.com/replaceWithDropboxLink
Another hint: To my knowledge, Dropbox allows to share direct, temporary links to files which do not require further authentication, see e.g. https://help.dropbox.com/files-folders/share/set-link-permissions - if this is not a security threat, and you can influence your counter-party to use these, everything is easy.
References and further reading
Superuser: How to download dropbox files using wget
https://www.apharmony.com/software-sagacity/2014/10/using-wget-with-cookies/ on how to use wget with cookies
Download Folder including Subfolder via wget from Dropbox link to Unix Server
How to save cookies and load it in another puppeteer session?
https://stackoverflow.com/a/56515357/6189678 - how to store and reuse cookies (suggestion by V.Kostenko)

chrome : unable to read local files by using `-allow-file-access-from-files `

Introduction
Honestry,I'm not native English user,so my English is not better.
If you have any question, ask me by comment.
what I wanna do
As a premise,when you try to access local files by web site link,IE or FireFox can access( by simple setting) ,but Chrome can't access for their security poricy.
I found the ways to do this, like below:
add --allow-file-access-from-files option when you execute chrome.exe
build a local server like nginx or Apache HTTP Server,which meditate to access local data
Use chrome extentionEnable local file links
Due to business restrictions,I should resolve this problem by method 1. but I can't enable loacl data link though I set option correctly.
I googled why I can't,but I couldn't find why it won't work.
(In the first place,I can't even find Google official description of --allow-file-access-from-files.How predecessors find that method ... )
While searching、I found person who have the same question、Google Forum.
I'm convinced that this is not my simple miss,but more deep problem,so I decided to make this question.
what I tried
I rebooted computer,killed all process related to chrome,then change directory where contains chrome executable file,and execute chrome.exe --allow-file-access-from-files
I rebooted computer,killed all process related to chrome 、create shortcut on my Desktop ,which destination is "/path/to/chrome/chrome.exe"--allow-file-access-from-files
After execute chrome,I checked chrome boot option to accesschrome://version/.Both method, I could boot chrome by add --allow-file-access-from-files option,but can't achieve first purpose to access local file access.
what I wanna know
lacking or similar to --allow-file-access-from-filesoption、
(solution similar to method 2 or method 3,like "Give up option solution,and use nginx as local server" is unnecessary)
or other solution
Additionary,if you know below info,please teach me.
- Google official documentation which descriptionsallow-file-access-from-files
( Indended to Oracle official Javadoc of Java.)
Using Tool
Chrome Version : 78.0.3904.87
OS : WIndows 7 SP1

Google Chrome localhost | NET::ERR_CERT_AUTHORITY_INVALID

All of a sudden I seem to have an issue with Google Chrome using localhost.
I'm trying to access any of my development sites (using Ampps) and I get the following error:-
Your connection is not private Attackers might be trying to steal your
information from website.dev (for example, passwords, messages
or credit cards). Learn more NET::ERR_CERT_AUTHORITY_INVALID
When I visit any of the dev sites it is redirecting from http://website.dev to https://website.dev automatically. I'm not having any issue in Safari or Firefox so I don't understand what is going on.
I've tried re-installing Google Chrome, resetting it to the factory default settings...
I think it could be an issue with Keychain Access --> Certificates but wouldn't that mean it wouldn't work in Firefox and Safari if that was the case?
I've spent a while trying to find a solution but so far nothing has worked so I would appreciate some suggestions on how I can fix this. I can't even proceed passed this warning as I don't get the proceed link (insecure) as shown below:-
Navigate to
chrome://flags/#allow-insecure-localhost
and set this to enabled.
After playing around, I came up with one kind of a solution.
First, lets talk about the problem: the cause of this error is that both of us used a .dev domain for our local development. If you go here you will find out that root .dev domain is owned by Google and applying HSTS in Chrome they enforce https-redirect for this domain. Since we use .dev domains, we get redirected to https version and at the same time we don't have any actual certificates installed. So, we see this annoying error. If you go to chrome://net-internals/#hsts you can check your .dev domain and you will actually find out that
static_sts_domain: dev
static_upgrade_mode: FORCE_HTTPS
static_sts_include_subdomains: true
which confirms that HSTS is enforced on *.dev indeed. The policy type is static and, as I understand, it's kind of hard-coded to https-redirect .dev domains.
So, there are at least 2 ways - get and set up an actual certificate somehow or just use another (not .dev) root domain in httpd-vhosts.conf for your local development (also don't forget to update /etc/hosts and launch apache again). I went another root domain route and it solved this issue.
This is really annoying to deal with, but mapping the local website to something other than .dev (I personally use .devo) does work and fixes the problem in chrome. Also, you can add an exception for the page in Mozilla Firefox and not deal with this at all. It's only a problem on Chrome 63+
Best solution is to not use .dev because it owned by Google.
Here you can find an updated list of all claimed TLDs : https://www.rfc-editor.org/rfc/rfc6761
To be safe, choose an unclaimed TLD like .test or .localhost. You can read a useful blog post here : https://iyware.com/dont-use-dev-for-development/
I got same bug, because the CRL file is out of date, and the solution is that update the crl file
you need to add remote site certificate to your local key store
To download certificate from remote site, you will require keytool, open gitbash as admin and run below command to generate the certificate
openssl s_client -showcerts -connect host:port
save the value of above command to from -----BEGIN CERTIFICATE----- to -----END CERTIFICATE----- to the .crt file.
to add certificate to your local key store, run below command
keytool -import -noprompt -trustcacerts -alias name_of_certificate -file "path_of_dot_crt_file" -keystore "C:\Program Files\Java\jdk1.8.0_192\jre\lib\security\cacerts" -storepass changeit
you can also add downloaded certificate to your browser.
In my case, in order to solve the issue so that I can "Proceed to unsafe",
I needed to go to:
chrome://net-internals/#sockets
then click: "Close idle sockets", "Flush socket pools"
Afterwards go to:
chrome://net-internals/#dns
Then click "Clear host cache"
If it still doesn't work and you are a mac user, try to repeat above after removing localhost certificates using Apple's Keychain Access tool

Unable to connect to https://myserver/VaultService

I have just installed Vault on my machine, I intend touse it single user.
When trying to use it for the first time, I get the error message below.
Is there anything I should enable?
I have the correct passwords.
I am using the Admin account.
I have Windows * 64 bits, Vault is also 64 bits.
Thanks to Beth's answer below, now I can at least visualise the following page, but still not be able to follow the links in it.
This is the result I get when I click on the links(Picture below).
Error messages are below, but the formatting is not good.
Most likely causes: The request matched a wildcard mime map.
The request is mapped to the static file handler. If there were different pre-conditions, the request will map to a different handler.
Things you can try: If you want to serve this content as a static file, add an explicit MIME map.
Detailed Error Information: Module StaticFileModule Notification ExecuteRequestHandler Handler StaticFile Error Code 0x80070032
Requested URL http://vrindavana:80/VaultService/Admin/default.aspx Physical Path C:\inetpub\wwwroot\VaultService\Admin\default.aspx
Logon Method Anonymous
Logon User Anonymous
IIS:
I have IIS installed and if I browse it I see what's on the picture below.
First try reaching the Vault server using a browser. Try the URL http://<your Vault server name>/vaultservice/index.html. If you can not reach that page, then change to either the FQDN or the IP address.
If you still cannot reach the index.html page, then try reaching it using a browser right on the server and use localhost. The URL would then be http://localhost/vaultservice/index.html.
Let me know the results of these URLs first and then I'll give you next steps.
Beth
SourceGear Technical Support
Thanks for the update.
The error seems to indicate a problem with the .NET Framework. Here are the next steps you should try.
1) Register the .NET framework with IIS. Run a command-prompt as administrator and then change directories until you get to the highest version of the 4.x .NET framework. It might look something like cd C:\Windows\Microsoft.net\framework64\v4.0.30319, but I'm not sure of the entire path on your machine. Then run the command aspnet_regiis -i.
Then try your web page again.
2) If you then get a 404.2 error, then open your IIS Manager, click on the machine name and then click on ISAPI and CGI restrictions. Make sure the .NET frameworks are set to Allow there.

Couchbase Mobile (Sync Gateway) sample TODOlite application doesn't replicate; complains _facebook doesn't exist

My objective: get https://github.com/couchbaselabs/ToDoLite-iOS syncing with a Couchbase Server and sync gateway on localhost rather than the default demo URL.
I run sync gateway like so: bin/sync_gateway -url http://localhost:8091
And then the only thing I changed in the example is:
-#define kSyncGatewayUrl #"http://demo.mobile.couchbase.com/todolite"
+#define kSyncGatewayUrl #"http://localhost:4984/sync_gateway/"
And when I run
Error: Error Domain=CBLHTTP Code=404 "404 not_found" UserInfo=0x7ff11941fb50 {NSURL=http://localhost:4984/sync_gateway/_facebook, NSLocalizedFailureReason=not_found, NSLocalizedDescription=404 not_found}
How do I fix this?
I solved it. The reason is that I ran sync_gateway without enabling Facebook registration support.
Normally this is done in config.json file. In fact, this configuration file was supplied in ToDoLite all along.
It is crucial that you launch sync_gateway with this configuration file. The README actually states this but in a loose and casual way...
cd ToDoLite-iOS
sync_gateway -url http://localhost:8091 sync-gateway-config.json
NB: I assume above that sync_gateway has been made accessible through $PATH. It's a good idea to do that anyway.
Also, I didn't pay attention to the dbname. So you'll need to replace
#define kSyncGatewayUrl #"http://demo.mobile.couchbase.com/todolite"`
with
#define kSyncGatewayUrl #"http://localhost:4984/todos"
So, what's the complete sequence of steps to get it working?:
If you want to wipe everything on the server, rm -rf Library/Application\ Support/Couchbase and start over. Homebrew cask hides this setting somewhere else where it's hard to reset so a manual install is very recommended.
Install Couchbase Server
Set up login credentials if fresh install; otherwise just login
Create a bucket (a database) with name todos on the cluster. This is the dbname used by TODOLite.
Launch sync gateway. Be sure to pass in the replication URL AND the JSON config file.
bin/sync_gateway -url http://localhost:8091 sync-gateway-config.json; keep sync gateway running
In the TODOLite AppDelegate.m, change kSyncGatewayUrl:
#define kSyncGatewayUrl #"http://localhost:4984/todos". Notice the name of the database is necessary!
(Optionally) Access the administrator interface of the sync gateway by going to http://localhost:4985/_admin/db/sync_gateway/sync. You can set up the sync function here.
In case you're wondering where those port numbers came from, check out
ports Couchbase Server uses
ports Sync Gateway uses
4984 — SG API port
4985 — SG admin server
The default remote sync URL will be defined in different files depending on the version of the project you download (iOS, Android, PhoneGap, and Motion). To find the appropriate string to change simply search through your project for the URL "http://demo.mobile.couchbase.com/todolite" and replace it with the URL of your new sync gateway database.