I'm experiencing a console error on Chrome & Firefox after every few minutes but events are still coming through - this continues on for a while until eventually Chrome logs a 520 error & Firefox 'can't establish a connection to the server' resulting in the eventsource breaking completely.
I tested this locally and no issues at all in either browser. The difference in our production environment is that we are behind an nginx proxy and CloudFlare security.
These are the headers I'm using in the backend:
'Connection': 'keep-alive',
'Content-Type': 'text/event-stream',
'Cache-Control': 'no-cache',
'X-Accel-Buffering': 'no'
This is the code in the backend (node JS)
res.writeHead(200, headers);
res.setTimeout(1000 * 60 * 120)
res.flushHeaders()
req.on('close', () => {
res.end()
})
res.on('timeout', () => res.end())
And these are the nginx configurations I have tried:
proxy_set_header Connection '';
proxy_http_version 1.1;
chunked_transfer_encoding off;
proxy_buffering off;
proxy_cache off;
http2_max_field_size 64k;
http2_max_header_size 512k;
proxy_read_timeout 12h;
Front-end code
let source = new EventSource('/events')
source.onerror = (e) => console.log(e)
source.onmessage = (msg) => console.log(msg)
Chrome Console:
Appreciate any advice. Thanks
Firefox console:
I've went through this myself and it turned out to be caused by CloudFlare proxy. Seems CloudFlare will terminate the connection after some time.
To solve this simply turn off the CloudFlare proxy as described here: https://docs.readme.com/docs/turn-off-cloudflare-proxy
Make sure to create a subdomain so you don't disable proxying for your entire website.
Related
Subsequent HTTP requests GET and POST will sometimes take exactly 60s to get a response and other times requests get a response under 100 ms. How should I try to fix this? I think it has something to do with my nodejs backend and mysql connection and not necessarily nginx reverse proxy and docker containers that these both live in.
This is the nginx error that I am getting fairly frequently:
2022/01/10 16:43:15 [error] 33#33: *985 upstream timed out (110: Connection timed out) while connecting to upstream, client: xxx.xxx.xxx.xxx, server: sub.domain.tld, request: "GET /getTemp?api_key=xxxxxxx HTTP/1.1", upstream: "http://127.0.0.1:8080/getTemp?api_key=xxxxxxx", host: "sub.domain.tld"
I have tried to increase nginx proxy_read_timeout to 120s as suggested on other posts but that isn't the fix here. I have also switched from mysql.createConnection() to mysql.createPool() but that didn't fix it.
Here is my nodejs connection to mysql database:
let pool = mysql.createPool({
connectionLimit: 10,
host: process.env.DB_HOST,
user: process.env.DB_USERNAME,
password: process.env.DB_PASSWORD,
database: process.env.DB_DATABASE,
timezone: 'Europe/Helsinki'
});
Here is snippet of getTemp endpoint:
app.get('/getTemp', (req, res) =>{
console.log(req.path);
let ip = req.header('x-forwarded-for') || req.socket.remoteAddress;
let ua = req.get('user-agent');
console.log(ua);
let {api_key} = req.query;
console.log(req.query);
if(api_key == process.env.API_KEY){
console.log(`Authorized access from IP: '${ip}'`);
// if one of the variables are undefined then send 400 status code to the client
if(api_key == undefined){
return res.sendStatus(400).send({message:"One or more variables are undefined"});
}
sql_query = "SELECT * FROM tempData ORDER BY id DESC";
sql_query = mysql.format(sql_query);
// attempt to query mysql server with the sql_query
pool.query(sql_query, (error, result) =>{
if (error){
// on error log the error to console and send 500 status code to client
console.log(error);
return res.sendStatus(500);
};
// if temp is found we send 200 status code to client
if(result.length > 0){
let currentTemp = result[0].currentTemp;
console.log(`Temp: ${currentTemp}`);
return res.status(200).send({temperature:currentTemp});
}else{
// incase temp is not found then we send 500 status code to client
console.log('Temperature not found');
return res.status(500).send({error:"Temperature not found"});
}
});
}else{
// if client sends invalid api key then send 403 status code to the client
console.log(`Unauthorized access using API key '${api_key}' from IP: '${ip}'`);
return res.sendStatus(403);
}
})
And here is my nginx config:
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
events {
worker_connections 4096;
}
http {
server {
listen 80;
listen [::]:80;
listen 443 default_server ssl;
listen [::]:443 ssl;
ssl_certificate /cert.pem;
ssl_certificate_key /cert.key;
server_name sub.domain.tld;
location / {
proxy_read_timeout 120;
proxy_pass http://localhost:8080;
proxy_set_header X-Real-IP $http_cf_connecting_ip;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-NginX-Proxy true;
}
}
}
I assigned static value to the endpoint so it wont query anything from the database and it also would take a minute to respond so it wasn't the database connection or the endpoint that was causing issues. I disabled rate limiting in the app and I didn't timeout anymore. So the reason for my timeouts was a misconfigured rate limit.
I'm trying to build a nginx-based maintenance mode application, that catches all requests to my applications and returns a predefined response as a 503.
I currently have applications requesting json responses as well as users accessing the pages with their browsers. So in case the request contains the header Accept: application/json, I want to respond with the content of a json file maintenance.json, otherwise with an html file maintenance.html.
My current nginx config looks like this:
server {
listen 8080;
root /usr/share/nginx/maintenance;
server_tokens off;
error_page 503 = #unavailable;
location ^~ / {
return 503;
}
location #unavailable {
set $maintenanceContentType text/html;
set $maintenanceFile /maintenance.html;
if ($http_accept = 'application/json') {
set $maintenanceContentType application/json;
set $maintenanceFile /maintenance.json;
}
default_type $maintenanceContentType;
try_files $uri $maintenanceFile;
}
}
For browser requests to any path this works out fine: "https://maintenance.my-domain.local/some-path". I get the response code and the html content.
But for requests with header Accept: application/json I get a 404 html page. And the nginx log shows [error] 21#21: *1 open() "/usr/share/nginx/maintenance/some-path" failed (2: No such file or directory), client: 10.244.2.65, server: , request: "GET /asd HTTP/1.1", host: "maintenance.my-domain.local".
It seems like json requests are ignoring my location for some reason. When I remove the directive to set the appropriate file and just always return the html this also works for json-requests.
Anyone any idea?
I'm not necessarily looking for a fix for this specific config, but rather for something that fits my needs of responding with different "error pages" based on the Accept header.
Thanks in advance!
EDIT: For some reason this now results in an HTTP 200 instead of a 503. Don't know what I changed..
EDIT2: Managed to fix a part of it:
server {
listen 8080;
root /usr/share/nginx/maintenance;
server_tokens off;
location ^~ / {
if ($http_accept = 'application/json') {
return 503;
}
try_files /maintenance.html =404;
}
error_page 503 /maintenance.json;
location = /maintenance.json {
internal;
}
}
With this config I now get the maintenance page when using the browser and the maintenance json, when defining the header Accept: application/json. The browser response code is 200 now though...
Ok, I found the solution to my problem.
# map the incoming Accept header to a file extension
map $http_accept $accept_ext {
default html;
application/json json;
}
server {
listen 8080;
root /usr/share/nginx/maintenance;
server_tokens off;
# return 503 for all incoming requests
location ^~ / {
return 503;
}
# a 503 redirects to the internal location `#maintenance`. the
# extension of the returned file is decided by the Accept header map
# above (404 in case the file is not found).
error_page 503 #maintenance;
location #maintenance {
internal;
try_files /maintenance.$accept_ext =404;
}
}
Key was the map on the top. I just added application/json there and mapped everything else to the html file by default. But you could add multiple other files/file types there of course.
I'm trying to send some sensor data to MySQL database using Arduino and sim800l GSM module. And it's working perfectly when I use HTTP for the URL and remove the SSL certificate from my website. But when I have enabled the SSL certificate on my website it shows HTTP response code 606 in the serial monitor.
Here is my working code without SSL certificate(this code works perfectly)
#include <SoftwareSerial.h>
SoftwareSerial gprsSerial(3, 2);
void setup()
{
gprsSerial.begin(19200);
Serial.begin(19200);
Serial.println("Config SIM900...");
delay(2000);
Serial.println("Done!...");
gprsSerial.flush();
Serial.flush();
// attach or detach from GPRS service
gprsSerial.println("AT+CGATT?");
delay(100);
toSerial();
// bearer settings
gprsSerial.println("AT+SAPBR=3,1,\"CONTYPE\",\"GPRS\"");
delay(2000);
toSerial();
// bearer settings
gprsSerial.println("AT+SAPBR=3,1,\"APN\",\"myapn\"");
delay(2000);
toSerial();
// bearer settings
gprsSerial.println("AT+SAPBR=0,1");
delay(2000);
gprsSerial.println("AT+SAPBR=1,1");
delay(2000);
toSerial();
}
void loop()
{
// initialize http service
gprsSerial.println("AT+HTTPINIT");
delay(2000);
toSerial();
// set http param value
gprsSerial.println("AT+HTTPPARA=\"URL\",\"http://www.website123456.com/data/index.php?data1=2.88&data2=2.93\"");
delay(20000);
toSerial();
// set http action type 0 = GET, 1 = POST, 2 = HEAD
gprsSerial.println("AT+HTTPACTION=0");
delay(6000);
toSerial();
// read server response
gprsSerial.println("AT+HTTPREAD");
delay(10000);
toSerial();
gprsSerial.println("");
gprsSerial.println("AT+HTTPTERM");
toSerial();
delay(300);
gprsSerial.println("");
delay(10000);
}
void toSerial()
{
while(gprsSerial.available()!=0)
{
Serial.write(gprsSerial.read());
}
}
Here is the output when I replace the URL with https and enable my website ssl. (as you can see it shows http response code 606 "Not Acceptable
The user's agent was contacted successfully but some aspects of the session description such as the requested media, bandwidth, or addressing style were not acceptable"
23:16:48.974 -> Config SIM800L...
23:16:50.946 -> Done!...
23:16:51.079 -> AT+CGATT?
23:16:51.079 -> +CGATT: 1
23:16:51.079 ->
23:16:51.079 -> OK
23:16:53.107 -> AT+SAPBR=3,1,"CONTYPE","GPRS"
23:16:53.107 -> OK
23:16:55.100 -> AT+SAPBR=3,1,"APN","dialogbb"
23:16:55.133 -> OK
23:16:59.122 -> AT+SAPBR=0,1
23:16:59.155 -> OK
23:16:59.155 -> AT+SAPBR=1,1
23:17:01.151 -> OK
23:17:03.185 -> AT+HTTPPARA="URL","https://www.mywebsitttte.com/index.php?dAT+HTTPSSL=1
23:17:08.207 -> OK
23:17:14.221 -> AT+HTTPACTION=0
23:17:14.255 -> OK
23:17:14.255 ->
23:17:14.255 -> +HTTPACTION: 0,606,0
23:17:24.266 -> AT+HTTPREAD
23:17:24.266 -> OK
23:17:36.595 ->
AT+HTTPTERM
23:17:36.595 -> OK
So what are the modifications I should do to this code, to be able to upload data to the website with SSL certificate? I have seen people say I need to enter to SSL mode using this code AT+HTTPSSL=1 where should I put this code. Thanks
I have figured out it finally, It happens because this sim800 only supports TLS 1.0. But it's a deprecated version and many websites and host services have disabled it. You can check your server SSL version from here cdn77.com/tls-test
Check your SIM800 firmware first, I had old firmware on my 808 module that had no SSL and I only bought it a few weeks ago. you can check using this AT command:
AT+HTTPSSL=?
If you get error then check and update the firmware.
Otherwise if working this is the order I use:
client.println("AT+HTTPINIT");
getresponse();
delay(1000);
client.println("AT+HTTPSSL=1"); // set SSL for HTTPS
getresponse();
delay(1000);
client.println("AT+HTTPPARA=\"CID\",1");
getresponse();
delay(1000);
I can reliably HTTPS connect to my Firebase database.
I was trying to incorporate IDM (Docker) latest, and pep-proxy (git example running with node server).
When I started pep-proxy, everything was working as intended.
I've got the following messages:
INFO: Server - Starting PEP proxy in port 80. IdM authentication...
Server - Success authenticating PEP proxy. Proxy Auth-token: d9badf48-16fa-423d-884c-a3e155578791
Now a problem happens. When I enter the wrong token I get this error.
ERROR: IDM-Client - Error validating token.
Proxy not authorized in keystone. Keystone authentication ...
ERROR: Server - Caught exception:
SyntaxError: Unexpected token u in JSON at position 0
As far as I understand I am expecting some return like invalid token, etc.. instead I get this error in pep-proxy and my curl command show->(52) Empty reply from server.
My config.json of pep-proxy:
var config = {};
// Used only if https is disabled
config.pep_port = 80;
// Set this var to undefined if you don't want the server to listen on HTTPS
config.https = {
enabled: false,
cert_file: 'cert/cert.crt',
key_file: 'cert/key.key',
port: 443
};
config.idm = {
host: 'localhost',
port: 3000,
ssl: false
}
config.app = {
host: 'www.google.es',
port: '80',
ssl: false // Use true if the app server listens in https
}
// Credentials obtained when registering PEP Proxy in app_id in Account Portal
config.pep = {
app_id: 'xxxxxx',
username: 'xxxxxx',
password: 'xxxxxx',
trusted_apps : []
}
// in seconds
config.cache_time = 300;
// if enabled PEP checks permissions with AuthZForce GE.
// only compatible with oauth2 tokens engine
//
// you can use custom policy checks by including programatic scripts
// in policies folder. An script template is included there
config.azf = {
enabled: true,
protocol: 'http',
host: 'localhost',
port: 8080,
custom_policy: undefined // use undefined to default policy checks (HTTP verb + path).
};
// list of paths that will not check authentication/authorization
// example: ['/public/*', '/static/css/']
config.public_paths = [];
config.magic_key = 'undefined';
module.exports = config;
IDM logs:
fiware-idm_1 | GET
/user?access_token=7cb25729577c2e01dc337314dcd912ec981dc49b 401 4.445 ms - 116
fiware-idm_1 | Executing (default): SELECT email, 'user' as Source FROM
user WHERE email='pep_proxy_edf60435-7de7-4875-85a9-cf68b8838b8c'
fiware-idm_1 | UNION ALL
fiware-idm_1 | SELECT id, 'pep_proxy' as Source FROM
pep_proxy WHERE id='pep_proxy_edf60435-7de7-4875-85a9-cf68b8838b8c';
fiware-idm_1 | Executing (default): SELECT `id`, `password`,
`oauth_client_id` FROM `pep_proxy` AS `PepProxy` WHERE `PepProxy`.`id` =
'pep_proxy_edf60435-7de7-4875-85a9-cf68b8838b8c';
fiware-idm_1 | Executing (default): INSERT INTO `auth_token`
(`access_token`,`expires`,`valid`,`pep_proxy_id`) VALUES ('a0d54a6f-
8461-4000-bb80-5fb60193bcb4','2018-05-04
11:45:21',true,'pep_proxy_edf60435-7de7-4875-85a9-cf68b8838b8c');
fiware-idm_1 | POST /v3/auth/tokens 201 13.733 ms - 74
The error "SyntaxError: Unexpected token u in JSON at position 0", as stated here, is probably due to some place at the code where JSON.parse is called with an undefined parameter. You are getting this message because the error was not properly treated and the exception is being thrown (exception not treated).
In the Wilma PEP Proxy github, we can see the latest changes at the code and we can guess/infer where this error comes from.
I think you can open an issue at github.
I am developing Android mobile application on Windows(OS). I want to send data to couchbase server.
I am making mistake in URL for sync gateway. I am running services on my machine as well. I have already set up couchbase server.
My config.json file:
{
"log": ["HTTP+"],
"adminInterface": "127.0.0.1:4985",
"interface": "0.0.0.0:4984",
"databases": {
"db": {
"server": "http://localhost:8091",
"bucket": "mobile_data",
"users": {
"GUEST": {"disabled": false, "admin_channels": ["*"] }
}
}
}
}
My Android App code:
private void initCouchbase() {
// Create replicators to push & pull changes to & from Sync Gateway.
URL url = null;
try {
url = new URL("http://127.0.0.1:4984/db/");
} catch (MalformedURLException e) {
e.printStackTrace();
}
Replication push = database.createPushReplication(url);
Replication pull = database.createPullReplication(url);
push.setContinuous(true);
pull.setContinuous(true);
// Start replicators
push.start();
pull.start();
}
I am using my mobile for testing.
I am getting an error in connection with the URL.
Logcat:
01-28 19:21:23.851 28672-28718/com.example.yumnaasim.couchbaseapp W/RemoteRequest: com.couchbase.lite.replicator.RemoteRequest {GET, http://127.0.0.1:4984/db/_local/3259f53711e089457eaed7b5c61d82403d1a98e4}: executeRequest() Exception: java.net.ConnectException: Failed to connect to /127.0.0.1:4984. url: http://127.0.0.1:4984/db/_local/3259f53711e089457eaed7b5c61d82403d1a98e4
java.net.ConnectException: Failed to connect to /127.0.0.1:4984
at okhttp3.internal.io.RealConnection.connectSocket(RealConnection.java:187)
at okhttp3.internal.io.RealConnection.buildConnection(RealConnection.java:170)
at okhttp3.internal.io.RealConnection.connect(RealConnection.java:111)
at okhttp3.internal.http.StreamAllocation.findConnection(StreamAllocation.java:187)
at okhttp3.internal.http.StreamAllocation.findHealthyConnection(StreamAllocation.java:123)
at okhttp3.internal.http.StreamAllocation.newStream(StreamAllocation.java:93)
at okhttp3.internal.http.HttpEngine.connect(HttpEngine.java:296)
at okhttp3.internal.http.HttpEngine.sendRequest(HttpEngine.java:248)
at okhttp3.RealCall.getResponse(RealCall.java:243)
at okhttp3.RealCall$ApplicationInterceptorChain.proceed(RealCall.java:201)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:163)
at okhttp3.RealCall.execute(RealCall.java:57)
at com.couchbase.lite.replicator.RemoteRequest.executeRequest(RemoteRequest.java:261)
at com.couchbase.lite.replicator.RemoteRequest.execute(RemoteRequest.java:165)
at com.couchbase.lite.replicator.RemoteRequest.run(RemoteRequest.java:105)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:423)
at java.util.concurrent.FutureTask.run(FutureTask.java:237)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:269)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1113)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:588)
at java.lang.Thread.run(Thread.java:818)
01-28 19:21:23.853 28672-28719/com.example.yumnaasim.couchbaseapp W/RemoteRequest: com.couchbase.lite.replicator.RemoteRequest {GET, http://127.0.0.1:4984/db/_local/3410a851b84016993416af638a20280537838364}: executeRequest() Exception: java.net.ConnectException: Failed to connect to /127.0.0.1:4984. url: http://127.0.0.1:4984/db/_local/3410a851b84016993416af638a20280537838364
java.net.ConnectException: Failed to connect to /127.0.0.1:4984
at okhttp3.internal.io.RealConnection.connectSocket(RealConnection.java:187)
at okhttp3.internal.io.RealConnection.buildConnection(RealConnection.java:170)
at okhttp3.internal.io.RealConnection.connect(RealConnection.java:111)
at okhttp3.internal.http.StreamAllocation.findConnection(StreamAllocation.java:187)
at okhttp3.internal.http.StreamAllocation.findHealthyConnection(StreamAllocation.java:123)
at okhttp3.internal.http.StreamAllocation.newStream(StreamAllocation.java:93)
at okhttp3.internal.http.HttpEngine.connect(HttpEngine.java:296)
at okhttp3.internal.http.HttpEngine.sendRequest(HttpEngine.java:248)
at okhttp3.RealCall.getResponse(RealCall.java:243)
at okhttp3.RealCall$ApplicationInterceptorChain.proceed(RealCall.java:201)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:163)
at okhttp3.RealCall.execute(RealCall.java:57)
at com.couchbase.lite.replicator.RemoteRequest.executeRequest(RemoteRequest.java:261)
at com.couchbase.lite.replicator.RemoteRequest.execute(RemoteRequest.java:165)
at com.couchbase.lite.replicator.RemoteRequest.run(RemoteRequest.java:105)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:423)
at java.util.concurrent.FutureTask.run(FutureTask.java:237)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:269)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1113)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:588)
at java.lang.Thread.run(Thread.java:818)
01-28 19:21:25.877 28450-28450/? I/Finsky: [1] com.google.android.finsky.scheduler.ak.a(146): onJobSchedulerWakeup
The problem is with your Android app URL. You have http://127.0.0.1:4984/db/. This tries to connect to the Android device itself.
You need to have the external IP address of your server there instead. You configured Sync Gateway correctly with "interface": "0.0.0.0:4984". Beware, though, that you have guest access on ("GUEST": {"disabled": false, "admin_channels": ["*"] }, so you have no security as it stands.
You can reach on localhost with adb reverse port forwarding:
adb reverse tcp:4984 tcp:4984
After port forwarding, that is certain that you will encounter ssl problem. You have three options to solve this issue:
1- Add network security config for permit cleartext on localhost
Easy to apply: https://developer.android.com/training/articles/security-config
Maybe security vulnerability
2- Create and install custom ssl and push emulator/physical devices via adb
Every developer should install certificate to their machine, emulator
and physical devices(Maybe scripting needed)
All service projects which used by Android should serve with ssl
Complexity
3- Permit cleartext globally while debugging
Quick to apply
You shouldn't forget in the prod environment.
Adb Extra:
If you have connected multiple devices, you can select device "-s {deviceId}" parameter
For removing port forwarding, adb forward -remove tcp: {specifiedPort} or -remove-all