CRITICAL keystonemiddleware.auth_token [-] Unable to validate token: Failed to fetch token data from identity server - openstack-nova

I am building openstack (Yoga version on Ubuntu 22.04) high availability using ssl configuration. I was able to get other services to work using https (except neutron, cinder and dashboard), but Nova throws the error in /var/log/nova/nova-api.log below:
CRITICAL keystonemiddleware.auth_token [-] Unable to validate token: Failed to fetch token data from identity server: keystonemiddleware.auth_token._exceptions.ServiceError: Failed to fetch token data from identity server
When I run the command below to get token for user "nova" I am able to get a token:
openstack --os-auth-url https://controller:5000/v3 --os-project-domain-name Default --os-user-domain-name Default --os-project-name service --os-username nova --os-password token issue
Controller is the virtual hostname for all controllers (x3). I have all nodes (controller and Compute nodes) configured in /etc/hosts file.
My configuration is as follows:
admin-openrc
export OS_USER_DOMAIN_NAME=default
export OS_PROJECT_NAME=admin
export OS_USERNAME=admin
export OS_PASSWORD=<admin-password>
export OS_AUTH_URL=https://controller:5000/v3
#export OS_SERVICE_TOKEN=
export OS_IDENTITY_API_VERSION=3
export OS_IMAGE_API_VERSION=2
/etc/nova/nova.conf
[keystone_authtoken]
www_authenticate_uri = https://controller:5000
auth_url = https://controller:5000
memcached_servers = 192.168.120.11:11211,192.168.120.12:11211,192.168.120.13:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = nova
password = <nova-passwd>
Your assistance is highly appreciated. Please let me know if you require more info.
Thank you

Thank you for your assistance in advance. I discovered that when I use the config below, everything works fine
frontend glance-api-front
bind 192.168.100.10:9292
default_backend glance-api-back
backend glance-api-back
balance source
option tcpka
option httpchk
# option tcplog
server controller1 192.168.100.11:9292 check inter 2000 rise 2 fall 5
server controller2 192.168.100.12:9292 check backup inter 2000 rise 2 fall 5
server controller3 192.168.100.13:9292 check backup inter 2000 rise 2 fall 5
but when I try to simulate failure of active controller node, I get the error below:
"An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-8d4979ac-c0f0-4900-94b8-814b855c5853)"
not sure how to configure HA to failover to backup controller nodes
Thank you

Related

GCP deployment fails on "Updating service"

I have asp.net core application hosted on GCP App Engine. When I try to deploy the application it fails on last step:
Updating service [name] (this may take several minutes)... ...failed
ERROR: (gcloud.app.deploy) Error Response: [9] An internal error occurred while processing task /app-engine-flex/flex_await_healthy/flex_await_healthy>blablabla.wm.1
The exception stack trace show that service running in background couldn't find MySQL table (that table obviously exists).
my app.yaml file:
service: XXX
runtime: custom
env: flex
automatic_scaling:
max_concurrent_requests: 80
min_num_instances: 1
max_num_instances: 1
resources:
cpu: XXX
memory_gb: XXX
beta_settings:
cloud_sql_instances: "XXX:XXXX:XXXX=tcp:3306"
It looks like the application is deployed properly despite the error. This is the only error and backgroud service desn't throw any exceptions at later point. In fact it works properly and can connect to the database.
My guess was that maybe GCP is checking health while the application is not connected do database. So I tried to add liveness_check and readiness_check to app.yaml and configured dedicated /healthcheck endpoint in my application but it didn't make any change.
Any ideas how to fix it and what might be a cause?
Deploying app with new version fixed the issue

Getting error while sending email from GCP composer airflow

I am trying to configure email but getting following error.
when running task getting following error: -
I am using Python3 with airflow 10.3 versions in GCP Composer need help.
My airflow.cfg
[email]
email_backend = airflow.contrib.utils.sendgrid.send_email
[smtp]
smtp_host = smtp.gmail.com
smtp_starttls = True
smtp_ssl = False
smtp_user = airflow
smtp_port = 587
smtp_password = mypassword
smtp_mail_from = myemail#gamil.com
in my dag file I have created task :-
dag.py file:-
from airflow.operators.email_operator import EmailOperator
email_task=EmailOperator(task_id='email_task',to="email#gamil.com", subject="test", html_content="<h1>Most important heading here</h1>", files=None, cc=None, bcc=None, mime_subtype='mixed', mime_charset='us_ascii', dag=dag)
Getting error:
ERROR - HTTP Error 401: Unauthorized
python_http_client.exceptions.UnauthorizedError: HTTP Error 401: Unauthorize
The error is an Authorization error, so you need to check that you're setting your Sendgrid API Key (or password, if using a smtp-server) correctly.
Based on your airflow.cfg, it seems to me that you are trying to use both Sendgrid and a third-party smtp server.
The email flag determines which one to use. In this case, the flag email_backend = airflow.contrib.utils.sendgrid.send_email is specifying that you are going to use Sendgrid.
To configure Sendgrid as your email server, you need to obtain your SENDGRID_API_KEY and set both the key and SENDGRID_MAIL_FROM as environment variables.
On the other hand, if you want to use another smtp server, you have to change the email flag to email_backend = airflow.utils.email.send_email_smtp. In this case, you need to override the smtp user and password.
With Google Cloud Composer, you must set the:
SENDGRID_MAIL_FROM
and the:
SENDGRID_API_KEY
Composer "ENVIRONMENT VARIABLES"

Connecting to CloudSQL Mysql over ssl from external application

I am trying to get a sample java application to connect to a Mysql gen2 instance I have in GCP. I use SSL and the ip address is whitelisted. I have confirmed connectivity to the instance using the mysql command line and passing in the client-cert.pem, client-key.pem and the server-ca.pem. Now inorder to connect to it from the spring boot java application, I did the following:
created a p12 file from the client cert and key and added it to keystore.jks
created a truststore with the server-ca.pem file.
Added this code in the main before the connection is created:
System.setProperty("javax.net.debug", "all");
System.setProperty("javax.net.ssl.trustStore", TRUST_STORE_PATH);
System.setProperty("javax.net.ssl.trustStorePassword", "fake_password");
System.setProperty("javax.net.ssl.keyStore", KEY_STORE_PATH);
System.setProperty("javax.net.ssl.keyStorePassword", "fake_password");
For the jdbc url, I used : jdbc:mysql://1.1.1.1:3306/sampledb?useSSL=true&requireSSL=true
However I am unable to connect to the instance and see this error from the java ssl debug:
restartedMain, RECV TLSv1.1 ALERT: fatal, unknown_ca
%% Invalidated: [Session-2, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA]
restartedMain, called closeSocket()
restartedMain, handling exception: javax.net.ssl.SSLHandshakeException: Received fatal alert: unknown_ca
restartedMain, called close()
restartedMain, called closeInternal(true)
I also tried to run
openssl verify -CAfile server-ca.pem client-cert.pem`
and got this output:
error 20 at 0 depth lookup:unable to get local issuer certificate`
Any ideas on what I might be doing wrong?

Salesforce connection with proxy not working

I've a big problem getting the connection to Salesforce test environment (https://test.salesforce.com/services/Soap/u/37.0).
I'm using Wildfly 10.0 as webserver and I've also tried to set the proxy in the standalone configuration.
This is my code:
ConnectorConfig config = new ConnectorConfig();
config.setUsername(username);
config.setPassword(password);
config.setAuthEndpoint(endpoint);
config.setServiceEndpoint(endpoint);
config.setProxy(proxy_host, Integer.parseInt(proxy_port));
config.setProxyUsername(proxy_username);
config.setProxyPassword(proxy_password);
this.connection = Connector.newConnection(config);
In my working environment I've no problem with the connection because we have a proxy without authentication and it connects configuring just proxy_host and proxy_port.
In the customer environment, instead, I have the following exception despite I've configured all the proxy parameters correctly (including username and password):
com.sforce.ws.ConnectionException: Failed to send request to https://test.salesforce.com/services/Soap/u/37.0
at com.sforce.ws.transport.SoapConnection.send(SoapConnection.java:121)
at com.sforce.soap.partner.PartnerConnection.login(PartnerConnection.java:1426)
at com.sforce.soap.partner.PartnerConnection.<init>(PartnerConnection.java:406)
at com.sforce.soap.partner.Connector.newConnection(Connector.java:27)
.....
Caused by: java.io.IOException: Unable to tunnel through proxy. Proxy returns "HTTP/1.1 407 authenticationrequired"
at sun.net.www.protocol.http.HttpURLConnection.doTunneling(HttpURLConnection.java:2124)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:183)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1316)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1291)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
at com.sforce.ws.transport.JdkHttpTransport.connectRaw(JdkHttpTransport.java:136)
at com.sforce.ws.transport.JdkHttpTransport.connectLocal(JdkHttpTransport.java:100)
at com.sforce.ws.transport.JdkHttpTransport.connectLocal(JdkHttpTransport.java:95)
at com.sforce.ws.transport.JdkHttpTransport.connect(JdkHttpTransport.java:91)
at com.sforce.ws.transport.SoapConnection.send(SoapConnection.java:95)
The proxy configuration is working properly in telnet for istance.
Any idea?
Thanks in advance!!

Why does my openshift app timeout when I try to access the URL?

I am trying to set up a BrowserQuest server that runs in openshift
I've been following this readme. Everything seems to go fine, I get to the end and run rhc app show bq and get the following output:
bq # http://bq-plantagenet.rhcloud.com/ (uuid: 55e4311189f5cf028d0000fc)
------------------------------------------------------------------------
Domain: plantagenet
Created: 8:18 AM
Gears: 1 (defaults to small)
Git URL: ssh://55e4311189f5cf028d0000fc#bq-plantagenet.rhcloud.com/~/git/bq.git/
SSH: 55e4311189f5cf028d0000fc#bq-plantagenet.rhcloud.com
Deployment: auto (on git push)
nodejs-0.10 (Node.js 0.10)
--------------------------
Gears: Located with smarterclayton-redis-2.6
smarterclayton-redis-2.6 (Redis)
--------------------------------
From: http://cartreflect-claytondev.rhcloud.com/reflect?github=smarterclayton/openshift-redis-cart
Website: https://github.com/smarterclayton/openshift-redis-cart
Gears: Located with nodejs-0.10
But when I try to access http://bq-plantagenet.rhcloud.com:8080/ in a browser, I get:
The connection has timed out
The server at bq-plantagenet.rhcloud.com is taking too long to respond
My questions are what is going wrong and how can I fix it? Many thanks for your consideration in reading through this and any suggestions you might have for resolving it
You need to access http://bq-plantagenet.rhcloud.com, leave off the port 8080, that is the port you listen on internally. You should also try checking your log files (https://developers.openshift.com/en/managing-log-files.html) to see what errors your application is producing.