I have deployed my service running on spring boot in openshift. The spring boot service initializes good and we see the below logs which is good.
2020-05-06 19:32:33.930 INFO 1 --- [ main] c.a.r.l.MyApplication : Started MyApplication in 44.227 seconds (JVM running for 67.578)
2020-05-06 19:32:38.706 INFO 1 --- [nio-8198-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring DispatcherServlet 'dispatcherServlet'
2020-05-06 19:32:38.709 INFO 1 --- [nio-8198-exec-1] o.s.web.servlet.DispatcherServlet : Initializing Servlet 'dispatcherServlet'
2020-05-06 19:32:38.802 INFO 1 --- [nio-8198-exec-1] o.s.web.servlet.DispatcherServlet : Completed initialization in 93 ms
However the containers ready status shows 0/1 and after 5 mins i see this warning and the pod restarts.
The container has been running for more than five minutes and has not passed its readiness check
and i see
Readiness probe failed: HTTP probe failed with statuscode: 404
What could be wrong?
Readiness probe failed: HTTP probe failed with statuscode: 404
This indicates that the URL that you specified for your readinessProbe does not exist (HTTP 404). So check how your readinessProbe is defined (which URI is called) and make sure there is a valid response.
For Spring Boot, there is an actuator available for Health endpoints, see the following documentation: https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-features.html
Here is an example:
[..]
spec:
containers:
- args:
image: k8s.gcr.io/readiness
readinessProbe:
httpGet:
path: /healthz
In the example above, make sure that the /healthz endpoint exists.
You can find more information on how to configure Readiness Probes in the OpenShift documentation.
Related
how should I check the root cause for this issue, using Basic Authentication for login.
2022-07-13 17:03:10.180 DEBUG 1 --- [nio-9999-exec-5] o.s.s.w.a.www.BasicAuthenticationFilter : Basic Authentication Authorization header found for user 'superadmin'
2022-07-13 17:03:10.180 DEBUG 1 --- [nio-9999-exec-5] o.s.s.authentication.ProviderManager : Authentication attempt using org.springframework.security.authentication.dao.DaoAuthenticationProvider
2022-07-13 17:03:10.278 DEBUG 1 --- [nio-9999-exec-5] o.s.s.a.dao.DaoAuthenticationProvider : User 'superadmin' not found
2022-07-13 17:03:10.279 DEBUG 1 --- [nio-9999-exec-5] o.s.s.w.a.www.BasicAuthenticationFilter : Authentication request for failed: org.springframework.security.authentication.BadCredentialsException: Bad credentials
2022-07-13 17:03:10.279 DEBUG 1 --- [nio-9999-exec-5] s.s.w.c.SecurityContextPersistenceFilter : SecurityContextHolder now cleared, as request processing completed
On a Windows Server 2012 machine, I have a local DTC and a clustered DTC, as you can see here:
Here you can see the clustered DTC in the Failover Cluster Manager:
I have enabled WS-AT with the following command on the clustered DTC:
wsatconfig -network:enable -endpointCert:7c6361568413852afb471d5f8b92604cdde530dd -accountsCerts:3bcf068b0b984d2af9d2efa03e8a489c8483ba11 -virtualServer:ftsappdev -restart
For the endpointCert, I gave the thumbprint of the certificate for ftsappdev (the cluster role), and for accountscerts, I gave the thumbprint of the certificate of a JBOSS server.
I also have configured WS-AT for the local DTC through the WS-AT tab in Component Services:
In Failover Cluster Manager, when I take the clustered DTC resource offline and then online, I get the following entry in the Eventviewer/Application:
The MSDTC WS-AT protocol failed at the beginning of recovery. As a result, WS-AT functionality will be disabled.
Protocol ID: c05b9cad-ab24-4bb3-9440-3548fa7b4b1b
Protocol Name: WS-AtomicTransaction 1.1
Exception: Microsoft.Transactions.Bridge.PluggableProtocolException: A channel factory could not be opened. ---> Microsoft.Transactions.Wsat.Messaging.MessagingInitializationException: A channel factory could not be opened. ---> System.ServiceModel.AddressAccessDeniedException: HTTP could not register URL https://+:2372/WsatService/. Your process does not have access rights to this namespace (see http://go.microsoft.com/fwlink/?LinkId=70353 for details). ---> System.Net.HttpListenerException: Access is denied
at System.Net.HttpListener.AddAllPrefixes()
at System.Net.HttpListener.Start()
at System.ServiceModel.Channels.SharedHttpTransportManager.OnOpen()
--- End of inner exception stack trace ---
at System.ServiceModel.Channels.SharedHttpTransportManager.OnOpen()
at System.ServiceModel.Channels.TransportManager.Open(TransportChannelListener channelListener)
at System.ServiceModel.Channels.TransportManagerContainer.Open(SelectTransportManagersCallback selectTransportManagerCallback)
at System.ServiceModel.Channels.TransportChannelListener.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.HttpChannelListener`1.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.LayeredChannelListener`1.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.DatagramChannelDemuxer`2.OnOuterListenerOpen(ChannelDemuxerFilter filter, IChannelListener listener, TimeSpan timeout)
at System.ServiceModel.Channels.SingletonChannelListener`3.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.InternalDuplexChannelFactory.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelFactory.TypedServiceChannelFactory`1.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.ChannelFactory.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at Microsoft.Transactions.Wsat.Messaging.CoordinationService.OpenChannelFactory[T](ChannelFactory`1 cf)
--- End of inner exception stack trace ---
at Microsoft.Transactions.Wsat.Messaging.CoordinationService.OpenChannelFactory[T](ChannelFactory`1 cf)
at Microsoft.Transactions.Wsat.Messaging.CoordinationService.Initialize(CoordinationServiceConfiguration config)
at Microsoft.Transactions.Wsat.Messaging.CoordinationService..ctor(CoordinationServiceConfiguration config, ProtocolVersion protocolVersion)
at Microsoft.Transactions.Wsat.Protocol.ProtocolState.RecoveryBeginning()
--- End of inner exception stack trace ---
at Microsoft.Transactions.Wsat.Protocol.ProtocolState.RecoveryBeginning()
at Microsoft.Transactions.Wsat.InputOutput.TransactionManagerReceive.RecoveryBeginning()
Process Name: msdtc
Process ID: 12248
In Component Services, when I restart the local DTC I get the following entry in the Eventviewer/Application:
The WS-AT protocol service successfully completed startup and recovery.
Protocol ID: cc228cf4-a9c8-43fc-8281-8565eb5889f2
Protocol Name: WS-AtomicTransaction 1.0
Process Name: msdtc
Process ID: 7744
Both DTCs run under the user Network Service:
Why does the clustered DTC not have access rights to this namespace, whereas the local DTC has? Both run under the same user.
How can I make the clustered DTC to register the URL https://+:2372/WsatService/ successfully?
I finally used port 8444. I had to reserve it with the command:
netsh http add urlacl url=https://+:8444/ user=Everyone
and then I ran wsatonfig specifying port 8444:
wsatconfig -network:enable -port:8444 -accounts:Everyone -endpointcert:7c6361568413852afb471d5f8b92604cdde530dd -accountsCerts:7c6361568413852afb471d5f8b92604cdde530dd,83112f9b598c4341b3975aba413bf04eb71eb679 -traceLevel:ALL -restart
Another time, it helped to disable and reenable the Network DTC Access in the properties of the Local DTC and the Cluster DTC:
Disable Local DTC, Apply and OK:
Enable Local DTC, Apply and OK:
Disable Cluster DTC, Apply and OK:
Enable Cluster DTC, Apply and OK:
I'm using Horizontal Pod Autoscaler to scale my pods in an OpenShift environment. I have a web application running in pods. As the pod scales, I got an HTTP status code 404 error in the first few seconds of an HTTP request. Is this because routes is sending a request to a pod that is in the process of being launched? If so, is there any way to prevent the error? I've tried setting router.openshift.io/haproxy.health.check.interval to a small value, but I still can't avoid this error.
It seems you did not configure your readiness checks correctly. Check the documentation on how to add readiness and liveness checks to your Deployment.
A readiness probe determines if a container is ready to accept service requests.
A liveness probe determines if a container is still running.
In newer versions of OpenShift / Kubernetes there is now also the startupProbe, which may help you in your case.
Here is an example of a Deployment with a liveness and a readiness probe:
kind: Deployment
apiVersion: apps/v1
...
spec:
...
template:
spec:
containers:
- name: example
readinessProbe:
tcpSocket:
port: 8080
livenessProbe:
tcpSocket:
port: 8080
...
I have a camel application which is running in spring boot 2 and camel routes are visualized through hawtio. And all actuator endpoints are exposed including jolokia endpoint
this application is completely working in local and
when I try to access actuator endpoints in local http://localhost:8080/actuator/, i could see below endpoint along with other 16 endpoints (such as health, info and so on)
"jolokia": {
"href": "http://localhost:8080/actuator/jolokia",
"templated": false
}
I have deployed the in same Openshift, but when i try to access actuator endpoints in Openshift
i could see only 16 endpoints without jolokia endpoint
Application start up logs in local
INFO : Initializing Spring embedded WebApplicationContext
INFO : Root WebApplicationContext: initialization completed in 3543 ms
INFO : Registered '/actuator/jolokia' to jolokia-actuator-endpoint
INFO : Initialising hawtio services
INFO : Configuration will be discovered via system properties
INFO : Welcome to Hawtio 2.10.0
INFO : Starting hawtio authentication filter, JAAS authentication disabled
INFO : Initializing ExecutorService 'applicationTaskExecutor'
INFO : Detected and using LURCacheFactory: camel-caffeine-lrucache
INFO : Exposing 17 endpoint(s) beneath base path '/actuator'
Application start up logs in Openshift
INFO : Initializing Spring embedded WebApplicationContext
INFO : Root WebApplicationContext: initialization completed in 3543 ms
INFO : Initialising hawtio services
INFO : Configuration will be discovered via system properties
INFO : Welcome to Hawtio 2.10.0
INFO : Starting hawtio authentication filter, JAAS authentication disabled
INFO : Initializing ExecutorService 'applicationTaskExecutor'
INFO : Detected and using LURCacheFactory: camel-caffeine-lrucache
INFO : Exposing 16 endpoint(s) beneath base path '/actuator'
INFO : Registered '/actuator/jolokia' to jolokia-actuator-endpoint is missing in Openshift logs,
so clearly its not registered with spring boot actuator
Any idea why jolokia is not exposed via spring boot actuator ?
because of this hawtio is not able to access camel routes (JMX).
Issue is resolved
Solution : Disable the default OpenJDK8 jolokia in Openshift
In Local
Application is not running in Openshift / Docker as an image, running as normal spring boot application in tomcat so I didn't face this issue.
In Openshift
Application is running in openshift / Docker as an image instance.
The image is created with base openJDK 8 which has default jolokia enabled
Red hat openshift reference
I have disabled it by overriding it with AB_JOLOKIA_OFF:true in openshift environment variables.
or either if you are using maven fabric8 plugin in pom for building image then you can override jolokia properties it in pom itself (i haven't try it but its possible as per documents).
I am currently trying to work with the geth and I want to start my private Ethereum Network so I can test my applications. However, when I try to use geth --datadir=./chaindata/ but that's only giving me some error in the terminal which I have shown at the bottom of this question. I am aware that there are other users that are having the same problem on Mac OS, which is what I'm using as well.
Here is the terminal output:
Steves-MBP:assignment_1 stevesahayadarlin$ geth --datadir=./chaindata/
WARN [01-06|22:12:18] No etherbase set and no accounts found as default
INFO [01-06|22:12:18] Starting peer-to-peer node instance=Geth/v1.7.3-stable/darwin-amd64/go1.9.2
INFO [01-06|22:12:18] Allocated cache and file handles database=/Users/stevesahayadarlin/Desktop/distributed_exchange_truffle_class_3-master/assignment_1/chaindata/geth/chaindata cache=128 handles=1024
INFO [01-06|22:12:18] Initialised chain configuration config="{ChainID: 15 Homestead: 0 DAO: <nil> DAOSupport: false EIP150: <nil> EIP155: 0 EIP158: 0 Byzantium: <nil> Engine: unknown}"
INFO [01-06|22:12:18] Disk storage enabled for ethash caches dir=/Users/stevesahayadarlin/Desktop/distributed_exchange_truffle_class_3-master/assignment_1/chaindata/geth/ethash count=3
INFO [01-06|22:12:18] Disk storage enabled for ethash DAGs dir=/Users/stevesahayadarlin/.ethash count=2
INFO [01-06|22:12:18] Initialising Ethereum protocol versions="[63 62]" network=1
INFO [01-06|22:12:18] Loaded most recent local header number=0 hash=9b8d4a…9021ba td=131072
INFO [01-06|22:12:18] Loaded most recent local full block number=0 hash=9b8d4a…9021ba td=131072
INFO [01-06|22:12:18] Loaded most recent local fast block number=0 hash=9b8d4a…9021ba td=131072
INFO [01-06|22:12:18] Loaded local transaction journal transactions=0 dropped=0
INFO [01-06|22:12:18] Regenerated local transaction journal transactions=0 accounts=0
INFO [01-06|22:12:18] Starting P2P networking
INFO [01-06|22:12:20] UDP listener up self=enode://258e1a8136fd23d47b97404139841059a37e95751182dde366adc4a22bab88b9580eb53bfb1de937016645817f071d0766a3be66e7e056c8f6afe0a450bb221d#70.106.232.168:30303
INFO [01-06|22:12:20] RLPx listener up self=enode://258e1a8136fd23d47b97404139841059a37e95751182dde366adc4a22bab88b9580eb53bfb1de937016645817f071d0766a3be66e7e056c8f6afe0a450bb221d#70.106.232.168:30303
INFO [01-06|22:12:20] Blockchain manager stopped
INFO [01-06|22:12:20] Stopping Ethereum protocol
INFO [01-06|22:12:20] Ethereum protocol stopped
INFO [01-06|22:12:20] Transaction pool stopped
INFO [01-06|22:12:20] Database closed database=/Users/stevesahayadarlin/Desktop/distributed_exchange_truffle_class_3-master/assignment_1/chaindata/geth/chaindata
INFO [01-06|22:12:20] Mapped network port proto=udp extport=30303 intport=30303 interface="UPNP IGDv1-IP1"
INFO [01-06|22:12:20] Mapped network port proto=tcp extport=30303 intport=30303 interface="UPNP IGDv1-IP1"
Fatal: Error starting protocol stack: listen unix /Users/stevesahayadarlin/Desktop/distributed_exchange_truffle_class_3-master/assignment_1/chaindata/geth.ipc: bind: invalid argument
Steves-MBP:assignment_1 stevesahayadarlin$