In Kilo code release, nova boot failed on keystoneclient with 500 error - openstack-nova

Question

I have kilo rc1 installed on CentOS7 with following rpm pkgs.
# rpm -qa | grep -Ei 'nova|urllib3|request|six'
requests-2.6.0-1.noarch
python-novaclient-2.23.0.post13-1.noarch
nova-2015.1.0rc1-1.noarch
six-1.9.0-1.noarch
nova-docker-0.0.0.post183-1.noarch
urllib3-1.10.1-1.noarch
Keystone, Glance and Neutron work as expected plus nova GET method such as nova service-list. All password and configuration files have been verified. when issuing a nova boot cmd, I got http 500 error detailed below.
--- Get the token with neutron/password ----
# openstack token issue
+------------+----------------------------------+
| Field | Value |
+------------+----------------------------------+
| expires | 2015-04-22T21:12:14Z |
| id | 24e6a5e2546c41c98865c946f10f7ddb |
| project_id | 0ece3f1cc56a4a0bba1906b43d1faceb |
| user_id | ef18eea137ed4dabad1f92f4a393fd70 |
+------------+----------------------------------+
-------------- Used the issued token to call nova API ---------------
# curl -g -i -X GET http://10.0.0.244:8774/v2/0ece3f1cc56a4a0bba1906b43d1faceb/os-services -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: 24e6a5e2546c41c98865c946f10f7ddb"
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 1285
Date: Wed, 22 Apr 2015 20:14:57 GMT
{"services": [{"status": "enabled", "binary": "nova-scheduler", "zone": "internal", "state": "up", "updated_at": "2015-04-22T20:14:53.000000", "host": "ctrail72", "disabled_reason": null, "id": 1}, {"status": "enabled", "binary": "nova-conductor", "zone": "internal", "state": "up", "updated_at": "2015-04-22T20:14:55.000000", "host": "ctrail72", "disabled_reason": null, "id": 2}, {"status": "enabled", "binary": "nova-console", "zone": "internal", "state": "up", "updated_at": "2015-04-22T20:14:47.000000", "host": "ctrail72", "disabled_reason": null, "id": 3}, {"status": "enabled", "binary": "nova-consoleauth", "zone": "internal", "state": "up", "updated_at": "2015-04-22T20:14:48.000000", "host": "ctrail72", "disabled_reason": null, "id": 4}, {"status": "enabled", "binary": "nova-cert", "zone": "internal", "state": "up", "updated_at": "2015-04-22T20:14:50.000000", "host": "ctrail72", "disabled_reason": null, "id": 5}, {"status": "enabled", "binary": "nova-compute", "zone": "nova", "state": "up", "updated_at": "2015-04-22T20:14:51.000000", "host": "ctrail72", "disabled_reason": null, "id": 6}, {"status": "enabled", "binary": "nova-compute", "zone": "nova", "state": "up", "updated_at": "2015-04-22T20:14:54.000000", "host": "comp70-1", "disabled_reason": null, "id": 7}]}[
---------- Same token works for neutron API ------
# curl -g -i -X GET http://10.0.0.244:9696/v2.0/subnets.json -H "X-Auth- Token: 24e6a5e2546c41c98865c946f10f7ddb"
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 898
X-Openstack-Request-Id: req-4f547520-4cb2-4f59-835a-711c933664d1
Date: Wed, 22 Apr 2015 20:16:15 GMT
{"subnets": [{"name": "sub101", "enable_dhcp": true, "network_id": "40945ae1-344c-4ebd-a25b-2776feb0f409", "tenant_id": "959d7f7e020b48509aea18dcec819491", "dns_nameservers": [], "gateway_ip": "10.0.0.1", "ipv6_ra_mode": ...
-------- nova boot failed with 500 error code ------
nova --debug boot --flavor 1 --image dockerc7 --nic net-id=40945ae1-344c-4ebd-a25b-2776feb0f409 d01
.......
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 911, in main
OpenStackComputeShell().main(argv)
File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 838, in main
args.func(self.cs, args)
File "/usr/lib/python2.7/site-packages/novaclient/v2/shell.py", line 500, in do_boot
server = cs.servers.create(*boot_args, **boot_kwargs)
File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 929, in create
**boot_kwargs)
File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 557, in _boot
return_raw=return_raw, **kwargs)
File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 152, in _create
_resp, body = self.api.client.post(url, body=body)
File "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 171, in post
return self.request(url, 'POST', **kwargs)
File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 97, in request
raise exceptions.from_response(resp, body, url, method)
ClientException: The server has either erred or is incapable of performing
the requested operation. (HTTP 500)
ERROR (ClientException): The server has either erred or is incapable of performing the requested operation. (HTTP 500)
Further trace back it found that self._sock.recv in /usr/lib64/python2.7/socket.py returns 500 error.
> /usr/lib64/python2.7/socket.py(481)readline()
-> if not data:
(Pdb) l
476 data = self._sock.recv(self._rbufsize)
477 except error, e:
478 if e.args[0] == EINTR:
479 continue
480 raise
481 -> if not data:
482 break
483 left = size - buf_len
484 # did we just receive a newline?
485 nl = data.find('\n', 0, left)
486 if nl >= 0:
(Pdb) p data
'HTTP/1.1 500 Internal Server Error\r\nContent-Length: 128\r\nContent-Type: application/json; charset=UTF-8\r\nDate: Wed, 22 Apr 2015 18:21:28 GMT\r\nConnection: keep-alive\r\n\r\n{"computeFault": {"message": "The server has either erred or is incapable of performing the requested operation.", "code": 500}}'
Any suggestion and tips are appreciated!