Python GCP-API permissions issue: compute.instances.list - google-compute-engine

I have a 5 line Python script which is failing.
GCP indicates a problem with a GCP-API permissions issue.
I'm not sure though it is telling me the truth.
Here is the Python syntax:
import googleapiclient.discovery
compute = googleapiclient.discovery.build('compute', 'v1')
project_s = 'shining-sol-241621'
zone_s = 'us-central1-a'
result = compute.instances().list(project=project_s, zone=zone_s).execute()
The above syntax depends on this env variable setting:
export GOOGLE_APPLICATION_CREDENTIALS=$HOME/secret.json
When I run the Python syntax I see:
$ python3
Python 3.5.2 (default, Nov 12 2018, 13:43:14)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import googleapiclient.discovery
>>> compute = googleapiclient.discovery.build('compute', 'v1')
>>> project_s = 'shining-sol-241621'
>>> zone_s = 'us-central1-a'
>>> result = compute.instances().list(project=project_s, zone=zone_s).execute()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.5/dist-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
return wrapped(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/googleapiclient/http.py", line 851, in execute
raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 403 when requesting https://www.googleapis.com/compute/v1/projects/shining-sol-241621/zones/us-central1-a/instances?alt=json returned "Required 'compute.instances.list' permission for 'projects/shining-sol-241621'">
>>> >>>
Above, I see this msg in the error:
"Required 'compute.instances.list' permission for 'projects/shining-sol-241621'"
At this URL:
https://console.cloud.google.com/iam-admin/iam?project=shining-sol-241621
When I look at the service-account permissions attached to my secret.json file I see that I have attached these permissions to the account:
Compute Admin
Compute Instance Admin (beta)
Compute Instance Admin (v1)
Compute Viewer
Security Admin
Owner
When I attached permissions to the service-account, I saw no permission available which looked like:
'compute.instances.list'
I did note that the Compute Admin permission has this description:
"Full control of all Compute Engine resources"
That description suggests to me that the permission should be able to do: 'compute.instances.list'
I am suspicious that GCP is telling me that I have a permissions problem but the problem is somewhere else.
Based on my description, what would be some good questions to ask in order to move past this bad API behavior?

I was able to fix this problem after I assumed that the GCP server was caching bad info.
I created a service account with a name I had not used in the past.
I gave it just one permission: Project - Owner
I asked it to create a JSON-key.
I copied the key into $HOME/secret.json
I issued shell command:
export GOOGLE_APPLICATION_CREDENTIALS=$HOME/secret.json
I tried my 6 lines of Python:
$ python3
Python 3.5.2 (default, Nov 12 2018, 13:43:14)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>>
>>> import googleapiclient.discovery
>>> compute = googleapiclient.discovery.build('compute', 'v1')
>>> project_s = 'shining-sol-241621'
>>> zone_s = 'us-central1-a'
>>> result = compute.instances().list(project=project_s, zone=zone_s).execute()
>>> result['items'][0]['status']
'RUNNING'
>>>
>>>
As you can see, it worked.
Yay!

Related

Selenium python library via docker, Chrome error failed to start: exited abnormally

I am trying to run some python scripts with the selenium library from within a docker container based on miniconda/anaconda, but I keep getting this error: selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: exited abnormally. I am also using a python wrapper for xvfb to avoid opening a real Chrome window.
To reproduce this (from a running docker container):
root#304ccd3bae83:/opt# python
Python 3.6.4 |Anaconda, Inc.| (default, Jan 16 2018, 18:10:19)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>>
>>> from selenium import webdriver
>>> from xvfbwrapper import Xvfb
>>>
>>> with Xvfb(width=1366, height=768) as xvfb:
... my_driver = webdriver.Chrome('/opt/chromedriver/2.33/chromedriver')
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
File "/opt/conda/lib/python3.6/site-packages/selenium/webdriver/chrome/webdriver.py", line 69, in __init__
desired_capabilities=desired_capabilities)
File "/opt/conda/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 151, in __init__
self.start_session(desired_capabilities, browser_profile)
File "/opt/conda/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 240, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/opt/conda/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 308, in execute
self.error_handler.check_response(response)
File "/opt/conda/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 194, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: exited abnormally
(Driver info: chromedriver=2.33.506092 (733a02544d189eeb751fe0d7ddca79a0ee28cce4),platform=Linux 4.4.0-116-generic x86_64)
According to this: https://sites.google.com/a/chromium.org/chromedriver/help/chrome-doesn-t-start it seems someone may need to use a stand-alone version of Chrome that works for all users, but I am not sure how the docker build works, I guess the docker image is built as root, and all the code inside it is executed as root, so there should not be any issue with different users controlling Chrome.
This python code works fine on a normal Ubuntu laptop with X windows. I need to carefully pick both the versions of Chrome and chromedriver, at the moment when checking from within the running docker container:
root#304ccd3bae83:/opt# /opt/chromedriver/2.33/chromedriver --version
ChromeDriver 2.33.506092 (733a02544d189eeb751fe0d7ddca79a0ee28cce4)
root#304ccd3bae83:/opt# google-chrome-stable --version
Google Chrome 62.0.3202.75
These options helped solving the issue.
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument("--disable-setuid-sandbox")
One of them is needed when seeing Chrome failed to start: crashed.
Also: make sure there are no zombies (from previous executions) for the chrome-driver process using ps aux | grep chrome-driver to find the PIDs to kill.
Bare in mind that if you are using the Python multiprocessing library to spawn many processes involving their own instance of the Chrome browser, then you can not use Docker (which is supposed to start just one Python process, unless using stuff like supervisor), so you may see: selenium.common.exceptions.WebDriverException: Message: chrome not reachable if you try anyway.

root able to import python module, but user can't

I need to import simplemysql module. But getting ImportError. Import doesn't give error for root user.
Can you please help me with this issue?
$ python
Python 2.7.11 |Anaconda 2.5.0 (32-bit)| (default, Dec 6 2015, 18:08:45)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> from simplemysql import SimpleMysql
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named simplemysql
>>>
$ su
Password:
root#mtk-Inspiron-1525:/home/mtk/.anaconda2# python
Python 2.7.10 (default, Oct 14 2015, 16:09:02)
[GCC 5.2.1 20151010] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from simplemysql import SimpleMysql
>>>
These are different Pythons called in these two cases:
Python 2.7.11 |Anaconda 2.5.0 (32-bit)| (default, Dec 6 2015, 18:08:45)
Python 2.7.10 (default, Oct 14 2015, 16:09:02)
Check that you have simplemysql installed into the 2.7.11 Anaconda one.
I had a similiar problem and i found out that i can only work with the Module if i import MySQLdb first. Installation of simplemysql (as far as i remember) only worked with a cmd ran as admin on Windows 8.

I get an error message when I try FreqDist() in NLTK -- NameError: name 'nltk' is not defined

I'm learning about the NLTK and my mac
is working fine except I have trouble with the FreqDist(). (I saw another question about FreqDist() but he was getting a different error message. TypeError: unhashable type: 'list')
Here's an example:
>>> from nltk.corpus import brown
>>> news_text = brown.words(categories='news')
>>> fdist = nltk.FreqDist([w.lower() for w in news_text])
Traceback (most recent call last):
` File "<stdin>", line 1, in <module>`
`NameError: name 'nltk' is not defined`
This error message is pretty consistent. I get this message every time I try the FreqDist(). Other commands like - >>> brown.fileids() are fine.
Thanks for your help!
Before you can use FreqDist, you need to import it.
Add a line as follows:
import nltk
or if you just want to use FreqDist you should try this:
>>> from nltk.corpus import brown
>>> from nltk import FreqDist
>>> news_text = brown.words(categories='news')
>>> fdist = FreqDist([w.lower() for w in news_text])
which means you haven't installed nltk.
follow these steps to install nltk:
1:go to this link https://pypi.python.org/pypi/setuptools at the end of page you find setuptools-7.0.zip (md5) download it, then unzip it. you can find easy_install.py python script.
2:use the command sudo easy_install pip. By this time pip will be installed ready to use, (make sure you are in the directory where you can find easy_install script file).
3:use this command sudo pip install -U nltk. successful execution ensure that nltk is now installed.
4:open the IDLE then you type the following:
import nltk
if nltk is installed properly then you will be returned with console.
setuptools are required for older versions of Python. There is no need for the same if you are running 3.2+
You can easily download the same from https://pypi.python.org/pypi/nltk
For more information on http://www.nltk.org/install.html
nltk requires data you need to download first.
Then run the following code:
import nltk
nltk.download('stopwords')
from nltk.corpus import stopwords
stopwords.words("english")

calling legacy c functions from python

I want to call legacy c third party functions from python.
I created a C api to make the function calls simpler.
In my python file
I tried to import the *.so for the api which links with the legacy code library.
But I kept seeing a libxxx ( the legacy c library) not found error.
I tried to import both ans still saw the same error.
I then tried using ctypes, but get the error ctypes not found.
Any suggestions ?
# python
Python 2.4.2 (#1, Apr 20 2012, 19:31:49)
[GCC 4.1.2 20070115 (SUSE Linux)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes
Traceback (most recent call last):
File "<stdin>", line 1, in ?
ImportError: No module named ctypes
>>> quit
ctypes only made it into the standard library in version 2.5, so 2.4 does not have it. I've also heard that some people exclude it when they build Python for a distribution or for embedding in a specific application, as you can easily crash your process using it, or even corrupt the host application's heap. But I have never encountered this myself, and I doubt a full installation does this.
As for not finding your C library, we'll need more details on that.

Using MySQL with dev_appserver (Google App Engine) and Google SQL Service

I am using Google App Engine and Google SQL Service, and would like to use the option
dev_appserver.py --mysql_user=username myapp
in order to use a local MySQL database for development purposes, and SQL Service in prod environment.
I have MySQLdb installed and working:
/usr/bin>>python
Python 2.7.2 (v2.7.2:8527427914a2, Jun 11 2011, 15:22:34)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import MySQLdb
>>>
However, when I run the dev_appserver command, I get the following error:
zipimporter('/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/MySQL_python-1.2.3-py2.7-macosx-10.6-intel.egg', 'MySQLdb/')
ERROR 2011-09-12 09:34:53,541 rdbms_mysqldb.py:90] The rdbms API is not available because the MySQLdb library could not be loaded.
I have verified that the MySQL_python-1.2.3-py2.7-macosx-10.6-intel.egg is in the location (/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/) and that it is accessible.
Any idea what might be causing this error?
Full stacktrace provided below for information:
/Users/eddieboyd>>dev_appserver.py --debug --mysql_user=mysql eddiehelloworld
Warning: You are using a Python runtime (2.7) that is more recent than the production runtime environment (2.5). Your application may use features that are not available in the production environment and may not work correctly when deployed to production.
INFO 2011-09-12 09:34:53,290 appengine_rpc.py:159] Server: appengine.google.com
INFO 2011-09-12 09:34:53,304 appcfg.py:449] Checking for updates to the SDK.
DEBUG 2011-09-12 09:34:53,305 appengine_rpc.py:364] Sending HTTPS request:
POST /api/updatecheck?release=1.5.3&timestamp=1311108376&api_versions=%5B%271%27%5D HTTPS/1.1
Host: appengine.google.com
X-appcfg-api-version: 1
Content-type: application/octet-stream
User-agent: appcfg_py/1.5.3 Darwin/10.8.0 Python/2.7.2.final.0
INFO 2011-09-12 09:34:53,465 appcfg.py:466] The SDK is up to date.
WARNING 2011-09-12 09:34:53,465 datastore_file_stub.py:512] Could not read datastore data from /var/folders/++/++71vE++6+0++4RjPqRgNE+0Eyo/-Tmp-/dev_appserver.datastore
INFO 2011-09-12 09:34:53,478 py_zipimport.py:148] zipimporter('/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/MySQL_python-1.2.3-py2.7-macosx-10.6-intel.egg', 'MySQLdb/')
ERROR 2011-09-12 09:34:53,541 rdbms_mysqldb.py:90] The rdbms API is not available because the MySQLdb library could not be loaded.
ERROR 2011-09-12 09:34:53,541 dev_appserver_main.py:638] <type 'exceptions.NotImplementedError'>: Unable to find the MySQLdb library. Please see the SDK documentation for installation instructions.
DEBUG 2011-09-12 09:34:53,543 dev_appserver_main.py:640] Traceback (most recent call last):
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver_main.py", line 635, in main
dev_appserver.SetupStubs(appinfo.application, **option_dict)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 4668, in SetupStubs
rdbms_mysqldb.connect(database='')
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/rdbms_mysqldb.py", line 96, in connect
'Unable to find the MySQLdb library. Please see the SDK '
NotImplementedError: Unable to find the MySQLdb library. Please see the SDK documentation for installation instructions.
Poor man's solution.
I added
"import MySQLdb"
in dev_appserver.py.
It works now.
Ensure to use the latest version of the AppEngine SDK which currently is the version 1.6.2.
After that install the package for python mysql support: python-mysqldb.
Maybe your system does not have the MySQL module required. On Fedora 16 64 bits, I solved this by installing the MySQL python module:
yum install MySQL-python