Some internet url or IP are not reachable through Cloud Function - google-cloud-functions

I tried to make some http requests through GCP's Cloud Function with Python 3.10 runtime, some went well, and some went wrong.
To find out the reason, I ping the IP of each url, and the IP I need has no response:
ping ip(173.194.217.106) of url(https://www.google.com): True
ping ip(69.147.92.11) of url(https://www.yahoo.com): True
ping ip(13.107.42.14) of url(https://www.linkedin.com): True
ping ip(117.56.7.114) of url(https://data.moi.gov.tw): False
Is there any way to make a successful request to https://data.moi.gov.tw through Cloud Function?
Here are the materials to reproduce the results with Cloud Function (Gen1):
main.py:
import platform # For getting the operating system name
import subprocess # For executing a shell command
import requests
def ping(host):
"""
Returns True if host (str) responds to a ping request.
Remember that a host may not respond to a ping (ICMP) request even if the host name is valid.
"""
# Option for the number of packets as a function of
param = '-n' if platform.system().lower()=='windows' else '-c'
# Building the command. Ex: "ping -c 1 google.com"
command = ['ping', param, '1', host]
return subprocess.call(command) == 0
def main(event):
d_ip_url = {
'173.194.217.106' : 'https://www.google.com',
'69.147.92.11' : 'https://www.yahoo.com',
'13.107.42.14' : 'https://www.linkedin.com',
'117.56.7.114' : 'https://data.moi.gov.tw',
}
for ip, url in d_ip_url.items():
print(f'ping ip({ip}) of url({url}):', ping(ip))
requirements.txt:
# Function dependencies, for example:
# package>=version
requests
The cloud Function settings:
{
"name": "projects/corgis-361708/locations/asia-east1/functions/ping-test",
"httpsTrigger": {
"url": "https://asia-east1-corgis-361708.cloudfunctions.net/ping-test",
"securityLevel": "SECURE_ALWAYS"
},
"status": "ACTIVE",
"entryPoint": "main",
"timeout": "60s",
"availableMemoryMb": 256,
"serviceAccountEmail": "corgis-361708#appspot.gserviceaccount.com",
"updateTime": "2022-09-21T06:04:31.746Z",
"versionId": "2",
"labels": {
"deployment-tool": "console-cloud"
},
"sourceUploadUrl": "https://storage.googleapis.com/uploads-918581105162.asia-east1.cloudfunctions.appspot.com/78ad8f77-d16c-412c-843f-51238703fbbf.zip",
"runtime": "python310",
"maxInstances": 1,
"ingressSettings": "ALLOW_ALL",
"buildId": "0c8bf5d0-3467-4516-8fea-c39d0e093c2e",
"buildName": "projects/647355426154/locations/asia-east1/builds/0c8bf5d0-3467-4516-8fea-c39d0e093c2e",
"dockerRegistry": "CONTAINER_REGISTRY"
}

Related

Opaque error with self-hosted Sourcegraph and Google Workspace SMTP relay config

Finally deployed a self-hosted Sourcegraph, v3.39.1. I'm running on Google Compute Engine on a Container Optimized OS VM. Got everything working except I'm having difficulty getting SMTP set up though Google Workspace's smtp-relay. When I run a sendTestEmail command, I get back an opaque error:
{
"data": {
"sendTestEmail": "Failed to send test email: EOF"
}
}
Here is a redacted snippet of my config:
"email.address": "myemailaddress#mydomain.com",
"email.smtp": {
"authentication": "PLAIN",
"domain": "mydomain.com",
"username": "myemailaddress#mydomain.com",
"password": "REDACTED",
"host": "smtp-relay.gmail.com",
"port": 587
}
For a quick smoke test, I ran netcat (nc smtp-relay.gmail.com 587) from the container and did sent EHLO mydomain.com. The smtp-relay responded with a "at your service" so I assume that worked. Any one have any other tips? Any logs I can check? (I didn't see anything obvious, but I'm new to Sourcegraph)

Disabling the Consul HTTP endpoints

We have enabled ACL's and TLS for Consul cluster in our environment. We have disabled the UI as well. But when I use the URL: http://<consul_agent>:8500/v1/coordinate/datacenters. How can disable the URL's as this?
I tested with adding the following to the consulConfig.json:
"ports":{
"http": -1
}
this did not solve the problem.
Apart from the suggestion provided to use "http_config": { "block_endpoints": I am trying to use the ACL Policy if that can solve.
I enabled the ACL's first
I created a policy using the command: consul acl policy create -name "urlblock" -description "Url Block Policy" -rules #service_block.hcl -token <tokenvalue>
contents of the service_block.hcl: service_prefix "/v1/status/leader" { policy = "deny" }
I created a agent token for this using the command: consul acl token create -description "Block Policy Token" -policy-name "urlblock" -token <tokenvalue>
I copied the agent token from the output of the above command and pasted that in the consul_config.json file in the acl -> tokens section as "tokens": { "agent": "<agenttokenvalue>"}
I restarted the consul agents (did the same in the consul client also).
Still I am able to access the endpoint /v1/status/leader. Any ideas as what is wrong with this approach?
That configuration should properly disable the HTTP server. I was able to validate this works using the following config with Consul 1.9.5.
Disabling Consul's HTTP server
Create config.json in the agent's configuration directory which completely disables the HTTP API port.
config.json
{
"ports": {
"http": -1
}
}
Start the Consul agent
$ consul agent -dev -config-file=config.json
==> Starting Consul agent...
Version: '1.9.5'
Node ID: 'ed7f0050-8191-999c-a53f-9ac48fd03f7e'
Node name: 'b1000.local'
Datacenter: 'dc1' (Segment: '<all>')
Server: true (Bootstrap: false)
Client Addr: [127.0.0.1] (HTTP: -1, HTTPS: -1, gRPC: 8502, DNS: 8600)
Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
==> Log data will now stream in as it occurs:
...
Note the HTTP port is set to "-1" on the Client Addr line. The port is now inaccessible.
Test connectivity to HTTP API
$ curl localhost:8500
curl: (7) Failed to connect to localhost port 8500: Connection refused
Blocking access to specific API endpoints
Alternatively you can block access to specific API endpoints, without completely disabling the HTTP API, by using the http_config.block_endpoints configuration option.
For example:
Create a config named block-endpoints.json
{
"http_config": {
"block_endpoints": [
"/v1/catalog/datacenters",
"/v1/coordinate/datacenters",
"/v1/status/leader",
"/v1/status/peers"
]
}
}
Start Consul with this config
consul agent -dev -config-file=block-endpoints.json
==> Starting Consul agent...
Version: '1.9.5'
Node ID: '8ff15668-8624-47b5-6e83-7a8bfd715a56'
Node name: 'b1000.local'
Datacenter: 'dc1' (Segment: '<all>')
Server: true (Bootstrap: false)
Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600)
Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
==> Log data will now stream in as it occurs:
...
In this example, the HTTP API is enabled and listening on port 8500.
Test connectivity to HTTP API
If you issue a request to one of the blocked endpoints, the following error will be returned.
$ curl localhost:8500/v1/status/peers
Endpoint is blocked by agent configuration
However, access to other endpoints are still permitted.
$ curl localhost:8500/v1/agent/members
[
{
"Name": "b1000.local",
"Addr": "127.0.0.1",
"Port": 8301,
"Tags": {
"acls": "0",
"build": "1.9.5:3c1c2267",
"dc": "dc1",
"ft_fs": "1",
"ft_si": "1",
"id": "6d157a1b-c893-3903-9037-2e2bd0e6f973",
"port": "8300",
"raft_vsn": "3",
"role": "consul",
"segment": "",
"vsn": "2",
"vsn_max": "3",
"vsn_min": "2",
"wan_join_port": "8302"
},
"Status": 1,
"ProtocolMin": 1,
"ProtocolMax": 5,
"ProtocolCur": 2,
"DelegateMin": 2,
"DelegateMax": 5,
"DelegateCur": 4
}
]

Chrome Failed to Start Native Messaging Host on Mac

Just to preface, I'm very new to all of this so please let me know if I'm missing anything fundamental or if I'm even going about this the right way.
I'm trying to make a chrome extension that reads the sender of an email in Gmail, then starts a python script (fmconnect.py) and sends the name of that sender to the script. I am able to retrieve the name of the sender using gmail.js, but when I try to send it to the python script, I keep receiving the error:
Unchecked runtime.lastError: Failed to start native messaging host.
Here are all the relevant files:
background.js
chrome.runtime.onMessage.addListener(function(request, sender, sendResponse) {
console.log(request);
sendResponse({message: "Message received"});
if (request.length > 0) {
const hostName = "com.google.chrome.example.echo";
var port = chrome.runtime.connectNative(hostName);
if (port) {
port.postMessage({text: request});
}
}
return true;
});
relevant part of manifest.json
"permissions": [
"https://*/*",
"http://localhost/",
"nativeMessaging",
"background",
"tabs"
]
fmconnect.py
def read_thread_func():
text_length_bytes = sys.stdin.read(4)
sys.stdout.write(text_length_bytes)
def Main():
read_thread_func()
sys.exit(0)
if __name__ == '__main__':
Main()
com.google.chrome.example.echo.json
{
"name": "com.google.chrome.example.echo",
"description": "Chrome Native Messaging API Host",
"path": "HOST_PATH",
"type": "stdio",
"allowed_origins": [
"chrome-extension://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/"
]
}
install_host.sh
set -e
DIR="$( cd "$( dirname "$0" )" && pwd )"
if [ "$(uname -s)" = "Darwin" ]; then
if [ "$(whoami)" = "root" ]; then
TARGET_DIR="/Library/Google/Chrome/NativeMessagingHosts"
else
TARGET_DIR="$HOME/Library/Application Support/Google/Chrome/NativeMessagingHosts"
fi
else
if [ "$(whoami)" = "root" ]; then
TARGET_DIR="/etc/opt/chrome/native-messaging-hosts"
else
TARGET_DIR="$HOME/.config/google-chrome/NativeMessagingHosts"
fi
fi
HOST_NAME=com.google.chrome.example.echo
# Create directory to store native messaging host.
mkdir -p "$TARGET_DIR"
# Copy native messaging host manifest.
cp "$DIR/$HOST_NAME.json" "$TARGET_DIR"
# Update host path in the manifest.
HOST_PATH=$DIR/fmconnect.py
ESCAPED_HOST_PATH=${HOST_PATH////\\/}
sed -i -e "s/HOST_PATH/$ESCAPED_HOST_PATH/" "$TARGET_DIR/$HOST_NAME.json"
# Set permissions for the manifest so that all users can read it.
chmod o+r "$TARGET_DIR/$HOST_NAME.json"
echo "Native messaging host $HOST_NAME has been installed."
I know that on the native messaging host documentation, it says to check whether I have sufficient permissions to start the file. How exactly would I do this?

What is wrong in the following Lambda code that throws up module error?

Using the following code to use make an API that connects to Amazon AWS. This is the AMazon Lambda code that I use-
import boto3
import json
import requests
from requests_aws4auth import AWS4Auth
region = 'us-east-1'
service = 'es'
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region,
service, session_token=credentials.token)
host = 'XXX.com'
index = 'items'
url = 'https://' + host + '/' + index + '/_search'
# Lambda execution starts here
def handler(event, context):
# Put the user query into the query DSL for more accurate search results.
# Note that certain fields are boosted (^).
query = {
"query": {
"multi_match": {
"query": event['queryStringParameters']['q'],
}
}
}
# ES 6.x requires an explicit Content-Type header
headers = { "Content-Type": "application/json" }
# Make the signed HTTP request
r = requests.get(url, auth=awsauth, headers=headers,
data=json.dumps(query))
# Create the response and add some extra content to support CORS
response = {
"statusCode": 200,
"headers": {
"Access-Control-Allow-Origin": '*'
},
"isBase64Encoded": False
}
# Add the search results to the response
response['body'] = r.text
return response
This should connect to an AWS ES cluster with endpoint XXX.com
Getting output when trying to test -
START RequestId: f640016e-e4d6-469f-b74d-838b9402968b Version: $LATEST
Unable to import module 'index': Error
at Function.Module._resolveFilename (module.js:547:15)
at Function.Module._load (module.js:474:25)
at Module.require (module.js:596:17)
at require (internal/module.js:11:18)
END RequestId: f640016e-e4d6-469f-b74d-838b9402968b
REPORT RequestId: f640016e-e4d6-469f-b74d-838b9402968b Duration:
44.49 ms Billed Duration: 100 ms Memory Size: 128 MB Max
Memory Used: 58 MB
While creating a Lambda function, we need to specify a handler, which is a function in your code, that the AWS Lambda service can invoke when the given Lambda function is executed.
By default, a Python Lambda function is created with handler as lambda_function.lambda_handler which signifies that the service must invoke lambda_handler function contained inside lambda_function module.
From the error you're receiving, it seems that the handler is incorrectly configured to something like index.<something>, and since there's no Python module called index in your deployment package, Lambda is unable to import the same in order to start the execution.
If i am getting things correctly to connect to a AWS ES cluster you need something of this sort
import gitlab
import logging
from elasticsearch import Elasticsearch, RequestsHttpConnection
from requests_aws4auth import AWS4Auth
import boto3
#from aws_requests_auth.aws_auth import AWSRequestsAuth
LOGGER = logging.getLogger()
ES_HOST = {'host':'search-testelasticsearch-xxxxxxxxxx.eu-west-2.es.amazonaws.com', 'port': 443}
def lambda_handler(event, context):
LOGGER.info('started')
dump2={
'number': 9
}
service = 'es'
credentials = boto3.Session().get_credentials()
print('-------------------------------------------')
print(credentials.access_key)
print(credentials.secret_key)
print('--------------------------------------------------------')
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, "eu-west-2", service, session_token=credentials.token)
es = Elasticsearch(hosts=[ES_HOST], http_auth = awsauth, use_ssl = True, verify_certs = True, connection_class = RequestsHttpConnection)
DAVID_INDEX = 'test_index'
response = es.index(index=DAVID_INDEX, doc_type='is_this_important?', body=dump2, id='4')

Problems with cosmos auth and Identity manager integration

I want to integrate cosmos-auth with Idm GE.
Config for node.js application is:
{
"host": "192.168.4.180",
"port": 13000,
"private_key_file": "key.pem",
"certificate_file": "cert.pem",
"idm": {
"host": "192.168.4.33",
"port": "443",
"path": "/oauth2/token"
},
"cosmos_app": {
"client_id": "0434fdf60897479588c3c31cfc957b6d",
"client_secret": "a7c3540aa5de4de3a0b1c52a606b82df"
},
"log": {
"file_name": "/var/log/cosmos/cosmos-auth/cosmos-auth.log",
"date_pattern": ".dd-MM-yyyy"
}
}
When i send HTTP POST request directly to IDM GE to url
https://192.168.4.33:443/oauth2/token
with required parameters i get ok results:
{
access_token: "LyZT5DRGSn0F8IKqYU8EmRFTLo1iPJ"
token_type: "Bearer"
expires_in: 3600
refresh_token: "XiyfKCHrIVyludabjaCyGqVsTkx8Sf"
}
But when i curl the cosmos-auth node.js application
curl -X POST "https://192.168.4.180:13000/cosmos-auth/v1/token" -H
"Content-Type: application/x-www-form-urlencoded" -d
"grant_type=password&username=idm&password=idm" -k
I get next result:
{"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred"}
Has anyone encountered something similar?
What could be the problem?
The error i made was using unsigned certificate.How clumsy of me.
So either sign the certificate or insert additional element in options object (rejectUnauthorized: false)
var options = {
host : host,
port : port,
path : path,
method : method,
headers: headers,
rejectUnauthorized: false
};
or in the beginning of the file insert:
process.env.NODE_TLS_REJECT_UNAUTHORIZED = '0';
Ofcourse this is only temporary solution until we use fully signed cert.
Anyways error handling and logs in cosmos-auth node.js app should show a little bit more.