Problems with cosmos auth and Identity manager integration - fiware

I want to integrate cosmos-auth with Idm GE.
Config for node.js application is:
{
"host": "192.168.4.180",
"port": 13000,
"private_key_file": "key.pem",
"certificate_file": "cert.pem",
"idm": {
"host": "192.168.4.33",
"port": "443",
"path": "/oauth2/token"
},
"cosmos_app": {
"client_id": "0434fdf60897479588c3c31cfc957b6d",
"client_secret": "a7c3540aa5de4de3a0b1c52a606b82df"
},
"log": {
"file_name": "/var/log/cosmos/cosmos-auth/cosmos-auth.log",
"date_pattern": ".dd-MM-yyyy"
}
}
When i send HTTP POST request directly to IDM GE to url
https://192.168.4.33:443/oauth2/token
with required parameters i get ok results:
{
access_token: "LyZT5DRGSn0F8IKqYU8EmRFTLo1iPJ"
token_type: "Bearer"
expires_in: 3600
refresh_token: "XiyfKCHrIVyludabjaCyGqVsTkx8Sf"
}
But when i curl the cosmos-auth node.js application
curl -X POST "https://192.168.4.180:13000/cosmos-auth/v1/token" -H
"Content-Type: application/x-www-form-urlencoded" -d
"grant_type=password&username=idm&password=idm" -k
I get next result:
{"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred"}
Has anyone encountered something similar?
What could be the problem?

The error i made was using unsigned certificate.How clumsy of me.
So either sign the certificate or insert additional element in options object (rejectUnauthorized: false)
var options = {
host : host,
port : port,
path : path,
method : method,
headers: headers,
rejectUnauthorized: false
};
or in the beginning of the file insert:
process.env.NODE_TLS_REJECT_UNAUTHORIZED = '0';
Ofcourse this is only temporary solution until we use fully signed cert.
Anyways error handling and logs in cosmos-auth node.js app should show a little bit more.

Related

Some internet url or IP are not reachable through Cloud Function

I tried to make some http requests through GCP's Cloud Function with Python 3.10 runtime, some went well, and some went wrong.
To find out the reason, I ping the IP of each url, and the IP I need has no response:
ping ip(173.194.217.106) of url(https://www.google.com): True
ping ip(69.147.92.11) of url(https://www.yahoo.com): True
ping ip(13.107.42.14) of url(https://www.linkedin.com): True
ping ip(117.56.7.114) of url(https://data.moi.gov.tw): False
Is there any way to make a successful request to https://data.moi.gov.tw through Cloud Function?
Here are the materials to reproduce the results with Cloud Function (Gen1):
main.py:
import platform # For getting the operating system name
import subprocess # For executing a shell command
import requests
def ping(host):
"""
Returns True if host (str) responds to a ping request.
Remember that a host may not respond to a ping (ICMP) request even if the host name is valid.
"""
# Option for the number of packets as a function of
param = '-n' if platform.system().lower()=='windows' else '-c'
# Building the command. Ex: "ping -c 1 google.com"
command = ['ping', param, '1', host]
return subprocess.call(command) == 0
def main(event):
d_ip_url = {
'173.194.217.106' : 'https://www.google.com',
'69.147.92.11' : 'https://www.yahoo.com',
'13.107.42.14' : 'https://www.linkedin.com',
'117.56.7.114' : 'https://data.moi.gov.tw',
}
for ip, url in d_ip_url.items():
print(f'ping ip({ip}) of url({url}):', ping(ip))
requirements.txt:
# Function dependencies, for example:
# package>=version
requests
The cloud Function settings:
{
"name": "projects/corgis-361708/locations/asia-east1/functions/ping-test",
"httpsTrigger": {
"url": "https://asia-east1-corgis-361708.cloudfunctions.net/ping-test",
"securityLevel": "SECURE_ALWAYS"
},
"status": "ACTIVE",
"entryPoint": "main",
"timeout": "60s",
"availableMemoryMb": 256,
"serviceAccountEmail": "corgis-361708#appspot.gserviceaccount.com",
"updateTime": "2022-09-21T06:04:31.746Z",
"versionId": "2",
"labels": {
"deployment-tool": "console-cloud"
},
"sourceUploadUrl": "https://storage.googleapis.com/uploads-918581105162.asia-east1.cloudfunctions.appspot.com/78ad8f77-d16c-412c-843f-51238703fbbf.zip",
"runtime": "python310",
"maxInstances": 1,
"ingressSettings": "ALLOW_ALL",
"buildId": "0c8bf5d0-3467-4516-8fea-c39d0e093c2e",
"buildName": "projects/647355426154/locations/asia-east1/builds/0c8bf5d0-3467-4516-8fea-c39d0e093c2e",
"dockerRegistry": "CONTAINER_REGISTRY"
}

Opaque error with self-hosted Sourcegraph and Google Workspace SMTP relay config

Finally deployed a self-hosted Sourcegraph, v3.39.1. I'm running on Google Compute Engine on a Container Optimized OS VM. Got everything working except I'm having difficulty getting SMTP set up though Google Workspace's smtp-relay. When I run a sendTestEmail command, I get back an opaque error:
{
"data": {
"sendTestEmail": "Failed to send test email: EOF"
}
}
Here is a redacted snippet of my config:
"email.address": "myemailaddress#mydomain.com",
"email.smtp": {
"authentication": "PLAIN",
"domain": "mydomain.com",
"username": "myemailaddress#mydomain.com",
"password": "REDACTED",
"host": "smtp-relay.gmail.com",
"port": 587
}
For a quick smoke test, I ran netcat (nc smtp-relay.gmail.com 587) from the container and did sent EHLO mydomain.com. The smtp-relay responded with a "at your service" so I assume that worked. Any one have any other tips? Any logs I can check? (I didn't see anything obvious, but I'm new to Sourcegraph)

Disabling the Consul HTTP endpoints

We have enabled ACL's and TLS for Consul cluster in our environment. We have disabled the UI as well. But when I use the URL: http://<consul_agent>:8500/v1/coordinate/datacenters. How can disable the URL's as this?
I tested with adding the following to the consulConfig.json:
"ports":{
"http": -1
}
this did not solve the problem.
Apart from the suggestion provided to use "http_config": { "block_endpoints": I am trying to use the ACL Policy if that can solve.
I enabled the ACL's first
I created a policy using the command: consul acl policy create -name "urlblock" -description "Url Block Policy" -rules #service_block.hcl -token <tokenvalue>
contents of the service_block.hcl: service_prefix "/v1/status/leader" { policy = "deny" }
I created a agent token for this using the command: consul acl token create -description "Block Policy Token" -policy-name "urlblock" -token <tokenvalue>
I copied the agent token from the output of the above command and pasted that in the consul_config.json file in the acl -> tokens section as "tokens": { "agent": "<agenttokenvalue>"}
I restarted the consul agents (did the same in the consul client also).
Still I am able to access the endpoint /v1/status/leader. Any ideas as what is wrong with this approach?
That configuration should properly disable the HTTP server. I was able to validate this works using the following config with Consul 1.9.5.
Disabling Consul's HTTP server
Create config.json in the agent's configuration directory which completely disables the HTTP API port.
config.json
{
"ports": {
"http": -1
}
}
Start the Consul agent
$ consul agent -dev -config-file=config.json
==> Starting Consul agent...
Version: '1.9.5'
Node ID: 'ed7f0050-8191-999c-a53f-9ac48fd03f7e'
Node name: 'b1000.local'
Datacenter: 'dc1' (Segment: '<all>')
Server: true (Bootstrap: false)
Client Addr: [127.0.0.1] (HTTP: -1, HTTPS: -1, gRPC: 8502, DNS: 8600)
Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
==> Log data will now stream in as it occurs:
...
Note the HTTP port is set to "-1" on the Client Addr line. The port is now inaccessible.
Test connectivity to HTTP API
$ curl localhost:8500
curl: (7) Failed to connect to localhost port 8500: Connection refused
Blocking access to specific API endpoints
Alternatively you can block access to specific API endpoints, without completely disabling the HTTP API, by using the http_config.block_endpoints configuration option.
For example:
Create a config named block-endpoints.json
{
"http_config": {
"block_endpoints": [
"/v1/catalog/datacenters",
"/v1/coordinate/datacenters",
"/v1/status/leader",
"/v1/status/peers"
]
}
}
Start Consul with this config
consul agent -dev -config-file=block-endpoints.json
==> Starting Consul agent...
Version: '1.9.5'
Node ID: '8ff15668-8624-47b5-6e83-7a8bfd715a56'
Node name: 'b1000.local'
Datacenter: 'dc1' (Segment: '<all>')
Server: true (Bootstrap: false)
Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600)
Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
==> Log data will now stream in as it occurs:
...
In this example, the HTTP API is enabled and listening on port 8500.
Test connectivity to HTTP API
If you issue a request to one of the blocked endpoints, the following error will be returned.
$ curl localhost:8500/v1/status/peers
Endpoint is blocked by agent configuration
However, access to other endpoints are still permitted.
$ curl localhost:8500/v1/agent/members
[
{
"Name": "b1000.local",
"Addr": "127.0.0.1",
"Port": 8301,
"Tags": {
"acls": "0",
"build": "1.9.5:3c1c2267",
"dc": "dc1",
"ft_fs": "1",
"ft_si": "1",
"id": "6d157a1b-c893-3903-9037-2e2bd0e6f973",
"port": "8300",
"raft_vsn": "3",
"role": "consul",
"segment": "",
"vsn": "2",
"vsn_max": "3",
"vsn_min": "2",
"wan_join_port": "8302"
},
"Status": 1,
"ProtocolMin": 1,
"ProtocolMax": 5,
"ProtocolCur": 2,
"DelegateMin": 2,
"DelegateMax": 5,
"DelegateCur": 4
}
]

read Local audio file STT Google speech-to-text

I don't find how to send a local file using the Google STT in curl command line.
As followed in their tuto, here my request.json :
{
"config": {
"encoding":"FLAC",
"sample_rate": 16000,
"language_code": "fr-FR"
},
"audio": {
"uri":"audio-file.flac"
}
}
my command line is:
curl -s -X POST -H "Content-Type: application/json" --data-binary #request.json "https://speech.googleapis.com/v1beta1/speech:syncrecognize?key=MY_API_KEY"
But I receive the error result:
{
"error": {
"code": 400,
"message": "RecognitionAudio not set.",
"status": "INVALID_ARGUMENT"
}
}
How can I write the request.json to send a local audio file ??
Thanks for help
:)
You need to use the base64 in the content section of the request.json file. It should look something like this except with a lot more base64 data:
{
'config': {
"encoding":"FLAC",
"sample_rate": 16000,
"language_code": "fr-FR"
},
'audio': {
'content':'UklGRuRDFQBXQVZFZm10IBAAAAABAAEAQB8AAIA+AAACABAAZGF0YcBDFQAIAAgACAAIAAgA
CAAIAAgACAAIAAgACAAIAAgACAAIAAgACAAIAAgACAAIAAgACAAIAAgACAAIAAgACAAIAAgA
CAAIAAgACAAIAAgACAAIAAgACAAIAAgACAAIAAgACAAIAAgACAAIAAgACAAIAAgACAAIAAgA
CAAIAAgACAAIAAgACAAIAAgACAAIAAgACAAIAAgACAD4/wgACAD4//j/+P8IAPj/+P8IAAgA
+P8IAAgACAAIAAgA+
'
}
}
Google doesn't provide an option to specify local file path in json object while using curl command.We are require to convert audio file into base64 and pass the output into json object
base64 source_audio_file -w 0 > dest_base64_audio_file_content
Alternatively one of the best approach to pass localfile is to use gcloud ml command.
gcloud ml speech recognize 'Clip48_Mike.wav' --language-code='en-US'
Example:
NOTE:
Following command to activate gcloud account if not already done.
gcloud auth activate-service-account --key-file /tmp/account-name.json

PEP proxy config file for integration of IDM GE, PEP proxy and Cosmos big data

I have a question regarding PEP proxy file.
My keystone service is running on 192.168.4.33:5000.
My horizon service is running on 192.168.4.33:443.
My WebHDFS service is running on 192.168.4.180:50070
and i intend to run PEP Proxy on 192.168.4.180:80
But what i don't get is what should i put in place of config.account_host?
Inside mysql database for keyrock manager there is "idm" user with "idm" password and every request i make via curl on Identity manager works.
But with this config:
config.account_host = 'https://192.168.4.33:443';
config.keystone_host = '192.168.4.33';
config.keystone_port = 5000;
config.app_host = '192.168.4.180';
config.app_port = '50070';
config.username = 'idm';
config.password = 'idm';
when i start pep-proxy with:
sudo node server.js
i get next error:
Starting PEP proxy in port 80. Keystone authentication ...
Error in keystone communication {"error": {"message": "The request you
have made requires authentication.", "code": 401, "title":
"Unauthorized"}}
First, I wouldn't type the port at your config.account_host, as it is not required there, but this doesn't interfere the operation.
My guessing is that you are using your own KeyRock FIWARE Identity Manager with the default provision of roles.
If you check the code, PEP Proxy sends a Domain Scoped request against KeyRock, as stands in the Keystone v3 API.
So the thing is, the idm user you are using to authenticate PEP, probably doesn't have any domain roles. The workaround to check it would be:
Try the Domain Scoped request:
curl -i \
-H "Content-Type: application/json" \
-d '
{ "auth": {
"identity": {
"methods": ["password"],
"password": {
"user": {
"name": "idm",
"domain": { "id": "default" },
"password": "idm"
}
}
},
"scope": {
"domain": {
"id": "default"
}
}
}
}' \
http://192.168.4.33:5000/v3/auth/tokens ; echo
If you get a 401 code, you are not authorized to make Domain Scoped requests.
Check if the user has any role in this domain. For this you will need to get an Auth token using the Default Scope request:
curl -i -H "Content-Type: application/json" -d '
{ "auth": {
"identity": {
"methods": ["password"],
"password": {
"user": {
"name": "idm",
"domain": { "id": "default" },
"password": "idm"
}
}
}
}
}' http://192.168.4.33:5000/v3/auth/tokens ; echo
This will return a X-Subject-Token that you will need for the workaround.
With that token, we will send a request to the default domain using the user we selected before, idm, to check if we have assigned any roles there:
curl -i \
-H "X-Auth-Token:<retrieved_token>" \
-H "Content-type: application/json" \
http://192.168.4.33:5000/v3/domains/default/users/idm/roles
And probably, this request will give you a response like:
{"links": {"self": "http://192.168.4.33:5000/v3/domains/default/users/idm/roles", "previous": null, "next": null}, "roles": []}
In that case, you will need to create a role for that user. To create it, you will need to assing a role to the user idm in the default domain. For that, you will need to retrieve the role id of the role you want to assign. You can do this by sending the following request:
curl -i \
-H "X-Auth-Token:<retrieved_token>" \
-H "Content-type: application/json" \
http://192.168.4.33:5000/v3/roles
It will return a JSON with all the available roles and its ids.
Assign a role to the user idm in the default domain. There are 6 available: member, owner, trial, basic, community and admin. As idm is the main administrator, I would chose the admin id. So finally, with the admin id, we assign the role by doing:
curl -s -X PUT \
-H "X-Auth-Token:<retrieved_token>" \
-H "Content-type: application/json" \
http://192.168.4.33:5000/v3/domains/default/users/idm/roles/<role_id>
Now you can try again Step 1, and if everything works, you should be able to start the PEP proxy:
sudo node server.js
Let me know how it goes!