When attempting to export from Cloud SQL using a RESTful call to the Google API we've been encountering random failures, approximately 1 in 10 attempts. The log extract below has been taken from Stackdriver and doesn't offer much insight into the root cause:
{
insertId: "55C4B416A7642.A1B5D8E.570939C6"
logName: "projects/my-dev-project/logs/cloudaudit.googleapis.com%2Fdata_access"
protoPayload: {
#type: "type.googleapis.com/google.cloud.audit.AuditLog"
authenticationInfo: {
principalEmail: "jsmith#mycompany.com"
}
authorizationInfo: [
0: {
authorizationLoggingOptions: {
permissionType: "DATA_READ"
}
granted: true
permission: "cloudsql.instances.export"
resource: "instances/my-db-instance"
}
]
methodName: "cloudsql.instances.export"
request: {
#type: "type.googleapis.com/cloudsql.admin.InstancesExportRequest"
exportContext: {
database: [
0: "business_db"
]
fileType: "CSV"
schemaOnly: true
selectQuery:
"SELECT
replace(IFNULL(`ID`,'<NULL_VALUE>'), '', '') AS `ID`,
replace(IFNULL(`NAME`,'<NULL_VALUE>'), '', '') AS `NAME`,
replace(IFNULL(`ADDRESS`,'<NULL_VALUE>'), '', '') AS `ADDRESS`,
replace(IFNULL(`DOB`,'<NULL_VALUE>'), '', '') AS `DOB`
FROM CUSTOMERS"
table: [
0: "CUSTOMERS"
]
uri: "gs://my-dev-project-stage/CUSTOMERS_full.csv"
}
instanceName: {
fullProjectId: "my-dev-project"
instanceId: "my-db-instance"
}
}
requestMetadata: {
callerIp: "107.178.192.158"
}
resourceName: "instances/my-db-instance"
serviceName: "cloudsql.googleapis.com"
status: {
code: 2
message: "UNKNOWN"
}
}
receiveTimestamp: "2017-10-24T13:52:53.738754270Z"
resource: {
labels: {
database_id: "my-dev-project:my-db-instance"
project_id: "my-dev-project"
region: "europe"
}
type: "cloudsql_database"
}
severity: "ERROR"
timestamp: "2017-10-24T13:52:53.317Z"
}
The only difference between this log entry and a successful export are the three property values mentioned in the post title. In a successful entry they are as follows:
status: {
}
}
receiveTimestamp: "2017-10-24T14:21:44.997050352Z"
resource: {
labels: {
database_id: "my-dev-project:my-db-instance"
project_id: "my-dev-project"
region: "europe"
}
type: "cloudsql_database"
}
severity: "INFO"
timestamp: "2017-10-24T14:21:43.757Z"
}
The intermittent nature suggests the root cause is an external factor. I.e. the exact same request with the same parameters; connection details, output file location and SQL statement will work a few minutes later. There's no consistency to which exports are failing.
We've already discovered that Cloud SQL does not support multiple simultaneous exports and we've implemented a queuing system to avoid this issue.
The data volumes don't seem to be an influencing factor either. An export will work for a 100,000 row file and then fail for 500 row file. In fact smaller or even empty data sets seem to be more susceptible to failure.
If anyone has any idea's or suggestions as to a solution or avenue of investigation I'd be interested to hear them. Thanks.
Related
I´m trying to specify the API used by ansibles dynamic inventory.
Does anyone have experience with solving the incompatibility?
For a single host it might work as following:
The Ansible json output will be like this:
{
"ansible_host": "172.16.19.123",
"proxy": "somehost.domain.fake"
}
The openapi.yml
paths:
/api/inventory/host/:
get:
summary: Gets One Ansible Host
parameters:
- in: query
name: hostname
schema:
type: string
required: true
responses:
'200':
description: OK
content:
application/json:
schema:
$ref: '#/components/schemas/SingleHost'
components:
schemas:
SingleHost: # A Single Host for ansible-inventory --host [hostname] requests
type: object
properties:
ansible_host:
type: string
description: IP-Address of the host
ansible_port:
type: integer
description: ssh-port number of host
But when specifiyng the endpoint for ansible-inventory --list it gets tricky already..
example of a ansible inventory yml
{
"HostGroupName-A": {
"hosts": [
"Host-A"
]
},
"HostGroupName-B": {
"hosts": [
"Host-A",
"Host-B"
]
}
}
Should I just avoid using openapi to specify this?
Using guidance from a previous SO post: [https://stackoverflow.com/questions/43266506/is-it-possible-to-execute-a-cloudformation-file-in-terraform]:
I am trying to pass two parameters created in my Terraform template into a Cloudformation template as parameters:
My terraform (v.0.13.5) code:
resource "aws_cloudformation_stack" "cloudwatch-synthetics-canary" {
provider = aws.primary
name = "cloudwatch-synthetics"
parameters = {
CloudWatchSyntheticsRole = aws_iam_role.cloudwatch_synthetics_role.arn,
ResultsBucket = aws_s3_bucket.results_bucket.arn
}
template_body = file("${path.module}/cloudwatch_canary.yml")
}
resource "aws_iam_role" "cloudwatch_synthetics_role" {
provider = aws.primary
name = "CloudWatchSyntheticsRole"
description = "Allows Cloudwatch Lambda to operate in this account."
assume_role_policy = <<POLICY
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowLambdaAssumeRole",
"Effect": "Allow",
"Principal": {
"Service": "lambda.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
POLICY
}
resource "aws_iam_policy" "cloudwatch_synthetics_role_policy" {
provider = aws.primary
name = "CloudWatchSyntheticsRolePolicy"
path = "/"
description = "Addtional allowances for the synthetics role"
policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
# --- SNIPPED FOR BREVITY ---
]
}
EOF
}
resource "aws_iam_role_policy_attachment" "cloudwatch_synthetics_role_policy_attachment" {
provider = aws.primary
role = aws_iam_role.cloudwatch_synthetics_role.name
policy_arn = aws_iam_policy.cloudwatch_synthetics_role_policy.arn
}
My Cloudformation code:
Parameters:
CanaryName:
Type: String
Default: my-canary
MaxLength: 21
HostName:
Type: String
Default: api.myhost.net
MaxLength: 128
Path:
Type: String
Default: /v1/status
MaxLength: 256
Port:
Type: Number
Default: 443
CloudWatchSyntheticsRole:
Type: AWS::IAM::Role
ResultsBucket:
Type: AWS::S3::Bucket
...
Resources:
ExecutionRoleArn:
Fn::GetAtt:
- CloudWatchSyntheticsRole
- Arn # <-- TRIED WITH AND WITHOUT THIS
In Terraform.io, the error is as follows:
Error: Creating CloudFormation stack failed: ValidationError: Template error: instance of Fn::GetAtt references undefined resource CloudWatchSyntheticsRole
status code: 400, request id: 694c...
I tried making the types in the CFT as Strings, but that didn't seem to work, either. The Fn::GetAtt: also had '- Arn' below the CloudwatchSyntheticsRole to refer to the Arn, but since I'm passing that in directly, I tried removing it.
I feel like I'm close, but missing something and just need some help from some fresh eyes.
It looks like the solution, found by a co-worker of mine is pretty simple:
Make sure ALL CFT variables are in Terraform and pass any default values through Terraform as literals to Cloudformation
Remove all default parameters from CloudFormation, but still keep the Name and Type of variable at a minimum. Other constraints like 'MaxLength' are ok, too.
Example:
Terraform.tf:
resource "aws_cloudformation_stack" "cloudwatch-synthetics-canary" {
provider = aws.primary
name = "cloudwatch-synthetics"
parameters = {
CanaryName = "my-canary",
HostName = "api.server.net",
Path = "/v1/status",
Port = 443,
RoleArn = aws_iam_role.cloudwatch_synthetics_role.arn,
S3Location = "s3://${aws_s3_bucket.results_bucket.id}"
}
template_body = file("${path.module}/cloudwatch_canary.yml")
}
Cloudformation.yml:
Parameters:
CanaryName:
Type: String
MaxLength: 21
HostName:
Type: String
MaxLength: 128
Path:
Type: String
MaxLength: 256
Port:
Type: Number
RoleArn:
Type: String
S3Location:
Type: String
MaxLength: 1024
This is the sync-gateway-config.json file that I am using to authenticate:
**sync-gateway-config.json** file
{
“interface”:":4984",
“logging”: {
“log_file_path”: “C:/Program Files/Couchbase/Sync Gateway/var/lib/couchbase/logs/sg_logs”,
“console”: {
“enabled”: true,
“log_level”: “info”,
“log_keys”: [
“*”
]
}
},
“databases”:{
“db”:{
“use_views”: true,
“cacertpath”: “C:/Program Files/servercertfiles/ca.pem”,
“certpath”: “C:/Program Files/servercertfiles/clientcertfiles/seedata.pem”,
“keypath”: “C:/Program Files/servercertfiles/clientcertfiles/seedata.key”,
“server”: “couchbases://127.0.0.1”,
“bucket”: “bucketname”,
“username”: “sync_gateway”
}
}
}
But I am getting following error:
[INF] GoCBCustomSGTranscoder Opening Couchbase database bucketname on couchbases://127.0.0.1 as user “sync_gateway”
2020-07-13T15:38:20.566+05:30 [INF] Error opening bucket bucketname: failed to connect to any of the specified hosts
Links followed :*https://docs.couchbase.com/sync-gateway/2.7/security.html#x-509-certificates*
I have deployed an API-Platform app using JWT token to ElasticBeanstalk which, as usual, works fine in my local server.
On EB though it is denying access to logged in users despite the correct BearerToken being provided.
This is the error thrown:
{
"errors": [
{
"message": "Access Denied.",
"extensions": {
"category": "graphql"
},
"locations": [
{
"line": 6,
"column": 9
}
],
"path": [
"retrievedQueryUser"
]
}
],
"data": {
"retrievedQueryUser": null
}
}
The query in question attempts to retrieve user profile info through the below graphql config:
* "retrievedQuery"={
* "item_query"=UserProfileResolver::class,
* "normalization_context"={"groups"={"get-owner"}},
* "security"="is_granted('IS_AUTHENTICATED_FULLY') and object == user"
* },
So, it should be a simple matter of checking if the users IS_AUTHENTICATED_FULLY and if it is the user him/herself trying to execute the query.
Far as I could tell, by dump below on /vendor/symfony/security-core/Authorization/AuthorizationChecker.php, it's failing to retrieve a token.
var_dump($this->tokenStorage->getToken()->getUser()->getUsername());
I did a cursory comparison of phpinfo() between my local installation and the one at AWS-EB and could not find any obvious mismatch.
This is the config for JWT at /config/packages/lexik_jwt_authentication.yaml.
lexik_jwt_authentication:
secret_key: '%env(resolve:JWT_SECRET_KEY)%'
public_key: '%env(resolve:JWT_PUBLIC_KEY)%'
pass_phrase: '%env(JWT_PASSPHRASE)%'
user_identity_field: email
token_ttl: 1800
Just to confirm that the users are able to login. It's passing through the isGranted() check that fails.
Any ideas?
EDIT - add `/config/packages/security.yaml
security:
# https://symfony.com/doc/current/security.html#where-do-users-come-from-user-providers
encoders:
App\Entity\User:
algorithm: auto
#algorithm: bcrypt
#algorithm: argon2i
cost: 12
providers:
database:
entity:
class: App\Entity\User
property: email
firewalls:
dev:
pattern: ^/(_(profiler|wdt)|css|images|js)/
security: false
refresh:
pattern: ^/api/token/refresh
stateless: true
anonymous: true
api:
pattern: ^/api
stateless: true
anonymous: true
json_login:
check_path: /api/login_check
success_handler: lexik_jwt_authentication.handler.authentication_success
failure_handler: lexik_jwt_authentication.handler.authentication_failure
guard:
authenticators:
- app.google_login_authenticator
- App\Security\TokenAuthenticator
entry_point: App\Security\TokenAuthenticator
user_checker: App\Security\UserEnabledChecker
access_control:
- { path: ^/login, roles: IS_AUTHENTICATED_ANONYMOUSLY }
- { path: ^/admin, roles: ROLE_SUPERADMIN }
- { path: ^/api/token/refresh, roles: IS_AUTHENTICATED_ANONYMOUSLY }
- { path: ^/api, roles: IS_AUTHENTICATED_ANONYMOUSLY }
role_hierarchy:
ROLE_PROVIDER: ROLE_USER
ROLE_ADMIN: [ROLE_PROVIDER, ROLE_EDITOR]
ROLE_SUPERADMIN: ROLE_ADMIN
Upon further research I found out that Apache was stripping the authorization token from the request.
On the method supports of /lexik/jwt-authenticator-bundle/Security/Guard/JWTTokenAuthenticator, the dump as below will not include the token on AWS:
var_dump($request->headers->all());
var_dump($_SERVER);
As per this question, this is an issue of Apache configuration which is not accepting the authorization headers.
The indicated solution is to add the following to .htaccess:
SetEnvIf Authorization "(.*)" HTTP_AUTHORIZATION=$1
This resolves the issue, though one should note that the local Apache installation works fine without the above edit to .htaccess.
So, it should also be possible to change Apache config directly, but I could not find how to go about it.
EDIT: Later I found a specific instruction on 'JWT-Token' docs as follows, that confirm that solution on this link.
I have a inventory file which has a RDS endpoint as :
[ems_db]
syd01-devops.ce4l9ofvbl4z.ap-southeast-2.rds.amazonaws.com
I wrote the following play book to create a Cloudwatch ALARM :
---
- name: Get instance ec2 facts
debug: var=groups.ems_db[0].split('.')[0]
register: ems_db_name
- name: Display
debug: var=ems_db_name
- name: Create CPU utilization metric alarm
ec2_metric_alarm:
state: present
region: "{{aws_region}}"
name: "{{ems_db_name}}-cpu-util"
metric: "CPUUtilization"
namespace: "AWS/RDS"
statistic: Average
comparison: ">="
unit: "Percent"
period: 300
description: "It will be triggered when CPU utilization is more than 80% for 5 minutes"
dimensions: { 'DBInstanceIdentifier' : ems_db_name }
alarm_actions: arn:aws:sns:ap-southeast-2:493552970418:cloudwatch_test
ok_actions: arn:aws:sns:ap-southeast-2:493552970418:cloudwatch_test
But this results in
TASK: [cloudwatch | Get instance ec2 facts] ***********************************
ok: [127.0.0.1] => {
"var": {
"groups.ems_db[0].split('.')[0]": "syd01-devops"
}
}
TASK: [cloudwatch | Display] **************************************************
ok: [127.0.0.1] => {
"var": {
"ems_db_name": {
"invocation": {
"module_args": "var=groups.ems_db[0].split('.')[0]",
"module_complex_args": {},
"module_name": "debug"
},
"var": {
"groups.ems_db[0].split('.')[0]": "syd01-devops"
},
"verbose_always": true
}
}
}
TASK: [cloudwatch | Create CPU utilization metric alarm] **********************
failed: [127.0.0.1] => {"failed": true}
msg: BotoServerError: 400 Bad Request
<ErrorResponse xmlns="http://monitoring.amazonaws.com/doc/2010-08-01/">
<Error>
<Type>Sender</Type>
<Code>MalformedInput</Code>
</Error>
<RequestId>f30470a3-2d65-11e6-b7cb-cdbbbb30b60b</RequestId>
</ErrorResponse>
FATAL: all hosts have already failed -- aborting
What is wrong here? What can i do to solve this ? I am new to this but surely this seems some syntax issue with me or the way i am picking up the inventory endpoint split.
The variable from debug isn't being assigned in the first debug statement, though you may be able to if you change it to a message and enclose it with quotes and double braces (untested):
- name: Get instance ec2 facts
debug: msg="{{groups.ems_db[0].split('.')[0]}}"
register: ems_db_name
However, I would use the set_fact module in that task (instead of debug) and assign that value to it. That way, you can reuse it in this and subsequent calls to a play.
- name: Get instance ec2 facts
set_fact: ems_db_name="{{groups.ems_db[0].split('.')[0]}}"
UPDATE: Add a threshold: 80.0 to the last task, and the dimensions needs to use the instance id encapsulated with double braces.