Related
I am trying to deploy a Cloud Function from a Cloud Source Repository placed in a different project, but getting the following error: Failed to retrieve function source code (see full proto below).
Project-A contains the cloud function and service accounts listed below.
Project-B contains the source repository.
I have successfully deployed the function on Project-B.
I've tried giving the following service accounts the Source Repository Administrator role on the cloud source repository, but that did not help.
{project_A_number}#cloudservices.gserviceaccount.com
{project_A_number}-compute#developer.gserviceaccount.com
{project_A_number}#cloudbuild.gserviceaccount.com
Project-A#appspot.gserviceaccount.com
I have also tried disabling the Cloud Functions API on Project-A and turning it back on again.
I am not sure what is going wrong - if anyone has a clue as to where to further look, I would appreciate it - thanks in advance!
The deployment creates two entries in monitoring - a NOTICE followed by an ERROR:
The ERROR log:
{
"protoPayload": {
"#type": "type.googleapis.com/google.cloud.audit.AuditLog",
"status": {
"code": 5,
"message": "Failed to retrieve function source code"
},
"authenticationInfo": {
"principalEmail": "***#***.**"
},
"serviceName": "cloudfunctions.googleapis.com",
"methodName": "google.cloud.functions.v1.CloudFunctionsService.UpdateFunction",
"resourceName": "projects/Project-A/locations/europe-west1/functions/pubsub-to-gcs"
},
"insertId": "-vmfbt4cd54",
"resource": {
"type": "cloud_function",
"labels": {
"function_name": "pubsub-to-gcs",
"region": "europe-west1",
"project_id": "Project-A"
}
},
"timestamp": "2021-10-20T12:21:45.352043Z",
"severity": "ERROR",
"logName": "projects/Project-A/logs/cloudaudit.googleapis.com%2Factivity",
"operation": {
"id": "operations/cm9ldHotbGlmZS1kYXRhLXRlc3QvZXVyb3BlLXdlc3QxL3B1YnN1Yi10by1nY3MvVEhFbUQtLTZITWM",
"producer": "cloudfunctions.googleapis.com",
"last": true
},
"receiveTimestamp": "2021-10-20T12:21:45.781856467Z"
}
The NOTICE log (logged right before the ERROR):
{
"protoPayload": {
"#type": "type.googleapis.com/google.cloud.audit.AuditLog",
"authenticationInfo": {
"principalEmail": "***#****.**"
},
"requestMetadata": {
"callerIp": "35.205.252.75",
"callerSuppliedUserAgent": "google-cloud-sdk gcloud/360.0.0 command/gcloud.functions.deploy invocation-id/917d697431e84b91bfa2bd9f9cc4f302 environment/devshell environment-version/None interactive/True from-script/False python/3.7.3 term/screen (Linux 5.4.144+),gzip(gfe),gzip(gfe)",
"requestAttributes": {
"time": "2021-10-20T12:21:44.909430Z",
"auth": {}
},
"destinationAttributes": {}
},
"serviceName": "cloudfunctions.googleapis.com",
"methodName": "google.cloud.functions.v1.CloudFunctionsService.UpdateFunction",
"authorizationInfo": [
{
"resource": "projects/Project-A/locations/europe-west1/functions/pubsub-to-gcs",
"permission": "cloudfunctions.functions.update",
"granted": true,
"resourceAttributes": {}
}
],
"resourceName": "projects/Project-A/locations/europe-west1/functions/pubsub-to-gcs",
"request": {
"#type": "type.googleapis.com/google.cloud.functions.v1.UpdateFunctionRequest",
"function": {
"timeout": "60s",
"status": "UNKNOWN",
"serviceAccountEmail": "Project-A#appspot.gserviceaccount.com",
"availableMemoryMb": 256,
"name": "projects/Project-A/locations/europe-west1/functions/pubsub-to-gcs",
"runtime": "python39",
"labels": {
"deployment-tool": "cli-gcloud"
},
"entryPoint": "pubsub-to-gcs",
"updateTime": "2021-10-20T12:21:40.149Z",
"sourceRepository": {
"url": "https://source.developers.google.com/projects/Project-B/repos/my-repo/moveable-aliases/master/paths/my-folder"
},
"httpsTrigger": {},
"ingressSettings": "ALLOW_ALL",
"versionId": "1"
},
"updateMask": "eventTrigger,httpsTrigger,runtime,sourceRepository"
},
"resourceLocation": {
"currentLocations": [
"europe-west1"
]
}
},
"insertId": "1xdbim3e16pgu",
"resource": {
"type": "cloud_function",
"labels": {
"function_name": "pubsub-to-gcs",
"region": "europe-west1",
"project_id": "Project-A"
}
},
"timestamp": "2021-10-20T12:21:44.650257Z",
"severity": "NOTICE",
"logName": "projects/Project-A/logs/cloudaudit.googleapis.com%2Factivity",
"operation": {
"id": "operations/cm9ldHotbGlmZS1kYXRhLXRlc3QvZXVyb3BlLXdlc3QxL3B1YnN1Yi10by1nY3MvVEhFbUQtLTZITWM",
"producer": "cloudfunctions.googleapis.com",
"first": true
},
"receiveTimestamp": "2021-10-20T12:21:45.832588036Z"
}
Turns out it wasn't an IAM issue: I've tried deploying the function from the UI, but that's not possible when deploying from a source repo in a different project.
Deploying using gcloud function deploy solved the issue.
I was searching a way to run ecs task. I already have a cluster and task definition settings. I just wanted to trigger a task using CloudFormation template. I know that I can run a task by clicking on the console and it works fine. For cfn, approach needs to be define properly.
Check the attached screenshots. I wanted to run that task using CloudFormation and pass container override environment variables. As per my current templates, it is not allowing me to do same like I can do using console. Using console I just need to select the following options
1. Launch type
2. Task Definition
Family
Revision
3. VPC and security groups
4. Environment variable overrides rest of the things automatically selected
It starts working with console but with cloudformaton template how can we do that. Is it possible to do or there is no such feature?
"taskdefinition": {
"Type" : "AWS::ECS::TaskDefinition",
"DependsOn": "DatabaseMaster",
"Properties" : {
"ContainerDefinitions" : [{
"Environment" : [
{
"Name" : "TARGET_DATABASE",
"Value" : {"Ref":"DBName"}
},
{
"Name" : "TARGET_HOST",
"Value" : {"Fn::GetAtt": ["DatabaseMaster", "Endpoint.Address"]}
}
]
}],
"ExecutionRoleArn" : "arn:aws:iam::xxxxxxxxxx:role/ecsTaskExecutionRole",
"Family" : "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"TaskRoleArn" : "arn:aws:iam::xxxxxxxxxxxxxxx:role/xxxxxxxxxxxxxxx-XXXXXXXXX"
}
},
"EcsService": {
"Type" : "AWS::ECS::Service",
"Properties" : {
"Cluster" : "xxxxxxxxxxxxxxxxx",
"LaunchType" : "FARGATE",
"NetworkConfiguration" : {
"AwsvpcConfiguration" : {
"SecurityGroups" : ["sg-xxxxxxxxxxx"],
"Subnets" : ["subnet-xxxxxxxxxxxxxx"]
}
},
"TaskDefinition" : "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
}
}
There is no validity error in the code however, I am talking about the approach. I added image name container name but now it is asking for memory and cpu, it should not ask as it is already defined we just need to run a task.
Edited
I wanted to run a task after creation of my database and wanted to pass those database values to the task to run and complete a job.
Here is the working example of what you can do if you wanted to pass variable and run a task. In my case, I wanted to run a task after creation of my database but with environment variables, directly AWS does not provide any feature to do so, this is the solution which can help to trigger you ecs task.
"IAMRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"events.amazonaws.com"
]
},
"Action": [
"sts:AssumeRole"
]
}
]
},
"Description": "Allow CloudWatch Events to trigger ECS task",
"Policies": [
{
"PolicyName": "Allow-ECS-Access",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ecs:*",
"iam:PassRole",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "*"
}
]
}
}
],
"RoleName": { "Fn::Join": [ "", ["CloudWatchTriggerECSRole-", { "Ref": "DBInstanceIdentifier" }]]}
}
},
"DummyParameter": {
"Type" : "AWS::SSM::Parameter",
"Properties" : {
"Name" : {"Fn::Sub": "${AWS::StackName}-${DatabaseMaster}-EndpointAddress"},
"Type" : "String",
"Value" : {"Fn::GetAtt": "DatabaseMaster.Endpoint.Address"}
},
"DependsOn": "TaskSchedule"
},
"TaskSchedule": {
"Type": "AWS::Events::Rule",
"Properties": {
"Description": "Trigger ECS task upon creation of DB instance",
"Name": { "Fn::Join": [ "", ["ECSTaskTrigger-", { "Ref": "DBName" }]]},
"RoleArn": {"Fn::GetAtt": "IAMRole.Arn"},
"EventPattern": {
"source": [ "aws.ssm" ],
"detail-type": ["Parameter Store Change"] ,
"resources": [{"Fn::Sub":"arn:aws:ssm:eu-west-1:XXXXXXX:parameter/${AWS::StackName}-${DatabaseMaster}-EndpointAddress"}],
"detail": {
"operation": ["Create"],
"name": [{"Fn::Sub": "${AWS::StackName}-${DatabaseMaster}-EndpointAddress"}],
"type": ["String"]
}
},
"State": "ENABLED",
"Targets": [
{
"Arn": "arn:aws:ecs:eu-west-1:xxxxxxxx:cluster/NameOf-demo",
"Id": "NameOf-demo",
"RoleArn": {"Fn::GetAtt": "IAMRole.Arn"},
"EcsParameters": {
"LaunchType": "FARGATE",
"NetworkConfiguration": {
"AwsVpcConfiguration": {
"SecurityGroups": {"Ref":"VPCSecurityGroups"},
"Subnets": {"Ref":"DBSubnetName"}
}
},
"PlatformVersion": "LATEST",
"TaskDefinitionArn": "arn:aws:ecs:eu-west-1:XXXXXXXX:task-definition/NameXXXXXXXXX:1"
},
"Input": {"Fn::Sub": [
"{\"containerOverrides\":[{\"name\":\"MyContainerName\",\"environment\":[{\"name\":\"VAR1\",\"value\":\"${TargetDatabase}\"},{\"name\":\"VAR2\",\"value\":\"${TargetHost}\"},{\"name\":\"VAR3\",\"value\":\"${TargetHostPassword}\"},{\"name\":\"VAR4\",\"value\":\"${TargetPort}\"},{\"name\":\"VAR5\",\"value\":\"${TargetUser}\"},{\"name\":\"VAR6\",\"value\":\"${TargetLocation}\"},{\"name\":\"VAR7\",\"value\":\"${TargetRegion}\"}]}]}",
{
"VAR1": {"Ref":"DBName"},
"VAR2": {"Fn::GetAtt": ["DatabaseMaster", "Endpoint.Address"]},
"VAR3": {"Ref":"DBPassword"},
"VAR4": "5432",
"VAR5": {"Ref":"DBUser"},
"VAR6": "value6",
"VAR7": "eu-west-2"
}
]}
}
]
}
}
For Fargate task, we need to specify in CPU in Task Definition. and memory or memory reservation in either task or container definition.
and environment variables should be passed to each container as ContainerDefinitions and overrided when task is run from ecs task-run from console or cli.
{
"ContainerTaskdefinition": {
"Type": "AWS::ECS::TaskDefinition",
"Properties": {
"Family": "SomeFamily",
"ExecutionRoleArn": !Ref RoleArn,
"TaskRoleArn": !Ref TaskRoleArn,
"Cpu": "256",
"Memory": "1GB",
"NetworkMode": "awsvpc",
"RequiresCompatibilities": [
"EC2",
"FARGATE"
],
"ContainerDefinitions": [
{
"Name": "container name",
"Cpu": 256,
"Essential": "true",
"Image": !Ref EcsImage,
"Memory": "1024",
"LogConfiguration": {
"LogDriver": "awslogs",
"Options": {
"awslogs-group": null,
"awslogs-region": null,
"awslogs-stream-prefix": "ecs"
}
},
"Environment": [
{
"Name": "ENV_ONE_KEY",
"Value": "Valu1"
},
{
"Name": "ENV_TWO_KEY",
"Value": "Valu2"
}
]
}
]
}
}
}
EDIT(from discussion in comments):
ECS Task Run is not a cloud-formation resource, it can only be run from console or CLI.
But if we choose to run from a cloudformation resource, it can be done using cloudformation custom resource. But once task ends, we now have a resource in cloudformation without an actual resource behind. So, custom resource needs to do:
on create: run the task.
on delete: do nothing.
on update: re-run the task
Force an update by changing an attribute or logical id, every time we need to run the task.
When using the "Open Folder" functionality of Visual Studio, the IDE searches for project settings and configurations in a special json file. For CPP projects, this could be CppProperties.json. For CMake projects, this could be CMakeSettings.json.
This json file contains a collection of one or more "configurations," such as "Debug" or "Release". I will use a recent CMake project as an example:
"configurations": [
{
"name": "ARM-Debug",
"generator": "Ninja",
"configurationType": "Debug",
"inheritEnvironments": [
"gcc-arm"
],
"buildRoot": "${env.USERPROFILE}\\CMakeBuilds\\${workspaceHash}\\build\\${name}",
"installRoot": "${env.USERPROFILE}\\CMakeBuilds\\${workspaceHash}\\install\\${name}",
"cmakeCommandArgs": "",
"buildCommandArgs": "-v",
"ctestCommandArgs": "",
"intelliSenseMode": "linux-gcc-arm",
"variables": [
{
"name": "CMAKE_TOOLCHAIN_FILE",
"value": "${workspaceRoot}/cmake/arm-none-eabi-toolchain.cmake"
}
]
},
{
"name": "ARM-Release",
"generator": "Ninja",
"configurationType": "Release",
"inheritEnvironments": [
"gcc-arm"
],
"buildRoot": "${env.USERPROFILE}\\CMakeBuilds\\${workspaceHash}\\build\\${name}",
"installRoot": "${env.USERPROFILE}\\CMakeBuilds\\${workspaceHash}\\install\\${name}",
"cmakeCommandArgs": "",
"buildCommandArgs": "-v",
"ctestCommandArgs": "",
"intelliSenseMode": "linux-gcc-arm",
"variables": [
{
"name": "CMAKE_TOOLCHAIN_FILE",
"value": "${workspaceRoot}/cmake/arm-none-eabi-toolchain.cmake"
}
]
}
As you can see, I have two configurations with nearly identical properties.
My question: is it possible to define these common/shared properties once, in such a way as to allow the configurations to inherit them and avoid repeating myself?
The easier way is to define an environment at global level (outside of any configuration), such as:
{
"environments": [
{
"namespace" : "env",
"varName": "varValue"
}
],
Then you can reuse that wherever you need to, e.g.:
"cmakeCommandArgs": "${env.varName}",
You can also have multiple environments, and reuse them, like this:
{
"environments": [
{
"environment": "env1",
"namespace": "env",
"varName": "varValueEnv1"
},
{
"environment": "env2",
"namespace": "env",
"varName": "varValueEnv2"
}
],
"configurations": [
{
"name": "x64-Release",
"inheritEnvironments": [
"msvc_x64_x64", "env2"
],
"cmakeCommandArgs": "${env.varName}",
.....
}
]
the 'x64-Release' will inherit the variables's value in the environment called "env2" (namespace 'env')
I'm trying to compile c++ inside VS Code.
I have MinGW installed.
I've followed the steps in this video: https://www.youtube.com/watch?v=rFdJ68WbkdQ
And the steps at the "getting started" docs https://code.visualstudio.com/docs/languages/cpp
Actually, my config shows like this:
{
"configurations": [
{
"name": "Mac",
"includePath": [
"/usr/include",
"/usr/local/include",
"${workspaceRoot}"
],
"defines": [],
"intelliSenseMode": "clang-x64",
"browse": {
"path": [
"/usr/include",
"/usr/local/include",
"${workspaceRoot}"
],
"limitSymbolsToIncludedHeaders": true,
"databaseFilename": ""
},
"macFrameworkPath": [
"/System/Library/Frameworks",
"/Library/Frameworks"
]
},
{
"name": "Linux",
"includePath": [
"/usr/include",
"/usr/local/include",
"${workspaceRoot}"
],
"defines": [],
"intelliSenseMode": "clang-x64",
"browse": {
"path": [
"/usr/include",
"/usr/local/include",
"${workspaceRoot}"
],
"limitSymbolsToIncludedHeaders": true,
"databaseFilename": ""
}
},
{
"name": "Win32",
"includePath": [
"C:/Program Files (x86)/Microsoft Visual Studio/2017/Community/VC/Tools/MSVC/14.11.25503/include/*",
"C:/Program Files (x86)/Microsoft Visual Studio/2017/Community/VC/Tools/MSVC/14.11.25503/atlmfc/include/*",
"C:/Program Files (x86)/Windows Kits/10/Include/10.0.16299.0/um",
"C:/Program Files (x86)/Windows Kits/10/Include/10.0.16299.0/ucrt",
"C:/Program Files (x86)/Windows Kits/10/Include/10.0.16299.0/shared",
"C:/Program Files (x86)/Windows Kits/10/Include/10.0.16299.0/winrt",
"${workspaceRoot}"
],
"defines": [
"_DEBUG",
"UNICODE"
],
"intelliSenseMode": "msvc-x64",
"browse": {
"path": [
"${workspaceRoot}",
"C:\\MinGW\\lib\\gcc\\mingw32\\6.3.0\\include\\c++",
"C:\\MinGW\\bin"
],
"limitSymbolsToIncludedHeaders": true,
"databaseFilename": ""
}
}
],
"version": 3
}
And the "tasks.json" file has the following:
{
// See https://go.microsoft.com/fwlink/?LinkId=733558
// for the documentation about the tasks.json format
"version": "2.0.0",
"tasks": [
{
"label": "build",
"type": "shell",
"command": "g++",
"args": [
"-g", "Calculator.cpp", "-o","Calculator"
],
"group": {
"kind": "build",
"isDefault": true
},
"problemMatcher":"$gcc"
}
]
}
But when I hit "run main task" it prompts:
"> Executing task: g++ -g Calculator.cpp -o Calculator <
'g++' is not recognized as an internal or external command,
operable program or batch file.
The terminal process terminated with exit code: 1"
How can I get gcc detected?
I'm using VS Code with Windows 10 machine BTW.
If you have not added the folder path to windows, try that first.
If that still does not work try adding the full path in "task.json" instead of just g++.
Something like this:
C:/MinGW/bin/g++
In my composer setup I load packages from 2 repos. One composer repo and a private gitlab repo. When I try to do a composer install on my local windows machine, I get the following error messages:
GitLab: The project you were looking for could not be found. fatal:
Could not read from remote repository. Please make sure you have the
correct access rights and the repository exists. ... The requested
package sv-test/testextension could not be found in any version, there
may be a typo in the package name.
Authentification is done via lokal ssh-key and password, that doesnt seem to be the problem. Whats wrong with my setup?
The composer.json of the project looks like this:
{
"repositories": [
{
"type": "composer",
"url": "https://composer.typo3.org/"
},
{
"type": "git",
"url": "git#gitlab.xydevbox.de:sv-test/Testproject.git"
}
],
"name": "svdev/master-dev-box",
"description": "",
"type": "project",
"license": "MIT",
"homepage": "https://www.xydevbox.de/",
"authors": [
{
"name": "Sacha Vorbeck",
"email": "sacha.vorbeck#xydevbox.de",
"homepage": "https://www.xydevbox.de/",
"role": "Developer"
}
],
"require": {
"sv-testbox/testextension": "*",
"helhum/typo3-console": "^4.5",
"typo3/cms": "^8.7"
},
"config": {
"sort-packages": true,
"process-timeout": 2000,
"preferred-install": {
"typo3/cms": "source",
"*": "dist"
}
},
"extra": {
"typo3/cms": {
"cms-package-dir": "{$vendor-dir}/typo3/cms",
"web-dir": "web"
}
}
}
The composer.json from the package to be included from the gitlab private repo looks like this:
{
"name": "sv-testbox/testextension",
"type": "typo3-cms-extension",
"description": "",
"homepage": "https://www.xydevbox.de/",
"license": ["GPL-2.0+"],
"keywords": ["TYPO3 CMS"],
"version": "master",
"dist": {
"url": "git#gitlab.xydevbox.de:sv-test/Testproject.git",
"type": "git"
}
}
Maybe the latest composer version 1.5.2 is something for you if you take a look at the release notes
Fixed GitLabDriver looping endlessly in some conditions
Fixed
GitLabDriver support for unauthenticated requests
Fixed GitLab zip
downloads not triggering credentials prompt if unauthenticated
Fixed
path repository support of COMPOSER_ROOT_VERSION, it now applies to
all path repos within the same git repository
Fixed path repository
handling of copies to avoid copying VCS files and others
Fixed
sub-directory call to ignore list and create-project commands as well
as calls to Composer using --working-dir
Fixed invalid warning
appearing when calling remove on an non-stable package
https://github.com/composer/composer/releases
Thank you Georg and NextThursday. With some help on TYPO3 slack I finally got it running. The replace part was missing. I also learned that one should not edit composer.json files manually - always use the command line options to modify it. This example: https://github.com/TYPO3-Console/TYPO3-Console/blob/master/composer.json was also helpful.
project composer.json:
{
"repositories": [
{
"type": "composer",
"url": "https://composer.typo3.org/"
},
{
"type": "vcs",
"url": "https://github.com/svorbeck/masterconfig"
}
],
"name": "svorbeck/demo",
"description": "",
"type": "project",
"license": "MIT",
"homepage": "https://xydevbox.de/",
"authors": [
{
"name": "Sacha Vorbeck",
"email": "sacha.vorbeck#xydevbox.de",
"role": "Developer"
}
],
"require": {
"svorbeck/masterconfig": "dev-master",
"typo3/cms": "^8.7"
},
"config": {
"sort-packages": true,
"process-timeout": 2000,
"preferred-install": {
"typo3/cms": "source",
"svorbeck/masterconfig": "source",
"*": "dist"
}
},
"extra": {
"typo3/cms": {
"cms-package-dir": "{$vendor-dir}/typo3/cms",
"web-dir": "web"
},
"helhum/typo3-console": {
"install-extension-dummy": "0"
}
}
}
ext composer.json:
{
"name": "svorbeck/masterconfig",
"type": "typo3-cms-extension",
"description": "svorbeck master configuration",
"require": {
"typo3/cms-core": "^8.7"
},
"replace": {
"masterconfig": "self.version",
"svorbeck/masterconfig": "self.version"
}
}