I am trying to generalize a resource policy for an aws secret that multiple lambdas will grab from (project based). Currently what i have below works
{
"Version" : "2012-10-17",
"Statement" : [ {
"Effect" : "Allow",
"Principal" : {
"AWS" : "arn:aws:iam::123456789999:role/lambda-a",
"AWS" : "arn:aws:iam::123456789999:role/lambda-b"
},
"Action" : "secretsmanager:*",
"Resource" : "*"
} ]
}
The problem is that I need a variable amount of lambdas. I could write in 30 or so of these but wanted to know how to use wildcards to do this efficiently.
When I try below I get the error This resource policy contains a syntax error.
{
"Version" : "2012-10-17",
"Statement" : [ {
"Effect" : "Allow",
"Principal" : {
"AWS" : "arn:aws:iam::123456789999:role/lambda-*",
},
"Action" : "secretsmanager:*",
"Resource" : "*"
} ]
}
And when I try to just eliminate the wildcard and specific path I get This resource policy contains an unsupported principal.
{
"Version" : "2012-10-17",
"Statement" : [ {
"Effect" : "Allow",
"Principal" : {
"AWS" : "arn:aws:iam::123456789999"
},
"Action" : "secretsmanager:*",
"Resource" : "*"
} ]
}
Any ideas how to do this cleanly?
Related
We are sending logs directly from Filebeats to Elasticsearch without Logstash.
Logs can contain JSON in different fields that also need to be parsed. I have created a pipeline to parse logs, tested it in the developer console, and output was as expected. I have set Filebeat to send logs to this pipeline by adding 'pipeline: application_pipeline' to filebeat.yml. But in Index Management, I see only my docs.
How to check if Filebeat is sending these logs to the pipeline?
log example:
{"level":"info","message":"Webhook DeletePrice-{\"_headers\":{\"x-forwarded-proto\":[\"https\"],\"x-requested-with\":[\"\"],\"x-client-ip\":[\"93.84.120.32\"],\"user-agent\":[\"1C+Enterprise\\/8.3\"],\"accept\":[\"application\\/json\"],\"host\":[\"host.com\"],\"content-length\":[\"\"],\"content-type\":[\"\"]},\"company_id\":\"10248103\",\"service_id\":\"102.01.02S\",\"service_type\":\"clientApi\"}","service":"servicename","project":"someproject.com","event_id":"255A854BED569B8D4C21B5DE6D8E109C","payload":[],"date_server":"2020-07-24T11:45:48+00:00","date_unix":1595591148.966919}
{"level":"error","message":"NO service integration","service":"servicename","project":"someproject.com","event_id":"D3986456E5A42AF8574230C29D1D474D","payload":{"exception":{"class":"\\Ship\\Exceptions\\IntegrationException","message":"NO service integration","code":0,"file":"/var/www/builds/someproject.com/build.lab.service-public-api.2020_07_22_12_17_45/app/Containers/Price/UI/API/Controllers/Controller.php:406"}},"date_server":"2020-07-24T08:40:34+00:00","date_unix":1595580034.975073}
{"level":"info","message":"No photo in priceId-3696930","service":"service-private-api","project":"someproject.com","event_id":"FBEDA2C9600BFE11523592114B32BAEB","payload":[],"date_server":"2020-07-24T12:16:40+00:00","date_unix":1595593000.97212}
{"level":"error","message":"C404HttpException: 404 \u0421\u0442\u0440\u0430\u043d\u0438\u0446\u0430 \u043d\u0435 \u043d\u0430\u0439\u0434\u0435\u043d\u0430 in \/var\/www\/builds\/build.lab.classified-platform.2020_07_29_12_13_54\/htdocs\/protected\/modules\/personal\/controllers\/RobotsController.php:65\nStack trace:\n#0 \/var\/www\/builds\/build.artox-lab.classified-platform.2020_07_29_12_13_54\/htdocs\/protected\/vendor\/yiisoft\/yii\/framework\/yiilite.php(4226): RobotsController->actionIndex()\n#1 \/var\/www\/builds\/build.lab.classified-platform.2020_07_29_12_13_54\/htdocs\/protected\/vendor\/yiisoft\/yii\/framework\/yiilite.php(3739): CInlineAction->runWithParams(Array)\n#2 \/var\/www\/builds\/build.lab.classified-platform.2020_07_29_12_13_54\/htdocs\/protected\/vendor\/yiisoft\/yii\/framework\/yiilite.php(3724): CController->runAction(Object(CInlineAction))\n#3 \/var\/www\/builds\/build.lab.classified-platform.2020_07_29_12_13_54\/htdocs\/protected\/vendor\/yiisoft\/yii\/framework\/yiilite.php(3714): CController->runActionWithFilters(Object(CInlineAction), Array)\n#4 \/var\/www\/builds\/build.lab.classified-platform.2020_07_29_12_13_54\/htdocs\/protected\/vendor\/yiisoft\/yii\/framework\/yiilite.php(1799): CController->run('index')\n#5 \/var\/www\/builds\/build.lab.classified-platform.2020_07_29_12_13_54\/htdocs\/protected\/vendor\/yiisoft\/yii\/framework\/yiilite.php(1719): CWebApplication->runController('personal\/robots...')\n#6 \/var\/www\/builds\/build.lab.classified-platform.2020_07_29_12_13_54\/htdocs\/protected\/vendor\/yiisoft\/yii\/framework\/yiilite.php(1236): CWebApplication->processRequest()\n#7 \/var\/www\/builds\/build.lab.classified-platform.2020_07_29_12_13_54\/htdocs\/index.php(22): CApplication->run()\n#8 {main}\nREQUEST_URI=\/robots.txt\n---","service":"artox-lab\/classified-platform","project":"someproject.com","event_id":"91a10782a3566a74d5abefa9589c926c","payload":"exception.C404HttpException.404","date_server":"2020-07-29T14:25:34+03:00","date_unix":1596021934.218448}
pipeline example:
PUT _ingest/pipeline/application_pipeline
{
"description" : "Pipeline for parsing application.log for services",
"processors" : [
{
"grok" : {
"field" : "message",
"patterns" : [
"%{JSON:json_message_payload}"
],
"pattern_definitions" : {
"JSON" : "{.*$"
},
"ignore_failure" : true,
"ignore_missing" : true
}
},
{
"remove" : {
"field" : "json_message_payload",
"ignore_failure" : true
}
}
]
}
}
output:
{
"_index" : "application_index",
"_type" : "_doc",
"_id" : "6",
"_version" : 1,
"_seq_no" : 3,
"_primary_term" : 1,
"found" : true,
"_source" : {
"date_server" : "2020-07-29T15:16:17+03:00",
"level" : "error",
"project" : "103by",
"message" : """
C404HttpException: 404 Страница не найдена in /var/www/builds/build.artox-lab.classified-platform.2020_07_29_12_13_54/htdocs/protected/modules/personal/components/PersonalController.php:140
Stack trace:
#0 /var/www/builds/build.artox-lab.classified-platform.2020_07_29_12_13_54/htdocs/protected/vendor/yiisoft/yii/framework/yiilite.php(3737): PersonalController->beforeAction(Object(ShowGalleryPhotoAction))
#1 /var/www/builds/build.artox-lab.classified-platform.2020_07_29_12_13_54/htdocs/protected/vendor/yiisoft/yii/framework/yiilite.php(3724): CController->runAction(Object(ShowGalleryPhotoAction))
#2 /var/www/builds/build.artox-lab.classified-platform.2020_07_29_12_13_54/htdocs/protected/vendor/yiisoft/yii/framework/yiilite.php(3714): CController->runActionWithFilters(Object(ShowGalleryPhotoAction), Array)
#3 /var/www/builds/build.artox-lab.classified-platform.2020_07_29_12_13_54/htdocs/protected/vendor/yiisoft/yii/framework/yiilite.php(1799): CController->run('showGalleryPhot...')
#4 /var/www/builds/build.artox-lab.classified-platform.2020_07_29_12_13_54/htdocs/protected/vendor/yiisoft/yii/framework/yiilite.php(1719): CWebApplication->runController('personal/galler...')
#5 /var/www/builds/build.artox-lab.classified-platform.2020_07_29_12_13_54/htdocs/protected/vendor/yiisoft/yii/framework/yiilite.php(1236): CWebApplication->processRequest()
#6 /var/www/builds/build.artox-lab.classified-platform.2020_07_29_12_13_54/htdocs/index.php(22): CApplication->run()
#7 {main}
REQUEST_URI=/gallery/23609/1439643/
HTTP_REFERER=http://rnpcomr.103.by/gallery/23609/1439643/
---
""",
"date_unix" : 1.596024977817727E9,
"event_id" : "b75c7a1ef2f8780986931b038d2f8599",
"payload" : "exception.C404HttpException.404",
"service" : "artox-lab/classified-platform"
}
}
Filebeat config:
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["elk.artoxlab.com:9200"]
pipeline: application_pipeline
If you run GET _nodes/stats/ingest, you're going to see the usage statistics for your pipeline in nodes.xyz.ingest.pipelines.application_pipeline
Another thing worth noting is that you could also do the same thing in Filebeat itself without resorting to using an ingest pipeline simply by defining a decode_json_fields processor, like this:
processors:
- decode_json_fields:
fields: ["message"]
process_array: true
max_depth: 2
target: ""
overwrite_keys: true
add_error_key: false
UPDATE: if you still don't see your data being indexed, what I suggest to do is to build some failure handling into your pipeline. Change it to this, son on case the indexing fails for some reason, you can see the document in the failed-xyz index with the reason for the error.
PUT _ingest/pipeline/application_pipeline
{
"description": "Pipeline for parsing application.log for services",
"processors": [
{
"grok": {
"field": "message",
"patterns": [
"%{JSON:json_message_payload}"
],
"pattern_definitions": {
"JSON": "{.*$"
},
"ignore_failure": true,
"ignore_missing": true
}
},
{
"remove": {
"field": "json_message_payload",
"ignore_failure": true
}
}
],
"on_failure": [
{
"append": {
"field": "meta.errors",
"value": "{{ _ingest.on_failure_message }}, {{ _ingest.on_failure_processor_type }}, {{ _ingest.on_failure_processor_tag }}"
}
},
{
"set": {
"field": "_index",
"value": "failed-{{ _index }}"
}
}
]
}
I can configure keycloak user in my users.json file:
{
"realm" : "my_realm",
"users" : [ {
"id" : "my_id",
"createdTimestamp" : 1568195319743,
"username" : "my_user_name",
"enabled" : true,
"totp" : false,
"emailVerified" : false,
"credentials" : [ {
"type" : "password",
"hashedSaltedValue" : "XXXXX",
"salt" : "XXXXX",
"hashIterations" : 27500,
"counter" : 0,
"algorithm" : "pbkdf2-sha256",
"digits" : 0,
"period" : 0,
"createdDate" : 1568195377551,
"config" : { }
} ],
"disableableCredentialTypes" : [ "password" ],
"requiredActions" : [ ],
"realmRoles" : [ "offline_access", "uma_authorization" ],
"clientRoles" : {
"account" : [ "manage-account", "view-profile" ]
},
"notBefore" : 0,
"groups" : [ ]
}]
}
When I start the keycloak container, those get loaded into keycloak database.
Can I script user groups the same way (in this file or some other file)?
And map the users to the groups?
Turned out it's simple (but also not documented):
{
"realm" : "my_realm",
......................
"groups": [
{
"id": "<my group id>",
"name": "<my_group_name>",
"path": "/<my_group_name>",
"attributes": {},
"realmRoles": [],
"clientRoles": {},
"subGroups": []
}
]
}
}
I figured this out by creating groups in keycloak admin interface, clicking "Export" and reading the downloaded file.
I am trying to find a value in the json file and based on that I need to get the entire json data instead of that particular block.
Here is my sample json
[{
"name" : "Redirect to Website 1",
"behaviors" : [ {
"name" : "redirect",
"options" : {
"mobileDefaultChoice" : "DEFAULT",
"destinationProtocol" : "HTTPS",
"destinationHostname" : "SAME_AS_REQUEST",
"responseCode" : 302
}
} ],
"criteria" : [ {
"name" : "requestProtocol",
"options" : {
"value" : "HTTP"
}
} ],
"criteriaMustSatisfy" : "all"
},
{
"name" : "Redirect to Website 2",
"behaviors" : [ {
"name" : "redirect",
"options" : {
"mobileDefaultChoice" : "DEFAULT",
"destinationProtocol" : "HTTPS",
"destinationHostname" : "SAME_AS_REQUEST",
"responseCode" : 301
}
} ],
"criteria" : [ {
"name" : "contentType",
"options" : {
"matchOperator" : "IS_ONE_OF",
"values" : [ "text/html*", "text/css*", "application/x-javascript*" ],
}
} ],
"criteriaMustSatisfy" : "all"
}]
I am trying to match for "name" : "redirect" inside each behaviors array and if it matches then I need the entire block including the "criteria" section, as you can see its under same block {}
I managed to find the values using select methods but not able to get the parent section.
https://jqplay.org/s/BWJwVdO3Zv
Any help is much appreciated!
To avoid unwanted duplication:
.[]
| first(select(.behaviors[].name == "redirect"))
Equivalently:
.[]
| select(any(.behaviors[]; .name == "redirect"))
You can try this jq command:
<file jq 'select(.[].behaviors[].name=="redirect")'
What I trying to connect is Loadbalancer DNS name to to Route53.
Lets look on example.
Here is Loadbabancer from template in Resource:
"RestELB" : {
"Type" : "AWS::ElasticLoadBalancing::LoadBalancer",
"DependsOn": "AttachGateway",
"Properties": {
"LoadBalancerName": {"Fn::Join": ["",["Rest-ELB-", {"Ref": "VPC"}]]},
"CrossZone" : "true",
"Subnets": [{ "Ref": "PublicSubnet1" },{ "Ref": "PublicSubnet2" }],
"Listeners" : [
{"LoadBalancerPort" : "80", "InstancePort" : "80","Protocol" : "HTTP"},
{"LoadBalancerPort" : "6060", "InstancePort" : "6060","Protocol" : "HTTP"}
],
}
},
And Here is Route53:
"ApiRecordSet" : {
"Type" : "AWS::Route53::RecordSet",
"Properties" : {
"AliasTarget" :{
"DNSName" : [
{"Fn::Join": ["", [{"ElasticLoadBalancer": "DNSName"},"."]]}
],
"EvaluateTargetHealth" : "Boolean",
"HostedZoneId" : "String"
},
"HostedZoneName" : "example.net.",
"Comment" : "A records for my frontends.",
"Name" : "api.example.net.",
"Type" : "A",
"TTL" : "900",
}
}
Just to put {"ElasticLoadBalancer": "DNSName"} didn't work. Can someone to suggest or give me correct way to add this?
Thanks!
Most likely you want to get the attribute DNSName for the LoadBalancer whose reference is RestELB. So you will need something with Fn::GetAtt like (untested)
"ApiRecordSet" : {
"Type" : "AWS::Route53::RecordSet",
"Properties" : {
"AliasTarget" :{
"DNSName" : { "Fn::GetAtt" : [ "RestELB", "DNSName" ]},
"EvaluateTargetHealth" : "Boolean",
"HostedZoneId" : "String"
},
"HostedZoneName" : "example.net.",
"Comment" : "A records for my frontends.",
"Name" : "api.example.net.",
"Type" : "A"
}
}
For anyone reading this answer in 2018, I got mine working using CanonicalHostedZoneNameID and not CanonicalHostedZoneID
"MyRecordSet": {
"Type": "AWS::Route53::RecordSet",
"Properties": {
"HostedZoneName" : "example.com.",
"Name": "abc.example.com.",
"Type": "A",
"AliasTarget": {
"HostedZoneId" : {"Fn::GetAtt": ["MyELB", "CanonicalHostedZoneNameID"]},
"DNSName": {"Fn::GetAtt": ["MyELB", "DNSName"]},
"EvaluateTargetHealth": "false"
}
}
}
Be sure to read the CloudFormation documentation on the AWS::Route53::Recordset AliasTarget type:
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-route53-aliastarget.html
This is how it looks in my CloudFormation when creating an alias target for an ELB:
"Route53LoadBalancerAlias" : {
"Type" : "AWS::Route53::RecordSet",
"Properties" : {
"AliasTarget" : {
"DNSName" : { "Fn::GetAtt" : [ "ELB", "DNSName" ]},
"EvaluateTargetHealth" : False,
"HostedZoneId" : { "Fn::GetAtt" : [ "ELB", "CanonicalHostedZoneID" ]}
},
For load balancers, use the canonical hosted zone ID of the load balancer. For Amazon S3, use the hosted zone ID for your bucket's website endpoint. For CloudFront, use Z2FDTNDATAQYW2. For a list of hosted zone IDs of other services, see the relevant service in the AWS Regions and Endpoints.
YAML for deploying a RecordSet referencing an ELB deployed in the same template.
Route53RecordSet:
Type: AWS::Route53::RecordSet
Properties:
Name: !Ref HostName
HostedZoneId: !Ref HostedZoneId
Type: A
AliasTarget:
DNSName: !GetAtt ElasticLoadBalancer.DNSName
HostedZoneId: !GetAtt ElasticLoadBalancer.CanonicalHostedZoneIDe
I'm building a Cloud Formation JSON to define EC2 Instances and Security Groups.
I need to create a security Group that allows every instance that belongs in it to share any data between each other.
My JSON was like that:
"InternalSecurityGroup" : {
"Type" : "AWS::EC2::SecurityGroup",
"Properties" : {
"VpcId" : {"Ref" : "myVPC"},
"GroupDescription" : "Allow the machines in this group to share all kinds of traffic between each other",
"SecurityGroupIngress" : [
{
"IpProtocol" : "-1",
"FromPort": "-1",
"ToPort": "-1",
"SourceSecurityGroupId" : { "Ref" : "InternalSecurityGroup" }
}
],
"SecurityGroupEgress" : [
{
"IpProtocol" : "-1",
"FromPort": "-1",
"ToPort": "-1",
"DestinationSecurityGroupId" : { "Ref" : "InternalSecurityGroup" }
}
]
}
},
But this shows me the following error:
A client error (ValidationError) occurred when calling the CreateStack
operation: Circular dependency between resources
To fix it I changed my code to CidrIp instead of SourceSecurityGroupId, defining the subnet the instances are in.
Is it possible to reference the same Security Group? What's the best (or correct) way to achieve what I want?
As noted in the documentation, you can use the AWS::EC2::SecurityGroupEgress and AWS::EC2::SecurityGroupIngress resources to define self-referencing security group rules:
Important
If you want to cross-reference two security groups in the ingress and egress rules of those security groups, use the AWS::EC2::SecurityGroupEgress and AWS::EC2::SecurityGroupIngress resources to define your rules. Do not use the embedded ingress and egress rules in the AWS::EC2::SecurityGroup. If you do, it causes a circular dependency, which AWS CloudFormation doesn't allow.
The result looks like this:
{
"Resources":{
"myVPC":{
"Type":"AWS::EC2::VPC",
"Properties":{
"CidrBlock":"10.0.0.0/16"
}
},
"InternalSecurityGroup":{
"Type":"AWS::EC2::SecurityGroup",
"Properties":{
"VpcId":{
"Ref":"myVPC"
},
"GroupDescription":"Allow the machines in this group to share all kinds of traffic between each other"
}
},
"InternalSecurityGroupIngress":{
"Type":"AWS::EC2::SecurityGroupIngress",
"Properties":{
"IpProtocol":"-1",
"FromPort":"-1",
"ToPort":"-1",
"SourceSecurityGroupId":{
"Ref":"InternalSecurityGroup"
},
"GroupId":{
"Ref":"InternalSecurityGroup"
}
}
},
"InternalSecurityGroupEgress":{
"Type":"AWS::EC2::SecurityGroupEgress",
"Properties":{
"IpProtocol":"-1",
"FromPort":"-1",
"ToPort":"-1",
"DestinationSecurityGroupId":{
"Ref":"InternalSecurityGroup"
},
"GroupId":{
"Ref":"InternalSecurityGroup"
}
}
}
}
}
Define two security groups, this should work a little better:
"InternalSecurityGroup1" : {
"Type" : "AWS::EC2::SecurityGroup",
"Properties" : {
"VpcId" : {"Ref" : "myVPC"},
"GroupDescription" : "Allow the machines in this group to share all kinds of traffic between each other",
"SecurityGroupIngress" : [ {
"IpProtocol" : "-1",
"FromPort": "-1",
"ToPort": "-1",
"SourceSecurityGroupId" : { "Ref" : "InternalSecurityGroup2" }
}
]
}
}
"InternalSecurityGroup2" : {
"Type" : "AWS::EC2::SecurityGroup",
"Properties" : {
"VpcId" : {"Ref" : "myVPC"},
"GroupDescription" : "Allow the machines in this group to share all kinds of traffic between each other",
"SecurityGroupIngress" : [ {
"IpProtocol" : "-1",
"FromPort": "-1",
"ToPort": "-1",
"SourceSecurityGroupId" : { "Ref" : "InternalSecurityGroup1" }
}
]
}
}