Is there a way in the json template under AWS::EC2::Instance to specify the number of instances?
you can use auto scaling group with fixed size:
"MyFixedSizeGroup":{
"Type":"AWS::AutoScaling::AutoScalingGroup",
"Properties":{
"LaunchConfigurationName":{"Ref":"GlobalWorkersSmallLaunchConf"},
"AvailabilityZones" : [ "us-east-1a" ],
"MinSize":"4",
"MaxSize":"4",
"DesiredCapacity":"4",
"Tags":[{"Key":"Name", "Value":"worker instance", "PropagateAtLaunch":"true"}]
}
}
and the desired launch configuration, for example:
"GlobalWorkersSmallLaunchConf":{
"Type":"AWS::AutoScaling::LaunchConfiguration",
"Properties":{"KeyName":{"Ref":"MyKeyName"},
"ImageId":"ami-SomeAmi",
"UserData":{"Fn::Base64":{"Fn::Join":["",[{"Ref":"SomeInitScript"}]]}},
"SecurityGroups":[{"Ref":"InstanceSecurityGroup"}],
"InstanceType":"m1.small",
"InstanceMonitoring":"false"
}
}
BTW- this wasn't available through the dashboard until last week.
CloudFormation does not provide any feature which you cannot do from AWS Console. Can you provide number of instance to be created when you are creating it from the AWS Console? No, you cannot.
There is an option in AWS Console to specify number of instances to be created.
But, There is no such option Cloudforamtion for AWS::EC2::Instance.
Related
I'm trying to update the binlog_format parameter in my Aurora mysql 5.6.10 (Data API enabled) instance to ROW but I'm not able to change it.
I've updated my custom parameter group accordingly but those changes do not reflect on the cluster when I run show variables like 'binlog_format'.
Right after changing the parameter group, the cluster goes into Modifying state but after that finishes the parameter hasn't been updated.
I can't seem to find an option to reboot or stop the cluster on the AWS UI.
Using the CLI, I get this error trying to stop the cluster: An error occurred (InvalidDBClusterStateFault) when calling the StopDBCluster operation: Stop-db-cluster is not supported for these configurations.
Tried changing the capacity settings but that didn't do anything.
Is there any other way I'm missing?
You'll have to check if the specific property modification is supported by serverless engine or not by running this command:
aws rds describe-db-cluster-parameters --db-cluster-parameter-group-name <param-group-name>
If you read the output from above statement, it says 'provisioned' for SupportedEngineModes:
{
"ParameterName": "binlog_format",
"ParameterValue": "OFF",
"Description": "Binary logging format for replication",
"Source": "system",
"ApplyType": "static",
"DataType": "string",
"AllowedValues": "ROW,STATEMENT,MIXED,OFF",
"IsModifiable": true,
"ApplyMethod": "pending-reboot",
"SupportedEngineModes": [
"provisioned"
]
}
Ideal state is something like this for a modifiable parameter:
{
"ParameterName": "character_set_server",
"Description": "The server's default character set.",
"Source": "engine-default",
"ApplyType": "dynamic",
"DataType": "string",
"AllowedValues": "big5,dec8,cp850,hp8,koi8r,latin1,latin2,swe7,ascii,ujis,sjis,hebrew,tis620,euckr,koi8u,gb2312,greek,cp1250,gbk,latin5,armscii8,utf8,ucs2,cp866,keybcs2,macce,macroman,cp852,latin7,utf8mb4,cp1251,utf16,cp1256,cp1257,utf32,binary,geostd8,cp932,eucjpms",
"IsModifiable": true,
"ApplyMethod": "pending-reboot",
"SupportedEngineModes": [
"provisioned",
"serverless"
]
},
Aurora does support Start and Stop APIs now, so I'm surprised that you were not able to use it.
https://aws.amazon.com/about-aws/whats-new/2018/09/amazon-aurora-stop-and-start/
Can you try using them through CLI?
On a separate note, if you just want to reboot the engine for the param change to flow in, you just need to use the reboot-db-instance API.
https://docs.aws.amazon.com/cli/latest/reference/rds/reboot-db-instance.html
In Airflow, we've created several DAGS. Some of which share common properties, for example the directory to read files from. Currently, these properties are listed as a property in each separate DAG, which will obviously become problematic in the future. Say if the directory name was to change, we'd have to go into each DAG and update this piece of code (possibly even missing one).
I was looking into creating some sort of a configuration file, which can be parsed into Airflow and used by the various DAGS when a certain property is required, but I cannot seem to find any sort of documentation or guide on how to do this. Most I could find was the documentation on setting up Connection ID's, but that does not meet my use case.
The question to my post, is it possible to do the above scenario and how?
Thanks in advance.
There are a few ways you can accomplish this based on your setup:
You can use a DagFactory type approach where you have a function generate DAGs. You can find an example of what that looks like here
You can store a JSON config as an Airflow Variable, and parse through that to generate a DAG. You can store something like this in a Admin -> Variables:
[
{
"table": "users",
"schema": "app_one",
"s3_bucket": "etl_bucket",
"s3_key": "app_one_users",
"redshift_conn_id": "postgres_default"
},
{
"table": "users",
"schema": "app_two",
"s3_bucket": "etl_bucket",
"s3_key": "app_two_users",
"redshift_conn_id": "postgres_default"
}
]
Your DAG could get generated as:
sync_config = json.loads(Variable.get("sync_config"))
with dag:
start = DummyOperator(task_id='begin_dag')
for table in sync_config:
d1 = RedshiftToS3Transfer(
task_id='{0}'.format(table['s3_key']),
table=table['table'],
schema=table['schema'],
s3_bucket=table['s3_bucket'],
s3_key=table['s3_key'],
redshift_conn_id=table['redshift_conn_id']
)
start >> d1
Similarly, you can just store that config as a local file and open it as you would any other file. Keep in mind the best answer to this will depend on your infrastructure and use case.
I have a public-input type app using Firebase, with anonymous auth. The user data is used to create points on a map. Each anonymous user can only edit the data inside the node matching their auth id - via security rules.
However, my app depends on a certain database structure. How do I ensure my database structure/integrity using anonymous auth, since the database url is client-side readable?
I think it is possible with security and validation rules, but I'm not sure. Maybe deny children creation in a node? This would be necessary to ensure the schema is followed.
Each auth node can have many key nodes, but I would want to limit this Firebase-side. And each key node must follow the schema below (so I can pull out the geojson easily). Below is my current setup - wondering what is missing?
"features" : {
"5AGxfaK2q8hjJsmsO3PUxUs09Sz1" : {
"-KS3R4sWPdcDkrxyIFX6" : {
"geometry" : {
"coordinates" : [ -81.88247680664062, 38.884619201291905 ],
"type" : "Point"
},
"properties" : {
"color" : "#2be",
"title" : ""
},
"type" : "Feature"
},
Authentication and database schema are completely separate topics. You ensure database schema by using a combination of .write and .validate rules in your security doc, not by anything to do with your authentication provider (i.e. Anonymous authentication).
This is described in detail in our database security guide.
A quick summary:
hasChildren: specify required fields
newData: refer to the data being written
data: refer to data already in the database
.validate: specify data schema using things like newData.isString() or newData.val() == data.val() + 1
Keep in mind that .validate rules are only run for non-null values. Thus, if you want to try something like !data.exists() (i.e. you can only write to this path once and can't modify it later) or newData.exists() (i.e. you can't delete this data) then you need to specify those in a .write rule.
Refer to the guide for more detail.
With the gcloud command line tool I can do:
$ gcloud compute instances list --filter='tags.items:development'
The documentation claims: "..you can also filter on nested fields. For example, you could filter on instances that have set the scheduling.automaticRestart field to true. Use filtering on nested fields to take advantage of labels to organize and search for results based on label values." But no examples are provided, so it's not clear how one actually goes about this.
I've tried labels.development eq *.*, labels eq *development*, labels:development et al.. I've also tried setting the verbosity of the of the command line client to info and looking through the output, as well as monitoring requests that go to the API from the Compute Engine web console, but neither has gotten me anywhere.
Finding Tags with regular expression filters
I'm struggling with the same issue but I think that regular expressions solve the problem.
I have many instances with multiple tags but I can search across all tags with the '~' operator e.g. to find all servers with the production tag:
gcloud compute instances list --filter='tags.items~^production$'
For many servers the 'production' tag is the third entry in tags.items yet the regexp finds it.
This seems to work but I can't find any documentation that specifically says that it should work. The nearest is the section on topic filters which mentions this
key ~ value True if key matches the RE (regular expression) pattern
value.
You can also search for multiple tags
gcloud compute instances list --filter='tags.items~^production$ AND tags.items~^european$'
which would find all servers with the two tags 'production' and 'european'
Tags v Custom metadata
If you want something a bit more flexible than tags (which can only be present or missing), you can attach your own custom multi-valued metadata to an instance (via the UI, command-line or API). You can then search for particular values of that item.
For example suppose I have different instances supporting eCommerce for different brands, I could attach a custom 'brand' metadata item to each server and then find all of the servers which run my "Coca-Cola" brands via ..
gcloud compute instances list --filter="metadata.items.key['brand']['value']='Coca-Cola'"
... and my 'Pepsi Cola' servers with ...
gcloud compute instances list --filter="metadata.items.key['brand']['value']='Pepsi Cola'"
Finding Metadata with regular expression filters
You probably guessed this already but the regular expression operator also works with metadata filters so you can do
gcloud compute instances list --filter="metadata.items.key['brand']['value']~'Cola'"
As explained here you can use the following syntax using gcloud cli:
gcloud compute instances list --filter labels.env=dev
Multi Filtering example that I'm using:
gcloud compute instances list --filter="zone:( europe-west1-d )" --filter="name:( testvm )" --filter labels.group=devops --filter labels.environment_type=production
Add instance metadata MyKey=MyValue for gke-cluster-asia-eas-default-pool-dc8f484c-knbs:
gcloud compute instances add-metadata gke-cluster-asia-eas-default-pool-dc8f484c-knbs --metadata=MyKey=MyValue
Display all instances with MyKey=MyValue:
gcloud compute instances list --filter="metadata.items.key['MyKey'][value]='MyValue'"
Display all instances that belong to the cluster cluster-asia-east1-a:
gcloud compute instances list --filter="metadata.items.key['cluster-name'][value]='cluster-asia-east1-a'"
NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS
gke-cluster-asia-eas-default-pool-dc8f484c-knbs asia-east1-a n1-standard-1 10.140.0.2 104.155.227.25 RUNNING
gke-cluster-asia-eas-default-pool-dc8f484c-x8cv asia-east1-a n1-standard-1 10.140.0.3 104.199.226.16 RUNNING
gke-cluster-asia-eas-default-pool-dc8f484c-z5wv asia-east1-a n1-standard-1 10.140.0.4 104.199.134.9 RUNNING
Should be a pretty simple thing, but I couldn't find a clue anywhere. I want to run get_all_dhcp_options() but set filters so that only those that match a certain domain-name are returned.
If anyone has better way, please let me know.
I understand to get the answer directly as #garnaat did is simple, but on how to get the filter key is not that direct.
What I did here is the instruction to find out the key/value easily. This is a general way for finding any keys in boto, not only vpc, same for ec2, s3, etc.
$ python
>>> import boto.vpc
>>> c = boto.vpc.connect_to_region('us-west-2') # or whatever
>>> c.get_all_dhcp_options()
[DhcpOptions:dopt-12dc23d1, DhcpOptions:dopt-426e82c7]
>>> for dhcp in c.get_all_dhcp_options():
... print dir(dhcp)
...
['__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', 'add_tag', 'add_tags', 'connection', 'endElement', 'id', 'item', 'options', 'region', 'remove_tag', 'remove_tags', 'startElement', 'tags']
It makes sense that the domain name should be in options, but you can verify it by login aws management console, click VPC => DHCP Options Sets, the domain name is in column options. So continue the debug:
>>> for dhcp in c.get_all_dhcp_options():
... print dhcp.options
...
{u'domain-name': [u'us-west-2.compute.internal'], u'domain-name-servers': [u'AmazonProvidedDNS']}
{u'domain-name': [u'abc.example.com xyz.example.com'], u'domain-name-servers': [u'10.0.0.1', u' 10.0.0.2'], u'ntp-servers': [u'10.0.0.1', u' 10.0.0.2']}
...
So now you get the key which is domain-name, and you also know the other keys domain-name-servers, ntp-servers. Then you should confidently put the filter now.
>>> c.get_all_dhcp_options(filters={'key': 'domain-name', 'value': 'us-west-2.compute.internal'})
[DhcpOptions:dopt-426e82c7]
After you get the result properly, you can add above steps in your codes.
Good luck.
This should work for you:
import boto.vpc
c = boto.vpc.connect_to_region('us-west-2') # or whatever
c.get_all_dhcp_options(filters={'key': 'domain-name', 'value': 'us-west-2.compute.internal'})
Obviously, use whatever domain name makes sense in your situation.
Finding out which filter key/values are supported by a given API call is another problem. Unfortunately, boto does not provide a way to do this mainly because there is no API that provides this information. However, if you have the AWSCLI, it has help information that comes directly from the service documentation so it does list the available filters.
To find what filters are supported you have to know that the get_all_* method calls in boto map to the Describe* API calls in EC2 and those, in turn, are available in the AWSCLI as describe-* commands. So:
aws ec2 describe-dhcp-options help
will display all of the help for the API call including the supported filters. Its a roundabout way of getting the info but its better than looking it up in the API docs.