I am playing with Kubernetes on google compute engines, and I would like to do something that will stop multiple instances at the same time.
I reason of this is, Kubernetes will create a cluster which contains partial common names, for example,
kubernetes-minion-group-1
kubernetes-minion-group-2
......
Kubernetes has the option to delete all the instances, but I couldn't found the option to shutdown them. I don't want to delete all the instance and recreate the next time.
I understand that the following command will delete one instance,
gcloud compute instances stop kubernetes-minion-group-1
but how to stop all of them at the same time in one line or maybe in a bash script? like,
gcloud compute instances stop (instances contains 'kubernetes')
Thank you in advance.
I'm not sure if there's a cleaner way, but something based on the below using instances list might work:
gcloud compute instances list --filter 'name:kubernetes' --format="csvno-heading"
...which you'd have to parse/grep to get the list. I told it to give a CSV of just the name attributes that match, with no title row (you might be able to play with different topic formats to get one that works exactly like a spaced list. Then:
gcloud computes instances stop [list of names]
It seems 'stop' won't take a filter or wildcard. :/
Related
I have a MySQL database in Amazon RDS setup right now that needs to be able to act as a database and also be able to store some flat files.
It was working just fine for a while until I noticed it wasn't storing anything over 1MB... and I couldn't figure out why. So I dove deeper into RDS and learned about parameter groups. It seems to be a subset of configurations for the database itself, and so I figured it was the max_allowed_packet value was the problem and I set it to a higher value.
However, I was still unable to make uploads over 1MB so then I realized there was another parameter by the name of mysqlx_max_allowed_packet and its value is set to about 1MB, but I am unable to change it.
Does anyone have any idea how to get around this or if it is possible?
I hope these steps help.
Go to your RDS Dashboard and click Parameter Groups.
Click Create DB Parameter Group, name it something like 'LargeImport', (making sure the DB Parameter Group Family you select matches your instance version) and edit the parameters.
Increase the 'max_allowed_packet' on 'LargeImport' to accommodate your import size (Valid values are 1024-1073741824).
Increase the 'wait_timeout' parameter to accommodate your import size. (Valid values are 1-31536000 seconds).
Save your changes.
Click Instances in the left column and select your instance.
Click Instance Actions and choose Modify.
Change the Parameter Group to your new 'LargeImport' group and click Continue.
Click 'Modify DB Instance'.
Once the change has completed, click Instance Actions again and reboot your instance.
Once your instance has rebooted, you should be able to do larger SQL imports.
Once you've completed your import, switch your instance parameter group back to the default parameter group and reboot it again.
I reccomend you to test if your change take effect , so go on mysqlworkbench on your mysql instance and launch the query :
show variables like 'max_allowed_packet';
If it isn't then you can start change it to 64 MB for example ( tune the parameter to your requirements but take in mind that 1GB for aws is the max limitation). Remember also after modify RDS instance you should reboot to apply your changes.
I am new in PDI (passing from SSIS) and I am having some troubles by handling the variables issue.
I would like to perform this:
From a sql select query I would like to save the result into a variable.
For that reason I have created one job and two transformations, given that in pentaho every step is executed in parallel.
The first transformation is going to be on charge of setting the variable and the second transformation is going to use this result as an input.
But in the first transformation I am having troubles by setting the variable, I do not understand where do I have to instanciate this variable to implement the "set season variable" step. And then how to get this result in the next transformation.
If anyone knows about this, or if you could recommend any link with a good example, I'll really appreciate it.
This can indeed be confusing for SSIS users. In PDI, you don't create a recordset variable as you do in SSIS. Simply creating a job creates one for you. Each job has two different types of "Results". One for recordset rows and one for filenames.
These variables are not directly accessible; they are just part of the job. There are steps that interact with them directly. For example under the "Job" branch when you're creating a transform, there is a Get rows from results step and a Copy rows to results step. They work directly with the job's row results.
Be aware that you must manually manage the metadata for the results. This is a pain, but over-all I find PDI's method of doing this more intuitive and easier than SSIS. I find SSIS more flexible in this regard.
There are also Get files from result and Set files in result. These interact with the job's built in file results. This is simply a list of every file touched by any step configured in the job. On the job tab there are tasks that deal with it directly such as Process result filenames, Add filenames to result and Delete filenames from results. These tasks operate on the built in file results list for the job and provide an easy way to, say, archive all the files loaded by the transform you just ran.
Be aware when using these steps that they record EVERY file touched by EVERY step in the job. If you look through most of the steps in transformations (data flows) that deal with files, there's usually an "Add files to results" checkbox that is checked by default. If you uncheck this, it will not add the file names to the jobs file results. You can also delete specific files from the file results with the Delete filenames from result step.
From your Job, start a Transformation:
Overload transformation variable into global variable in your job and use it:
How do I get access to the MySQL logs (primarily to take a look at the insert/update/delete statements) from an Amazon RDS instance?
Basically you have to enable the "general_log" parameter in the parameter group of your RDS instance
$ rds-modify-db-parameter-group mydbparametergroup --parameters "name=general_log,value=ON,method=immediate"
In case you did not apply the parametergroup to the instance:
$ rds-modify-db-instance mydbinstance --db-parameter-group-name mydbparametergroup
Then access your mysql instance using root:
mysql> select * from mysql.general_log;
See:
AWS Developer Forum - Re: general query log
AWS RDS - Working with DB Parameter Groups
EDIT: 4 years have past since I posted this answer, and it still seems valid. I hope someone from Amazon RDS documentation team would read it and update their documentation.
I had a really hard time to figure such a simple thing out, because all online information in this regard seems outdated including one in Amazon Docs. Amazon has obviously changed how you do things since now the default parameters cannot be modified, and you need to create a custom set of parameters in order to modify them, including general_log. It is an obvious bug that you can still click the Edit button for default parameters, but when you try to save them, you get an error that default parameters can't be changed.
How you do it now, is that in the Parameters Groups, click on Create DB Parameter Group, and create a new group and select the same DB in 'DB Parameter Group Family' as in the default parameter group. See the attached screen shot. Once done, it'll create a copy identical to the default parameter group. Now edit the parameters, e.g. change general_log to '1'. According to the Docs is should be '0' by default but it is neither '0' nor '1' by default.
Now save it, go back to your instance, click on 'Instance Actions', select 'Modify' and in the setting which will appear, change 'Parameter Group' to your new custom parameter group. It'll take a few moments to apply it, after which you'll need to restart your DB instance.
This is how it is till June 2014. But there is no guarantee that it'll stay like this in future too, since in the technology industry things keep getting updated too fast (many times unnecessarily) but documents and tutorials don't get updated as fast.
I'm using php with MySQL on macOS.
I would like to select a large amount of emails from a database and perform a dns lookup for each email in my selection using a dig command from the terminal/shell, something like: "dig gmail.com" .
Of course, I can loop this select through php but it will be very slow compared to looping cursor on MySQL.
How to send terminal commands from mysql to the terminal and retrieve answer on macOS?
You can't execute shell commands from within an SQL query (thank god), or else it would be a horrible security vulnerability... You would have to do it from php.
P.S. It is however possible to execute shell commands from the MySQL command line utility
\! ls
...but if I understand your question, it won't help solve your current problem.
(I'm assuming that you really mean ADDR_SPEC when you're talking about email addresses)
but it will be very slow in compare with looping cursor on mysql
No not really. The only difference is that depending on how you implement this the PHP approach requires that you retrieve the entire result set before you start iterating through it. However breaking this up into smaller result sets is trivial.
Also, the limitation on the performance of your algorithm is the speed of DNS lookups - and that's all about latency - if your objective is to make this go faster then you should be running multple requests in parallel.
The next thing you should consider is that you've probably got multiple mailboxes for each MX, e.g. user1#gmail.com, user2#gmail.com.... While if you've got DNS caching setup properly there will be less overhead than going to the source each time, if you're working with a very large data set or will be doing this more than once, it makes a lot more sense to just work with unique MX host values, e.g.
SELECT DISTINCT SUBSTR(addr_spec FROM LOCATE('#', addr_spec)) AS mx2chk
FROM yourtable
WHERE addr_spec LIKE '%#%'
AND (email_checked IS NULL
OR email_checked<NOW() - INTERVAL 300 DAY )
;
Indeed, if your flagging the data then you can use your own database to verify the MX.
using a dig command from terminal/shell
Please don't tell me that you're running a shell from a PHP controlling process to do a DNS lookup?
I need to update a particular field in the database based on some timestamps in the same table. Now are there time based triggers which can be scripted to look at the value and do things? much like the linux 'at' command.
I know one way is to execute a script using the at command. I am asking about something at the database level itself.
MySQL has an event scheduler.
See
http://dev.mysql.com/doc/refman/5.5/en/events.html