On my AWS account I have four file systems. In creating each of these file systems, I have given each on a tag Name. I would like to use the AWS CLI to filter out the one with a specific name, and retrieve its file system id.
I have already tried looking in JMESPath in order to filter out the one specific file system, but it seems the structure of my query as well as the structure of the output is a bit difficult for me to work with.
Here is my query so far:
aws efs describe-file-systems --query FileSystems[?Name=='dev-four'].FileSystemId --region us-east-1 --output text
Ideally, I would get the file system id as a single string, but I keep getting the error:
Bad value for --query FileSystems[?Name==dev-sg].FileSystemId: Bad jmespath expression: Unknown token '-':
FileSystems[?Name==dev-four].FileSystemId
I am not sure how to do the JMES a different way.
I think you should escape the query. Instead of using single-quotes around dev-four, try using escaped back-ticks. jmespath can be quirky about escape sequence.
--query FileSystems[?Name == \`dev-spring\`].FileSystemId
On my mac, I need the back slashes. Try the query with and without them, it may affect the result. More detail here
Related
I am working on a batch script where I am parsing IIS logs using Log Parser.
So, I have 2 questions here:
Q1. What I want is, not to see the entries having username written in a text file. So, one can update that text file without worrying about the code syntax.
In other words, Instead of putting every username(could be around 30-50) in 'WHERE' clause using 'AND' again & again, I will have a text file having list of usernames.
Code Example:
"LogParser.exe" -i:csv "SELECT DISTINCT date, cs-username, clientun, cs-uri-stem FROM D:\temp.csv WHERE NOT cs-username IN ('NULL';'abc';'def';'hij';'and_so_on')" >D:\final_output.txt -o:NAT -rtp:-1
I was trying to do this via sub-query first, but it is not supported in Log parser it seems. I found OPENROWSET as a solution here but that doesn't seems to be working for me or may be I am not getting how to make it work.
Other solution I found were for SQL and were not working for log parser.
Q2. I want the logs to be read between the dates mentioned. So, I am taking a start and an end date from user in YYYY-MM-DD format and putting them in query
"LogParser.exe" -i:iisw3c "SELECT DISTINCT cs-username, REVERSEDNS(C-IP), date, cs-uri-stem, FROM \logs\location\* WHERE date BETWEEN %date_1% AND %Date_2%" >D:\temp.csv -o:csv
The error I get here is:
Error: WHERE clause: Semantic Error: left interval of BETWEEN operator ("'2016-02-15'") has a different type than operand ("date")
Please note, the output file generated from the Q2 is used as input in Q1.
Q1: Instead of filtering the data using logparser use findstr and the /g:file /v switches to filter input files or output lines (depending on the case)
Q2: Strings are not timestamps. Use
BETWEEN TO_TIMESTAMP('%date_1%','yyyy-MM-dd') AND TO_TIMESTAMP('%date_2%','yyyy-MM-dd')
not sure how far I'm going to get with this, but I'm going through a database removing certain bits and pieces in preparation for a conversion to different software.
I'm struggling with the image tags as on the site they currently look like
[img:<string>]<image url>[/img:<string>]
those strings are in another field called bbcode_uid
The query I'm running to make the changes so far is
UPDATE phpbb_posts SET post_text = REPLACE(post_text, '[img:]', '');
So my actual question, is there any way of pulling in each string from bbcode_uid inside of that SQL query so that I don't have to run the same command 10,000+ times, changing the unique string every time.
Alternatively could I include something inside [img:] to also include the next 8 characters, whatever they may be, as that is the length of the string that is used.
Hoping to save time with this, otherwise I might have to think of another way of doing it.
As requested.
The text I wish to replace would be
[img:1nynnywx]http://i.imgur.com/Tgfrd3x.jpg[/img:1nynnywx]
I want to end up with just
http://i.imgur.com/Tgfrd3x.jpg
Just removing the code around the URL, however each post_text has a different string which is contained inside bbcode_uid.
Method 1
LIB_MYSQLUDF_PREG
If you want more regular expression power in your database, you can consider using LIB_MYSQLUDF_PREG. This is an open source library of MySQL user functions that imports the PCRE library. LIB_MYSQLUDF_PREG is delivered in source code form only. To use it, you'll need to be able to compile it and install it into your MySQL server. Installing this library does not change MySQL's built-in regex support in any way. It merely makes the following additional functions available:
PREG_CAPTURE extracts a regex match from a string. PREG_POSITION returns the position at which a regular expression matches a string. PREG_REPLACE performs a search-and-replace on a string. PREG_RLIKE tests whether a regex matches a string.
All these functions take a regular expression as their first parameter. This regular expression must be formatted like a Perl regular expression operator. E.g. to test if regex matches the subject case insensitively, you'd use the MySQL code PREG_RLIKE('/regex/i', subject). This is similar to PHP's preg functions, which also require the extra // delimiters for regular expressions inside the PHP string
you can refer this link :github.com/hholzgra/mysql-udf-regexp
Method 2
Use php program, fetch records one by one , use php preg_replace
refer : www.php.net/preg_replace
reference:http://www.online-ebooks.info/article/MySql_Regular_Expression_Replace.html
You might be able to do this with substring_index().
The following will work on your example:
select substring_index(substring_index(post_text, '[/img:', 1), ']', -1)
I have a MySQL database containing a table with a binary-typed column. I'd like to be able to project that column without having to run it through, e.g., HEX(). Does the mysql CLI tool have a configuration option or other means to display a representation of binary data in a manner that won't output arbitrary bytes for my console to interpret in hilarious/annoying ways?
Start MySQL CLI with param --binary-as-hex
Example:
mysql --binary-as-hex
Set mysql client options in /etc/my.cnf works for me:
[client]
binary-as-hex = true
[mysql]
binary-as-hex = true
Since you want to look at the table mostly for convenience, create a view:
CREATE OR REPLACE VIEW myview AS
SELECT col1, HEX(col2) AS col2, col3, etc....
FROM table;
Then, all you have to do is reference myview instead of table:
SELECT * FROM myview;
The behavior of the MySQL command line client when viewing result sets with binary data has always been an annoyance to me, in fact I found this page because I was once again annoyed by the MySQL command line client (dumping binary data into my terminal when looking at a result set with binary UUID columns) and I wanted to solve the issue once and for all :-)
Creating views really isn't an option for me (I'm looking at dozens of tables with binary UUID columns) and I also found that it's really annoying to switch from SELECT * to typing out all of the column names instead (just so HEX() can be applied to the value of one column).
Eventually I came up with a creative hack that provides inspiration for alternative solutions to this annoyance: Using a custom pager command to sanitize output for terminal rendering. Here's how it works:
Create an executable (chmod +x) Python script with the following contents:
#!/usr/bin/python
import binascii, string, sys
for line in sys.stdin:
line = line.rstrip()
column, _, value = line.partition(': ')
if any(c not in string.printable for c in value):
sys.stdout.write("%s: %s\n" % (column, binascii.hexlify(value)))
else:
sys.stdout.write("%s\n" % line)
Start the MySQL command line client as follows:
$ mysql --pager=/home/peter/binary-filter.py --vertical ...
Change the pathname of the Python script as applicable. You can also put the script in your $PATH, in that case you can just pass the name to the --pager option (similar to how you would use less as a pager for the MySQL client).
Now when you SELECT ..., any line that shows a column whose value contains non-printable characters is rewritten so that the complete value is rendered as hexadecimal characters, similar to the results of MySQL's HEX() function.
Disclaimer: This is far from a complete solution, for example the Python snippet I showed expects SELECT ... \G format output (hence the --vertical option) and I tested it for all of five minutes so it's bound to contain bugs.
My point was to show that the problem can be solved on the side of the MySQL command line client, because that's where the problem is! (this is why it feels backwards for me to define server side views - only to make a command line client more user friendly :-P)
For me there is no problem with database size, so I will use two different column in every table, one as binary(16), and the second as char(32) without indexing. both of them will have the same value.
when I need to search I will use binary column, and when I need to read I will use char(32).
is there any problem with this scenario?
I have a list of a million or urls in an mysql table.
I need to cleanse the data (extract domains) so I can be confident about DISTINCT type queries.
Data is in several different types: -
www.domain.tld
domain.tld
http://domain.tld
https://vhost.domain.tld
domain.tld/
There are invalid domains and empty data.
Ideally I'd like to do something along the lines of : -
UPDATE table1 SET domain = website REGEXP '^(https?://)?[a-zA-Z0-9\\\\.\\\\-]+(/|$|\\\\?)'
domain being a new empty field, website being the original url.
You can't use regex like that in MySQL as is, but apparently you can some some UDFs that implement it. See:
How to do a regular expression replace in MySQL?
https://launchpad.net/mysql-udf-regexp
http://www.mysqludf.org/lib_mysqludf_preg/
This question already has an answer here:
Closed 11 years ago.
Possible Duplicate:
Simulating regex capture groups in mysql
Good day,
I have many rows of data stored in a MySQL table. A typical value could look something like this:
::image-gallery::
::gallery-entry::images/01.jpg::/gallery-entry::
::/image-gallery::
Is there a way - by means of a regular expression that I can a) extract the term image gallery from the first line (it could be any phrase, not just image-gallery) and then extract the center line as two separate values like this:
gallery-entry and then images/01.jpg
There could be many lines of ::gallery-entry:: values, and they could be called anything as well. A more complete example would be:
::image-gallery::
::title::MY GALLERY::/title::
::date::2011-05-20::/date::
::gallery-entry::images/01.jpg::/gallery-entry::
::/image-gallery::
In essence I want this information: The content type (image-gallery) in the above case, first line and last line. Then I need the title as a key value style pair, so title as the key and MY GALLERY as the value. Then, subsequently, I would need all the rows of fields thereafter (gallery-entry) as key value pairs too.
This is for a migration script where data from an old system will be migrated over to a new system with different syntax.
If MySQL select statements would not work, would it be easier to parse the results with a PHP script for data extraction?
Any and all help is always appreciated.
Kind regards,
Simon
Try this regex:
::image-gallery::\s+::title::(.*?)::/title::.*?::gallery-entry::(.*?)::/gallery-entry::\s+::/image-gallery::
Use single-line mode (/pattern/s) so the .*? chews up newlines.
Your key-value pairs will be:
title: $1 (matching group 1)
gallery-entry: $2 (matching group 2)
From simulating-regex-capture-groups-in-mysql there does not seem to be a way to easily capture groups with a regex in mysql. The reason is that MySQL does not natively support capture groups in a regex. If you want that functionality you can use a server side extension like lib_mysqludf_preg to add that capability to MySQL.
The easiest way is to extract the whole column with SQL and then do the text matching in another language (such as php).
In my tests kenbritton's regex didn't work, but building off of it the following regex worked on your test data:
::image-gallery::\s+::title::(.*?)::\/title::\s+(?:.*\s+)*::gallery-entry::(.*?)::\/gallery-entry::\s+::\/image-gallery::