I track a lot of parameters on my Server and the only thing I can't realy put in perspective is the IOstat. It is a MySQL Server, is this a good result, or should I worry?
root:/var/lib/mysql# iostat -xc
Linux 2.6.28-11-server () 07/25/2009 _x86_64_ (8 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
3.66 0.19 0.45 1.04 0.00 94.69
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 2.55 871.36 1.46 27.67 392.40 7200.45 260.64 1.02 34.85 2.48 7.22
sda1 0.18 0.61 0.03 0.01 3.60 4.98 215.91 0.01 185.95 19.25 0.08
sda2 0.01 0.00 0.00 0.00 1.03 0.02 919.32 0.00 21.36 6.94 0.00
sda3 2.36 870.75 1.43 27.66 387.76 7195.46 260.68 1.01 34.65 2.48 7.21
sdb 2.37 871.36 1.63 27.67 392.69 7200.45 259.12 0.65 22.07 2.51 7.35
sdb1 0.17 0.61 0.04 0.01 3.59 4.98 187.33 0.01 110.67 12.54 0.06
sdb2 0.00 0.00 0.00 0.00 1.03 0.02 256.48 0.00 2.36 1.50 0.00
sdb3 2.19 870.75 1.60 27.66 388.06 7195.46 259.23 0.64 21.93 2.51 7.34
md0 0.00 0.00 0.38 0.62 3.06 4.96 8.00 0.00 0.00 0.00 0.00
md1 0.00 0.00 0.00 0.00 0.00 0.02 8.36 0.00 0.00 0.00 0.00
md2 0.00 0.00 2.01 898.28 62.49 7186.28 8.05 0.00 0.00 0.00 0.00
Also what war options for decreasing read / write activity?
delay_______key_______writes
memory based Tables
less indicies
The write load is quite high on the tables.
If anyone worrying about the disk IO bottlenecks, please have a check with the following command.
iostat
If this tool is not installed then,
apt-get install sysstat
on Debian based servers.
yum install sysstat
on Redhat/CentOS based servers.
Then,
iostat -x -d sda
-here "sda" denotes your HDD
Output:
root#forum.innovationframes.com:~# iostat -x -d sda
Linux 2.6.32-24-server (forum.innovationframes.com) 10/01/2011 _x86_64_ (1 CPU)
Dev: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm **%util**
sda 0.01 0.04 0.06 0.03 1.34 0.51 21.77 0.00 5.23 0.30 **0.00**
Note:
If Util shows more than 75-80% then you should keep an eye on your HDD.
Related
I have the following html
<table id="statTableHTML" class="table hidden"><thead><tr><th>Characteristic</th><th>GCI Score</th><th>Legal</th><th>Technical</th><th>Organizational</th><th>Capacity Building</th><th>Cooperation</th></tr></thead><tbody><tr><td>United Kingdom</td><td>0.93</td><td>0.2</td><td>0.19</td><td>0.2</td><td>0.19</td><td>0.15</td></tr><tr><td>United States</td><td>0.93</td><td>0.2</td><td>0.18</td><td>0.2</td><td>0.19</td><td>0.15</td></tr><tr><td>France</td><td>0.92</td><td>0.2</td><td>0.19</td><td>0.2</td><td>0.19</td><td>0.14</td></tr><tr><td>Lithuania</td><td>0.91</td><td>0.2</td><td>0.17</td><td>0.2</td><td>0.19</td><td>0.16</td></tr><tr><td>Estonia</td><td>0.91</td><td>0.2</td><td>0.2</td><td>0.19</td><td>0.17</td><td>0.15</td></tr><tr><td>Singapore</td><td>0.9</td><td>0.2</td><td>0.19</td><td>0.19</td><td>0.2</td><td>0.13</td></tr><tr><td>Spain</td><td>0.9</td><td>0.2</td><td>0.18</td><td>0.2</td><td>0.17</td><td>0.15</td></tr><tr><td>Malaysia</td><td>0.89</td><td>0.18</td><td>0.2</td><td>0.2</td><td>0.2</td><td>0.12</td></tr><tr><td>Norway</td><td>0.89</td><td>0.19</td><td>0.2</td><td>0.18</td><td>0.19</td><td>0.14</td></tr><tr><td>Canada</td><td>0.89</td><td>0.2</td><td>0.19</td><td>0.2</td><td>0.17</td><td>0.14</td></tr><tr><td>Australia</td><td>0.89</td><td>0.2</td><td>0.17</td><td>0.2</td><td>0.18</td><td>0.14</td></tr></tbody></table>
I would like to get all the values within the tr, td elements.
I have the following
awk -F '</*td>|</*tr>' '{print $5, $7, $9, $11, $13, $15, $17}'
however it only outputs the first row
United Kingdom 0.93 0.2 0.19 0.2 0.19 0.15
how can i write this so I get all the values of the rows?
thanks
With GNU awk for multi-char RS:
$ awk -v RS='<tr><td>|</td></tr>' -F'(</?td>)+' -v OFS='\t' 'NF>1{$1=$1; print}' file
United Kingdom 0.93 0.2 0.19 0.2 0.19 0.15
United States 0.93 0.2 0.18 0.2 0.19 0.15
France 0.92 0.2 0.19 0.2 0.19 0.14
Lithuania 0.91 0.2 0.17 0.2 0.19 0.16
Estonia 0.91 0.2 0.2 0.19 0.17 0.15
Singapore 0.9 0.2 0.19 0.19 0.2 0.13
Spain 0.9 0.2 0.18 0.2 0.17 0.15
Malaysia 0.89 0.18 0.2 0.2 0.2 0.12
Norway 0.89 0.19 0.2 0.18 0.19 0.14
Canada 0.89 0.2 0.19 0.2 0.17 0.14
Australia 0.89 0.2 0.17 0.2 0.18 0.14
or if you prefer tabular instead of tab-separated output just pipe it to column:
$ awk -v RS='<tr><td>|</td></tr>' -F'(</?td>)+' -v OFS='\t' 'NF>1{$1=$1; print}' file | column -s$'\t' -t
United Kingdom 0.93 0.2 0.19 0.2 0.19 0.15
United States 0.93 0.2 0.18 0.2 0.19 0.15
France 0.92 0.2 0.19 0.2 0.19 0.14
Lithuania 0.91 0.2 0.17 0.2 0.19 0.16
Estonia 0.91 0.2 0.2 0.19 0.17 0.15
Singapore 0.9 0.2 0.19 0.19 0.2 0.13
Spain 0.9 0.2 0.18 0.2 0.17 0.15
Malaysia 0.89 0.18 0.2 0.2 0.2 0.12
Norway 0.89 0.19 0.2 0.18 0.19 0.14
Canada 0.89 0.2 0.19 0.2 0.17 0.14
Australia 0.89 0.2 0.17 0.2 0.18 0.14
Something like:
awk -F '</*td>|</*tr>' '{ for (i=5;i<=NF-1;i+=2) print $i }' file
should get you the values, but they will be output in a single column so you will need to format the column further with e.g.
awk -F '</*td>|</*tr>' '{ for (i=5;i<=NF-1;i+=2) print $i }' test | awk -v RS= '{gsub(/\n/, "\t")}1'
United Kingdom 0.93 0.2 0.19 0.2 0.19 0.15
United States 0.93 0.2 0.18 0.2 0.19 0.15
France 0.92 0.2 0.19 0.2 0.19 0.14
Lithuania 0.91 0.2 0.17 0.2 0.19 0.16
Estonia 0.91 0.2 0.2 0.19 0.17 0.15
Singapore 0.9 0.2 0.19 0.19 0.2 0.13
Spain 0.9 0.2 0.18 0.2 0.17 0.15
Malaysia 0.89 0.18 0.2 0.2 0.2 0.12
Norway 0.89 0.19 0.2 0.18 0.19 0.14
Canada 0.89 0.2 0.19 0.2 0.17 0.14
Australia 0.89 0.2 0.17 0.2 0.18 0.14
Check how your regex is matching each field. Each line of the output starts with the number of the field you came up with. Empty lines mean <tr> and <td> were side by side.
You could try this, instead:
BEGIN{RS="<\/?t[dr]><\/?t[dr]>";ORS=" "}
$0 && !/^</
END{printf "\n"}
Try it online!
It removes all trs and tds, regarding them as record separators. The output record separator becomes a space. Then, only prints non-blank records that do not start with <, i.e., that do not start with a html tag. Also prints a trailing newline, for the sake of POSIX definition of a line.
Based on your attempt, seems like number of columns per row is known to be 7. So, you can use tools like xpath or xmllint and then use pr to format the output.
$ # same as: xmllint --xpath '//tr/td/text()'
$ xpath -q -e '//tr/td/text()' ip.html | pr -7ats' '
United Kingdom 0.93 0.2 0.19 0.2 0.19 0.15
United States 0.93 0.2 0.18 0.2 0.19 0.15
France 0.92 0.2 0.19 0.2 0.19 0.14
Lithuania 0.91 0.2 0.17 0.2 0.19 0.16
Estonia 0.91 0.2 0.2 0.19 0.17 0.15
Singapore 0.9 0.2 0.19 0.19 0.2 0.13
Spain 0.9 0.2 0.18 0.2 0.17 0.15
Malaysia 0.89 0.18 0.2 0.2 0.2 0.12
Norway 0.89 0.19 0.2 0.18 0.19 0.14
Canada 0.89 0.2 0.19 0.2 0.17 0.14
Australia 0.89 0.2 0.17 0.2 0.18 0.14
I have MySQL 8 docker installation installed on an edge device which has the following two tables to write to
video_paths | CREATE TABLE `video_paths` (
`entry` int(11) NOT NULL AUTO_INCREMENT,
`timestamp` bigint(20) NOT NULL,
`duration` int(11) NOT NULL,
`path` varchar(255) NOT NULL,
`motion` int(11) NOT NULL DEFAULT '0',
`cam_id` varchar(255) NOT NULL DEFAULT '',
`hd` tinyint(1) NOT NULL DEFAULT '0',
PRIMARY KEY (`entry`),
KEY `cam_id` (`cam_id`),
KEY `timestamp` (`timestamp`)
) ENGINE=InnoDB AUTO_INCREMENT=7342309 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
AND
CREATE TABLE `tracker` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`table_name` varchar(255) NOT NULL,
`primary_key_name` varchar(255) NOT NULL,
`pointer` int(11) NOT NULL DEFAULT '0',
PRIMARY KEY (`id`),
UNIQUE KEY `table_name` (`table_name`)
) ENGINE=InnoDB AUTO_INCREMENT=4 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
The following queries are run every few secs for up to 32 cameras and are taking a lot of time as indicated by the slow query log.
UPDATE tracker SET pointer = 7342046 WHERE table_name = 'video_paths'
INSERT INTO video_paths (timestamp,duration,path,cam_id,hd) VALUES (1597548365000,5000,'/s/ss/x-0/v/2020-08-16/3/1.ts','x-1',1)
Most of the time is spent in the waiting for handler commit state
The total size of my data (tables + index) is ~1GB and I have the following settings enabled to optimise for write
skip-log-bin - Disabled the bin log because I don't have a replica and therefore no use for it
innodb_flush_log_at_trx_commit =2 - I am Optimising for performance rather than consistency here.
range_optimizer_max_mem_size =0 As mention in this question, I have allowed max memory to range optimiser.
inndo_buffer_pool_size= 512Mb - This should be enough for my data?.
innodb_log_file_size= 96Mb *2 files
I am seeing queries that are taking up to 90-100 secs sometimes.
SET timestamp=1597549337;
INSERT INTO video_paths (timestamp,duration,path,cam_id,hd) VALUES (1597548365000,5000,'/s/ss/x-0/v/2020-08-16/3/1.ts','x-1',1);
# Time: 2020-08-16T03:42:24.533408Z
# Query_time: 96.712976 Lock_time: 0.000033 Rows_sent: 0 Rows_examined: 0
---UPDATE---
Here's the complete my.cnf file
my.cnf
[mysqld]
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
datadir = /var/lib/mysql
secure-file-priv= NULL
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
skip-log-bin
innodb_buffer_pool_size=536870912
innodb_log_file_size=100663296
# Custom config should go here
!includedir /etc/mysql/conf.d/
conf.d/docker.cnf
[mysqld]
skip-host-cache
skip-name-resolve
The docker container is using the host mode so complete 15GB memory is available to the container.
--- UPDATE 2 ---
After increasing the innodb_buffer_poo_size to 2GB as suggested by #fyrye, the statements have now started getting stuck on STATE = UPDATE instead of waiting for handler commit.
---- UPDATE 3 ---
Looks like the CPU is causing the bottleneck
** ---- UPDATE 4 ---- **
Additional info
Ram Size
total used free shared buff/cache available
Mem: 15909 1711 9385 2491 4813 11600
Swap: 0 0 0
No SSD/NVMe devices attached
SHOW GLOBAL STATUS - https://pastebin.com/vtWi0PUq
SHOW GLOBAL VARIABLES - https://pastebin.com/MUZeG959
SHOW FULL PROCESSLIST - https://pastebin.com/eebEcYk7
htop - htop here is for the edge system which has 4 other containers running which include the main app, ffmpeg, mqtt, etc.
ulimit -a:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 62576
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 62576
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
opstat -xm 5 4
Linux 4.15.0-106-generic (xxxx) 08/18/2020 _x86_64_ (4 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
26.97 0.00 22.36 22.53 0.00 28.14
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
loop0 0.00 0.00 0.00 0.00 0.00 0.00 3.20 0.00 2.40 2.40 0.00 0.00 0.00
sda 13.78 9.89 32.24 11.44 0.37 4.10 209.51 47.52 1079.07 44.07 3994.87 22.39 97.81
avg-cpu: %user %nice %system %iowait %steal %idle
19.71 0.00 27.85 40.87 0.00 11.57
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
loop0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda 0.00 0.00 1.40 4.60 0.03 2.71 934.93 142.66 24221.33 666.29 31390.26 166.67 100.00
avg-cpu: %user %nice %system %iowait %steal %idle
20.16 0.00 26.77 28.30 0.00 24.77
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
loop0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda 0.00 0.00 8.80 5.60 0.03 3.45 496.11 141.28 12507.78 194.00 31858.00 69.44 100.00
mpstat -P ALL 5 3
Linux 4.15.0-106-generic (sn-1f0ce8) 08/18/2020 _x86_64_ (4 CPU)
02:15:47 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
02:15:52 PM all 21.48 0.00 20.40 29.01 0.00 7.94 0.00 0.00 0.00 21.17
02:15:52 PM 0 24.95 0.00 20.86 5.32 0.00 0.61 0.00 0.00 0.00 48.26
02:15:52 PM 1 17.59 0.00 18.81 57.67 0.00 5.93 0.00 0.00 0.00 0.00
02:15:52 PM 2 21.28 0.00 17.36 0.21 0.00 24.79 0.00 0.00 0.00 36.36
02:15:52 PM 3 22.34 0.00 24.59 52.46 0.00 0.61 0.00 0.00 0.00 0.00
02:15:52 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
02:15:57 PM all 20.56 0.00 20.00 28.26 0.00 7.08 0.00 0.00 0.00 24.10
02:15:57 PM 0 24.44 0.00 18.89 12.32 0.00 0.21 0.00 0.00 0.00 44.15
02:15:57 PM 1 17.73 0.00 15.46 33.20 0.00 4.95 0.00 0.00 0.00 28.66
02:15:57 PM 2 18.93 0.00 22.22 12.35 0.00 22.84 0.00 0.00 0.00 23.66
02:15:57 PM 3 21.06 0.00 23.31 55.21 0.00 0.41 0.00 0.00 0.00 0.00
02:15:57 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
02:16:02 PM all 21.81 0.00 18.32 26.42 0.00 7.03 0.00 0.00 0.00 26.42
02:16:02 PM 0 26.43 0.00 19.67 0.20 0.00 0.41 0.00 0.00 0.00 53.28
02:16:02 PM 1 20.57 0.00 17.11 45.21 0.00 5.30 0.00 0.00 0.00 11.81
02:16:02 PM 2 19.67 0.00 16.74 0.21 0.00 21.97 0.00 0.00 0.00 41.42
02:16:02 PM 3 20.45 0.00 19.84 58.91 0.00 0.81 0.00 0.00 0.00 0.00
Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
Average: all 21.28 0.00 19.57 27.90 0.00 7.35 0.00 0.00 0.00 23.90
Average: 0 25.27 0.00 19.81 5.94 0.00 0.41 0.00 0.00 0.00 48.57
Average: 1 18.63 0.00 17.13 45.39 0.00 5.39 0.00 0.00 0.00 13.45
Average: 2 19.96 0.00 18.78 4.28 0.00 23.20 0.00 0.00 0.00 33.77
Average: 3 21.28 0.00 22.57 55.54 0.00 0.61 0.00 0.00 0.00 0.00
Suggestions to consider for your my.cnf [mysqld] section
log_error=/var/lib/mysql/sn-1f0ce8-error.log # from stderr to have a visible error log
innodb_lru_scan_depth=100 # from 1024 to conserve 90% CPU cycles used for function
innodb_io_capacity=1900 # from 200 to allow more IOPSecond to your storage device
innodb_flush_neighbors=2 # from 0 to expedite writing to current extent
innodb_max_dirty_pages_pct_lwm=1 # from 10 percent to expedite writes
innodb_max_dirty_pages_pct=1 # from 90 percent to reduce innodb_buffer_pool_pages_dirty count
innodb_change_buffer_max_size=50 # from 25 percent to expedite your high volume activity
You will find these suggestions will reduce CPU busy and expedite query completion.
For additional suggestions view profile, Network profile for contact information and free downloadable Utility Scripts to assist with performance improvements.
I have 8 RTX GPUs. When run p2pBandwidthLatencyTest, The latencies between GPU0 and GPU1, GPU2 and GPU3, GPU4 and GPU5, GPU6 and GPU7 is 40,000 times slower than other pairs:
P2P=Enabled Latency (P2P Writes) Matrix (us)
GPU 0 1 2 3 4 5 6 7
0 1.80 49354.72 1.70 1.70 1.74 1.74 1.74 1.72
1 49354.84 1.37 1.70 1.69 1.74 1.76 1.73 1.72
2 1.88 1.81 1.73 49355.00 1.79 1.76 1.76 1.75
3 1.88 1.79 49354.85 1.33 3.79 3.84 3.88 3.91
4 1.89 1.88 1.90 1.87 1.72 49354.96 3.49 3.56
5 2.30 1.93 1.88 1.89 49354.89 1.32 3.63 3.60
6 2.55 2.53 2.37 2.29 2.24 2.26 3.50 49354.77
7 2.30 2.27 2.29 1.87 1.82 1.83 49354.85 1.36
Compare it with when peer-to-peer is disabled:
P2P=Disabled Latency Matrix (us)
GPU 0 1 2 3 4 5 6 7
0 1.80 14.31 13.86 13.49 14.52 13.89 13.58 13.58
1 13.71 1.82 14.44 13.95 14.65 13.62 15.05 15.20
2 13.38 14.23 1.73 16.59 13.77 15.44 14.10 13.64
3 12.68 15.62 12.50 1.77 14.92 15.01 15.17 14.87
4 13.51 13.60 15.09 13.40 1.27 12.48 12.68 19.47
5 14.92 13.84 13.42 13.42 16.53 1.30 16.37 16.60
6 14.29 13.62 14.66 13.62 14.90 13.70 1.32 14.33
7 14.26 13.42 14.35 13.53 16.89 14.26 17.03 1.36
Is this normal?
It turns out the super slow peer-to-peer is abnormal.
After I disable IOMMU (Intel VT-d) in the BIOS, the problem is gone:
P2P=Enabled Latency (P2P Writes) Matrix (us)
GPU 0 1 2 3 4 5 6 7
0 1.34 1.22 1.68 1.69 1.71 1.70 1.75 1.73
1 1.20 1.38 1.70 1.67 1.71 1.75 1.75 1.72
2 1.69 1.67 1.29 1.20 1.73 1.75 1.75 1.75
3 1.69 1.66 1.17 1.29 1.74 1.75 1.72 1.73
4 1.72 1.76 1.74 1.70 1.32 1.13 1.66 1.70
5 1.74 1.73 1.75 1.74 1.18 1.28 1.67 1.69
6 1.75 1.74 1.74 1.72 1.67 1.68 1.31 1.19
7 1.76 1.75 1.73 1.73 1.67 1.69 1.18 1.32
It seems the problem is the same as or is very similar to discussions in:
https://github.com/pytorch/pytorch/issues/1637
https://github.com/pytorch/pytorch/issues/24081
A few possible solutions are mentioned in the discussions:
Disable IOMMU:
https://github.com/pytorch/pytorch/issues/1637#issuecomment-338268158
https://github.com/pytorch/pytorch/issues/1637#issuecomment-401422046
Disable ACS:
https://github.com/pytorch/pytorch/issues/24081#issuecomment-557074611
https://github.com/pytorch/pytorch/issues/24081#issuecomment-547976127
My system having the problem only had IOMMU enabled in the BIOS. ACS was not turned on as lspci -vvv | grep ACS got back nothing.
==============================
Background on I/O MMU:
https://en.wikipedia.org/wiki/X86_virtualization#I/O_MMU_virtualization_(AMD-Vi_and_Intel_VT-d)
It's part of the x86 virtualization. It's the virtualization done by the chipset. Besides the name IOMMU, it's also called AMD-Vi or Intel VT-d. Not to be confused with AMD-V and Intel VT-x which are virtualization via the CPU.
The I/O wait time is increasing for one of my servers. I haven't seen any long running queries in the process list. Can someone help me identify the cause?
The server is running Cent OS Linux 7.7, 4 core CPU , MySQL version 5.6.42
iotop output
Actual DISK READ: 57.85 M/s | Actual DISK WRITE: 74.90 K/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
31529 be/4 mysql 58.08 M/s 0.00 B/s 0.00 % 84.45 % mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib~p.geolearning.com.pid --socket=/var/lib/mysql/mysql.sock --port=3306
12331 be/4 mysql 7.68 K/s 61.46 K/s 0.00 % 0.66 % mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib~p.geolearning.com.pid --socket=/var/lib/mysql/mysql.sock --port=3306
6819 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.02 % [kworker/2:16]
1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % systemd --switched-root --system --deserialize 22
2 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kthreadd]
4 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/0:0H]
6 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/0]
7 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/0]
8 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_bh]
9 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_sched]
10 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [lru-add-drain]
Processlist:
+---------+----------------+------------------+---------+---------+------+--------------+------------------------------------------------------------------------------------------------------+-----------+---------------+
| Id | User | Host | db | Command | Time | State | Info | Rows_sent | Rows_examined |
+---------+----------------+------------------+---------+---------+------+--------------+------------------------------------------------------------------------------------------------------+-----------+---------------+
| 2128874 | lpaauser | 10.7.40.7:64197 | prodsup | Query | 0 | Sending data | REPLACE INTO course_auto_assignments_compute8 (domain_id,lms_user_id,filter_set_id)
S | 0 | 0 |
| 2129350 | mveeranki | localhost | | Query | 0 | init | show processlist | 0 | 0 |
+---------+----------------+------------------+---------+---------+------+--------------+------------------------------------------------------------------------------------------------------+-----
I would like to download the 10-year federal note yield from the treasury website: https://www.treasury.gov/resource-center/data-chart-center/interest-rates/Pages/TextView.aspx?data=yield
To parse the above webpage and retrieve the most recent 10 yr treasury note yield, I used to follow the instructions provided here:
Parsing 10-year federal note yield from the website
library(httr)
URL = "https://www.treasury.gov/resource-center/data-chart-center/interest-rates/Pages/TextView.aspx?data=yield"
urldata <- GET(URL)
data <- readHTMLTable(rawToChar(urldata$content),
stringsAsFactors = FALSE)
data <- as.data.frame((data[69]))
names(data) <- gsub("NULL.","", names(data)) # Take out "NULL."
But it no longer works.
Any thoughts what may be wrong or alternative suggestions?
This does not answer the specific question of why your code no longer works. Here is an alternative using rvest package which simplifies many scraping operations. In particular below, the selection of the table is made with the class id CSS selector .t-chart. This makes it much more tolerant of page formatting changes. The chaining with operator %>% makes for very compact code.
library(rvest)
t_url = "https://www.treasury.gov/resource-center/data-chart-center/interest-rates/Pages/TextView.aspx?data=yield"
rates <- read_html(t_url) %>%
html_node(".t-chart") %>%
html_table()
rates
# Date 1 mo 3 mo 6 mo 1 yr 2 yr 3 yr 5 yr 7 yr 10 yr 20 yr 30 yr
# 1 04/03/17 0.73 0.79 0.92 1.02 1.24 1.47 1.88 2.16 2.35 2.71 2.98
# 2 04/04/17 0.77 0.79 0.92 1.03 1.25 1.47 1.88 2.16 2.36 2.72 2.99
# 3 04/05/17 0.77 0.80 0.93 1.03 1.24 1.44 1.85 2.14 2.34 2.71 2.98
# 4 04/06/17 0.78 0.79 0.94 1.05 1.24 1.45 1.87 2.15 2.34 2.72 2.99
# 5 04/07/17 0.77 0.82 0.95 1.08 1.29 1.52 1.92 2.20 2.38 2.74 3.00
# 6 04/10/17 0.77 0.82 0.97 1.07 1.29 1.52 1.91 2.18 2.37 2.72 2.99
# 7 04/11/17 0.74 0.82 0.94 1.05 1.24 1.45 1.84 2.11 2.32 2.67 2.93
# 8 04/12/17 0.77 0.81 0.95 1.04 1.24 1.44 1.81 2.09 2.28 2.65 2.92
# 9 04/13/17 0.76 0.81 0.94 1.03 1.21 1.40 1.77 2.05 2.24 2.62 2.89