Webscraping Pokemon Data - html
I am trying to find out the number of moves each Pokemon (first generation) could learn.
I found the following website that contains this information: https://pokemondb.net/pokedex/game/red-blue-yellow
There are 151 Pokemon listed here - and for each of them, their move set is listed on a template page like this: https://pokemondb.net/pokedex/bulbasaur/moves/1
Since I am using R, I tried to get the website addresses for each of these 150 Pokemon (https://docs.google.com/document/d/1fH_n_BPbIk1bZCrK1hLAJrYPH2d5RTy9IgdR5Ck_lNw/edit#):
names = c("Bulbasaur","Ivysaur","Venusaur","Charmander","Charmeleon","Charizard","Squirtle","Wartortle","Blastoise","Caterpie","Metapod","Butterfree","Weedle","Kakuna","Beedrill",
"Pidgey","Pidgeotto","Pidgeot","Rattata","Raticate","Spearow","Fearow","Ekans","Arbok","Pikachu","Raichu","Sandshrew","Sandslash","Nidoran","Nidorina","Nidoqueen","Nidorino","Nidoking",
"Clefairy","Clefable","Vulpix","Ninetales","Jigglypuff","Wigglytuff","Zubat","Golbat","Oddish","Gloom","Vileplume","Paras","Parasect","Venonat","Venomoth","Diglett","Dugtrio","Meowth","Persian",
"Psyduck","Golduck","Mankey","Primeape","Growlithe","Arcanine","Poliwag","Poliwhirl","Poliwrath","Abra","Kadabra","Alakazam","Machop","Machoke","Machamp","Bellsprout","Weepinbell","Victreebel","Tentacool",
"Tentacruel","Geodude","Graveler","Golem","Ponyta","Rapidash","Slowpoke","Slowbro","Magnemite","Magneton","Farfetch’d","Doduo","Dodrio","Seel","Dewgong","Grimer","Muk","Shellder","Cloyster","Gastly","Haunter",
"Gengar","Onix","Drowzee","Hypno","Krabby","Kingler","Voltorb","Electrode","Exeggcute","Exeggutor","Cubone","Marowak","Hitmonlee","Hitmonchan","Lickitung","Koffing","Weezing","Rhyhorn","Rhydon","Chansey","Tangela",
"Kangaskhan","Horsea","Seadra","Goldeen","Seaking","Staryu","Starmie","Mr.Mime","Scyther","Jynx","Electabuzz","Magmar","Pinsir","Tauros","Magikarp","Gyarados","Lapras","Ditto"
,"Eevee","Vaporeon","Jolteon","Flareon","Porygon","Omanyte","Omastar","Kabuto","Kabutops","Aerodactyl","Snorlax","Articuno","Zapdos","Moltres","Dratini","Dragonair","Dragonite","Mewtwo","Mew")
template_1 = rep("https://pokemondb.net/pokedex/",150)
template_2 = rep("/moves/1",150)
pokemon_websites = data.frame(template_1, names, template_2)
pokemon_websites$full_website = paste(pokemon_websites$template_1, pokemon_websites$names, pokemon_websites$template_2)
Next, I remove all spaces:
library(stringr)
pokemon_websites$full_website = str_remove_all( pokemon_websites$full_website," ")
Now, I have a column with all the website names:
head(pokemon_websites)
template_1 names template_2 full_website
1 https://pokemondb.net/pokedex/ Bulbasaur /moves/1 https://pokemondb.net/pokedex/Bulbasaur/moves/1
2 https://pokemondb.net/pokedex/ Ivysaur /moves/1 https://pokemondb.net/pokedex/Ivysaur/moves/1
3 https://pokemondb.net/pokedex/ Venusaur /moves/1 https://pokemondb.net/pokedex/Venusaur/moves/1
4 https://pokemondb.net/pokedex/ Charmander /moves/1 https://pokemondb.net/pokedex/Charmander/moves/1
5 https://pokemondb.net/pokedex/ Charmeleon /moves/1 https://pokemondb.net/pokedex/Charmeleon/moves/1
6 https://pokemondb.net/pokedex/ Charizard /moves/1 https://pokemondb.net/pokedex/Charizard/moves/1
I would like to count the number of moves each of these 150 Pokemon can learn. For example, the first Pokemon "Bulbasaur" can learn 24 moves:
In the end, I would like to add a column to the earlier data frame that contains the number of moves each Pokemon can learn. For example, something that looks like this:
> head(pokemon_websites)
template_1 names template_2 full_website number_of_moves
1 https://pokemondb.net/pokedex/ Bulbasaur /moves/1 https://pokemondb.net/pokedex/Bulbasaur/moves/1 24
2 https://pokemondb.net/pokedex/ Ivysaur /moves/1 https://pokemondb.net/pokedex/Ivysaur/moves/1 ???
3 https://pokemondb.net/pokedex/ Venusaur /moves/1 https://pokemondb.net/pokedex/Venusaur/moves/1 ???
4 https://pokemondb.net/pokedex/ Charmander /moves/1 https://pokemondb.net/pokedex/Charmander/moves/1 ???
5 https://pokemondb.net/pokedex/ Charmeleon /moves/1 https://pokemondb.net/pokedex/Charmeleon/moves/1 ???
6 https://pokemondb.net/pokedex/ Charizard /moves/1 https://pokemondb.net/pokedex/Charizard/moves/1 ???
Is there a way to webscrape this data in R, count the number of moves for each of the 150 Pokemon, and then place this move count into a column?
Right now I am doing this by hand and it is taking a long time! Also, I have heard some websites do not allow for automated webscraping - if this website (https://pokemondb.net/pokedex/game/red-blue-yellow) does not allow webscraping, I can try to find another website that might allow it.
Thank you!
You can scrape all the tables for each of the pokemen using something like this:
tables =lapply(pokemon_websites$full_website,function(link) {
tryCatch(
read_html(link) %>% html_nodes("table") %>% html_table(),
error = function(e) {}, warning=function(w) {}
)
})
However, note that the number of tables returned differs for each of the pokemon. For example the first has 6 tables - the first three of those are for Red/Blue, the second three of those are for Yellow.
lengths(tables)
[1] 6 6 6 6 6 6 6 6 6 2 4 7 2 4 8 6 6 6 4 4 6 6 6 6 6 8 6 6 0 4 8 4 8 6 8 4 6 6 8 4 4 6 6 8 6 6 5 5 5 5 4 4 6 6 6
[56] 6 4 6 6 6 8 6 6 6 6 6 6 6 6 8 6 6 6 6 6 4 4 6 6 6 6 0 6 6 6 6 4 4 6 8 4 4 6 6 6 6 6 6 6 6 4 8 6 7 6 6 6 4 4 6
[111] 6 6 6 6 6 6 6 6 6 8 0 6 4 6 6 6 6 2 8 6 2 4 8 8 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6
Since the OP wants to count only the moves in Red/Blue tab we can do the following, (If you need moves from both the tabs follow #langtang answer)
tables1 =lapply(pokemon_websites$full_website, function(x){
tryCatch( x %>% read_html() %>% html_nodes('.active') %>% html_nodes('.resp-scroll') %>% html_table(),
error = function(e) NULL
)
})
moves= lapply(tables1, function(x) lapply(x, function(x) dim(x)[1]))
moves = lapply(moves, unlist, use.names=FALSE)
moves = lapply(moves, sum) %>% unlist()
[1] 24 25 27 32 33 37 32 33 37 2 3 30 2 3 26 22 23 25 24 27 21 23 24 26 29 30 27 29 0 28 43 28 44 41 42 22 23 40 41 19 22 21 23 25 23 26 22 29 20 23 24
[52] 27 31 34 31 34 23 25 25 36 37 25 34 35 29 31 32 23 24 26 28 31 30 31 33 22 26 37 46 23 26 0 23 26 25 28 22 24 27 29 20 20 32 25 33 36 25 27 24 27 24 29
[103] 32 35 26 26 37 19 21 27 42 44 24 36 22 24 25 27 33 34 0 21 34 35 30 23 27 2 34 34 1 19 32 32 28 30 22 28 22 30 24 43 25 25 22 30 32 36 45 60
Related
Can you explain this function and how does this work with examples?
I can't understand this function and I have checked the tutorial website for those unknown function
here is the code:
def print_formatted(number):
# your code goes here
for i in range(1,n + 1):
pad = n.bit_length()
dec = str(i).rjust(pad)
octs = str(oct(i)[2:]).rjust(pad)
hexx = str(hex(i)[2:]).rjust(pad).upper()
bina = str(bin(i)[2:]).rjust(pad)
print(f'{dec} {octs} {hexx} {bina}')
Thanking you in advance!
This is the output it gave when called upon with n = 17
1 1 1 1
2 2 2 10
3 3 3 11
4 4 4 100
5 5 5 101
6 6 6 110
7 7 7 111
8 10 8 1000
9 11 9 1001
10 12 A 1010
11 13 B 1011
12 14 C 1100
13 15 D 1101
14 16 E 1110
15 17 F 1111
16 20 10 10000
17 21 11 10001
so how does this above code makes sure that the output gives not the output like this:
1 1 1 1
2 2 2 10
3 3 3 11
4 4 4 100
5 5 5 101
6 6 6 110
7 7 7 111
8 10 8 1000
9 11 9 1001
10 12 A 1010
11 13 B 1011
12 14 C 1100
13 15 D 1101
14 16 E 1110
15 17 F 1111
16 20 10 10000
17 21 11 10001
Find if a value exists in a Google sheet on a certain column, in all the rows above the current row based on 2 criterias
I have the following scenario: columns from A-Z and 100 rows in each row for the Z column I want to find if the value in A column from the current row exists in the rows above in A column then if exists, I would like to find if the B column for the matching rows have the cell completed with a value for all the rows that are matching I would to receive the matching rows in an array list, not as rows or at least to be able to put a value like "mathing"/"not matching" this should be an array formula I've tried something like this, only for the first criteria, but somehow it checks only the current row. =ARRAYFORMULA( IF(ROW(Z2:Z)>2, IF(MATCH(A2:A,$A$2:A&ROW(A2:A)-1),"matching","not matching"),"not matching")) I check to see if it's the first row (as it has headers), and if it's the first row, then surely it can't have any data matching above It will be great to have it as a google sheet formula but if it's not possible it could also be a google app script
Try this:
function myfunk() {
const ss = SpreadsheetApp.getActive();
const sh = ss.getSheetByName("Sheet0");
const osh = ss.getSheetByName("Sheet1");
osh.clearContents();
const dsr = 2;
const vs = sh.getRange(dsr, 1, sh.getLastRow() - dsr + 1, sh.getLastColumn()).getDisplayValues();
let o = [];
vs.forEach((r, i) => {
if (i > 0) {
let as = vs.map(r => r[0]).slice(0, i);// suggested by DoubleUnary
let bs = vs.map(r => r[1]).slice(0, i);//suggested by DoubleUnary
let idx = as.indexOf(r[25]);
if (~idx && bs[idx]) {
o.push(['yes', dsr + i, dsr + idx, r[25], bs[idx]])
} else {
o.push(['no', dsr + i, ~idx ? as[idx] : '', r[25], ~idx ? bs[idx] : '']);
}
}
});
o.unshift(['Value', 'Test Row', 'Result Row', 'Z value', 'B value'])
Logger.log(JSON.stringify(o));
osh.getRange(1, 1, o.length, o[0].length).setValues(o);
}
My Data:
COL1
COL2
COL3
COL4
COL5
COL6
COL7
COL8
COL9
COL10
COL11
COL12
COL13
COL14
COL15
COL16
COL17
COL18
COL19
COL20
COL21
COL22
COL23
COL24
COL25
COL26
4
4
8
18
3
15
15
6
6
18
2
10
19
14
5
16
3
6
0
13
15
14
10
13
19
7
14
5
18
12
12
3
5
5
12
0
0
4
19
17
13
14
2
6
2
0
18
15
16
1
1
15
14
8
18
19
18
19
14
11
9
2
12
4
19
8
7
17
2
5
17
12
3
18
6
15
12
17
12
15
1
11
2
14
4
12
15
4
2
7
13
12
4
10
0
2
9
2
15
12
18
7
10
6
15
8
3
11
3
11
8
2
0
12
18
12
17
3
3
10
5
18
0
6
19
12
11
2
3
5
16
16
7
14
12
3
1
9
0
1
9
4
17
11
18
2
4
16
13
4
1
3
4
13
9
8
11
18
9
9
10
17
6
16
8
10
15
10
18
1
2
9
10
18
13
0
11
4
7
2
0
18
3
5
1
5
18
17
4
8
2
4
10
13
7
10
9
6
3
7
5
7
12
12
6
0
3
7
3
3
19
4
2
5
0
9
5
14
0
2
15
9
18
6
1
15
5
5
1
12
4
7
9
3
19
19
15
16
12
18
13
0
12
4
12
4
1
8
19
2
1
1
8
14
6
10
0
16
14
14
10
8
3
15
5
13
9
13
10
6
16
2
15
3
2
16
19
2
14
1
10
1
1
5
5
10
8
3
8
17
13
15
8
9
6
4
2
14
6
4
1
6
14
8
9
11
12
3
18
5
14
9
18
2
12
17
2
17
10
0
11
7
11
2
0
11
15
6
7
13
10
18
17
6
19
12
14
15
7
12
5
0
17
15
2
2
18
6
7
13
1
10
19
9
7
13
15
13
7
18
11
13
10
8
1
10
5
17
9
9
5
14
3
3
1
19
7
13
0
5
10
2
12
17
3
12
9
0
10
9
15
6
14
18
1
3
6
4
9
19
4
9
15
11
0
3
10
19
5
18
16
10
4
4
4
1
1
6
8
10
9
8
19
4
11
18
12
14
8
4
5
11
8
17
5
7
13
13
16
14
8
7
14
7
18
9
3
11
0
1
7
19
8
6
3
4
4
2
4
11
3
7
5
5
9
16
15
7
6
4
6
7
17
8
13
10
2
9
18
0
13
12
4
13
9
4
19
4
7
10
17
1
5
5
3
7
12
3
19
19
7
1
11
9
9
9
7
5
6
8
7
0
11
19
6
17
12
1
18
Results:
Value
Test Row
Result Row
Z value
B value
no
3
15
no
4
17
no
5
6
no
6
5
no
7
8
no
8
18
no
9
7
yes
10
9
3
5
yes
11
3
14
5
no
12
10
yes
13
3
14
5
yes
14
3
14
5
yes
15
12
10
8
yes
16
12
10
8
yes
17
2
4
4
no
18
8
8
yes
19
6
15
8
no
20
5
no
21
18
Google script to append multiple rows - select columns
How can I append multiple rows from source sheet to destination sheet; select columns like A,C,H ? source sheet columns A-Z, with multiple rows. destination sheet - append rows from source sheet - select columns A, C and H only. Thank you
Transfer Data from one sheet to another selecting only columns A,C and H
function xferdata() {
const ss = SpreadsheetApp.getActive();
const ssh = ss.getSheetByName("Sheet0");
const dsh = ss.getSheetByName("Sheet1");
const vs = ssh.getRange(2,1,ssh.getLastRow() - 1, 8).getValues();
let o = vs.map(([a,,c,,,,,h]) => [a,c,h]);
//Logger.log(JSON.stringify(o));
dsh.getRange(dsh.getLastRow() + 1,1,o.length,o[0].length).setValues(o);
}
Sheet0:
COL1
COL2
COL3
COL4
COL5
COL6
COL7
COL8
1
2
3
4
5
6
7
8
2
3
4
5
6
7
8
9
3
4
5
6
7
8
9
10
4
5
6
7
8
9
10
11
5
6
7
8
9
10
11
12
6
7
8
9
10
11
12
13
7
8
9
10
11
12
13
14
8
9
10
11
12
13
14
15
9
10
11
12
13
14
15
16
10
11
12
13
14
15
16
17
11
12
13
14
15
16
17
18
12
13
14
15
16
17
18
19
13
14
15
16
17
18
19
20
14
15
16
17
18
19
20
21
15
16
17
18
19
20
21
22
16
17
18
19
20
21
22
23
17
18
19
20
21
22
23
24
18
19
20
21
22
23
24
25
19
20
21
22
23
24
25
26
20
21
22
23
24
25
26
27
Sheet1:
A
B
C
1
3
8
2
4
9
3
5
10
4
6
11
5
7
12
6
8
13
7
9
14
8
10
15
9
11
16
10
12
17
11
13
18
12
14
19
13
15
20
14
16
21
15
17
22
16
18
23
17
19
24
18
20
25
19
21
26
20
22
27
Timed out script
I'm using the following code in order to avoid using an IMPORTRANGE() formula. The data I'm getting is over 50k rows so that's why I'm using App Script.
But now I'm getting the following error:
Exception: Service Spreadsheets timed out while accessing document with id
function Live_Data_importing() {
var raw_live = SpreadsheetApp.openByUrl("someURL").getSheetByName("Sheet1");
var selected_columns = raw_live.getRange("A:Q").getValues().map(([a,,,,e,f,g,,i,j,,l,,n,o,]) => [a,e,f,g,i,j,l,n,o] );
var FF_filtered = selected_columns.filter(row=>row[8]=="somedata");
var FF_Hourly_live_data = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("somesheet");
FF_Hourly_live_data.getRange(2, 1, FF_filtered.length, FF_filtered[0].length).setValues(FF_filtered);
}
How can I fix this error? I also made a copy of the file but script is still running the same error.
Tested this on smaller set of data
It works okay
function Live_Data_importing() {
const ss = SpreadsheetApp.getActive();
var raw_live = ss.getSheetByName("Sheet3");
var selected_columns = raw_live.getRange("A1:Q" + raw_live.getLastRow()).getValues().map(([a, , , , e, f, g, , i, j, , l, , n, o,]) => [a, e, f, g, i, j, l, n, o]);//fix this range
var FF_filtered = selected_columns.filter(row => row[8] == 0);//My data is all integers
var FF_Hourly_live_data = ss.getSheetByName("Sheet4");//output sheet
FF_Hourly_live_data.getRange(2, 1, FF_filtered.length, FF_filtered[0].length).setValues(FF_filtered);
}
Data:
COL1
COL2
COL3
COL4
COL5
COL6
COL7
COL8
COL9
COL10
COL11
COL12
COL13
COL14
COL15
COL16
COL17
COL18
COL19
COL20
0
1
18
13
0
7
5
11
15
2
1
17
0
8
11
18
0
16
1
8
0
16
16
9
5
0
16
8
14
0
7
4
3
15
8
19
18
4
18
10
9
14
7
16
17
13
17
4
8
6
19
18
9
18
12
1
8
19
12
10
13
17
4
2
10
3
7
16
14
13
1
5
16
12
2
3
14
3
0
9
4
17
14
5
9
13
18
0
4
1
14
1
12
7
10
1
15
4
14
5
1
17
13
17
3
1
17
8
19
7
14
18
2
17
1
16
2
19
13
15
7
4
9
0
11
14
11
7
12
14
3
19
14
13
18
12
16
19
19
6
7
6
10
5
15
8
0
8
2
16
4
14
18
14
2
16
16
5
15
16
4
5
9
6
6
2
1
15
14
8
19
8
4
10
12
12
4
6
10
12
11
15
3
4
1
3
17
19
10
4
11
2
10
16
12
1
6
3
0
3
11
0
14
8
13
13
4
2
16
18
10
14
5
3
7
4
7
9
5
15
0
6
5
1
2
18
1
1
11
13
7
13
5
15
5
13
17
18
14
19
0
17
10
6
10
16
16
0
18
19
12
8
15
1
11
4
19
4
17
14
4
14
14
6
16
8
15
4
5
2
4
5
14
14
16
9
0
16
0
4
8
3
8
5
12
15
5
19
14
0
1
6
12
2
19
10
10
19
13
19
0
5
14
7
2
17
3
10
2
14
2
5
0
18
5
1
13
13
3
13
10
18
18
4
18
11
0
7
13
9
18
13
9
2
0
16
15
3
9
14
12
15
15
7
16
13
14
1
4
12
18
9
6
14
18
11
16
2
18
15
15
0
2
4
6
16
5
18
2
6
14
18
11
1
9
15
13
8
8
19
11
19
14
0
7
15
1
10
8
9
14
14
15
3
11
13
4
10
5
16
Selecting or Updating with 2 group criteria for a max value on mysql [duplicate]
This question already has answers here: SQL select only rows with max value on a column [duplicate] (27 answers) Closed 3 years ago. I'm trying to figure out how to use group and max() on joins correctly, I'm doing some parsing of a moodle(Open source school software) mysql database. Students are allowed to retake the quizes indefinatly for this particular program, but I need to be able to update the course completion date to reflect the last time they took the test because a lot of other things depend on the completion fields. The mdl_quiz_attempts table stores all attempts for all quizes, the userid will have many of the same entries, but the attempt number is not unique to the table, but instead unique to both the student AND the key for the row. Meaning students have multiple entries. On the mdl_course_modules table, The instance field is the key for the mdl_quiz table, and the mdl_course_modules_completion coursemoduleid field is the key for mdl_course_modules. So what I want to do is this: given a student id UPDATE mdl_course_completion.timemodified to mdl_quize_attempts.timemodified WHERE the row on mdl_quiz_attempts is the max attempt by userid for each quiz.(the quiz field on the quiz_attempts has to be looked up through in course module instance table to get the, instance id for course completion module id) Here are example partial tables. mdl_quiz_attempts id quiz userid attempt timemodified 2 1 3 6 1365408901 6 1 4 1 1369873688 7 2 4 1 1369877532 8 7 4 1 1369881431 9 7 4 2 1369882897 12 5 4 1 1505165504 13 6 4 1 1369887643 17 8 4 1 1369958105 18 1 4 2 1374557701 22 7 4 3 1374639901 23 6 4 7 1374640202 24 5 4 2 1374639901 25 8 4 2 1374639901 26 2 4 2 1374639301 27 2 6 1 1376620469 29 2 12 1 1389915486 30 1 23 1 1390978667 31 1 23 2 1391030924 32 2 23 1 1392113103 33 2 23 2 1392696602 34 2 23 3 1392767435 35 7 12 1 1398914256 36 8 43 1 1405281193 37 1 50 1 1405522411 38 5 43 1 1505165504 mdl_course_modules id course module instance section 3 2 9 2 3 5 2 17 2 4 7 2 17 3 5 8 2 17 4 6 9 2 17 5 7 10 2 17 6 8 11 2 17 7 9 12 2 17 8 10 13 2 17 9 11 14 2 17 10 12 15 2 17 11 13 25 2 16 1 14 26 2 23 1 4 28 2 7 1 14 30 4 9 4 26 42 4 23 3 33 45 4 23 6 38 46 4 23 7 37 47 4 23 8 36 48 4 23 9 35 49 4 23 10 32 50 4 23 11 34 51 5 9 5 27 53 5 23 12 43 55 5 23 13 44 mdl_quiz id name 10 Unit 10 Quiz 11 Unit 2 Quiz 12 Unit 3 Quiz 13 Unit 5 Quiz 14 Unit 1 Quiz 15 Unit 8 Quiz 16 Unit 9 Quiz 17 Unit 7 Quiz 18 Unit 4 Quiz mdl_course_modules_completion id coursemoduleid userid completionstate viewed timemodified 14 25 2 0 1 0 15 25 6 0 1 0 67 25 4 1 1 1369873688 68 28 4 1 0 1369874483 69 192 4 1 0 1369875233 70 184 4 1 1 1369877532
Something like this ? update mdl_course_modules_completion c join mdl_quiz_attempts a on a.userid = c.userid join (select max(attempt) max_attempts from mdl_quiz_attempts group by userid) max on max.max_attempts = a.attempt set c.timemodified = a.timemodified where c.userid = :<USER_ID>