I am trying to read an html table using the package XML, but even though it looks easy, I haven’t managed to do it. I tried everything, but the names of the columns are always fixed by R as V1, V2, V3,…
This is the code:
require(XML)
tbl <- readHTMLTable("http://facedata.ornl.gov/ornl/npp_98-08.html”,
header = c("year","ring","CO2", "stem","root","leaf","fine root", "NPP"),
skip.rows=c(1,2),colClasses=c(rep("factor",3),rep("numeric",5)))
Many thanks for your help
The first row of the table is causing trouble. It maybe easiest to remove it:
library(XML)
appURL <- "http://facedata.ornl.gov/ornl/npp_98-08.html"
doc <- htmlParse(appURL)
removeNodes(doc["//table/tr[1]"]) # remove the first row with the troublesome header
myTable <- readHTMLTable(doc, which = 1)
> head(myTable)
Year Plot CO2 Stem Coarse Root Leaf Fine Root Total NPP
1 1998 1 elev 1540 127 362 168 2197
2 1998 2 elev 1487 139 418 175 2219
3 1998 3 amb 1085 112 333 231 1762
4 1998 4 amb 1204 113 368 185 1870
5 1998 5 amb 1136 109 382 56 1683
6 1999 1 elev 1218 98 475 295 2086
Related
I'm really new to working with JSON data, so I had a question about formatting.
Here's the link to the data I was trying to work with
I was using JSONlite and did this:
shot<-"http://stats.nba.com/stats/playerdashptshotlog?DateFrom=&DateTo=&
GameSegment=&LastNGames=0&LeagueID=00&Location=&Month=0&OpponentTeamID=0&
Outcome=&Period=0&PlayerID=202322&Season=2014-15&SeasonSegment=&
SeasonType=Regular+Season&TeamID=0&VsConference=&VsDivision="
I then did fromJSON:
json_data <- fromJSON(paste(readLines(shot), collapse=""))
This gives me the data in a list. My issue (although for all I know I messed up working towards this) is trying to create a data frame out of this info. I was able to make a data frame with code I read under similar questions on the site, but it is all of the data in just one column. Any recommendations would be appreciated!
Thanks
Normally, first thing to do when you get a JSON, you look at the structure.
str(json_data)
Doing so will reveal that your data has a very simple structure: is is a dataframe with rows, a line of headers, wrapped in some metadata about what each column means. Using the $ will allow you to address those specific components. In other words, your specific json is already a data frame structure, all you gotta to is take it out of json
library(jsonlite)
json_data <- fromJSON(paste(readLines(shot), collapse=""))
str(json_data)
mydf <- data.frame(json_data$resultSets$rowSet)
colnames(mydf) <- unlist(json_data$resultSets$headers)
You ought to get something like this:
head(mydf)
GAME_ID MATCHUP LOCATION W FINAL_MARGIN SHOT_NUMBER PERIOD
1 0021401215 APR 14, 2015 - WAS # IND A L -4 1 1
2 0021401215 APR 14, 2015 - WAS # IND A L -4 2 1
3 0021401215 APR 14, 2015 - WAS # IND A L -4 3 1
4 0021401215 APR 14, 2015 - WAS # IND A L -4 4 1
5 0021401215 APR 14, 2015 - WAS # IND A L -4 5 1
6 0021401215 APR 14, 2015 - WAS # IND A L -4 6 1
GAME_CLOCK SHOT_CLOCK DRIBBLES TOUCH_TIME SHOT_DIST PTS_TYPE SHOT_RESULT
1 10:33 7.7 0 1 25 3 missed
2 8:41 14 10 9.6 10.7 2 made
3 6:42 14.9 11 9.7 18.2 2 missed
4 5:16 19 3 3.5 4.2 2 made
5 4:45 19.8 3 3.7 3.3 2 missed
6 3:08 13.5 10 9.7 18 2 missed
CLOSEST_DEFENDER CLOSEST_DEFENDER_PLAYER_ID CLOSE_DEF_DIST FGM PTS
1 Hill, George 201588 4.3 0 0
2 Hill, George 201588 5.7 1 2
3 Hill, George 201588 3 0 0
4 Miles, CJ 101139 4 1 2
5 Hill, Solomon 203524 3 0 0
6 Hill, George 201588 4.5 0 0
I cbind a column "class" to a data frame and got a new tdm1, tdm1<- cbind(tdm1, class), it's all good
the content of class looks like this
1 715
2 715
3 707
4 705
5 704
6 701
7 701
...
Then after cbind, I want to get a look at the class column by using tdm1[,ncol(tdm1)], somehow i got 35 Levels: 156 174 205 250 295 324 335 340 343 345 348 349 361 370 375 381 382 428 439 451 455 701 704 705 706 ... 72 after the correct values for the entire column. it's like a summary of the column value. Idon't know where it came from. this additional information makes my later knn classification weird. how do i get rid of it?
Your object is a factor. Calling ?factor reveals:
factor returns an object of class "factor" which has a set of integer
codes the length of x with a "levels" attribute of mode character and
unique (!anyDuplicated(.))
The levels attribute being printed to your dismay reflects all the unique values contained within the object you are printing. To get rid of it, try:
as.numeric(as.characer(tdm1[,ncol(tdm1)]))
I have a list of data.frames (dfList) from which I'd like to generate directed weighted networks in Igraph in R.
The edge-weight variable is "ValueUSD", while vertices are identified by numbers with the columns "Reporter" and "Partner".
Here it follows a function I prepared called "write.graphs" to be used with lapply on my "dfList".
write.graphs<-function(filename){
d<-graph.data.frame(filename[c("Reporter", "Partner")], directed=TRUE)
d <- set.edge.attribute(d, "weight", value=filename$ValueUSD)
}
graphs<-lapply(dfList, write.graphs)
Every thing work perfectly. If I check for vertex names I get:
graphs[1]$names
NULL
But troubles emerge when I want to use names in place of numbers to identify my vertices, using the corresponding columns "ReporterN" and "PartnerN" in each of my data.frames in dfLIst.
Here you can see how my data.frames look like:
dfList[1]
$`Aug 2014`
Reporter YearPeriod Year Period Commodity Partner NetWeightKg ValueUSD Price PartnerN ReporterN
1 76 201408 2014 8 150910 0 4472917 22028271 4.924811 World Brazil
2 76 201408 2014 8 150910 32 380891 1533948 4.027262 Argentina Brazil
3 76 201408 2014 8 150910 152 239776 1336057 5.572105 Chile Brazil
4 76 201408 2014 8 150910 251 289 2164 7.487889 France Brazil
5 76 201408 2014 8 150910 300 27592 170658 6.185054 Greece Brazil
6
This is the message I get:
> grafi<-lapply(dfList, scrivi.grafi)
There were 50 or more warnings (use warnings() to see the first 50)
> warnings()
Warning messages:
1: In graph.data.frame(filename[c("ReporterN", "PartnerN")], ... :
In `d' `NA' elements were replaced with string "NA"
2: In graph.data.frame(filename[c("ReporterN", "PartnerN")], ... :
In `d' `NA' elements were replaced with string "NA"
3: In graph.data.frame(filename[c("ReporterN", "PartnerN")], ... :
If it helps, I checked:
class(dfList[1]$PartnerN)
[1] "NULL"
Any suggestion? Can anyone explain me what happens?
Thanks a lot, Umberto.
I have this group in a table. Where I need to display one value on the top and rest according to its alphabetical order.
Table
Column1 Value#1 Value#2
Alpha 12 26
Beta 65 745
Gamma 987 87
Pie 7 2
Non-Beta 132 426
Zeta 112 266
I want to sort it like this(Can anyone also tell me the real use of this other than Viewing Purpose)
Table
Column1 Value#1 Value#2
Non-Beta 132 426
Alpha 12 26
Beta 65 745
Gamma 987 87
Pie 7 2
Zeta 112 266
So the Non-Beta has to be displayed on the top and rest according to alphabetical order.
Edit
Thank you very much for the below reply Chris, Really appreciate and yes it works.
I have one more question from the over table format itself...How can I display it in the below format...
Table
Column1 Value#1 Value#2
Non-Beta 132 426
Alpha 12 26
Pie 7 2
Zeta 112 266
Total 263 720
Beta 65 745
Gamma 987 87
Total 1057 832
Thank you
Select the table, right-click the table handle, select Tablix Properties and select the Sorting tab. Press the Add button then click the fx button to open the expression editor. Enter the following expression:
=IIF(Fields!Column1.Value = "Non-Beta", "A" + Fields!Column1.Value, "B" + Fields!Column1.Value)
All we are doing is prefixing the special value field with something so it comes before the other fields.
I know that this is probably very simple but I have been trying to work this out for a while, and I need it for quite a few functions.
I have a DataFrame with 2 columns, both with share price data.
I would like to compute 2 new columns in a new dataframe called 'returns', with each column named as the same as in the first (i.e. 'AAPL' and 'GOOG').
I use this procedure to get the original data and create the 'data' dataframe:
names = ['AAPL', 'GOOG']
def get_data(stock, start, end):
return web.get_data_yahoo(stock, start, end)['Adj Close']
data = pd.DataFrame({n: get_data(n, '1/1/2009', '6/1/2012') for n in names})
I know that the returns could be generate using (from the pandas library):
returns = pd.DataFrame(index=data.index)
returns['*COLUMN A*'] = data['*COLUMN A*'].pct_change()
However I am guessing that I need to use some sort of loop to iterate over either 'names' or the columns but I cannot get anything to work.
Any help would be greatly appreciated. I am sorry if I have been rather vague, but this is my first question and I have searched for 30 minutes through the forum :)
You can use pct_change on the entire df
In [15]: df = DataFrame(np.random.randint(20,size=20).reshape(10,2),
columns=['AAPL','GOOG'],index=date_range('20130101',periods=10))+50
In [16]: df
Out[16]:
AAPL GOOG
2013-01-01 53 54
2013-01-02 66 64
2013-01-03 50 59
2013-01-04 53 57
2013-01-05 67 65
2013-01-06 61 55
2013-01-07 68 52
2013-01-08 64 65
2013-01-09 62 62
2013-01-10 66 50
In [17]: 100*df.pct_change()
Out[17]:
AAPL GOOG
2013-01-01 NaN NaN
2013-01-02 24.528302 18.518519
2013-01-03 -24.242424 -7.812500
2013-01-04 6.000000 -3.389831
2013-01-05 26.415094 14.035088
2013-01-06 -8.955224 -15.384615
2013-01-07 11.475410 -5.454545
2013-01-08 -5.882353 25.000000
2013-01-09 -3.125000 -4.615385
2013-01-10 6.451613 -19.354839