I am trying to have the following key value pairs in a table. The table is needed in Angular platform, so *ngFor loop would be something that would be used. I am looking for an efficient way where it there is no workaround such as creating 2 table and joining them.
{"Lawrence":{"Unallocated":215,"All":2,"Both":0,"Slabs":3,"Tiles":121},"West":{"Unallocated":82,"All":1,"Both":0,"Slabs":1,"Tiles":1},"King":{"Unallocated":388,"All":15,"Both":12,"Slabs":264,"Tiles":558}}
I am looking to have a table as such
Unallocated
All
Both
Slabs
Tiles
Lawrence
215
2
0
3
121
West
82
1
0
1
1
King
388
15
12
264
558
I have tried various ways from bootstrap but have not been successful. Any help would be greatly appreciated.
const data=JSON.parse(`{"Lawrence":{"Unallocated":215,"All":2,"Both":0,"Slabs":3,"Tiles":121},"West":{"Unallocated":82,"All":1,"Both":0,"Slabs":1,"Tiles":1},"King":{"Unallocated":388,"All":15,"Both":12,"Slabs":264,"Tiles":558}}`)
const newData=Object.entries(data).map(([key,value])=>{
const y={['title']:key}
return Object.assign({},y,value)
})
const headers=Object.keys(newData[0])
console.log(headers)
console.log(JSON.stringify(newData,null,4))
.as-console-wrapper { max-height: 100% !important; top: 0; }
Related
I need help with the specific code I will paste below. I am using the Ames Housing data set collected by Dean De Cock.
I am using a Python notebook and editing thru Anaconda's Jupyter Lab 2.1.5.
The code below is supposed to replace all np.nan or "None" values. For some reason,
after repeatedly calling a hand-made function inside a for loop, the columns of the resulting data frame get swapped around.
Note: I am aware I could do this with an "imputer." I plan to select numeric and object type features, impute them separately then put them back together. As a side-note, is there any way I can do that while having the details I output manually using text displayed or otherwise verified?
In the cell in question, the flow is:
Get and assign the number of data points in the data frame df_train.
Get and assign a series that lists the count of null values in df_train. The syntax is sr_null_counts = df_train.isnull().sum().
Create an empty list to which names of features that have 5% of their values equal to null are appended. They will be dropped later,
outside the for loop. I thought at first that this was the problem since the command to drop the columns of df_train in-place
used to be within the for-loop.
Repeatedly call a hand-made function to impute columns with null values not exceeding 5% of the row count for df_train.
I used a function that has a for-loop and nested try-except statements to:
Accept a series and, optionally, the series' name when it was a column in a dataframe. It assigns a copy of the passed series
to a local variable.
In the exact order, (a) try to replace all null (NaN or None) values with the mean of the passed series.
(b) If that fails, try to replace all null values with the median of the series.
(c) If even that fails, replace all null values with the mode of the series.
Return the edited copy of the series with all null values replaced. It should also print out strings that tell me what feature
was modified and what summary statistic was used to replace/impute the missing values.
The final line is to drop all the columns marked as having more than 5% missing values.
Here is the full code:
Splitting the main dataframe into a train and test set.
The full data-set was loaded thru df_housing = pd.read_csv(sep = '\t', filepath_or_buffer = "AmesHousing.tsv").
def make_traintest(df, train_fraction = 0.7, random_state_val = 88):
df = df.copy()
df_train = df.sample(frac = train_fraction, random_state = random_state_val)
bmask_istrain = df.index.isin(df_train.index.values)
df_test = df.loc[ ~bmask_istrain ]
return {
"train":df_train,
"test":df_test
}
dict_traintest = make_traintest(df = df_housing)
df_train = dict_traintest["train"]
df_test = dict_traintest["test"]
Get a List of Columns With Null Values
lst_have_nulls = []
for feature in df_housing.columns.values.tolist():
nullcount = df_housing[feature].isnull().sum()
if nullcount > 0:
lst_have_nulls.append(feature)
print(feature, "\n=====\nNull Count:\t", nullcount, '\n', df_housing[feature].value_counts(dropna = False),'\n*****')
Definition of the hand-made function:
def impute_series(sr_values, feature_name = ''):
sr_out = sr_values.copy()
try:
sr_out.fillna(value = sr_values.mean())
print("Feature", feature_name, "imputed with mean:", sr_values.mean())
except Exception as e:
print("Filling NaN values with mean of feature", feature_name, "caused an error:\n", e)
try:
sr_out.fillna(value = sr_values.median())
print("Feature", feature_name, "imputed with median:", sr_values.median())
except Exception as e:
print("Filling NaN values with median for feature", feature_name, "caused an error:\n", e)
sr_out.fillna(value = sr_values.mode())
print("Feature", feature_name, "imputed with mode:", sr_values.mode())
return sr_out
For-Loop
Getting the count of null values, defining the empty list of columns to drop to allow appending, and repeatedly
doing the following: For every column in lst_have_nulls, check if the column has equal, less or more than 5% missing values.
If more, append the column to lst_drop. Else, call the hand-made imputing function. After the for-loop, drop all columns in
lst_drop, in-place.
Where did I go wrong? In case you need the entire notebook, I have uploaded it to Kaggle. Here is a link.
https://www.kaggle.com/joachimrives/ames-housing-public-problem
Update: Problem Still Exists After Testing Anvar's Answer with Changes
When I tried the code of Anvar Kurmukov, my dataframe column values still got swapped. The change I made was adding int and float to the list of dtypes to check. The changes are inside the for-loop:
if dtype in [np.int64, np.float64, int, float].
It may be a problem with another part of my code in the full notebook. I will need to check where it is by calling df_train.info() cell by cell from the top. I tested the code in the notebook I made public. It is in cell 128. For some reason, after running Anvar's code, the df_train.info() method returned this:
1st Flr SF 2nd Flr SF 3Ssn Porch Alley Bedroom AbvGr Bldg Type Bsmt Cond Bsmt Exposure Bsmt Full Bath Bsmt Half Bath ... Roof Style SalePrice Screen Porch Street TotRms AbvGrd Total Bsmt SF Utilities Wood Deck SF Year Built Year Remod/Add
1222 1223 534453140 70 RL 50.0 4882 Pave NaN IR1 Bnk ... 0 0 0 0 0 NaN NaN NaN 0 87000
1642 1643 527256040 20 RL 81.0 13870 Pave NaN IR1 HLS ... 52 0 0 174 0 NaN NaN NaN 0 455000
1408 1409 905427050 50 RL 66.0 21780 Pave NaN Reg Lvl ... 36 0 0 144 0 NaN NaN NaN 0 185000
1729 1730 528218050 60 RL 65.0 10237 Pave NaN Reg Lvl ... 72 0 0 0 0 NaN NaN NaN 0 178900
1069 1070 528180110 120 RL 58.0 10110 Pave NaN IR1 Lvl ... 48 0 0 0 0 NaN NaN NaN 0 336860
tl;dr instead of try: except you should simply use if and check dtype of the column; you do not need to iterate over columns.
drop_columns = df.columns[df.isna().sum() / df.shape[0] > 0.05]
df.drop(drop_columns, axis=1)
num_columns = []
cat_columns = []
for col, dtype in df.dtypes.iteritems():
if dtype in [np.int64, np.float64]:
num_columns.append(col)
else:
cat_columns.append(col)
df[num_columns] = df[num_columns].fillna(df[num_columns].mean())
df[cat_columns] = df[cat_columns].fillna(df[cat_columns].mode())
Short comment on make_traintest function: I would simply return 2 separate DataFrames instead of a dictionary or use sklearn.model_selection.train_test_split.
upd. You can check for number of NaN values in a column, but it is unnecessary if your only goal is to impute NaNs.
Answer
I discovered the answer as to why my columns were being swapped. They were not actually being swapped. The original problem was that I had not set the "Order" column as the index column. To fix the problem on the notebook in my PC, I simply added the following paramater and value to pd.read_csv: index_col = "Order". That fixed the problem on my local notebook. When I tried it on the Kaggle notebook, however, it did not fix the problem
The version of the Ames Housing data set I first used on the notebook - for some reason - was also the cause for the column swapping.
Anvar's Code is fine. You may test the code I wrote, but to be safe, defer to Anvar's code. Mine is still to be tested.
Testing Done
I modified the Kaggle notebook I linked in my question. I used the data set I was actually working in with my PC. When I did that, the code given by Anvar Kurmukov's answer worked perfectly. I tested my own code and it seems fine, but test both versions before trying. I only reviewed the data sets using head() and manually checked the column inputs. If you want to check the notebook, here it is:
https://www.kaggle.com/joachimrives/ames-housing-public-problem/
To test if the data set was at fault, I created to data frames. One was taken directly from my local file uploaded to Kaggle. The other used the current version of the Ames Iowa Housing data set I had used as input. The columns were properly "aligned" with their expected input. To find the expected column values, I used this source:
http://jse.amstat.org/v19n3/decock/DataDocumentation.txt
Here are the screenshots of the different results I got when I swapped data sets:
With an uploaded copy of my local file:
With the original AmesHousing.csv From Notebook Version 1:
The data set I Used that Caused the Column-swap on the Kaggle Notebook
https://www.kaggle.com/marcopale/housing
I am trying to use tidyJSON to extract information from JSON, but I am open to any R package that can achieve my ends. I took a look at the documentation and vignittes and found the complex example was helpful. However, the information I want is nested inside of a non-key-value pair and I am not sure how to access it. I am interested in getting appid, name, developer, etc., but this information is within 570 and 730:
{"570":{"appid":570,"name":"Dota 2","developer":"Valve","publisher":"Valve","score_rank":71,"owners":102151578,"owners_variance":259003,"players_forever":102151578,"players_forever_variance":259003,"players_2weeks":9436299,"players_2weeks_variance":89979,"average_forever":11727,"average_2weeks":1229,"median_forever":277,"median_2weeks":662,"ccu":811259,"price":"0","tags":{"Free to Play":22678,"MOBA":7808,"Strategy":7415,"Multiplayer":6757,"Team-Based":4848,"Action":4602,"e-sports":4089,"Online Co-Op":3669,"Competitive":3553,"PvP":2655,"RTS":2267,"Difficult":2129,"RPG":2114,"Fantasy":2044,"Tower Defense":2024,"Co-op":1898,"Character Customization":1514,"Replay Value":1487,"Action RPG":1397,"Simulation":1024}},
"730":{"appid":730,"name":"Counter-Strike: Global Offensive","developer":"Valve","publisher":"Valve","score_rank":78,"owners":29225079,"owners_variance":154335,"players_forever":28552354,"players_forever_variance":152685,"players_2weeks":9102348,"players_2weeks_variance":88410,"average_forever":17648,"average_2weeks":791,"median_forever":5030,"median_2weeks":358,"ccu":543626,"price":"1499","tags":{"FPS":17082,"Multiplayer":13744,"Shooter":12833,"Action":10881,"Team-Based":10369,"Competitive":9664,"Tactical":8529,"First-Person":7329,"e-sports":6716,"PvP":6383,"Online Co-Op":5714,"Military":4621,"Co-op":4435,"Strategy":4424,"War":4361,"Realistic":3196,"Trading":3191,"Difficult":3158,"Fast-Paced":3100,"Moddable":2496}}
There are many thousands of such entries. Is there a way to skip the "top-level" and look within the nest?
The JSON information is from http://steamspy.com/api.php?request=top100in2weeks
This might be what you need:
library(jsonlite)
data = fromJSON("http://steamspy.com/api.php?request=top100in2weeks")
appid = lapply(data, function(x){x$appid})
name = lapply(data, function(x){x$name})
df = data.frame(appid = unlist(appid),
name = unlist(name),
stringsAsFactors = F)
Result:
> head(df)
appid name
570 570 Dota 2
730 730 Counter-Strike: Global Offensive
578080 578080 PLAYERUNKNOWN'S BATTLEGROUNDS
440 440 Team Fortress 2
271590 271590 Grand Theft Auto V
433850 433850 H1Z1: King of the Kill
I'll let you add the rest of the information
Edit: Adding arrays to a dataframe
Adding the tags information for each game in the data frame is possible. And the times tagged as well. For each game you must store an array of tag names in a column and the tag quantities in another.
After the definition of df add the following lines:
for(k in 1:nrow(d)){
d$tags[k] = list(names(data[[k]]$tags))
d$tagsQ[k] = list(unlist(data[[k]]$tags))
}
This will give you:
> d["570",]
appid name
570 570 Dota 2
tags
570 Free to Play, MOBA, Strategy, Multiplayer, Team-Based, Action, e-sports, Online Co-Op, Competitive, PvP, RTS, Difficult, RPG, Fantasy, Tower Defense, Co-op, Character Customization, Replay Value, Action RPG, Simulation
tagsQ
570 22686, 7810, 7420, 6759, 4850, 4603, 4092, 3672, 3555, 2657, 2267, 2130, 2116, 2045, 2024, 1898, 1514, 1487, 1397, 1023
In this situation, columns tags and tagsQ contain lists. To obtain the second tag and quantity for appid 570 do:
> df["570","tags"][[1]][2]
[1] "MOBA"
> d["570","tagsQ"][[1]][2]
MOBA
7810
I use this url for fetching map tile from google server
http://mts0.google.com/vt/lyrs=m#189000000&hl=en&src=app&x=41189&y=25680&z=16&s=Gal
I wonder if there is a way to customize this url, by adding some extra parameters to fetch tile without any label of streets or extra info or overlays.
something just like customizing map in map api v3.
any suggestion would be welcomed.
I didn't find a documentation about it, but there is a parameter apistyle
the value(must be urlencoded) to hide street-labels would be
s.t:3|s.e:l|p.v:off
The following is a guess because of a missing documentation:
s.t defines the feature-type, the value 3 seems to be road
s.e defines the element e.g. labels or geometry
p defines the style, v stands for visibility , the value off should be clear.
result:
https://mts0.google.com/vt/lyrs=m#289000001&hl=en&src=app&x=41189&y=25680&z=16&s=Gal&apistyle=s.t%3A3|s.e%3Al|p.v%3Aoff
You'll have to play around with the parameters to get the desired result. In the past it was possible to get the style e.g. by inspecting the tile-URL with developer-tools when using for example the Styled Map Wizard , but they have modified the tile-URLs used by the javascript-API , the parameters now will be encoded somehow.
A list of parameters and values:
FeatureTypes:s.t
all 0
administrative 1
administrative.country 17
administrative.land_parcel 21
administrative.locality 19
administrative.neighborhood 20
administrative.province 18
landscape 5
landscape.man_made 81
landscape.natural 82
poi 2
poi.attraction 37
poi.business 33
poi.government 34
poi.medical 36
poi.park 40
poi.place_of_worship 38
poi.school 35
poi.sports_complex 39
road 3
road.arterial 50
road.highway 49
road.local 51
transit 4
transit.line 65
transit.station 66
water 6
ElementType: s.e
geometry g
geometry.fill g.f
geometry.stroke g.s
labels l
labels.icon l.i
labels.text l.t
labels.text.fill l.t.f
labels.text.stroke l.t.s
Styler:
color p.c
RGBA hex-value #aarrggbb
gamma p.g
float between 0.01 and 10
hue p.h
RGB hex-value #rrggbb
invert_lightness p.il
true/false
lightness p.l
float between -100 and 100
saturation p.s
float between -100 and 100
visibility p.v
on/simplified/off
weight p.w
integer >=0
Implementation of what Dr. Molle found out:
function getEncodedStyles(styles){
var ret = "";
var styleparse_types = {"all":"0","administrative":"1","administrative.country":"17","administrative.land_parcel":"21","administrative.locality":"19","administrative.neighborhood":"20","administrative.province":"18","landscape":"5","landscape.man_made":"81","landscape.natural":"82","poi":"2","poi.attraction":"37","poi.business":"33","poi.government":"34","poi.medical":"36","poi.park":"40","poi.place_of_worship":"38","poi.school":"35","poi.sports_complex":"39","road":"3","road.arterial":"50","road.highway":"49","road.local":"51","transit":"4","transit.line":"65","transit.station":"66","water":"6"};
var styleparse_elements = {"all":"a","geometry":"g","geometry.fill":"g.f","geometry.stroke":"g.s","labels":"l","labels.icon":"l.i","labels.text":"l.t","labels.text.fill":"l.t.f","labels.text.stroke":"l.t.s"};
var styleparse_stylers = {"color":"p.c","gamma":"p.g","hue":"p.h","invert_lightness":"p.il","lightness":"p.l","saturation":"p.s","visibility":"p.v","weight":"p.w"};
for(i=0;i<styles.length;i++){
if(styles[i].featureType){
ret += "s.t:"+styleparse_types[styles[i].featureType]+"|";
}
if(styles[i].elementType){
if(!styleparse_elements[styles[i].elementType])
console.log("style element transcription unkown:"+styles[i].elementType);
ret += "s.e:"+styleparse_elements[styles[i].elementType]+"|";
}
if(styles[i].stylers){
for(u=0;u<styles[i].stylers.length;u++){
var keys = [];
var cstyler = styles[i].stylers[u]
for(var k in cstyler){
if(k=="color"){
if(cstyler[k].length==7)
cstyler[k] = "#ff"+cstyler[k].slice(1);
else if(cstyler[k].length!=9)
console.log("malformed color:"+cstyler[k]);
}
ret += styleparse_stylers[k]+":"+cstyler[k]+"|";
}
}
}
ret = ret.slice(0,ret.length-1);
ret += ","
}
return encodeURIComponent(ret.slice(0,ret.length-1));
}
Input is in this case a regular google maps styles array - A good wizard for that is Snazzy Maps
Anyways, thanks to Dr. Molle for saving hours!
After converting JSON data into a list using jsonlite, i end up with one of the list looking like following
In this case, 10th element contain a list of 9 columns (always fixed) and 2 rows (varies everytime).
mat <- lset$data$comments$data[10]
mat
[[1]]
id can_remove created_time from.id
1 10152663742099258_10152663749369258 TRUE 2014-07-01T11:10:29+0000 10203711779968366
2 10152663742099258_10152663842204258 TRUE 2014-07-01T12:15:57+0000 706804257
3 10152663742099258_10152663929639258 TRUE 2014-07-01T13:25:28+0000 10152738599744416
4 10152663742099258_10152663976344258 TRUE 2014-07-01T13:59:33+0000 706804257
from.name like_count
1 Aileen Yeow 1
2 Tejas Damania 0
3 Sandeep Kulkarni 1
4 Tejas Damania 0
message
1 Lame statement
2 Don't forget, people like you only because they don't know you! <ed><U+00A0><U+00BD><ed><U+00B8><U+00A1>
3 ...for a second I thought it's Accenture Singapore office with some new theme similar to its brand!
4 This is shanghai and nothing to do with firm I work for <ed><U+00A0><U+00BD><ed><U+00B8><U+008E>
user_likes
1 FALSE
2 FALSE
3 TRUE
4 FALSE
Whole mat shows us as a list of [1]
As you can see, it contains list (within a list?). When i print mat then it shows a structure as seen above.
typeof(mat)
[1] "list"
substring(mat,1,100)
[1] "list(id = c(\"10152663742099258_10152663749369258\", \"10152663742099258_10152663842204258\", \"101526637"
I cant access specific elements (say message) from this. Nor I am able to convert this into a matrix of strings so I can access the elements in structured way.
I changed the fromJSON call parameter to by setting simplifyVector = FALSE (which is default set to true)
lset <- fromJSON(jsonobj, simplifyVector = F, flatten=TRUE, unicode = TRUE)
this changes the way mat is formed, code maintain nesting all the way down to each leaf element. I can keep going deeper using $ and find the string value only at leaf element!
lset$data[[x]]$comments$data[[y]]$from$name
That works for now! thanks for all the help
On my (Cordova, jQuery-Mobile) iOS mobile application I receive some JSON values on the first page.
To store them I use arrays.
To pass them and reuse them on the second page I use a each function, count up an integer(i) on every run and push the received data into several arrays, the integer(i) I store in a DIV in data-id. (<div data-id'"+i+"'>)
As soon as the User navigates to the second page by tapping on one of the DIVs I use the integer(i) in data-id to get every value from the arrays and display on page 2.
Example
Database:
ID NAME SURNAME AGE USERID
1 Michael Douglas 44 uid3242
2 Eric Cartman 11 uid9275
3 Felix Woodspuck 38 uid3852
4 Amadeus Mozart 158 uid1120
Script:
ID = [];
NAME = [];
SURNAME = [];
AGE = [];
USERID = [];
for example - If the user clicks on the DIV with data-id=2
iInt = $(this).attr('data-id'); // iInt=2
$("#abc").text(NAME[iInt];) // NAME[2]; --> Eric
$("#xyz").text(SURNAME[iInt];) // SURNAME[2]; --> Cartman
...
This is just an example, please ignore the values.
But in my App I am using a lot of variables on a lot of pages, and it gets a little messy in code, so what I am wondering about is if there is a more efficent way to save the values I get from JSON to use them on the second page? How do you do it?
Thank you.
I've been tucking the JSON string into localstorage, then pulling it out later and restoring to an object.
localStorage.setItem("jsondata",mytextvar);
Then later:
var obj = JSON.parse(localStorage.getItem("jsondata"));