read csv with column-name x value pairs - csv

Question

I have a long (csv) file with "column-name x value" pairs which I would like to read into a pandas.DataFrame
user_id col val
00008901 1 55
00008901 2 66
00011501 1 77
00011501 3 88
00011501 4 99
The result should look like this:
1 2 3 4
00008901 55 66 0 0
00011501 77 0 88 99
I tried to read it into a list and create a DataFrame from it, but pandas crashed as I have 4.5 million elements.
What the best way to do that? Ideally directly with read_csv.

Answer 1

First use read_csv for create DataFrame:
df = pd.to_csv('file.csv')
Then need set_index with unstack:
df1 = df.set_index(['user_id','col'])['val'].unstack(fill_value=0)
print (df1)
col 1 2 3 4
user_id
8901 55 66 0 0
11501 77 0 88 99
Another solution with pivot, replacing NaN to 0 by fillna and last cast to int:
df1 = df.pivot(index='user_id', columns='col', values='val').fillna(0).astype(int)
print (df1)
col 1 2 3 4
user_id
8901 55 66 0 0
11501 77 0 88 99
If get error:
"ValueError: Index contains duplicate entries, cannot reshape"
It means you have some duplicates, so fastest solution is groupby with unstack and some aggreagte function like mean or sum:
print (df.groupby(['user_id','col'])['val'].mean().unstack(fill_value=0))
col 1 2 3 4
user_id
8901 55 66 0 0
11501 77 0 88 99
Better it see in a bit changed csv:
print (df)
user_id col val
0 8901 1 55
1 8901 2 66
2 11501 1 77 > duplicates -> 11501 and 1
3 11501 1 151 > duplicates -> 11501 and 1
4 11501 3 88
5 11501 4 99
print (df.groupby(['user_id','col'])['val'].mean().unstack(fill_value=0))
col 1 2 3 4
user_id
8901 55 66 0 0
11501 114 0 88 99
Actually I thought I had no duplicates, but found out that I really have some ...
I could not use ".mean" as it is categorial value, but solved the problem by first looking at the sorted table and then just keeping the last entry ... then applying the (great !) solution .. which I still have to fully understand ;-)
df.sort(columns=(['user_id','col']) ) # optional for debugging
df.drop_duplicates(subset=['user_id','col'], keep='last', inplace=True)
df_table = df.set_index(['user_id','col'])['val'].unstack(fill_value=0)

Answer 2

You can't directly read in the required structure using read_csv. But you can use pivot_table function to convert to the desired structure.
df = pd.read_csv('filepath/your.csv')
df = pd.pivot_table(df, index='user_id', columns='col', values='val, aggfunc='mean').reset_index()
The output will be like
1 2 3 4
00008901 55 66 0 0
00011501 77 0 88 99

Answer 3

I don't think it is possible to use read_csv to parse the csv file.
You can create a data structure such as dictionary and use it to create a dataframe:
import pandas as pd
from collections import defaultdict
import csv
data_dict = defaultdict(lambda: [0] * columns)
columns = 4
delimiter = ','
with open("my_csv.csv") as csv_file:
reader = csv.DictReader(csv_file,delimiter=delimiter)
for row in reader:
row_id = row["user_id"]
col = int(row["col"])-1
val = int(row["val"])
data_dict[row_id][col] = val
df = pd.DataFrame(data_dict.values(), index=data_dict.keys(), columns=range(1, columns+1))
For a csv file that contains:
user_id,col,val
00008901,1,55
00008901,2,66
00011501,1,77
00011501,3,88
00011501,4,99
The output is:
1 2 3 4
00008901 55 66 0 0
00011501 77 0 88 99