Python - filter column from a .dat file and return a given value from other columns - csv

Question

I'm new to Python and been practicing with sample data I created (150 rows) of student ID numbers, grades, age, class_code, area_code, etc. What I'm trying to do with the data is not only filtered by a certain column (by grade, age, etc), but then to also create a list of a different column from that row (student ID). I've manged to find how to isolate the column I need to find the certain value by, but then can't figure out how to create that list of the values I need to return.
So here's a sample of 5 rows of the data:
1/A/15/13/43214
2/I/15/21/58322
3/C/17/89/68470
4/I/18/6/57362
5/I/14/4/00000
6/A/16/23/34567
I need a list of the 1st column (student ID), based on filtering the second column (grade)...(and eventually the 3rd column, 4th column, etc. But if I see how it looks with just the 2nd, I think I can figure out the others) Also note: I didn't use headers in the .dat file.
I figured out how to isolate/view the 2nd column.
import numpy
data = numpy.genfromtxt('/testdata.dat', delimiter='/', dtype='unicode')
grades = data[:,1]
print (grades)
to print:
['A' 'I' 'C' 'I' 'I' 'A']
But now, how can I pull just the first column's that correspond to the A's, C's, I's into separate lists?
So I'd want to see a list, also with the commas between the integers of column 1 for the A's, C's, and I's
list from A = [1, 6]
list from C = [3]
list from I = [2, 4, 5]
Again, if I can just see how it's done with just the 2nd column, with just one of the values (say A's), I think I could figure out how to do it for B's, C's, D's, etc and probably the other columns. I just need to see one example to how the syntax would be applied and then like to play around with the rest.
Also, I've been using numpy, but also read about panda, csv and I think those libraries could be possibilities too. But like I said, been using numpy for the .dat files. I don't know if the other libraries would be easier to use?

Answer 1

You actually don't need any additional modules for such a simple task. Pure-Python solution would be reading file line-by-line and 'parsing' them using str.split() will give you your lists, and then you can pretty much filter on any parameter. Something like:
students = {} # store for our students by grade
with open("testdata.dat", "r") as f: # open the file
for line in f: # read the file line by line
row = line.strip().split("/") # split the line into individual columns
# you can now directly filter your row, or you can store the row in a list for later
# let's split them by grade:
grade = row[1] # second column of our row is the grade
# create/append the sublist in our `students` dict keyed by the grade
students[grade] = students.get(grade, []) + [row]
# now your students dict contains all students split by grade, e.g.:
a_students = students["A"]
# [['1', 'A', '15', '13', '43214'], ['6', 'A', '16', '23', '34567']]
# if you want only to collect the A-grade student IDs, you can get a list of them as:
student_ids = [entry[0] for entry in students["A"]]
# ['1', '6']
But let's go back a few steps - if you want a more generalized solution you should just store your list and then create a function to filter it by passed parameters, so:
# define a filter function
# filters should contain a list of filters whereas a filter would be defined as:
# [position, [values]]
# and you can define as many as you want
def filter_sublists(source, filters=None):
result = [] # store for our result
filters = filters or [] # in case no filter is returned
for element in source: # go through every element of our source data
try:
if all(element[f[0]] in f[1] for f in filters): # check if all our filters match
result.append(element) # add the element
except IndexError: # invalid filter position or data position, ignore
pass
return result # return the result
# now we can use it to filter our data, first lets load our data:
with open("testdata.dat", "r") as f: # open the file
students = [line.strip().split("/") for line in f] # store all our students as a list
# now we have all the data in the `students` list and we can filter it by any element
a_students = filter_sublists(students, [[1, ["A"]]])
# [['1', 'A', '15', '13', '43214'], ['6', 'A', '16', '23', '34567']]
# or again, if you just need the IDs:
a_student_ids = [entry[0] for entry in filter_sublists(students, [[1, ["A"]]])]
# ['1', '6']
# but you can filter by any parameter, for example:
age_15_students = filter_sublists(students, [[2, ["15"]]])
# [['1', 'A', '15', '13', '43214'], ['2', 'I', '15', '21', '58322']]
# or you can get all I-grade students aged 14 or 15:
i_students = filter_sublists(students, [[1, ["I"]], [2, ["14", "15"]]])
# [['2', 'I', '15', '21', '58322'], ['5', 'I', '14', '4', '00000']]

Answer 2

Pandas solution:
import pandas as pd
df = pd.read_csv('data.txt', header=None, sep='/')
dfs = {k:v for k,v in df.groupby(1)}
As a result we have a dictionary of DataFrames:
In [59]: dfs.keys()
Out[59]: dict_keys(['I', 'C', 'A'])
In [60]: dfs['I']
Out[60]:
0 1 2 3 4
1 2 I 15 21 58322
3 4 I 18 6 57362
4 5 I 14 4 0
In [61]: dfs['C']
Out[61]:
0 1 2 3 4
2 3 C 17 89 68470
In [62]: dfs['A']
Out[62]:
0 1 2 3 4
0 1 A 15 13 43214
5 6 A 16 23 34567
If you want to have groupped lists of first columns:
In [67]: dfs['I'].iloc[:, 0].tolist()
Out[67]: [2, 4, 5]
In [68]: dfs['C'].iloc[:, 0].tolist()
Out[68]: [3]
In [69]: dfs['A'].iloc[:, 0].tolist()
Out[69]: [1, 6]

Answer 3

You can go through the list and make a boolean to select arrays matching a particular grade. This may require some refinement.
import numpy as np
grades = np.genfromtxt('data.txt', delimiter='/', skip_header=0, dtype='unicode')
res = {}
for grade in set(grades[:, 1].tolist()):
res[grade] = grades[grades[:, 1]==grade][:,0].tolist()
print res