access VBA how to remove duplication from Collection - ms-access

Question

how do I remove duplication of objects in a Collection? This is what I tried:
dim unique_students as Collection
dim no_duplicate_student as cls_Student
dim no_duplication as boolean
For Each student as cls_Student In list_Student 'list_Students = original unsorted collection
no_duplication = True
Dim s As cls_Student
For Each s In unique_students
If s.name = student.name Then
no_duplication = False 'Duplication found
Exit For
End If
next s
If no_duplication Then
'Inserted into new sorted collection if no values matches that of the sorted collection
Set no_duplicate_student = New clsOverlap
no_duplicate_student.name = student.name
unique_students.Add no_duplicate_student
End If
Next student
This however still takes a long time (if list_Student.Count > 5000, then it'll take 30min+ to run). Is there a more efficient way (if possible, decrease time complexity) of removing duplication in a Collection?

Answer 1

Add the student names to a dictionary, which has the .Exists method to check if an item is already in the dictionary.
You can get some ideas from CollectionToDictionary in Collection And Dictionary Procedures
Something like this in your For Each student loop:
If Dict.Exists(Key:=student.name) Then
' is duplicate!
Else
Dict.Add Key:=student.name, Item:=student.name
' you could also do Item:=student if you want the de-duplicated list in a dictionary
End If
The inner loop is not needed. The function will probably run almost instantaneously.

Answer 2

I usually use a dictionary like what Andre451 suggested. Alternatively you could use an ArrayList like this. I'm not sure if there's much of a performance difference between the two but this method also produces a sorted list if that's desirable. The dictionary though can carry key/value pairs, so it just depends on what you're going after.
Sub Demo()
Set AL = CreateObject("System.Collections.ArrayList")
AL.Add "A"
AL.Add "B"
AL.Add "A"
AL.Add "A"
AL.Add "C"
'Sorting allows sequential comparisons to determine uniqueness
'You could also do something similar to the dictionary method with ArrayList.Contains
'but the evluation of ArrayList.Contains runs slower than this
AL.Sort
For i = 0 To AL.Count - 2
If AL(i) <> AL(i + 1) Then
'Prints unique values
Debug.Print AL(i)
End If
Next
If AL(i) <> AL(i - 1) Then
'Prints last value if unique by comparing to one before it
Debug.Print AL(i)
End If
End Sub
Edit: After testing I confirmed that the dictionary method is about twice as fast at 7.7 seconds versus 13 seconds per million. However, at the OP count of 5000 the difference is only 40 vs 80 ms.
Testing Code Here...
Public Declare Function GetTickCount Lib "kernel32.dll" () As Long
Sub DictionaryDemo()
Set D = CreateObject("Scripting.Dictionary")
Set AL = CreateObject("System.Collections.ArrayList")
For i = 0 To 10 ^ 6
AL.Add Round(Rnd * 10, 0)
Next
Start = GetTickCount
For i = 0 To AL.Count - 1
If Not (D.Exists(AL(i))) Then
D.Add AL(i), ""
Debug.Print AL(i)
End If
Next
Debug.Print GetTickCount - Start
End Sub
Sub ArrayListDemo()
Set AL = CreateObject("System.Collections.ArrayList")
For i = 0 To 10 ^ 6
AL.Add Round(Rnd * 10, 0)
Next
'Sorting allows sequential comparisons to determine uniqueness
Start = GetTickCount
AL.Sort
For i = 0 To AL.Count - 2
If AL(i) <> AL(i + 1) Then
'Prints unique values
Debug.Print AL(i)
End If
Next
If AL(i) <> AL(i - 1) Then
'Prints last value if unique by comparing to one before it
Debug.Print AL(i)
End If
Debug.Print GetTickCount - Start
End Sub
Edit Again: Ok, so I find this very interesting. What's most important appears to be the actual type itself. So for example, the testing above creates an ArrayList from which unique values are to be derived. If this is changed to a basic integer array Dim AL(10 ^ 6) As Integer, then the time is slashed from 7.7 to 0.8 seconds. Likewise, the ArrayList method can be slashed from 13 seconds down to 0.5 seconds simply by adding the line A = AL.ToArray after the sorting operation and looping through the array A instead.
This makes sense, since the memory allocation for arrays allows them to be processed really quick. It's also why some people prefer to create their own sorting and uniqueness algorithms rather than go with a less efficient but easy to use method employing a Dictionary or ArrayList as originally suggested here. The dictionary and ArrayLists are still powerful tools and, as mentioned above, they can still extract unique values from a 1 million length in a fraction of a second, but it's worth noting that when it comes to raw efficiency a simple array is wicked fast at looping.
The code below will extract unique values from a 1 million length array in about 0.3 seconds. It's not much different than the OP, but it's a lot more efficient. This is because looping through a collection is horribly slow, not because there was anything inefficient with the basic strategy. Also, notice that the efficiency will decrease as the number of unique values increases (this test only used the 10 unique of 1-10).
Sub ArrayDemo()
Dim A(10 ^ 6) As Integer
Dim B(10) As Integer
For i = 0 To 10 ^ 6
A(i) = Round(Rnd * 10, 0)
Next
Start = GetTickCount
k = 0
For i = 0 To 10 ^ 6
For j = 0 To k
If B(j) = A(i) Then GoTo nxt
Next
B(k) = A(i)
Debug.Print B(k)
k = k + 1
nxt:
Next
Debug.Print GetTickCount - Start
End Sub