expand and duplicates - duplicates

Question

I have a dataset as follows.
* Example generated by -dataex-. To install: ssc install dataex
clear
input str4 id str8 drug1 str3(drug2 drug3)
"pat" "thiazide" "BB" "CCB"
"ann" "thiazide" "ace" ""
"mary" "ace" "" ""
"john" "ace" "" ""
end
I want to create a separate row for each person for each drug they have. reshape definitely isn't what I want here: I've been experimenting with expand and think this is the solution.... , bar a few little things that I can't get right. I'm thinking I need to expand and then remove duplicates.
Step 1:
Here's the code I used to get what I want, and it works fine, except for pat: his third drug isn't copying into his third row.
expand 3
by id, sort: generate drug = cond(_n == 1,drug1, drug2, drug3)
* Example generated by -dataex-. To install: ssc install dataex
clear
input str4 id str8 drug1 str3(drug2 drug3) str8 drug
"ann" "thiazide" "ace" "" "thiazide"
"ann" "thiazide" "ace" "" "ace"
"ann" "thiazide" "ace" "" "ace"
"john" "ace" "" "" "ace"
"john" "ace" "" "" ""
"john" "ace" "" "" ""
"mary" "ace" "" "" "ace"
"mary" "ace" "" "" ""
"mary" "ace" "" "" ""
"pat" "thiazide" "BB" "CCB" "thiazide"
"pat" "thiazide" "BB" "CCB" "BB"
"pat" "thiazide" "BB" "CCB" "BB"
end
If anyone could instruct me on how to fix this that would be brilliant.
Step 2:
For step two (imagine that pat's rows are correct for this), I want to remove duplicates so that I am left only with the correct number of rows for each person according to their distinct number of drugs. For example, none of pat's rows should be duplicates, so I want to keep all his rows. but ann has a duplicate row that I need to remove.
This is what I have used:
bys id drug: gen dup2=cond(_N==1,0,_n)
drop if dup2>1
This is ok, but I am left with extra rows for mary and john. I deal with these using:
drop if drug==""
Is this the most efficient/least error prone approach?
Amendment
It turns out the my toy dataset was too simplistic to reflect my real data. My actual data are already long, so this is why reshape won't work here. I am very happy to be corrected, but I think expand might be the way to go. Except, now, when I try to expand on more complex data I can not figure out how to make the loop make the dataset I need (essentially, one observation per person per drug). Here is an example of what I have:
clear
input str4 id int day str8 drug1 str3(drug2 drug3)
"ann" 14 "thiazide" "ace" ""
"ann" 70 "thiazide" "ace" ""
"ann" 1 "CCB" "" ""
"ann" 35 "thiazide" "ace" ""
"ann " 30 "CCB" "" ""
"john" 1 "ace" "" ""
"john" 30 "CCB" "" ""
"john" 150 "ace" "" ""
"john" 60 "ace" "" ""
"john" 60 "CCB" "" ""
"john" 30 "ace" "" ""
"john" 1 "CCB" "" ""
"mary" 30 "ace" "" ""
"mary" 1 "ace" "" ""
"mary" 115 "thiazide" "" ""
"mary" 60 "ace" "" ""
"mary" 90 "ace" "" ""
"mary" 120 "ace" "" ""
"pat" 30 "thiazide" "BB" "CCB"
"pat" 1 "ace" "" ""
"pat" 30 "ace" "" ""
"pat" 1 "thiazide" "BB" "CCB"
end
After using:
expand 3
Here is an example of what I want, but am unsure of how to write the code to get this. I have tried using variations of Nick Cox's loop below; but am not getting it right.
clear
input str4 id int day str8 drug1 str3(drug2 drug3) str8 drug
"ann" 1 "CCB" "" "" "CCB"
"ann" 1 "CCB" "" "" ""
"ann" 1 "CCB" "" "" ""
"ann" 14 "thiazide" "ace" "" "thiazide"
"ann" 14 "thiazide" "ace" "" "ace"
"ann" 14 "thiazide" "ace" "" ""
"ann" 35 "thiazide" "ace" "" "thiazide"
"ann" 35 "thiazide" "ace" "" "ace"
"ann" 35 "thiazide" "ace" "" ""
"ann" 70 "thiazide" "ace" "" "thiazide"
"ann" 70 "thiazide" "ace" "" "ace"
"ann" 70 "thiazide" "ace" "" ""
"ann " 30 "CCB" "" "" "CCB"
"ann " 30 "CCB" "" "" ""
"ann " 30 "CCB" "" "" ""
"john" 1 "CCB" "" "" "CCB"
"john" 1 "CCB" "" "" ""
"john" 1 "CCB" "" "" ""
"john" 1 "ace" "" "" "ace"
"john" 1 "ace" "" "" ""
"john" 1 "ace" "" "" ""
"john" 30 "CCB" "" "" "CCB"
"john" 30 "CCB" "" "" ""
"john" 30 "CCB" "" "" ""
"john" 30 "ace" "" "" "ace"
"john" 30 "ace" "" "" ""
"john" 30 "ace" "" "" ""
"john" 60 "CCB" "" "" "CCB"
"john" 60 "CCB" "" "" ""
"john" 60 "CCB" "" "" ""
"john" 60 "ace" "" "" "ace"
"john" 60 "ace" "" "" ""
"john" 60 "ace" "" "" ""
"john" 150 "ace" "" "" "ace"
"john" 150 "ace" "" "" ""
"john" 150 "ace" "" "" ""
"mary" 1 "ace" "" "" "ace"
"mary" 1 "ace" "" "" ""
"mary" 1 "ace" "" "" ""
"mary" 30 "ace" "" "" "ace"
"mary" 30 "ace" "" "" ""
"mary" 30 "ace" "" "" ""
"mary" 60 "ace" "" "" "ace"
"mary" 60 "ace" "" "" ""
"mary" 60 "ace" "" "" ""
"mary" 90 "ace" "" "" "ace"
"mary" 90 "ace" "" "" ""
"mary" 90 "ace" "" "" ""
"mary" 115 "thiazide" "" "" "thiazide"
"mary" 115 "thiazide" "" "" ""
"mary" 115 "thiazide" "" "" ""
"mary" 120 "ace" "" "" "ace"
"mary" 120 "ace" "" "" ""
"mary" 120 "ace" "" "" ""
"pat" 1 "ace" "" "" "ace"
"pat" 1 "ace" "" "" ""
"pat" 1 "ace" "" "" ""
"pat" 1 "thiazide" "BB" "CCB" "thiazide"
"pat" 1 "thiazide" "BB" "CCB" "BB"
"pat" 1 "thiazide" "BB" "CCB" "CCB"
"pat" 30 "ace" "" "" "ace"
"pat" 30 "ace" "" "" ""
"pat" 30 "ace" "" "" ""
"pat" 30 "thiazide" "BB" "CCB" "thiazide"
"pat" 30 "thiazide" "BB" "CCB" "BB"
"pat" 30 "thiazide" "BB" "CCB" "CCB"
end
At this point I can remove the observations with missing values, and clean up the dataset to get the following:
drop if missing(drug)
drop drug?
clear
input str4 id int day str8 drug
"ann" 1 "CCB"
"ann" 14 "thiazide"
"ann" 14 "ace"
"ann" 35 "thiazide"
"ann" 35 "ace"
"ann" 70 "thiazide"
"ann" 70 "ace"
"ann " 30 "CCB"
"john" 1 "CCB"
"john" 1 "ace"
"john" 30 "CCB"
"john" 30 "ace"
"john" 60 "CCB"
"john" 60 "ace"
"john" 150 "ace"
"mary" 1 "ace"
"mary" 30 "ace"
"mary" 60 "ace"
"mary" 90 "ace"
"mary" 115 "thiazide"
"mary" 120 "ace"
"pat" 1 "ace"
"pat" 1 "thiazide"
"pat" 1 "BB"
"pat" 1 "CCB"
"pat" 30 "ace"
"pat" 30 "thiazide"
"pat" 30 "BB"
"pat" 30 "CCB"
end

Answer 1

I am mystified at the dismissal of reshape without argument or evidence. reshape gets you there directly except for one line to clean out missings.
* Example generated by -dataex-. To install: ssc install dataex
clear
input str4 id str8 drug1 str3(drug2 drug3)
"pat" "thiazide" "BB" "CCB"
"ann" "thiazide" "ace" ""
"mary" "ace" "" ""
"john" "ace" "" ""
end
reshape long drug, i(id) j(seq)
drop if missing(drug)
list, sepby(id)
+-----------------------+
| id seq drug |
|-----------------------|
1. | ann 1 thiazide |
2. | ann 2 ace |
|-----------------------|
3. | john 1 ace |
|-----------------------|
4. | mary 1 ace |
|-----------------------|
5. | pat 1 thiazide |
6. | pat 2 BB |
7. | pat 3 CCB |
+-----------------------+
EDIT:
Your idea of starting with expand can be made to work quite easily. Underneath the hood reshape is doing something similar.
clear
input str4 id str8 drug1 str3(drug2 drug3)
"pat" "thiazide" "BB" "CCB"
"ann" "thiazide" "ace" ""
"mary" "ace" "" ""
"john" "ace" "" ""
end
expand 3
sort id
gen drug = ""
quietly forval j = 1/3 {
by id: replace drug = drug`j' if _n == `j'
}
drop if missing(drug)
drop drug?
list, sepby(id)
EDIT 2
The extra complications are just that, complications, and don't imply a different approach. You need greater faith and to understand that reshape is more versatile than you think it is! See e.g. the FAQ here as well as the help and manual entry.
Trivially, I am going to assume that "Ann " is just a typo for "Ann". Then what we have is not just different days for the same people but also somehow duplicates for some people and days. All that means is to spell out the identifiers more fully; in fact we need one extra variable. The principle that sometimes a new identifier variable is needed to spell out a tacit order, even if arbitrarily, is discussed in the FAQ cited. The idea that "long long" layouts are possible is also a standard notion.
clear
input str4 id int day str8 drug1 str3(drug2 drug3)
"ann" 14 "thiazide" "ace" ""
"ann" 70 "thiazide" "ace" ""
"ann" 1 "CCB" "" ""
"ann" 35 "thiazide" "ace" ""
"ann " 30 "CCB" "" ""
"john" 1 "ace" "" ""
"john" 30 "CCB" "" ""
"john" 150 "ace" "" ""
"john" 60 "ace" "" ""
"john" 60 "CCB" "" ""
"john" 30 "ace" "" ""
"john" 1 "CCB" "" ""
"mary" 30 "ace" "" ""
"mary" 1 "ace" "" ""
"mary" 115 "thiazide" "" ""
"mary" 60 "ace" "" ""
"mary" 90 "ace" "" ""
"mary" 120 "ace" "" ""
"pat" 30 "thiazide" "BB" "CCB"
"pat" 1 "ace" "" ""
"pat" 30 "ace" "" ""
"pat" 1 "thiazide" "BB" "CCB"
end
replace id = trim(id)
bysort id day : gen SEQ = _n
reshape long drug, i(id day SEQ) j(seq)
drop if missing(drug)
list, sepby(id)
+-----------------------------------+
| id day SEQ seq drug |
|-----------------------------------|
1. | ann 1 1 1 CCB |
2. | ann 14 1 1 thiazide |
3. | ann 14 1 2 ace |
4. | ann 30 1 1 CCB |
5. | ann 35 1 1 thiazide |
6. | ann 35 1 2 ace |
7. | ann 70 1 1 thiazide |
8. | ann 70 1 2 ace |
|-----------------------------------|
9. | john 1 1 1 ace |
10. | john 1 2 1 CCB |
11. | john 30 1 1 ace |
12. | john 30 2 1 CCB |
13. | john 60 1 1 ace |
14. | john 60 2 1 CCB |
15. | john 150 1 1 ace |
|-----------------------------------|
16. | mary 1 1 1 ace |
17. | mary 30 1 1 ace |
18. | mary 60 1 1 ace |
19. | mary 90 1 1 ace |
20. | mary 115 1 1 thiazide |
21. | mary 120 1 1 ace |
|-----------------------------------|
22. | pat 1 1 1 ace |
23. | pat 1 2 1 thiazide |
24. | pat 1 2 2 BB |
25. | pat 1 2 3 CCB |
26. | pat 30 1 1 ace |
27. | pat 30 2 1 thiazide |
28. | pat 30 2 2 BB |
29. | pat 30 2 3 CCB |
+-----------------------------------+

Answer 2

Here's my effort at more complex data - seems to work ok, but happy to be corrected. Or if there is another better way of doing this, please do post!
Toy data here
clear
input str4 id int day str8 drug1 str3(drug2 drug3)
"pat" 1 "thiazide" "BB" "CCB"
"pat" 1 "ace" "" ""
"pat" 30 "ace" "" ""
"pat" 30 "thiazide" "BB" "CCB"
"ann" 1 "CCB" "" ""
"ann" 14 "thiazide" "ace" ""
"ann " 30 "CCB" "" ""
"ann" 35 "thiazide" "ace" ""
"ann" 70 "thiazide" "ace" ""
"mary" 1 "ace" "" ""
"mary" 30 "ace" "" ""
"mary" 60 "ace" "" ""
"mary" 90 "ace" "" ""
"mary" 115 "thiazide" "" ""
"mary" 120 "ace" "" ""
"john" 150 "ace" "" ""
"john" 1 "CCB" "" ""
"john" 1 "ace" "" ""
"john" 30 "CCB" "" ""
"john" 30 "ace" "" ""
"john" 60 "CCB" "" ""
"john" 60 "ace" "" ""
end
code here:
expand 3
gen drug=""
sort id day
egen group=group(id day drug1)
bys id group: gen count=_n
forval j = 1/3 {
bys id group: replace drug = drug`j' if count == `j'
}
drop if missing(drug)
drop drug? count group
NJC simplification:
expand 3
gen drug = ""
forval j = 1/3 {
by id day drug1: replace drug = drug`j' if _n == `j'
}
drop if missing(drug)
drop drug?