I am trying to create a new column in my DataFrame.
I want the new column to be a*b if the sum of a few other columns is == 0, 1 if the sum is == 1, and 0 otherwise.
The number of columns that I am summing across is dynamic in that it may be 3 columns I am summing across or it could be 100. I have a list of those column names (list_to_check) which could be of any length.
df = pd.DataFrame({'a': [1,2,3], 'b': [2,3,4], 'c':['dd','ee','ff'], 'd1':[5,0,1], 'd2':[5,0,1], 'dn':[5,0,1]})
list_to_check = ['d1','d2','dn']
def func(a,b,c):
if sum(c) == 0:
a*b
elif sum(c) == 1:
1
else:
0
df['new_column'] = np.vectorize(func)(df.a,df.b,df[list_to_check])
vals = df[list_to_check].sum(1)
df['new_col'] = 0
df.loc[vals == 0, 'new_col'] = df.a * df.b
df.loc[vals == 1, 'new_col'] = 1
Related
I am trying to list down all the prime numbers up till a specific number e.g. 1000. The code gets slower as the number increase. I am pretty sure it is because of the for loop where (number -1) is checked by all the prime_factors. Need some advise how I can decrease the processing time of the code for larger numbers. Thanks
import time
t0 = time.time()
prime_list = [2]
number = 0
is_not_prime = 0
count = 0
while number < 1000:
print(number)
for i in range (2,number):
count = 0
if (number%i) == 0:
is_not_prime = 1
if is_not_prime == 1:
for j in range (0,len(prime_list)):
if(number-1)%prime_list[j] != 0:
count += 1
if count == len(prime_list):
prime_list.append(number-1)
is_not_prime = 0
count = 0
break
number += 1
print(prime_list)
t1 = time.time()
total = t1-t0
print(total)
Your solution, on top of being confusing, is very inefficient - O(n^3). Please, use the Sieve of Eratosthenes. Also, learn how to use booleans.
Something like this (not optimal, just a mock-up). Essentially, you start with a list of all numbers, 1-1000. Then, you remove ones that are the multiple of something.
amount = 1000
numbers = range(1, amount)
i = 1
while i < len(numbers):
n = i + 1
while n < len(numbers):
if numbers[n] % numbers[i] == 0:
numbers.pop(n)
else:
n += 1
i += 1
print(numbers)
Finally, I was able to answer because your question isn't language-specific, but please tag the question with the language you're using in the example.
I want to test the hypothesis whether some 30 occurrences should fit a Poisson distribution.
#GNU Octave
X = [8 0 0 1 3 4 0 2 12 5 1 8 0 2 0 1 9 3 4 5 3 3 4 7 4 0 1 2 1 2]; #30 observations
bins = {0, 1, [2:3], [4:5], [6:20]}; #each bin can be single value or multiple values
I am trying to use Pearson's chi-square statistics here and coded the below function. I want a Poisson vector to contain corresponding Poisson probabilities for each bin and count the observations for each bin. I feel the loop is rather redundant and ugly. Can you please let me know how can I re-factor the function without the loop and make the whole calculation cleaner and more vectorized?
function result= poissonGoodnessOfFit(bins, observed)
assert(iscell(bins), "bins should be a cell array");
assert(all(cellfun("ismatrix", bins)) == 1, "bin entries either scalars or matrices");
assert(ismatrix(observed) && rows(observed) == 1, "observed data should be a 1xn matrix");
lambda_head = mean(observed); #poisson lambda parameter estimate
k = length(bins); #number of bin groups
n = length(observed); #number of observations
poisson_probability = []; #variable for poisson probability for each bin
observations = []; #variable for observation counts for each bin
for i=1:k
if isscalar(bins{1,i}) #this bin contains a single value
poisson_probability(1,i) = poisspdf(bins{1, i}, lambda_head);
observations(1, i) = histc(observed, bins{1, i});
else #this bin contains a range of values
inner_bins = bins{1, i}; #retrieve the range
inner_bins_k = length(inner_bins); #number of values inside
inner_poisson_probability = []; #variable to store individual probability of each value inside this bin
inner_observations = []; #variable to store observation counts of each value inside this bin
for j=1:inner_bins_k
inner_poisson_probability(1,j) = poisspdf(inner_bins(1, j), lambda_head);
inner_observations(1, j) = histc(observed, inner_bins(1, j));
endfor
poisson_probability(1, i) = sum(inner_poisson_probability, 2); #assign over the sum of all inner probabilities
observations(1, i) = sum(inner_observations, 2); #assign over the sum of all inner observation counts
endif
endfor
expected = n .* poisson_probability; #expected observations if indeed poisson using lambda_head
chisq = sum((observations - expected).^2 ./ expected, 2); #Pearson Chi-Square statistics
pvalue = 1 - chi2cdf(chisq, k-1-1);
result = struct("actual", observations, "expected", expected, "chi2", chisq, "pvalue", pvalue);
return;
endfunction
There's a couple of things worth noting in the code.
First, the 'scalar' case in your if block is actually identical to your 'range' case, since a scalar is simply a range of 1 element. So no special treatment is needed for it.
Second, you don't need to create such explicit subranges, your bin groups seem to be amenable to being used as indices into a larger result (as long as you add 1 to convert from 0-indexed to 1-indexed indices).
Therefore my approach would be to calculate the expected and observed numbers over the entire domain of interest (as inferred from your bin groups), and then use the bin groups themselves as 1-indices to obtain the desired subgroups, summing accordingly.
Here's an example code, written in the octave/matlab compatible subset of both languges:
function Result = poissonGoodnessOfFit( BinGroups, Observations )
% POISSONGOODNESSOFFIT( BinGroups, Observations) calculates the [... etc, etc.]
pkg load statistics; % only needed in octave; for matlab buy statistics toolbox.
assert( iscell( BinGroups ), 'Bins should be a cell array' );
assert( all( cellfun( #ismatrix, BinGroups ) ) == 1, 'Bin entries either scalars or matrices' );
assert( ismatrix( Observations ) && rows( Observations ) == 1, 'Observed data should be a 1xn matrix' );
% Define helpful variables
RangeMin = min( cellfun( #min, BinGroups ) );
RangeMax = max( cellfun( #max, BinGroups ) );
Domain = RangeMin : RangeMax;
LambdaEstimate = mean( Observations );
NBinGroups = length( BinGroups );
NObservations = length( Observations );
% Get expected and observed numbers per 'bin' (i.e. discrete value) over the *entire* domain.
Expected_Domain = NObservations * poisspdf( Domain, LambdaEstimate );
Observed_Domain = histc( Observations, Domain );
% Apply BinGroup values as indices
Expected_byBinGroup = cellfun( #(c) sum( Expected_Domain(c+1) ), BinGroups );
Observed_byBinGroup = cellfun( #(c) sum( Observed_Domain(c+1) ), BinGroups );
% Perform a Chi-Square test on the Bin-wise Expected and Observed outputs
O = Observed_byBinGroup; E = Expected_byBinGroup ; df = NBinGroups - 1 - 1;
ChiSquareTestStatistic = sum( (O - E) .^ 2 ./ E );
PValue = 1 - chi2cdf( ChiSquareTestStatistic, df );
Result = struct( 'actual', O, 'expected', E, 'chi2', ChiSquareTestStatistic, 'pvalue', PValue );
end
Running with your example gives:
X = [8 0 0 1 3 4 0 2 12 5 1 8 0 2 0 1 9 3 4 5 3 3 4 7 4 0 1 2 1 2]; % 30 observations
bins = {0, 1, [2:3], [4:5], [6:20]}; % each bin can be single value or multiple values
Result = poissonGoodnessOfFit( bins, X )
% Result =
% scalar structure containing the fields:
% actual = 6 5 8 6 5
% expected = 1.2643 4.0037 13.0304 8.6522 3.0493
% chi2 = 21.989
% pvalue = 0.000065574
A general comment about the code; it is always preferable to write self-explainable code, rather than code that does not make sense by itself in the absence of a comment. Comments generally should only be used to explain the 'why', rather than the 'how'.
I have tried to generalise my tic-tac-toe game for an NxN grid. I have everything working but am finding it hard to get the code needed to detect a win.
This is my function at the moment where I loop over the rows and columns of the board. I can't figure out why it's not working currently. Thanks
def check_win(array_board):
global winner
for row in range(N):
for i in range(N-1):
if array_board[row][i] != array_board[row][i+1] or array_board[row][i] == 0:
break
if i == N-1:
winner = array_board[row][0]
pygame.draw.line(board, (0, 0, 0), (75, (row * round(height / N) + 150)), (825, (row * round(height / N) + 150)), 3)
for col in range(N):
for j in range(N-1):
if array_board[j][col] == 0 or array_board[col][j] != array_board[col][i+1]:
break
if j == N - 1:
winner = array_board[0][col]
pygame.draw.line(board, (0, 0, 0), (col * round(width / N) + 150, 75), (col * round(width / N) + 150, 825), 3)
You don't specify in your question, so my noughts-and-crosses grid is a 2D array of characters, with some default "empty" string (a single space).
def getEmptyBoard( size, default=' ' ):
""" Create a 2D array <size> by <size> of empty string """
grid = []
for j in range( size ):
row = []
for i in range( size ): # makes a full empty row
row.append( default )
grid.append( row )
return ( size, grid )
So given a 2D grid of strings, how does one check for a noughts-and-crosses Win? This would be when the count of the same character in a particular row or column is equal to the size of the grid.
Thus if you have a 5x5 grid, any row with 5 of the same item (say 'x') is a winner. Similarly for a column... 5 lots of 'o' is a win.
So given a 2D array, how do you check for these conditions. One way to do this is to tally the number of occurrences of separate symbols in each cell. If that tally reaches the 5 (grid size), then whatever that symbol is, it's a winner.
def checkForWin( board, default=' ' ):
winner = None
size = board[0]
grid = board[1]
### Tally the row and column
for j in range( size ):
col_results = {}
### Count the symbols in this column
for i in range( size ):
value = grid[i][j]
if ( value in col_results.keys() ):
col_results[ value ] += 1
else:
col_results[ value ] = 1
### Check the tally for a winning count
for k in col_results.keys():
if ( k != default and col_results[k] >= size ):
winner = k # Found a win
print("Winner: column %d" % ( j ) )
break
if ( winner != None ):
break
# TODO: also implement for rows
# TODO: also implement for diagonals
return winner # returns None, or 'o', 'x' (or whatever used for symbols)
The above function uses two loops and a python dictionary to keep a list of what's been found. It's possible to check both the row and columns in the same loops, so it's not really row-by-row or column-by-column, just two loops of size.
Anyway, so during the loop when we first encounter an x, it will be added to the dictionary, with a value of 1. The next time we find an x, the dictionary is used to tally that occurrence, dict['x'] → 2, and so forth for the entire column.
At the end of the loop, we iterate through the dictionary keys (which might be , o, and x) checking the counts. When the count is the same size as a row or column, it's a winning line.
Obviously if there's no win found, we zero the tally and move to the next column/row with the outer-loop.
import csv
with open('Class1scores.csv') as inf:
for line in inf:
parts = line.split()
if len(parts) > 1:
print (parts[4])
f = open('Class1scores.csv')
csv_f = csv.reader(f)
newlist = []
for row in csv_f:
row[1] = int(row[1])
row[2] = int(row[2])
row[3] = int(row[3])
maximum = max(row[1:3])
row.append(maximum)
average = round(sum(row[1:3])/3)
row.append(average)
newlist.append(row[0:4])
averageScore = [[x[3], x[0]] for x in newlist]
print('\nStudents Average Scores From Highest to Lowest\n')
Here the code is meant to read the CSV file and in the first three rows (row 0 being the users name) it should add all the three scores and divide by three but it doesn't calculate a proper average, it just takes the score from the last column.
Basically you want statistics of each row. In general you should do something like this:
import csv
with open('data.csv', 'r') as f:
rows = csv.reader(f)
for row in rows:
name = row[0]
scores = row[1:]
# calculate statistics of scores
attributes = {
'NAME': name,
'MAX' : max(scores),
'MIN' : min(scores),
'AVE' : 1.0 * sum(scores) / len(scores)
}
output_mesg ="name: {NAME:s} \t high: {MAX:d} \t low: {MIN:d} \t ave: {AVE:f}"
print(output_mesg.format(**attributes))
Try not to consider if doing specific things is inefficient locally. A good Pythonic script should be as readable as possible to every one.
In your code, I spot two mistakes:
Appending to row won't change anything, since row is a local variable in for loop and will get garbage collected.
row[1:3] only gives the second and the third element. row[1:4] gives what you want, as well as row[1:]. Indexing in Python normally is end-exclusive.
And some questions for you to think about:
If I can open the file in Excel and it's not that big, why not just do it in Excel? Can I make use of all the tools I have to get work done as soon as possible with least effort? Can I get done with this task in 30 seconds?
Here is one way to do it. See both parts. First, we create a dictionary with names as the key and a list of results as values.
import csv
fileLineList = []
averageScoreDict = {}
with open('Class1scores.csv', newline='') as csvfile:
reader = csv.reader(csvfile)
for row in reader:
fileLineList.append(row)
for row in fileLineList:
highest = 0
lowest = 0
total = 0
average = 0
for column in row:
if column.isdigit():
column = int(column)
if column > highest:
highest = column
if column < lowest or lowest == 0:
lowest = column
total += column
average = total / 3
averageScoreDict[row[0]] = [highest, lowest, round(average)]
print(averageScoreDict)
Output:
{'Milky': [7, 4, 5], 'Billy': [6, 5, 6], 'Adam': [5, 2, 4], 'John': [10, 7, 9]}
Now that we have our dictionary, we can create your desired final output by sorting the list. See this updated code:
import csv
from operator import itemgetter
fileLineList = []
averageScoreDict = {} # Creating an empty dictionary here.
with open('Class1scores.csv', newline='') as csvfile:
reader = csv.reader(csvfile)
for row in reader:
fileLineList.append(row)
for row in fileLineList:
highest = 0
lowest = 0
total = 0
average = 0
for column in row:
if column.isdigit():
column = int(column)
if column > highest:
highest = column
if column < lowest or lowest == 0:
lowest = column
total += column
average = total / 3
# Here is where we put the emtpy dictinary created earlier to good use.
# We assign the key, in this case the contents of the first column of
# the CSV, to the list of values.
# For the first line of the file, the Key would be 'John'.
# We are assigning a list to John which is 3 integers:
# highest, lowest and average (which is a float we round)
averageScoreDict[row[0]] = [highest, lowest, round(average)]
averageScoreList = []
# Here we "unpack" the dictionary we have created and create a list of Keys.
# which are the names and single value we want, in this case the average.
for key, value in averageScoreDict.items():
averageScoreList.append([key, value[2]])
# Sorting the list using the value instead of the name.
averageScoreList.sort(key=itemgetter(1), reverse=True)
print('\nStudents Average Scores From Highest to Lowest\n')
print(averageScoreList)
Output:
Students Average Scores From Highest to Lowest
[['John', 9], ['Billy', 6], ['Milky', 5], ['Adam', 4]]
sql import file
--sources (id, name)---
1, "source one"
2, "source two"
3, "source three"
--first_related_items (value, source_id) ---
0, 1
--second_related_items (value, source_id) ---
0, 2
2, 3
--third_related_items (value, source_id) ---
1, 3
I have source rows and related items in multiple other tables. How do I get a computed column based off the other tables? The computed column would be for each row in source table. Like
SELECT
sources.name as name,
COMPUTED_VALUE(
first_related_items,
second_related_items,
third_related_items) as result
FROM sources
The computed column would be derived something like this
if(value for source row in first table == 0){
return 0
}
else if(value for source row in second table == 0){
if(value for source row in third table == 1){
return 1
} else {
return 0
}
} else if(value for source row in third table != NULL){
return value for source row in third table
} else {
return -1
}
You can join all four tables and compute this column as a case expression:
SELECT s.id,
CASE WHEN fri.value = 0 THEN 0
WHEN sri.value = 0 THEN CASE WHEN tri = 1 THEN 1 ELSE 0 END
WHEN tri IS NOT NULL THEN tri
ELSE -1
END AS computed_value
FROM sources s
LEFT JOIN first_related_items fri ON fri.source_id = s.id
LEFT JOIN second_related_items sri ON sri.source_id = s.id
LEFT JOIN third_related_items tri ON tri.source_id = s.id