Related
I'm currently working on a bioinformatic and modelling project where I need to do some pattern matching. Let's say I have a DNA fragment as follow 'atggcgtatagagc' and I split that fragment in micro-sequences of 8 nucleotides so that I have :
'atggcgta' 'tggcgtat' 'ggcgtata' 'gcgtatag' 'cgtataga' 'gtatagag' 'tatagagc'
And for each of these fragment I want to search in a whole genome and per chromosome the number of time they appear and the positions (starting positions) of the matches.
Here is how my code looks like :
you can download the genome fasta file here :
drive to the fasta file
import re
from Bio.SeqIO.FastaIO import FastaIterator
from Bio.Seq import Seq
def reverse_complement(sequence: str) -> str:
my_sequence = Seq(sequence)
return str(my_sequence.reverse_complement())
# you will need to unzip the file ant change the path below according to your working directory
path = '../data/Genome_S288c.fa'
genome = open(path, "r")
chr_sequences = {}
for record in FastaIterator(genome):
chr_id = record.id
seq = str(record.seq).lower()
rc_seq = reverse_complement(seq)
chr_sequences[chr_id] = {'5to3': seq, '3to5': rc_seq}
genome.close()
sequences = 'ATGACTAACGAAAAGGTCTGGATAGAGAAGTTGGATAATCCAACTCTTTCAGTGTTACCACATGACTTTTTACGCCCACAATCTTTAT'.lower()
micro_size = 8
micro_sequences = []
start = micro_size - 1
for i in range(start, len(sequences), 1):
current_micro_seq = sequences[i - start:i + 1]
micro_sequences.append(current_micro_seq)
genome_count = 0
chr_count = {}
chr_locations = {}
micro_fragment_stats = {}
for ii_micro, micro_seq in enumerate(micro_sequences):
for chr_idx in list(chr_sequences.keys()):
chr_counter = 0
seq = chr_sequences[chr_idx]['5to3']
pos = [m.start() for m in re.finditer(pattern=r'(?=(' + micro_seq + '))', string=seq)]
rc_seq = chr_sequences[chr_idx]['3to5']
rc_pos = [m.start() for m in re.finditer(pattern=r'(?=(' + micro_seq + '))', string=rc_seq)]
chr_locations[chr] = {'5to3': pos, '3to5': rc_pos}
chr_counter += len(pos) + len(rc_pos)
chr_count[chr_idx] = chr_counter
genome_count += chr_counter
micro_fragment_stats[ii_micro] = {'occurrences genome': genome_count,
'occurrences chromosomes': chr_count,
'locations chromosomes': chr_locations}
Actually my fragment is something like 2000bp long, so I took about 1 hour to compute all the micro-sequences. \
By the way, I use the r'(?=('+self.sequence+'))' to avoid the case of pattern that overlaps itself in the sequence, for instance :
pattern = 'aaggaaaaa'
string = 'aaggaaaaaggaaaaa'
expected output : (0, 7)
I am looking for a more efficient regex method that I can use for my case (in python if possible).
Thanks in advance
I would not recommend using regex for repetitive simple pattern matching. Outright comparison is expected to perform better. I did some basic testing and came up with the demo below.
import time
import re
import random
def compare(r1, r2, microseq_len, test_condition=1):
# condition 1: make microseqs/indexes from longer sequence and search against shorter
# condition 2: use regex to find position of microseq in reference sequence
# condition 3: use regex to find position of microseq in reference sequence after verifying if microseq in reference strain
start_time = time.time()
if test_condition == 1:
r1, r2 = r2, r1
# assemble dictionary containing microsequences and index positions
microseq_di = {}
for i in range(len(r1)-microseq_len):
microseq = r1[i:i+microseq_len]
if microseq not in microseq_di:
microseq_di[microseq] = []
microseq_di[microseq].append([i, i+microseq_len])
# mark for deletion
for microseq in microseq_di:
# condition 2
if test_condition == 2:
microseq_di[microseq] = [m.start() for m in re.finditer(pattern=r'(?=('+microseq+'))', string=r2)]
elif microseq not in r2:
microseq_di[microseq] = []
# condition 3
elif test_condition == 3:
microseq_di[microseq] = [m.start() for m in re.finditer(pattern=r'(?=('+microseq+'))', string=r2)]
print(time.time() - start_time) # run time
# delete and return
return({x:y for x, y in microseq_di.items() if y != []})
Input and Output:
r_short = "".join([random.choices(["A", "T", "G", "C"])[0] for x in range(500)])
r_long = "".join([random.choices(["A", "T", "G", "C"])[0] for x in range(100000)])
len(compare(r_short, r_long, 8, test_condition=1).keys())
0.19868111610412598
Out[1]: 400
len(compare(r_short, r_long, 8, test_condition=2).keys())
0.8831210136413574
Out[2]: 399
len(compare(r_short, r_long, 8, test_condition=3).keys())
0.7925639152526855
Out[3]: 399
Test condition 1 (microseqs from longer sequence) performed a lot better than the other two conditions using regex. Relative performance should improve with longer strings.
r_short = "".join([random.choices(["A", "T", "G", "C"])[0] for x in range(2000)])
r_long = "".join([random.choices(["A", "T", "G", "C"])[0] for x in range(1000000)])
len(compare3(r_short, r_long, 8, test_condition=1).keys())
2.2517480850219727
Out[4]: 1970
len(compare3(r_short, r_long, 8, test_condition=2).keys())
35.65084385871887
Out[5]: 1969
len(compare3(r_short, r_long, 8, test_condition=3).keys())
34.994577169418335
Out[6]: 1969
Note that condition 1 is not fully accommodating to your use-case since it doesn't exclude overlapping microseqs.
I've been playing with this question for a while and I end up with some ideas.
The algorithm is mainly divided in two parts: k-mer generation and k-mer searching in the reference.
For the k-mer generation part, I can see that your algorithm is quick, but it generates duplicates (that you have to filter afterwards when generating the dictionary). My approach has been to generate a deduplicated list directly. In my sample code I also modified your method to perform the deduplication at the same time, so you can avoid doing it later and, more important, allows for a fair time comparison with my approach.
You will see that using a set to keep the kmers offers us free deduplication, and is faster than using a list, as it has not to be traversed.
For the search of the kmer in the reference, given that you were doing exact searches, using a regex is overkill. It's far more cheaper to do a standard search. In this code, I used the methods provided by the Seq class: find and index. The idea is to find the first occurrence starting from the beginning, and the repeat the search starting with the next position after the last index found (if you want to avoid overlaps, then start after the last position found plus the k-mer size).
The code generated follows:
import re
from pathlib import Path
from timeit import timeit
from Bio.Seq import Seq
from Bio.SeqIO.FastaIO import FastaIterator
def reverse_complement(sequence: Seq) -> Seq:
return sequence.reverse_complement()
def generate_kmers(sequence: Seq, kmer_size: int) -> set[Seq]:
return {
Seq(sequence[i : i + kmer_size]) for i in range(len(sequence) - kmer_size + 1)
}
def generate_kmers_original(sequence: Seq, kmer_size: int) -> list[Seq]:
kmers: list[Seq] = []
start = kmer_size - 1
for i in range(start, len(sequence), 1):
current_micro_seq = Seq(sequence[i - start : i + 1])
# We had to add this check to avoid the duplication of k-mers
if current_micro_seq not in kmers:
kmers.append(current_micro_seq)
return kmers
def load_fasta(fasta_file: str) -> dict[str, dict[str, Seq]]:
fasta_dict: dict[str, dict[str, Seq]] = {}
with Path(fasta_file).open("r", encoding="UTF-8") as genome:
for record in FastaIterator(genome):
seq = record.seq.lower()
fasta_dict[record.id] = {"5to3": seq, "3to5": reverse_complement(seq)}
return fasta_dict
if __name__ == "__main__":
# Load the big fasta file
chr_sequences = load_fasta(
".../Saccharomyces_cerevisiae/S288c_R64/fasta/scerevisiae.S288c_R64.fasta"
)
# Generate the micro-sequences
micro_size = 8
sequences = Seq(
"ATGACTAACGAAAAGGTCTGGATAGAGAAGTTGGATAATCCAACTCTTTCAGTGTTACCACATGACTTTTTACGCCCACAATCTTTAT"
).lower()
micro_sequences = generate_kmers(sequences, micro_size)
# k-mer generation benchmark
test_size = 1000
kmer_generation_time = timeit(
"generate_kmers(sequences, micro_size)", number=test_size, globals=globals()
)
kmer_generation_original_time = timeit(
"generate_kmers_original(sequences, micro_size)",
number=test_size,
globals=globals(),
)
print(f"New k-mer generation time : {kmer_generation_time}")
print(f"Original k-mer generation time: {kmer_generation_original_time}")
print(f"There are {len(micro_sequences)} k-mers")
# Search for the kmers in the reference
def find_kmers_original(sequence: Seq, kmer: Seq) -> list[int]:
positions = [
m.start()
for m in re.finditer(
pattern=r"(?=(" + str(kmer) + "))", string=str(sequence)
)
]
return positions
def find_kmers_find(sequence: Seq, kmer: Seq) -> list[int]:
current = 0
positions: list[int] = []
while current < len(sequence):
index = sequence.find(kmer, current)
if index == -1:
break
positions.append(index)
current = index + 1
return positions
def find_kmers_index(sequence: Seq, kmer: Seq) -> list[int]:
positions: list[int] = []
current = 0
try:
while True:
index = sequence.index(kmer, current)
positions.append(index)
current = index + 1
except ValueError:
# Exception thrown when the kmer is not found
# This is our exit condition
pass
return positions
# k-mer search benchmark
test_size = 1000
haystack = next(iter(chr_sequences.values()))["5to3"]
needle = next(iter(micro_sequences))
search_original_time = timeit(
"find_kmers_original(haystack, needle)",
number=test_size,
globals=globals(),
)
search_find_time = timeit(
"find_kmers_find(haystack, needle)",
number=test_size,
globals=globals(),
)
search_index_time = timeit(
"find_kmers_index(haystack, needle)",
number=test_size,
globals=globals(),
)
print(f"Search with original time: {search_original_time}")
print(f"Search with find time : {search_find_time}")
print(f"Search with index time : {search_index_time}")
# Actual calculus
genome_count = 0
chr_count: dict[str, int] = {}
chr_locations: dict[str, dict[str, list[int]]] = {}
micro_fragment_stats: dict[
int, dict[str, int | dict[str, int] | dict[str, dict[str, list[int]]]]
] = {}
for ii_micro, micro_seq in enumerate(micro_sequences):
for chr_counter, (chromosome, contents) in enumerate(chr_sequences.items()):
pos = find_kmers_find(contents["5to3"], micro_seq)
rc_pos = find_kmers_find(contents["3to5"], micro_seq)
chr_locations[chromosome] = {"5to3": pos, "3to5": rc_pos}
chr_counter += len(pos) + len(rc_pos)
chr_count[chromosome] = chr_counter
genome_count += chr_counter
micro_fragment_stats[ii_micro] = {
"occurrences genome": genome_count,
"occurrences chromosomes": chr_count,
"locations chromosomes": chr_locations,
}
The output of this toy example is:
New k-mer generation time : 0.6696164240129292
Original k-mer generation time: 5.967410315992311
There are 81 k-mers
Search with original time: 3.1360475399997085
Search with find time : 0.5738343889825046
Search with index time : 0.5662875371053815
You can see that the k-mer generation is 9x faster and the search without the regex is around 5.5x faster.
In general, you will be better taking advantage of comprehensions and built-in data types (like the sets used here). And using the more simple approach also helps with performance. Regexes are powerful, but they need their time; if they are not required, better to avoid them. Specially in loops, where every small performance change is amplified.
Besides all of this benchmarking, you can also try to add the approach introduced by #Ghothi where the long and short sequences are exchanged. Maybe it could lead to some further improvement.
As a side note, Seq.find and Seq.index seems to offer the same performance, but I find it cleaner and more elegant the Seq.index version: you don't need a weird value to test against and the code intent is clearer. Also, the performance is slightly better, as it is avoiding a comparison in the loop, but this is a very minor improvement.
I have a (very) simple process to carry out many times, along the lines of:
import time
class Shop:
def __init__(self):
time.sleep(1.)
self.sale_price = 0
def update(self, weight, price):
self.sale_price = weight * price
my_list = [Shop() for _ in range(20)] #20 shops
weight, price = 2, 3
for o in my_list:
o.update(weight, price)
(The function update changes the internal state of o, without any return value)
my question: What's a simple way to make this loop/process run in parallel?
I've seen many examples here for parallel mathematical operations (e.g. matrix operations), but I can't get them working for my simple example. I'm using Python 3.6.5
Found a simple solution:
import time
from joblib import Parallel, delayed
import multiprocessing
class Shop:
def __init__(self):
time.sleep(1.)
self.sale_price = 0
def update(self, weight, price):
self.sale_price = weight * price
my_list = [Shop() for _ in range(20)] #20 shops
weight, price = 2, 3
### Non-parallel
for o in my_list:
o.update(weight, price)
### Parallel
#Helper function (doesn't take weight/price parameters - assumes these are already defined)
def update2(o):
o.update(weight, price)
return o
num_cores = multiprocessing.cpu_count()
my_list = Parallel(n_jobs=num_cores)(delayed(update2)(o) for o in my_list)
I realise this reassigns my_list at the end, but this is fine for my purpose.
I explain, I am trying to develop a program to optimize a system based on the parameters it receives. My program will have to vary these parameters to try to find the best possible combination.
here is a code to simplify my problem:
parameters=[["toto1","toto2","toto3"],["tutu1","tutu2","tutu3"],["titi1","titi2","titi3"],["tata1","tata2","tata3"]]
def MySysteme(param1,param2,param3,param4):
result=0
for i in range(0,len(param1)):
result+=ord(param1[i])
for i in range(0,len(param2)):
result+=ord(param1[i])
for i in range(0,len(param3)):
result+=ord(param1[i])
for i in range(0,len(param4)):
result+=ord(param1[i])
return result
print(MySysteme(parameters[0][0],parameters[1][2],parameters[2][2],parameters[3][0]))
print(MySysteme(parameters[1][0],parameters[2][2],parameters[3][2],parameters[0][0]))
print(MySysteme(parameters[3][1],parameters[1][2],parameters[2][2],parameters[0][0]))
#how to find the highest value?
I try to (try) find the highest number, without testing all the parameters naively. hence the use of a genetic algorithm. 1 parameter is a list contained in the list parameters, the contents of the list is a varariante of my parameter
knowing that in my function / my system, one should not have 2 times the same parameter, for example this should not happen: print (MySystem (parameters [1] [0], parameters [1] [0])) or this print (MySystem (parameters [2] [1], parameters [2] [0]))
on the other hand the number of parameters is included in 1 and 4 (there can be 1,2,3 or 4 parameters)
To solve my problem here is the data that I consider: Individual: it is a variant of parameter which carries a name ("toto1", "tata3", "toto2 = 12" ... etc.) Population: set of the variants of the parameters fitness : it is the result of the function according to the parameters a circuit: a set of parameters
but unlike the commercial traveler, I have no starting data => that is to say that I do not have GPS coordinates. and it is at this level that I am stuck for the resolution of my problem.
can anyone help me?
edit:
I have been looking some examples of how I could find the points at which a function achieves its maxium using a genetic algorithm approach in Python. I looked at this tutorial
https://lethain.com/genetic-algorithms-cool-name-damn-simple/
my objective is to found the smaller number to "mySysteme" function
i set a new code :
je re-explique mon probleme plus simplement. J’ai mets un code plus complet, plus clair avec un algo génétique.
from random import randint, random
from operator import add
from functools import reduce
parameters=[["toto123","toto27","toto3000"],["tu","tut","tutu378694245"],["t","choicezaert","titi3=78965"],["blabla","2","conjoncture_is_enable"]]
def individual(length, min, max):
return [ randint(min,max) for x in range(length) ]
def population(count, length, min, max):
return [ individual(length, min, max) for x in range(count) ]
def fitness(individual, target):
sum = reduce(add, individual, 0)
return abs(target-sum)
def grade(pop, target):
individu_number_parameters=randint(1, len(parameters)-1)
for j in range(0,individu_number_parameters):
position=randint(1, len(parameters)-1)
parameter=parameters[position]
if isinstance(parameter, list):
parameter=parameters[position][randint(1, len(parameters[position])-1)]
result=0
for i in range(0,len(parameter)):
result+=ord(parameter[i])
return result
def evolve(pop, target, retain=0.2, random_select=0.05, mutate=0.01):
graded = [ (fitness(x, target), x) for x in pop]
graded = [ x[1] for x in sorted(graded)]
retain_length = int(len(graded)*retain)
parents = graded[:retain_length]
for individual in graded[retain_length:]:
if random_select > random():
parents.append(individual)
for individual in parents:
if mutate > random():
pos_to_mutate = randint(0, len(individual)-1)
individual[pos_to_mutate] = randint(
min(individual), max(individual))
parents_length = len(parents)
desired_length = len(pop) - parents_length
children = []
while len(children) < desired_length:
male = randint(0, parents_length-1)
female = randint(0, parents_length-1)
if male != female:
male = parents[male]
female = parents[female]
half = int(len(male) / 2)
child = male[:half] + female[half:]
children.append(child)
parents.extend(children)
return parents
target = 0
p_count = 100
i_length = 6
i_min = 0
i_max = 100
p = population(p_count, i_length, i_min, i_max)
fitness_history = [grade(p, target),]
for i in range(1000):
p = evolve(p, target)
fitness_history.append(grade(p, target))
for datum in fitness_history:
print(datum)
print(len(fitness_history))
I updated with new code. My ask : i want that my program found smaller number
This is the problem I am trying to solve:
B: The Foxen's Treasure
There are N (1 ≤ N ≤ 4) Foxen guarding a certain valuable treasure,
which you'd love to get your hands on. The problem is, the Foxen
certainly aren't about to allow that - at least, not while they're
awake.
Fortunately, through careful observation, you've seen that each Fox
has a regular sleep cycle. In particular, the ith Fox stays awake for
Ai (1 ≤ Ai ≤ 23) hours, then sleeps for Si (1 ≤ Si ≤ 23) hours,
repeating this pattern indefinitely (2 ≤ Ai + Si ≤ 24). At the start
of your treasure-nabbing attempt, the ith Fox is
exactly Oi (0 ≤ Oi < Ai + Si) hours into its cycle.
There are T (1 ≤ T ≤ 20) scenarios as described above. For each one,
you'd like to determine how soon all of the Foxen will be
simultaneously asleep, allowing you to grab their treasure, or if this
will simply never happen.
Input
Line 1: 1 integer, T
For each scenario:
Line 1: 1 integer, N
Next N lines: 3 integers, Ai, Si, and Oi, for i = 1..N
Output
For each scenario:
Line 1: 1 integer, the minimum number of hours after the start to
wait until all of the Foxen are asleep during the same hour. If this
will never happen, output the string "Foxen are too powerful" (without
quotes) instead.
Sample Input
2
2
2 1 2
2 2 1
3
1 1 0
1 1 0
1 1 1
Sample Output
6
Foxen are too powerful
My Solution works as expected when I input the given sample case and get expected output. But when I submit the code to online judge it gives clipped error. Now there is no detail of the error which makes it difficult to find what the problem is.
Here is the solution which I have worked so far:
# ai is awake hours
# si is sleep hours.
# ai + si <= 24.
# False == sleep. True == awake.
datasets = int(raw_input());
foxen = [];
number_of_foxen = 0;
foxes = [];
class fox:
def __init__(self, a, s, i):
self.awake = a;
self.sleep = s;
self.current = i;
awake = 0;
sleep = 0;
current = 0;
def next(self):
if ( self.sleep + self.awake-1 > self.current ) :
self.current = self.current+1;
else:
self.current = 0;
return self.current;
def check(self):
if(self.current>=self.awake):
return False;
return True;
def printdata(self):
print "awake="+str(self.awake)+" sleep="+str(self.sleep)+" current="+str(self.current);
#return "awake="+str(self.awake)+" sleep="+str(self.sleep)+" current="+str(self.current);
for i in range(0, datasets):
number_of_foxen = int(raw_input());
for j in range(0, number_of_foxen):
foxen.append(raw_input());
x = foxen[j].split();
a = fox(int(x[0]), int(x[1]), int(x[2]));
foxes.append(a);
solution = False;
for j in range(0, 48):
#print "hour number = " + str(j);
#for k in range(0, len(foxes)):
#print "fox number="+ str(k)+" "+ foxes[k].printdata()+str(foxes[k].check());
count = 0 ;
for k in range(0, len(foxes)):
if(foxes[k].check()==False):
count+=1;
#print "count = "+str(count);
#print len(foxes);
if( (int(count) == int(len(foxes))) and (solution == False) ):
#print "this runs now *************";
solution = True;
number = j;
for k in range(0, len(foxes)):
foxes[k].next();
if(solution==True):
print number;
else:
print "Foxen are too powerful";
#print "Foxen are too powerful";
foxen = [];
number_of_foxen = 0;
foxes = [];
The biggest problem with your code is that it is unreadable. Indeed, it looks like it was written with little concept of Python's strengths. Here is my suggestion:
#!/usr/bin/env python3
"""
The Foxen's Treasure puzzle from http://wcipeg.com/problem/acmtryouts1b
"""
from sys import stdin
from itertools import cycle
from euclid import lcm
debug = True # set to False before submission to mechanical judge
class Fox:
"""A Fox cointains its defining integers and other derived
bindings such as its cycle and schedule."""
def __init__(self, trio):
(self.awake_count, self.sleep_count, self.skip_count) = trio
self.cycle = 'a' * self.awake_count + 's' * self.sleep_count
self.schedule = cycle(self.cycle)
if debug: print('<Fox: {}> cycle {}'.format(trio, self.cycle))
# handle skips by discarding the first elements
for _ in range(self.skip_count):
next(self.schedule)
def find_all_sleeping(foxes):
"""Return an hour number if all foxes are sleeping at that hour."""
# only examine the LCM of all fox periods. If not there it will never be.
lcm_period = 1
for fox in foxes:
lcm_period = lcm(lcm_period, len(fox.cycle))
for hour in range(lcm_period):
states = [next(fox.schedule) for fox in foxes]
if debug: print('{:2d} {}'.format(hour, ' '.join(states)))
if 'a' not in states:
return hour
return None
def read_trials(fp):
"""Reads the entire input at once. Returns a list of trials.
Each trial is a list of Fox."""
trials = list()
trial_count = int(fp.readline())
for trial in range(trial_count):
if debug: print('--Read trial {}'.format(trial))
foxes = list()
fox_count = int(fp.readline())
for _ in range(fox_count):
fox = Fox([int(x) for x in fp.readline().split()])
foxes.append(fox)
trials.append(foxes)
return trials
for trial, foxes in enumerate(read_trials(stdin)):
if debug: print('--Run trial {}'.format(trial))
hour = find_all_sleeping(foxes)
if hour is None:
print('Foxen are too powerful')
else:
print(hour)
I suspect that the first concern is that it looks much longer than the OP; that is true, but if you take out the debugging code which shows how things are happening, and the docstrings that explain why it is doing things, it's actually a few line shorter than the OP.
The main loop of the OP is too long to understand without significant study, and a whole bunch of bad variable names makes that even harder. In contrast, there are places here where a value is given a name only to make the code more explicit about what an input line means. You'll find a number of
for _ in range(trial)
to show that the loop value is not used. This is a frequent idiom when dealing with fixed format input.
The Fox representation keeps the inner workings in the problem space. As noted in the exercise page, it makes more sense to look at things as a concurrence between sequences:
--Read trial 0
<Fox: [2, 1, 2]> cycle aas
<Fox: [2, 2, 1]> cycle aass
the offsets skip_count are not shown here, but they are clear in the trial run.
The input from the datafile is all kept inside read_trials() instead of scattered through the code. This confines the mess to one place rather than distributing it through the code. We know from the puzzle instructions that the datafile will not be large enough to care about. read_trials(fp) also takes a file-like object which allows it to read from an actual file, a StringIO buffer, or the standard input.
Once the Fox schedule generator is initialized, itertools.cycle will give an unending supply of the next letter in the sequence; it does the wrap-around for you.
It is worth noting that the primary data structure trials is a plain old list because it doesn't need anything more than that.
I've gotten a little weary of bad code being answered with worse code. Sure, this could be considered way more than the needs of an electronic judge where only the output matters. Contrariwise, I'm still puzzled by bits like (solution == False), a main loop that is 42 lines long and split between the top and bottom of the file, variables like i and j which convey no intent, the memory burden of False == awake (or did I mix them up?), dead code, no-op code, `range(0, n) and a whole bunch of magic numbers throughout.
Sure, you can code like the code doesn't matter, but if you are teaching yourself to code it is good to practice good practice. Yeah, you might never look at this piece of code again, but if you ain't gonna learn it now, then when?
In case you feel it a cheat to have imported lcm() there's no reason to write it a second time, so I referenced a homebrew package of which the relevant lines are:
def gcd(a, b):
"""Return the Greatest Common Divisor of a and b."""
while b:
a, b = b, a % b
return a
def lcm(a, b):
"""Return the Least Common Multiple of a and b."""
return abs(a * b) // gcd(a, b)
Jorge was correct in his comment, there doesn't appear to be any problem with your algorithm other than the arbitrary 48 hour cuttoff.
However:
1) your print statements do not use the correct syntax for Python 3+. For example, your final print statement print "Foxen are too powerful"; must be changed to work in Python 3, try print ('Foxen are too powerful') instead.
2) I'm seeing some odd C/MatLab-like syntax as well, lines being ended by a semicolon, and double brackets surrounding conditions in your if statements. This probably isn't a problem, but depending on how picky the system you are submitting the answer to is, you may want to clean it up a little.
3) Definitely increase the cutoff time for your search. I'd recommend a reasonably large value, on the order of 10,000 hours, just to be sure that it won't be a factor.
I've taken the liberty of making all of the above changes so I'm posting the resultant code now:
# ai is awake hours
# si is sleep hours.
# ai + si <= 24.
# False == sleep. True == awake.
datasets = int(raw_input())
foxen = []
number_of_foxen = 0
foxes = []
class fox:
def __init__(self, a, s, i):
self.awake = a
self.sleep = s
self.current = i
awake = 0
sleep = 0
current = 0
def next(self):
if ( self.sleep + self.awake-1 > self.current ):
self.current = self.current+1
else:
self.current = 0
return self.current
def check(self):
if(self.current>=self.awake):
return False
return True
def printdata(self):
print ("awake="+str(self.awake)+" sleep="+str(self.sleep)+" current="+str(self.current))
#return ("awake="+str(self.awake)+" sleep="+str(self.sleep)+" current="+str(self.current))
for i in range(0, datasets):
number_of_foxen = int(raw_input())
for j in range(0, number_of_foxen):
foxen.append(raw_input())
x = foxen[j].split()
a = fox(int(x[0]), int(x[1]), int(x[2]))
foxes.append(a)
solution = False
for j in range(0, 10000):
#print ("hour number = " + str(j))
#for k in range(0, len(foxes)):
#print ("fox number="+ str(k)+" "+ foxes[k].printdata()+str(foxes[k].check()))
count = 0
for k in range(0, len(foxes)):
if(foxes[k].check()==False):
count+=1
#print ("count = "+str(count))
#print (len(foxes))
if (int(count) == int(len(foxes)) and (solution == False)):
#print ("this runs now *************")
solution = True
number = j
for k in range(0, len(foxes)):
foxes[k].next()
if(solution == True):
print (number)
else:
print ("Foxen are too powerful")
#print ("Foxen are too powerful")
foxen = []
number_of_foxen = 0
foxes = []
Enjoy and Good Luck!
Interesting problem, here's my code:
import sys
# Globals
debugLevel = 0
fileMode = True # True if loading data from a file.
# Constants
AWAKE = 0
ASLEEP = -1
def gcd(a, b):
"""Return greatest common divisor using Euclid's Algorithm."""
while b:
a, b = b, a % b
return a
def lcm(a, b):
"""Return lowest common multiple."""
return a * b // gcd(a, b)
def readData(f):
''' Read in the problem data and store in data structures
'''
numTrials = int(f.readline().strip())
if debugLevel >= 4:
print("Num trials: ", numTrials)
trialData = []
for _ in range(numTrials):
numFoxen = int(f.readline().strip())
allFoxenHoursInfo = []
for _ in range(numFoxen):
aFoxHoursInfo = f.readline().split()
aFoxHoursInfo = list(map(int, aFoxHoursInfo))
allFoxenHoursInfo.append(aFoxHoursInfo)
trialData.append((numFoxen, allFoxenHoursInfo))
if debugLevel >= 8:
print("Trial data\n", trialData)
return numTrials, trialData
def runTrials(trialData):
'''
Go through each lot of foxen, and their sleep/awake schedules and
See if there's a time that all of them will be asleep.
'''
global debugLevel
for trial in trialData:
numFoxen, allFoxenHoursInfo = trial
# Create a table of the status of each fox in each hour
row = [AWAKE] * (numFoxen+1)
row[0] = 0
hoursTable = [row]
# Cycle length for each fox is the number of hours they spend awake then asleep.
cycleLength = [0] * (numFoxen)
# This is the number of hours into the cycle each fox is at the start
startingPosInCycle= [0] * (numFoxen)
# Initialise the first row
for fox in range(numFoxen):
cycleLength[fox] = allFoxenHoursInfo[fox][0] + allFoxenHoursInfo[fox][1] # Time awake plus time asleep
startingPosInCycle[fox] = allFoxenHoursInfo[fox][2] # % cycleLength[fox]
if startingPosInCycle[fox] >= allFoxenHoursInfo[fox][0]:
hoursTable[0][fox+1] = ASLEEP
if debugLevel >= 4:
print("Initial table: ", hoursTable)
# lcm = lowest common multiple and it's implemented above.
# For this problem, we only need to look at the lcm of all the cycle lengths for the foxen.
numIterations = 1
for fox in range(numFoxen):
numIterations = lcm(numIterations, cycleLength[fox])
# Go around a loop adding a new row to the table for each new hour containing the updated
# statuses of each fox.
for hourNum in range(1, numIterations):
allFoxesSleeping = False
# Update our hours table by creating a new row and calculating the status of each fox
newRow = [AWAKE] * (numFoxen+1)
newRow[0] = hourNum
for fox in range(numFoxen):
currentPosInCycle = (startingPosInCycle[fox] + hourNum) % cycleLength[fox]
if currentPosInCycle >= allFoxenHoursInfo[fox][0]:
newRow[fox+1] = ASLEEP
hoursTable.append(newRow)
if debugLevel >= 4:
print("Hours table\n", hoursTable)
# See if all foxen are sleeping, if they are, success
numFoxesSleeping = hoursTable[hourNum].count(ASLEEP)
if numFoxesSleeping == numFoxen:
allFoxesSleeping = True
print(hourNum)
break
if not allFoxesSleeping:
print('Foxen are too powerful')
def main():
'''Reads, runs, and outputs problem specific data.'''
# Initialisation
#strDir = ".\\"
# if fileMode:
# dataSource = open(strDir + "DataFile.txt", 'r')
# else:
dataSource = sys.stdin
# Read in the input data.
numTrials, trialData = readData(dataSource)
# Run each trial, outputting the result of that trial
runTrials(trialData)
sys.stdout.flush()
# Cleanup
# if fileMode:
# dataSource.close()
if __name__ == '__main__':
main()
Unfortunately, it does not pass the judge either. I have no idea why. I get this output:
Test case #1: WA [0.178s, 3628K] (0/1) (Details)
Your Output (clipped)
6
Foxen are too powe
Final score: 0/1
Interesting problem. I have emailed the author, because there is an inconsistency in the problem definition and the sample input data. He says this: Oi (0 ≤ Oi < Ai + Si) but then gives 1 1 1 on the last line of sample input data. And 1 is not strictly less than 1+1.
So who knows what other data the judge might be using...
The bit at the bottom that is commented out re files, lets me work with files and an IPython console rather than a Python console and pasting the data in, which I find slow and annoying.
Also, it's a little strange I reckon to not be able to see the data the judge is using. Surely seeing the data you are working against would enable the problem to be run and debugged offline, then when it's going, a new online submit could be done.
I have some python code that has many classes. I used cProfile to find that the total time to run the program is 68 seconds. I found that the following function in a class called Buyers takes about 60 seconds of those 68 seconds. I have to run the program about 100 times, so any increase in speed will help. Can you suggest ways to increase the speed by modifying the code? If you need more information that will help, please let me know.
def qtyDemanded(self, timePd, priceVector):
'''Returns quantity demanded in period timePd. In addition,
also updates the list of customers and non-customers.
Inputs: timePd and priceVector
Output: count of people for whom priceVector[-1] < utility
'''
## Initialize count of customers to zero
## Set self.customers and self.nonCustomers to empty lists
price = priceVector[-1]
count = 0
self.customers = []
self.nonCustomers = []
for person in self.people:
if person.utility >= price:
person.customer = 1
self.customers.append(person)
else:
person.customer = 0
self.nonCustomers.append(person)
return len(self.customers)
self.people is a list of person objects. Each person has customer and utility as its attributes.
EDIT - responsed added
-------------------------------------
Thanks so much for the suggestions. Here is the
response to some questions and suggestions people have kindly
made. I have not tried them all, but will try others and write back later.
(1) #amber - the function is accessed 80,000 times.
(2) #gnibbler and others - self.people is a list of Person objects in memory. Not connected to a database.
(3) #Hugh Bothwell
cumtime taken by the original function - 60.8 s (accessed 80000 times)
cumtime taken by the new function with local function aliases as suggested - 56.4 s (accessed 80000 times)
(4) #rotoglup and #Martin Thomas
I have not tried your solutions yet. I need to check the rest of the code to see the places where I use self.customers before I can make the change of not appending the customers to self.customers list. But I will try this and write back.
(5) #TryPyPy - thanks for your kind offer to check the code.
Let me first read a little on the suggestions you have made to see if those will be feasible to use.
EDIT 2
Some suggested that since I am flagging the customers and noncustomers in the self.people, I should try without creating separate lists of self.customers and self.noncustomers using append. Instead, I should loop over the self.people to find the number of customers. I tried the following code and timed both functions below f_w_append and f_wo_append. I did find that the latter takes less time, but it is still 96% of the time taken by the former. That is, it is a very small increase in the speed.
#TryPyPy - The following piece of code is complete enough to check the bottleneck function, in case your offer is still there to check it with other compilers.
Thanks again to everyone who replied.
import numpy
class person(object):
def __init__(self, util):
self.utility = util
self.customer = 0
class population(object):
def __init__(self, numpeople):
self.people = []
self.cus = []
self.noncus = []
numpy.random.seed(1)
utils = numpy.random.uniform(0, 300, numpeople)
for u in utils:
per = person(u)
self.people.append(per)
popn = population(300)
def f_w_append():
'''Function with append'''
P = 75
cus = []
noncus = []
for per in popn.people:
if per.utility >= P:
per.customer = 1
cus.append(per)
else:
per.customer = 0
noncus.append(per)
return len(cus)
def f_wo_append():
'''Function without append'''
P = 75
for per in popn.people:
if per.utility >= P:
per.customer = 1
else:
per.customer = 0
numcustomers = 0
for per in popn.people:
if per.customer == 1:
numcustomers += 1
return numcustomers
EDIT 3: It seems numpy is the problem
This is in response to what John Machin said below. Below you see two ways of defining Population class. I ran the program below twice, once with each way of creating Population class. One uses numpy and one does not use numpy. The one without numpy takes similar time as John found in his runs. One with numpy takes much longer. What is not clear to me is that the popn instance is created before time recording begins (at least that is what it appears from the code). Then, why is numpy version taking longer. And, I thought numpy was supposed to be more efficient. Anyhow, the problem seems to be with numpy and not so much with the append, even though it does slow down things a little. Can someone please confirm with the code below? Thanks.
import random # instead of numpy
import numpy
import time
timer_func = time.time # using Mac OS X 10.5.8
class Person(object):
def __init__(self, util):
self.utility = util
self.customer = 0
class Population(object):
def __init__(self, numpeople):
random.seed(1)
self.people = [Person(random.uniform(0, 300)) for i in xrange(numpeople)]
self.cus = []
self.noncus = []
# Numpy based
# class Population(object):
# def __init__(self, numpeople):
# numpy.random.seed(1)
# utils = numpy.random.uniform(0, 300, numpeople)
# self.people = [Person(u) for u in utils]
# self.cus = []
# self.noncus = []
def f_wo_append(popn):
'''Function without append'''
P = 75
for per in popn.people:
if per.utility >= P:
per.customer = 1
else:
per.customer = 0
numcustomers = 0
for per in popn.people:
if per.customer == 1:
numcustomers += 1
return numcustomers
t0 = timer_func()
for i in xrange(20000):
x = f_wo_append(popn)
t1 = timer_func()
print t1-t0
Edit 4: See the answers by John Machin and TryPyPy
Since there have been so many edits and updates here, those who find themselves here for the first time may be a little confused. See the answers by John Machin and TryPyPy. Both of these can help in improving the speed of the code substantially. I am grateful to them and others who alerted me to slowness of append. Since, in this instance I am going to use John Machin's solution and not use numpy for generating utilities, I am accepting his response as an answer. However, I really appreciate the directions pointed out by TryPyPy also.
There are many things you can try after optimizing your Python code for speed. If this program doesn't need C extensions, you can run it under PyPy to benefit from its JIT compiler. You can try making a C extension for possibly huge speedups. Shed Skin will even allow you to convert your Python program to a standalone C++ binary.
I'm willing to time your program under these different optimization scenarios if you can provide enough code for benchmarking,
Edit: First of all, I have to agree with everyone else: are you sure you're measuring the time correctly? The example code runs 100 times in under 0.1 seconds here, so there is a good chance the either the time is wrong or you have a bottleneck (IO?) that isn't present in the code sample.
That said, I made it 300000 people so times were consistent. Here's the adapted code, shared by CPython (2.5), PyPy and Shed Skin:
from time import time
import random
import sys
class person(object):
def __init__(self, util):
self.utility = util
self.customer = 0
class population(object):
def __init__(self, numpeople, util):
self.people = []
self.cus = []
self.noncus = []
for u in util:
per = person(u)
self.people.append(per)
def f_w_append(popn):
'''Function with append'''
P = 75
cus = []
noncus = []
# Help CPython a bit
# cus_append, noncus_append = cus.append, noncus.append
for per in popn.people:
if per.utility >= P:
per.customer = 1
cus.append(per)
else:
per.customer = 0
noncus.append(per)
return len(cus)
def f_wo_append(popn):
'''Function without append'''
P = 75
for per in popn.people:
if per.utility >= P:
per.customer = 1
else:
per.customer = 0
numcustomers = 0
for per in popn.people:
if per.customer == 1:
numcustomers += 1
return numcustomers
def main():
try:
numpeople = int(sys.argv[1])
except:
numpeople = 300000
print "Running for %s people, 100 times." % numpeople
begin = time()
random.seed(1)
# Help CPython a bit
uniform = random.uniform
util = [uniform(0.0, 300.0) for _ in xrange(numpeople)]
# util = [random.uniform(0.0, 300.0) for _ in xrange(numpeople)]
popn1 = population(numpeople, util)
start = time()
for _ in xrange(100):
r = f_wo_append(popn1)
print r
print "Without append: %s" % (time() - start)
popn2 = population(numpeople, util)
start = time()
for _ in xrange(100):
r = f_w_append(popn2)
print r
print "With append: %s" % (time() - start)
print "\n\nTotal time: %s" % (time() - begin)
if __name__ == "__main__":
main()
Running with PyPy is as simple as running with CPython, you just type 'pypy' instead of 'python'. For Shed Skin, you must convert to C++, compile and run:
shedskin -e makefaster.py && make
# Check that you're using the makefaster.so file and run test
python -c "import makefaster; print makefaster.__file__; makefaster.main()"
And here is the Cython-ized code:
from time import time
import random
import sys
cdef class person:
cdef readonly int utility
cdef public int customer
def __init__(self, util):
self.utility = util
self.customer = 0
class population(object):
def __init__(self, numpeople, util):
self.people = []
self.cus = []
self.noncus = []
for u in util:
per = person(u)
self.people.append(per)
cdef int f_w_append(popn):
'''Function with append'''
cdef int P = 75
cdef person per
cus = []
noncus = []
# Help CPython a bit
# cus_append, noncus_append = cus.append, noncus.append
for per in popn.people:
if per.utility >= P:
per.customer = 1
cus.append(per)
else:
per.customer = 0
noncus.append(per)
cdef int lcus = len(cus)
return lcus
cdef int f_wo_append(popn):
'''Function without append'''
cdef int P = 75
cdef person per
for per in popn.people:
if per.utility >= P:
per.customer = 1
else:
per.customer = 0
cdef int numcustomers = 0
for per in popn.people:
if per.customer == 1:
numcustomers += 1
return numcustomers
def main():
cdef int i, r, numpeople
cdef double _0, _300
_0 = 0.0
_300 = 300.0
try:
numpeople = int(sys.argv[1])
except:
numpeople = 300000
print "Running for %s people, 100 times." % numpeople
begin = time()
random.seed(1)
# Help CPython a bit
uniform = random.uniform
util = [uniform(_0, _300) for i in xrange(numpeople)]
# util = [random.uniform(0.0, 300.0) for _ in xrange(numpeople)]
popn1 = population(numpeople, util)
start = time()
for i in xrange(100):
r = f_wo_append(popn1)
print r
print "Without append: %s" % (time() - start)
popn2 = population(numpeople, util)
start = time()
for i in xrange(100):
r = f_w_append(popn2)
print r
print "With append: %s" % (time() - start)
print "\n\nTotal time: %s" % (time() - begin)
if __name__ == "__main__":
main()
For building it, it's nice to have a setup.py like this one:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
ext_modules = [Extension("cymakefaster", ["makefaster.pyx"])]
setup(
name = 'Python code to speed up',
cmdclass = {'build_ext': build_ext},
ext_modules = ext_modules
)
You build it with:
python setupfaster.py build_ext --inplace
Then test:
python -c "import cymakefaster; print cymakefaster.file; cymakefaster.main()"
Timings were run five times for each version, with Cython being the fastest and easiest of the code generators to use (Shed Skin aims to be simpler, but cryptic error messages and implicit static typing made it harder here). As for best value, PyPy gives impressive speedup in the counter version with no code changes.
#Results (time in seconds for 30000 people, 100 calls for each function):
Mean Min Times
CPython 2.5.2
Without append: 35.037 34.518 35.124, 36.363, 34.518, 34.620, 34.559
With append: 29.251 29.126 29.339, 29.257, 29.259, 29.126, 29.272
Total time: 69.288 68.739 69.519, 70.614, 68.746, 68.739, 68.823
PyPy 1.4.1
Without append: 2.672 2.655 2.655, 2.670, 2.676, 2.690, 2.668
With append: 13.030 12.672 12.680, 12.725, 14.319, 12.755, 12.672
Total time: 16.551 16.194 16.196, 16.229, 17.840, 16.295, 16.194
Shed Skin 0.7 (gcc -O2)
Without append: 1.601 1.599 1.599, 1.605, 1.600, 1.602, 1.599
With append: 3.811 3.786 3.839, 3.795, 3.798, 3.786, 3.839
Total time: 5.704 5.677 5.715, 5.705, 5.699, 5.677, 5.726
Cython 0.14 (gcc -O2)
Without append: 1.692 1.673 1.673, 1.710, 1.678, 1.688, 1.711
With append: 3.087 3.067 3.079, 3.080, 3.119, 3.090, 3.067
Total time: 5.565 5.561 5.562, 5.561, 5.567, 5.562, 5.572
Edit: Aaaand more meaningful timings, for 80000 calls with 300 people each:
Results (time in seconds for 300 people, 80000 calls for each function):
Mean Min Times
CPython 2.5.2
Without append: 27.790 25.827 25.827, 27.315, 27.985, 28.211, 29.612
With append: 26.449 24.721 24.721, 27.017, 27.653, 25.576, 27.277
Total time: 54.243 50.550 50.550, 54.334, 55.652, 53.789, 56.892
Cython 0.14 (gcc -O2)
Without append: 1.819 1.760 1.760, 1.794, 1.843, 1.827, 1.871
With append: 2.089 2.063 2.100, 2.063, 2.098, 2.104, 2.078
Total time: 3.910 3.859 3.865, 3.859, 3.944, 3.934, 3.951
PyPy 1.4.1
Without append: 0.889 0.887 0.894, 0.888, 0.890, 0.888, 0.887
With append: 1.671 1.665 1.665, 1.666, 1.671, 1.673, 1.681
Total time: 2.561 2.555 2.560, 2.555, 2.561, 2.561, 2.569
Shed Skin 0.7 (g++ -O2)
Without append: 0.310 0.301 0.301, 0.308, 0.317, 0.320, 0.303
With append: 1.712 1.690 1.733, 1.700, 1.735, 1.690, 1.702
Total time: 2.027 2.008 2.035, 2.008, 2.052, 2.011, 2.029
Shed Skin becomes fastest, PyPy surpasses Cython. All three speed things up a lot compared to CPython.
Please consider trimming down your f_wo_append function:
def f_wo_append():
'''Function without append'''
P = 75
numcustomers = 0
for person in popn.people:
person.customer = iscust = person.utility >= P
numcustomers += iscust
return numcustomers
Edit in response to OP's comment """This made it a lot worse! The trimmed version takes 4 times more time than the version I have posted above. """
There is no way that that could take "4 times more" (5 times?) ... here is my code, which demonstrates a significant reduction in the "without append" case, as I suggested, and also introduces a significant improvement in the "with append" case.
import random # instead of numpy
import time
timer_func = time.clock # better on Windows, use time.time on *x platform
class Person(object):
def __init__(self, util):
self.utility = util
self.customer = 0
class Population(object):
def __init__(self, numpeople):
random.seed(1)
self.people = [Person(random.uniform(0, 300)) for i in xrange(numpeople)]
self.cus = []
self.noncus = []
def f_w_append(popn):
'''Function with append'''
P = 75
cus = []
noncus = []
for per in popn.people:
if per.utility >= P:
per.customer = 1
cus.append(per)
else:
per.customer = 0
noncus.append(per)
popn.cus = cus # omitted from OP's code
popn.noncus = noncus # omitted from OP's code
return len(cus)
def f_w_append2(popn):
'''Function with append'''
P = 75
popn.cus = []
popn.noncus = []
cusapp = popn.cus.append
noncusapp = popn.noncus.append
for per in popn.people:
if per.utility >= P:
per.customer = 1
cusapp(per)
else:
per.customer = 0
noncusapp(per)
return len(popn.cus)
def f_wo_append(popn):
'''Function without append'''
P = 75
for per in popn.people:
if per.utility >= P:
per.customer = 1
else:
per.customer = 0
numcustomers = 0
for per in popn.people:
if per.customer == 1:
numcustomers += 1
return numcustomers
def f_wo_append2(popn):
'''Function without append'''
P = 75
numcustomers = 0
for person in popn.people:
person.customer = iscust = person.utility >= P
numcustomers += iscust
return numcustomers
if __name__ == "__main__":
import sys
popsize, which, niter = map(int, sys.argv[1:4])
pop = Population(popsize)
func = (f_w_append, f_w_append2, f_wo_append, f_wo_append2)[which]
t0 = timer_func()
for _unused in xrange(niter):
nc = func(pop)
t1 = timer_func()
print "popsize=%d func=%s niter=%d nc=%d seconds=%.2f" % (
popsize, func.__name__, niter, nc, t1 - t0)
and here are the results of running it (Python 2.7.1, Windows 7 Pro, "Intel Core i3 CPU 540 # 3.07 GHz"):
C:\junk>\python27\python ncust.py 300 0 80000
popsize=300 func=f_w_append niter=80000 nc=218 seconds=5.48
C:\junk>\python27\python ncust.py 300 1 80000
popsize=300 func=f_w_append2 niter=80000 nc=218 seconds=4.62
C:\junk>\python27\python ncust.py 300 2 80000
popsize=300 func=f_wo_append niter=80000 nc=218 seconds=5.55
C:\junk>\python27\python ncust.py 300 3 80000
popsize=300 func=f_wo_append2 niter=80000 nc=218 seconds=4.29
Edit 3 Why numpy takes longer:
>>> import numpy
>>> utils = numpy.random.uniform(0, 300, 10)
>>> print repr(utils[0])
42.777972538362874
>>> type(utils[0])
<type 'numpy.float64'>
and here's why my f_wo_append2 function took 4 times longer:
>>> x = utils[0]
>>> type(x)
<type 'numpy.float64'>
>>> type(x >= 75)
<type 'numpy.bool_'> # iscust refers to a numpy.bool_
>>> type(0 + (x >= 75))
<type 'numpy.int32'> # numcustomers ends up referring to a numpy.int32
>>>
The empirical evidence is that these custom types aren't so fast when used as scalars ... perhaps because they need to reset the floating-point hardware each time they are used. OK for big arrays, not for scalars.
Are you using any other numpy functionality? If not, just use the random module. If you have other uses for numpy, you may wish to coerce the numpy.float64 to float during the population setup.
You can eliminate some lookups by using local function aliases:
def qtyDemanded(self, timePd, priceVector):
'''Returns quantity demanded in period timePd. In addition,
also updates the list of customers and non-customers.
Inputs: timePd and priceVector
Output: count of people for whom priceVector[-1] < utility
'''
price = priceVector[-1]
self.customers = []
self.nonCustomers = []
# local function aliases
addCust = self.customers.append
addNonCust = self.nonCustomers.append
for person in self.people:
if person.utility >= price:
person.customer = 1
addCust(person)
else:
person.customer = 0
addNonCust(person)
return len(self.customers)
This comment rings alarm bells:
'''Returns quantity demanded in period timePd. In addition,
also updates the list of customers and non-customers.
Aside from the fact that timePd is not used in the function, if you really want just to return the quantity, do just that in the function. Do the "in addition" stuff in a separate function.
Then profile again and see which of these two functions you are spending most of your time in.
I like to apply SRP to methods as well as classes: it makes them easier to test.
Depending on how often you add new elements to self.people or change person.utility, you could consider sorting self.people by the utility field.
Then you could use a bisect function to find the lower index i_pivot where the person[i_pivot].utility >= price condition is met. This would have a lower complexity ( O(log N) ) than your exhaustive loop ( O(N) )
With this information, you could then update your people list if needed :
Do you really need to update the utility field each time ? In the sorted case, you could easily deduce this value while iterating : for example, considering your list sorted in incresing order, utility = (index >= i_pivot)
Same question with customers and nonCustomers lists. Why do you need them? They could be replaced by slices of the original sorted list : for example, customers = self.people[0:i_pivot]
All this would allow you to reduce the complexity of your algorithm, and use more built-in (fast) Python functions, this could speedup your implementation.
Some curious things I noted:
timePd is passed as a parameter but never used
price is an array but you only ever use the last entry - why not pass the value there instead of passing the list?
count is initialized and never used
self.people contains multiple person objects which are then copied to either self.customers or self.noncustomers as well as having their customer flag set. Why not skip the copy operation and, on return, just iterate over the list, looking at the customer flag? This would save the expensive append.
Alternatively, try using psyco which can speed up pure Python, sometimes considerably.
It's surprising that the function shown is such a bottleneck because it's so relatively simple. For that reason, I'd double check my profiling procedure and results. However, if they're correct, the most time consuming part of your function has to be the for loop it contains, of course, so it makes sense to focus on speeding that up. One way to do this is by replacing the if/else with straight-line code. You can also reduce the attribute lookup for the append list method slightly. Here's how both of those things could be accomplished:
def qtyDemanded(self, timePd, priceVector):
'''Returns quantity demanded in period timePd. In addition,
also updates the list of customers and non-customers.
Inputs: timePd and priceVector
Output: count of people for whom priceVector[-1] < utility
'''
price = priceVector[-1] # last price
kinds = [[], []] # initialize sublists of noncustomers and customers
kindsAppend = [kinds[b].append for b in (False, True)] # append methods
for person in self.people:
person.customer = person.utility >= price # customer test
kindsAppend[person.customer](person) # add to proper list
self.nonCustomers = kinds[False]
self.customers = kinds[True]
return len(self.customers)
That said, I must add that it seems a little redundant to have both a customer flag in each person object and also put each of them into a separate list depending on that attribute. Not creating these two lists would of course speed the loop up further.
You're asking for guesses, and mostly you're getting guesses.
There's no need to guess. Here's an example.