Related
I do have a piece of code that compute partitions of a set of (potentialy duplicated) integers. But i am interested in the set of possible partition and there multiplicity.
You can for exemple launch the follwoing code :
import numpy as np
from collections import Counter
import pandas as pd
def _B(i):
# for a given multiindex i, we defined _B(i) as the set of integers containg i_j times the number j:
if len(i) != 1:
B = []
for j in range(len(i)):
B.extend(i[j]*[j])
else:
B = i*[0]
return B
def _partition(collection):
# from here: https://stackoverflow.com/a/62532969/8425270
if len(collection) == 1:
yield (collection,)
return
first = collection[0]
for smaller in _partition(collection[1:]):
# insert `first` in each of the subpartition's subsets
for n, subset in enumerate(smaller):
yield smaller[:n] + ((first,) + subset,) + smaller[n + 1 :]
# put `first` in its own subset
yield ((first,),) + smaller
def to_list(tpl):
# the final hierarchy is
return list(list(i) if isinstance(i, tuple) else i for i in tpl)
def _Pi(inst_B):
# inst_B must be a tuple
if type(inst_B) != tuple :
inst_B = tuple(inst_B)
pp = [tuple(sorted(p)) for p in _partition(inst_B)]
c = Counter(pp)
Pi = c.keys()
N = list()
for pi in Pi:
N.append(c[pi])
Pi = [to_list(pi) for pi in Pi]
return Pi, N
if __name__ == "__main__":
import cProfile
pr = cProfile.Profile()
pr.enable()
sh = (3, 3, 3)
rez = list()
rez_sorted= list()
rez_ref = list()
for idx in np.ndindex(sh):
if sum(idx) > 0:
print(idx)
Pi, N = _Pi(_B(idx))
print(pd.DataFrame({'Pi': Pi, 'N': N * np.array([np.math.factorial(len(pi) - 1) for pi in Pi])}))
pr.disable()
# after your program ends
pr.print_stats(sort="tottime")
This code computes, for several examples of tuples of integer numbers (generated by np.ndindex) the partitions and counts i need. Everything happens in the _partition and the _Pi functions, this is were you should look at.
If you look closely at how these two functions are working, you'll see that they comput eevery potential partition and THEN count up how many times they appeared. For small problems, this is fine, but if the size of the prolbme increase, this starts to take a looooot of time. Try setting sh = (5,5,5), you'll see what i mean;
So the problem is the following :
Is there a way to compute directly the partitions and there number of occurences instead ?
Edit: I cross-posted on mathoverflow there, and they propose a solution in this article, in corrolary 2.10 (page 10 of the pdf). The problem could be solved by implmenting the sets p(v,r) in this corrolary.
I was hoping, as in the univariate case, that those sets would have a nice recursive expression but i ould not find one yet.
More Edit : This problem is equivalent to finding all (multiset)-partitions of a multiset. If the solution for finding (set)-partitions of a set is given by Bell partial polynomials, here we need multivariate version of these polynomials.
This is the problem I am trying to solve:
B: The Foxen's Treasure
There are N (1 ≤ N ≤ 4) Foxen guarding a certain valuable treasure,
which you'd love to get your hands on. The problem is, the Foxen
certainly aren't about to allow that - at least, not while they're
awake.
Fortunately, through careful observation, you've seen that each Fox
has a regular sleep cycle. In particular, the ith Fox stays awake for
Ai (1 ≤ Ai ≤ 23) hours, then sleeps for Si (1 ≤ Si ≤ 23) hours,
repeating this pattern indefinitely (2 ≤ Ai + Si ≤ 24). At the start
of your treasure-nabbing attempt, the ith Fox is
exactly Oi (0 ≤ Oi < Ai + Si) hours into its cycle.
There are T (1 ≤ T ≤ 20) scenarios as described above. For each one,
you'd like to determine how soon all of the Foxen will be
simultaneously asleep, allowing you to grab their treasure, or if this
will simply never happen.
Input
Line 1: 1 integer, T
For each scenario:
Line 1: 1 integer, N
Next N lines: 3 integers, Ai, Si, and Oi, for i = 1..N
Output
For each scenario:
Line 1: 1 integer, the minimum number of hours after the start to
wait until all of the Foxen are asleep during the same hour. If this
will never happen, output the string "Foxen are too powerful" (without
quotes) instead.
Sample Input
2
2
2 1 2
2 2 1
3
1 1 0
1 1 0
1 1 1
Sample Output
6
Foxen are too powerful
My Solution works as expected when I input the given sample case and get expected output. But when I submit the code to online judge it gives clipped error. Now there is no detail of the error which makes it difficult to find what the problem is.
Here is the solution which I have worked so far:
# ai is awake hours
# si is sleep hours.
# ai + si <= 24.
# False == sleep. True == awake.
datasets = int(raw_input());
foxen = [];
number_of_foxen = 0;
foxes = [];
class fox:
def __init__(self, a, s, i):
self.awake = a;
self.sleep = s;
self.current = i;
awake = 0;
sleep = 0;
current = 0;
def next(self):
if ( self.sleep + self.awake-1 > self.current ) :
self.current = self.current+1;
else:
self.current = 0;
return self.current;
def check(self):
if(self.current>=self.awake):
return False;
return True;
def printdata(self):
print "awake="+str(self.awake)+" sleep="+str(self.sleep)+" current="+str(self.current);
#return "awake="+str(self.awake)+" sleep="+str(self.sleep)+" current="+str(self.current);
for i in range(0, datasets):
number_of_foxen = int(raw_input());
for j in range(0, number_of_foxen):
foxen.append(raw_input());
x = foxen[j].split();
a = fox(int(x[0]), int(x[1]), int(x[2]));
foxes.append(a);
solution = False;
for j in range(0, 48):
#print "hour number = " + str(j);
#for k in range(0, len(foxes)):
#print "fox number="+ str(k)+" "+ foxes[k].printdata()+str(foxes[k].check());
count = 0 ;
for k in range(0, len(foxes)):
if(foxes[k].check()==False):
count+=1;
#print "count = "+str(count);
#print len(foxes);
if( (int(count) == int(len(foxes))) and (solution == False) ):
#print "this runs now *************";
solution = True;
number = j;
for k in range(0, len(foxes)):
foxes[k].next();
if(solution==True):
print number;
else:
print "Foxen are too powerful";
#print "Foxen are too powerful";
foxen = [];
number_of_foxen = 0;
foxes = [];
The biggest problem with your code is that it is unreadable. Indeed, it looks like it was written with little concept of Python's strengths. Here is my suggestion:
#!/usr/bin/env python3
"""
The Foxen's Treasure puzzle from http://wcipeg.com/problem/acmtryouts1b
"""
from sys import stdin
from itertools import cycle
from euclid import lcm
debug = True # set to False before submission to mechanical judge
class Fox:
"""A Fox cointains its defining integers and other derived
bindings such as its cycle and schedule."""
def __init__(self, trio):
(self.awake_count, self.sleep_count, self.skip_count) = trio
self.cycle = 'a' * self.awake_count + 's' * self.sleep_count
self.schedule = cycle(self.cycle)
if debug: print('<Fox: {}> cycle {}'.format(trio, self.cycle))
# handle skips by discarding the first elements
for _ in range(self.skip_count):
next(self.schedule)
def find_all_sleeping(foxes):
"""Return an hour number if all foxes are sleeping at that hour."""
# only examine the LCM of all fox periods. If not there it will never be.
lcm_period = 1
for fox in foxes:
lcm_period = lcm(lcm_period, len(fox.cycle))
for hour in range(lcm_period):
states = [next(fox.schedule) for fox in foxes]
if debug: print('{:2d} {}'.format(hour, ' '.join(states)))
if 'a' not in states:
return hour
return None
def read_trials(fp):
"""Reads the entire input at once. Returns a list of trials.
Each trial is a list of Fox."""
trials = list()
trial_count = int(fp.readline())
for trial in range(trial_count):
if debug: print('--Read trial {}'.format(trial))
foxes = list()
fox_count = int(fp.readline())
for _ in range(fox_count):
fox = Fox([int(x) for x in fp.readline().split()])
foxes.append(fox)
trials.append(foxes)
return trials
for trial, foxes in enumerate(read_trials(stdin)):
if debug: print('--Run trial {}'.format(trial))
hour = find_all_sleeping(foxes)
if hour is None:
print('Foxen are too powerful')
else:
print(hour)
I suspect that the first concern is that it looks much longer than the OP; that is true, but if you take out the debugging code which shows how things are happening, and the docstrings that explain why it is doing things, it's actually a few line shorter than the OP.
The main loop of the OP is too long to understand without significant study, and a whole bunch of bad variable names makes that even harder. In contrast, there are places here where a value is given a name only to make the code more explicit about what an input line means. You'll find a number of
for _ in range(trial)
to show that the loop value is not used. This is a frequent idiom when dealing with fixed format input.
The Fox representation keeps the inner workings in the problem space. As noted in the exercise page, it makes more sense to look at things as a concurrence between sequences:
--Read trial 0
<Fox: [2, 1, 2]> cycle aas
<Fox: [2, 2, 1]> cycle aass
the offsets skip_count are not shown here, but they are clear in the trial run.
The input from the datafile is all kept inside read_trials() instead of scattered through the code. This confines the mess to one place rather than distributing it through the code. We know from the puzzle instructions that the datafile will not be large enough to care about. read_trials(fp) also takes a file-like object which allows it to read from an actual file, a StringIO buffer, or the standard input.
Once the Fox schedule generator is initialized, itertools.cycle will give an unending supply of the next letter in the sequence; it does the wrap-around for you.
It is worth noting that the primary data structure trials is a plain old list because it doesn't need anything more than that.
I've gotten a little weary of bad code being answered with worse code. Sure, this could be considered way more than the needs of an electronic judge where only the output matters. Contrariwise, I'm still puzzled by bits like (solution == False), a main loop that is 42 lines long and split between the top and bottom of the file, variables like i and j which convey no intent, the memory burden of False == awake (or did I mix them up?), dead code, no-op code, `range(0, n) and a whole bunch of magic numbers throughout.
Sure, you can code like the code doesn't matter, but if you are teaching yourself to code it is good to practice good practice. Yeah, you might never look at this piece of code again, but if you ain't gonna learn it now, then when?
In case you feel it a cheat to have imported lcm() there's no reason to write it a second time, so I referenced a homebrew package of which the relevant lines are:
def gcd(a, b):
"""Return the Greatest Common Divisor of a and b."""
while b:
a, b = b, a % b
return a
def lcm(a, b):
"""Return the Least Common Multiple of a and b."""
return abs(a * b) // gcd(a, b)
Jorge was correct in his comment, there doesn't appear to be any problem with your algorithm other than the arbitrary 48 hour cuttoff.
However:
1) your print statements do not use the correct syntax for Python 3+. For example, your final print statement print "Foxen are too powerful"; must be changed to work in Python 3, try print ('Foxen are too powerful') instead.
2) I'm seeing some odd C/MatLab-like syntax as well, lines being ended by a semicolon, and double brackets surrounding conditions in your if statements. This probably isn't a problem, but depending on how picky the system you are submitting the answer to is, you may want to clean it up a little.
3) Definitely increase the cutoff time for your search. I'd recommend a reasonably large value, on the order of 10,000 hours, just to be sure that it won't be a factor.
I've taken the liberty of making all of the above changes so I'm posting the resultant code now:
# ai is awake hours
# si is sleep hours.
# ai + si <= 24.
# False == sleep. True == awake.
datasets = int(raw_input())
foxen = []
number_of_foxen = 0
foxes = []
class fox:
def __init__(self, a, s, i):
self.awake = a
self.sleep = s
self.current = i
awake = 0
sleep = 0
current = 0
def next(self):
if ( self.sleep + self.awake-1 > self.current ):
self.current = self.current+1
else:
self.current = 0
return self.current
def check(self):
if(self.current>=self.awake):
return False
return True
def printdata(self):
print ("awake="+str(self.awake)+" sleep="+str(self.sleep)+" current="+str(self.current))
#return ("awake="+str(self.awake)+" sleep="+str(self.sleep)+" current="+str(self.current))
for i in range(0, datasets):
number_of_foxen = int(raw_input())
for j in range(0, number_of_foxen):
foxen.append(raw_input())
x = foxen[j].split()
a = fox(int(x[0]), int(x[1]), int(x[2]))
foxes.append(a)
solution = False
for j in range(0, 10000):
#print ("hour number = " + str(j))
#for k in range(0, len(foxes)):
#print ("fox number="+ str(k)+" "+ foxes[k].printdata()+str(foxes[k].check()))
count = 0
for k in range(0, len(foxes)):
if(foxes[k].check()==False):
count+=1
#print ("count = "+str(count))
#print (len(foxes))
if (int(count) == int(len(foxes)) and (solution == False)):
#print ("this runs now *************")
solution = True
number = j
for k in range(0, len(foxes)):
foxes[k].next()
if(solution == True):
print (number)
else:
print ("Foxen are too powerful")
#print ("Foxen are too powerful")
foxen = []
number_of_foxen = 0
foxes = []
Enjoy and Good Luck!
Interesting problem, here's my code:
import sys
# Globals
debugLevel = 0
fileMode = True # True if loading data from a file.
# Constants
AWAKE = 0
ASLEEP = -1
def gcd(a, b):
"""Return greatest common divisor using Euclid's Algorithm."""
while b:
a, b = b, a % b
return a
def lcm(a, b):
"""Return lowest common multiple."""
return a * b // gcd(a, b)
def readData(f):
''' Read in the problem data and store in data structures
'''
numTrials = int(f.readline().strip())
if debugLevel >= 4:
print("Num trials: ", numTrials)
trialData = []
for _ in range(numTrials):
numFoxen = int(f.readline().strip())
allFoxenHoursInfo = []
for _ in range(numFoxen):
aFoxHoursInfo = f.readline().split()
aFoxHoursInfo = list(map(int, aFoxHoursInfo))
allFoxenHoursInfo.append(aFoxHoursInfo)
trialData.append((numFoxen, allFoxenHoursInfo))
if debugLevel >= 8:
print("Trial data\n", trialData)
return numTrials, trialData
def runTrials(trialData):
'''
Go through each lot of foxen, and their sleep/awake schedules and
See if there's a time that all of them will be asleep.
'''
global debugLevel
for trial in trialData:
numFoxen, allFoxenHoursInfo = trial
# Create a table of the status of each fox in each hour
row = [AWAKE] * (numFoxen+1)
row[0] = 0
hoursTable = [row]
# Cycle length for each fox is the number of hours they spend awake then asleep.
cycleLength = [0] * (numFoxen)
# This is the number of hours into the cycle each fox is at the start
startingPosInCycle= [0] * (numFoxen)
# Initialise the first row
for fox in range(numFoxen):
cycleLength[fox] = allFoxenHoursInfo[fox][0] + allFoxenHoursInfo[fox][1] # Time awake plus time asleep
startingPosInCycle[fox] = allFoxenHoursInfo[fox][2] # % cycleLength[fox]
if startingPosInCycle[fox] >= allFoxenHoursInfo[fox][0]:
hoursTable[0][fox+1] = ASLEEP
if debugLevel >= 4:
print("Initial table: ", hoursTable)
# lcm = lowest common multiple and it's implemented above.
# For this problem, we only need to look at the lcm of all the cycle lengths for the foxen.
numIterations = 1
for fox in range(numFoxen):
numIterations = lcm(numIterations, cycleLength[fox])
# Go around a loop adding a new row to the table for each new hour containing the updated
# statuses of each fox.
for hourNum in range(1, numIterations):
allFoxesSleeping = False
# Update our hours table by creating a new row and calculating the status of each fox
newRow = [AWAKE] * (numFoxen+1)
newRow[0] = hourNum
for fox in range(numFoxen):
currentPosInCycle = (startingPosInCycle[fox] + hourNum) % cycleLength[fox]
if currentPosInCycle >= allFoxenHoursInfo[fox][0]:
newRow[fox+1] = ASLEEP
hoursTable.append(newRow)
if debugLevel >= 4:
print("Hours table\n", hoursTable)
# See if all foxen are sleeping, if they are, success
numFoxesSleeping = hoursTable[hourNum].count(ASLEEP)
if numFoxesSleeping == numFoxen:
allFoxesSleeping = True
print(hourNum)
break
if not allFoxesSleeping:
print('Foxen are too powerful')
def main():
'''Reads, runs, and outputs problem specific data.'''
# Initialisation
#strDir = ".\\"
# if fileMode:
# dataSource = open(strDir + "DataFile.txt", 'r')
# else:
dataSource = sys.stdin
# Read in the input data.
numTrials, trialData = readData(dataSource)
# Run each trial, outputting the result of that trial
runTrials(trialData)
sys.stdout.flush()
# Cleanup
# if fileMode:
# dataSource.close()
if __name__ == '__main__':
main()
Unfortunately, it does not pass the judge either. I have no idea why. I get this output:
Test case #1: WA [0.178s, 3628K] (0/1) (Details)
Your Output (clipped)
6
Foxen are too powe
Final score: 0/1
Interesting problem. I have emailed the author, because there is an inconsistency in the problem definition and the sample input data. He says this: Oi (0 ≤ Oi < Ai + Si) but then gives 1 1 1 on the last line of sample input data. And 1 is not strictly less than 1+1.
So who knows what other data the judge might be using...
The bit at the bottom that is commented out re files, lets me work with files and an IPython console rather than a Python console and pasting the data in, which I find slow and annoying.
Also, it's a little strange I reckon to not be able to see the data the judge is using. Surely seeing the data you are working against would enable the problem to be run and debugged offline, then when it's going, a new online submit could be done.
I've solved a problem on spoj, but it's still too slow for being accepted.
I've tried to make it use multiprocessing too, but I've failed because it's still slower.
The basic implemenation, even with pypy, returns "time limits exceeded" on spoj.
So, how can I improve it?
And what is wrong with the multiprocessing implementation?
# -- shipyard
from collections import defaultdict
#W = 100 total weight
#N = 2 number of types
#value | weight
#1 1
#30 50
# result -> 60 = minimum total value
#c = [1, 30]
#w = [1, 50]
def knap(W, N, c, w):
f = defaultdict(int)
g = defaultdict(bool)
g[0] = True
for i in xrange(N):
for j in xrange(W):
if g[j]:
g[j+w[i]] = True
#print "g("+str(j+w[i])+") = true"
if ( f[j+w[i]]==0 or f[j+w[i]]>f[j]+c[i]):
f[j+w[i]] = f[j]+c[i]
#print " f("+str(j+w[i])+") = ",f[j+w[i]]
if g[W]:
print f[W]
else:
print -1
def start():
while True:
num_test = int(raw_input())
for i in range(num_test):
totWeight = int(raw_input())
types = int(raw_input())
costs = defaultdict(int)
weights = defaultdict(int)
for t in range(int( types )):
costs[t], weights[t] = [int(i) for i in raw_input().split()]
knap(totWeight, types, costs, weights)
return
if __name__ == '__main__':
start()
And here is the multiprocessing version:
# -- shipyard
from multiprocessing import Process, Queue
from collections import defaultdict
from itertools import chain
W = 0
c = {} #[]
w = {} #[]
def knap(i, g, f, W, w, c, qG, qF):
for j in xrange( W ):
if g[j]:
g[j+w[i]] = True
#print "g("+str(j+w[i])+") = true"
if ( f[j+w[i]]==0 or f[j+w[i]]>f[j]+c[i]):
f[j+w[i]] = f[j]+c[i]
#print " f("+str(j+w[i])+") = ",f[j+w[i]]
qG.put( g)
qF.put( f)
def start():
global f, g, c, w, W
while True:
num_test = int(raw_input())
for _ in range(num_test):
qG = Queue()
qF = Queue()
W = int(raw_input())
N = int(raw_input()) # types
c = {} #[0 for i in range(N)]
w = {} #[0 for i in range(N)]
f = defaultdict(int)
g = defaultdict(bool)
g[0] = True
for t in range( N ):
c[t], w[t] = [int(i) for i in raw_input().split()]
# let's go parallel
for i in xrange(0, N, 2):
k1 = Process(target=knap, args=(i, g, f, W, w, c, qG, qF))
k2 = Process(target=knap, args=(i+1, g, f, W, w, c, qG, qF))
k1.start()
k2.start()
k1.join()
k2.join()
#while k1.is_alive(): # or k2.is_alive():
# None
#g2 = defaultdict(bool, chain( g.iteritems(), qG.get().iteritems(), qG.get().iteritems()))
#f2 = defaultdict(int, chain( f.iteritems(), qF.get().iteritems(), qF.get().iteritems()))
g2 = defaultdict(bool, g.items()+ qG.get().items()+ qG.get().items())
f2 = defaultdict(int, f.items()+ qF.get().items()+ qF.get().items())
g = g2
f = f2
print "\n g: ", len(g), "\n f: ", len(f),"\n"
if g[W]:
print f[W]
else:
print -1
return
if __name__ == '__main__':
start()
I probably haven't understood how to make two processes to work efficently on the same dictionary
Some programming contest will explicitly ban multithreading or block it so try to look elsewhere. My approach in those times is use a profiling tool to see where your code is struggling. You could try the built-in cProfile (python -m cProfile -o <outputfilename> <script-name> <options>) and then this wonderful visualization tool: http://www.vrplumber.com/programming/runsnakerun/
Once you have your visualization look around, dig into boxes. Some times there are things that are not directly evident but make sense once inspecting the running times. i.e. a common problem (not sure if it's your case) is checking for list membership. It's much faster to use a set in this case and often it pays to keep a separate list and set if you need the order later. There are many more tips regarding importing variables into local space, etc. You can check a list here: https://wiki.python.org/moin/PythonSpeed/PerformanceTips
Many people who use Python face the same problem on programming contest sites. I've found that its best to simply ditch Python altogether for problems which accept large inputs where you have to construct and iterate over a big data structure. Simply re-implement the same solution in C or C++. Python is known to be 10 to 100 times slower than well optimised C/C++ code.
Your code looks good and there's really little you can do to gain more speed (apart from big O improvements or going through hoops such as multiprocessing). If you have to use Python, try to avoid creating unnecessary large lists and use the most efficient algorithm you can think of. You can also try to generate and run large test cases prior to submitting your solution.
Okay, basically what I want is to compress a file by reusing code and then at runtime replace missing code. What I've come up with is really ugly and slow, at least it works. The problem is that the file has no specific structure, for example 'aGVsbG8=\n', as you can see it's base64 encoding. My function is really slow because the length of the file is 1700+ and it checks for patterns 1 character at the time. Please help me with new better code or at least help me with optimizing what I got :). Anything that helps is welcome! BTW i have already tried compression libraries but they didn't compress as good as my ugly function.
def c_long(inp, cap=False, b=5):
import re,string
if cap is False: cap = len(inp)
es = re.escape; le=len; ref = re.findall; ran = range; fi = string.find
c = b;inpc = inp;pattern = inpc[:b]; l=[]
rep = string.replace; ins = list.insert
while True:
if c == le(inpc) and le(inpc) > b+1: c = b; inpc = inpc[1:]; pattern = inpc[:b]
elif le(inpc) <= b: break
if c == cap: c = b; inpc = inpc[1:]; pattern = inpc[:b]
p = ref(es(pattern),inp)
pattern += inpc[c]
if le(p) > 1 and le(pattern) >= b+1:
if l == []: l = [[pattern,le(p)+le(pattern)]]
elif le(ref(es(inpc[:c+2]),inp))+le(inpc[:c+2]) < le(p)+le(pattern):
x = [pattern,le(p)+le(inpc[:c+1])]
for i in ran(le(l)):
if x[1] >= l[i][1] and x[0][:-1] not in l[i][0]: ins(l,i,x); break
elif x[1] >= l[i][1] and x[0][:-1] in l[i][0]: l[i] = x; break
inpc = inpc[:fi(inpc,x[0])] + inpc[le(x[0]):]
pattern = inpc[:b]
c = b-1
c += 1
d = {}; c = 0
s = ran(le(l))
for x in l: inp = rep(inp,x[0],'{%d}' % s[c]); d[str(s[c])] = x[0]; c += 1
return [inp,d]
def decompress(inp,l): return apply(inp.format, [l[str(x)] for x in sorted([int(x) for x in l.keys()])])
The easiest way to compress base64-encoded data is to first convert it to binary data -- this will already save 25 percent of the storage space:
>>> s = "YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXo=\n"
>>> t = s.decode("base64")
>>> len(s)
37
>>> len(t)
26
In most cases, you can compress the string even further using some compression algorithm, like t.encode("bz2") or t.encode("zlib").
A few remarks on your code: There are lots of factors that make the code hard to read: inconsistent spacing, overly long lines, meaningless variable names, unidiomatic code, etc. An example: Your decompress() function could be equivalently written as
def decompress(compressed_string, substitutions):
subst_list = [substitutions[k] for k in sorted(substitutions, key=int)]
return compressed_string.format(*subst_list)
Now it's already much more obvious what it does. You could go one step further: Why is substitutions a dictionary with the string keys "0", "1" etc.? Not only is it strange to use strings instead of integers -- you don't need the keys at all! A simple list will do, and decompress() will simplify to
def decompress(compressed_string, substitutions):
return compressed_string.format(*substitutions)
You might think all this is secondary, but if you make the rest of your code equally readable, you will find the bugs in your code yourself. (There are bugs -- it crashes for "abcdefgabcdefg" and many other strings.)
Typically one would pump the program through a compression algorithm optimized for text, then run that through exec, e.g.
code="""..."""
exec(somelib.decompress(code), globals=???, locals=???)
It may be the case that .pyc/.pyo files are compressed already, and one could check by creating one with x="""aaaaaaaa""", then increasing the length to x="""aaaaaaaaaaaaaaaaaaaaaaa...aaaa""" and seeing if the size changes appreciably.
I have some python code that has many classes. I used cProfile to find that the total time to run the program is 68 seconds. I found that the following function in a class called Buyers takes about 60 seconds of those 68 seconds. I have to run the program about 100 times, so any increase in speed will help. Can you suggest ways to increase the speed by modifying the code? If you need more information that will help, please let me know.
def qtyDemanded(self, timePd, priceVector):
'''Returns quantity demanded in period timePd. In addition,
also updates the list of customers and non-customers.
Inputs: timePd and priceVector
Output: count of people for whom priceVector[-1] < utility
'''
## Initialize count of customers to zero
## Set self.customers and self.nonCustomers to empty lists
price = priceVector[-1]
count = 0
self.customers = []
self.nonCustomers = []
for person in self.people:
if person.utility >= price:
person.customer = 1
self.customers.append(person)
else:
person.customer = 0
self.nonCustomers.append(person)
return len(self.customers)
self.people is a list of person objects. Each person has customer and utility as its attributes.
EDIT - responsed added
-------------------------------------
Thanks so much for the suggestions. Here is the
response to some questions and suggestions people have kindly
made. I have not tried them all, but will try others and write back later.
(1) #amber - the function is accessed 80,000 times.
(2) #gnibbler and others - self.people is a list of Person objects in memory. Not connected to a database.
(3) #Hugh Bothwell
cumtime taken by the original function - 60.8 s (accessed 80000 times)
cumtime taken by the new function with local function aliases as suggested - 56.4 s (accessed 80000 times)
(4) #rotoglup and #Martin Thomas
I have not tried your solutions yet. I need to check the rest of the code to see the places where I use self.customers before I can make the change of not appending the customers to self.customers list. But I will try this and write back.
(5) #TryPyPy - thanks for your kind offer to check the code.
Let me first read a little on the suggestions you have made to see if those will be feasible to use.
EDIT 2
Some suggested that since I am flagging the customers and noncustomers in the self.people, I should try without creating separate lists of self.customers and self.noncustomers using append. Instead, I should loop over the self.people to find the number of customers. I tried the following code and timed both functions below f_w_append and f_wo_append. I did find that the latter takes less time, but it is still 96% of the time taken by the former. That is, it is a very small increase in the speed.
#TryPyPy - The following piece of code is complete enough to check the bottleneck function, in case your offer is still there to check it with other compilers.
Thanks again to everyone who replied.
import numpy
class person(object):
def __init__(self, util):
self.utility = util
self.customer = 0
class population(object):
def __init__(self, numpeople):
self.people = []
self.cus = []
self.noncus = []
numpy.random.seed(1)
utils = numpy.random.uniform(0, 300, numpeople)
for u in utils:
per = person(u)
self.people.append(per)
popn = population(300)
def f_w_append():
'''Function with append'''
P = 75
cus = []
noncus = []
for per in popn.people:
if per.utility >= P:
per.customer = 1
cus.append(per)
else:
per.customer = 0
noncus.append(per)
return len(cus)
def f_wo_append():
'''Function without append'''
P = 75
for per in popn.people:
if per.utility >= P:
per.customer = 1
else:
per.customer = 0
numcustomers = 0
for per in popn.people:
if per.customer == 1:
numcustomers += 1
return numcustomers
EDIT 3: It seems numpy is the problem
This is in response to what John Machin said below. Below you see two ways of defining Population class. I ran the program below twice, once with each way of creating Population class. One uses numpy and one does not use numpy. The one without numpy takes similar time as John found in his runs. One with numpy takes much longer. What is not clear to me is that the popn instance is created before time recording begins (at least that is what it appears from the code). Then, why is numpy version taking longer. And, I thought numpy was supposed to be more efficient. Anyhow, the problem seems to be with numpy and not so much with the append, even though it does slow down things a little. Can someone please confirm with the code below? Thanks.
import random # instead of numpy
import numpy
import time
timer_func = time.time # using Mac OS X 10.5.8
class Person(object):
def __init__(self, util):
self.utility = util
self.customer = 0
class Population(object):
def __init__(self, numpeople):
random.seed(1)
self.people = [Person(random.uniform(0, 300)) for i in xrange(numpeople)]
self.cus = []
self.noncus = []
# Numpy based
# class Population(object):
# def __init__(self, numpeople):
# numpy.random.seed(1)
# utils = numpy.random.uniform(0, 300, numpeople)
# self.people = [Person(u) for u in utils]
# self.cus = []
# self.noncus = []
def f_wo_append(popn):
'''Function without append'''
P = 75
for per in popn.people:
if per.utility >= P:
per.customer = 1
else:
per.customer = 0
numcustomers = 0
for per in popn.people:
if per.customer == 1:
numcustomers += 1
return numcustomers
t0 = timer_func()
for i in xrange(20000):
x = f_wo_append(popn)
t1 = timer_func()
print t1-t0
Edit 4: See the answers by John Machin and TryPyPy
Since there have been so many edits and updates here, those who find themselves here for the first time may be a little confused. See the answers by John Machin and TryPyPy. Both of these can help in improving the speed of the code substantially. I am grateful to them and others who alerted me to slowness of append. Since, in this instance I am going to use John Machin's solution and not use numpy for generating utilities, I am accepting his response as an answer. However, I really appreciate the directions pointed out by TryPyPy also.
There are many things you can try after optimizing your Python code for speed. If this program doesn't need C extensions, you can run it under PyPy to benefit from its JIT compiler. You can try making a C extension for possibly huge speedups. Shed Skin will even allow you to convert your Python program to a standalone C++ binary.
I'm willing to time your program under these different optimization scenarios if you can provide enough code for benchmarking,
Edit: First of all, I have to agree with everyone else: are you sure you're measuring the time correctly? The example code runs 100 times in under 0.1 seconds here, so there is a good chance the either the time is wrong or you have a bottleneck (IO?) that isn't present in the code sample.
That said, I made it 300000 people so times were consistent. Here's the adapted code, shared by CPython (2.5), PyPy and Shed Skin:
from time import time
import random
import sys
class person(object):
def __init__(self, util):
self.utility = util
self.customer = 0
class population(object):
def __init__(self, numpeople, util):
self.people = []
self.cus = []
self.noncus = []
for u in util:
per = person(u)
self.people.append(per)
def f_w_append(popn):
'''Function with append'''
P = 75
cus = []
noncus = []
# Help CPython a bit
# cus_append, noncus_append = cus.append, noncus.append
for per in popn.people:
if per.utility >= P:
per.customer = 1
cus.append(per)
else:
per.customer = 0
noncus.append(per)
return len(cus)
def f_wo_append(popn):
'''Function without append'''
P = 75
for per in popn.people:
if per.utility >= P:
per.customer = 1
else:
per.customer = 0
numcustomers = 0
for per in popn.people:
if per.customer == 1:
numcustomers += 1
return numcustomers
def main():
try:
numpeople = int(sys.argv[1])
except:
numpeople = 300000
print "Running for %s people, 100 times." % numpeople
begin = time()
random.seed(1)
# Help CPython a bit
uniform = random.uniform
util = [uniform(0.0, 300.0) for _ in xrange(numpeople)]
# util = [random.uniform(0.0, 300.0) for _ in xrange(numpeople)]
popn1 = population(numpeople, util)
start = time()
for _ in xrange(100):
r = f_wo_append(popn1)
print r
print "Without append: %s" % (time() - start)
popn2 = population(numpeople, util)
start = time()
for _ in xrange(100):
r = f_w_append(popn2)
print r
print "With append: %s" % (time() - start)
print "\n\nTotal time: %s" % (time() - begin)
if __name__ == "__main__":
main()
Running with PyPy is as simple as running with CPython, you just type 'pypy' instead of 'python'. For Shed Skin, you must convert to C++, compile and run:
shedskin -e makefaster.py && make
# Check that you're using the makefaster.so file and run test
python -c "import makefaster; print makefaster.__file__; makefaster.main()"
And here is the Cython-ized code:
from time import time
import random
import sys
cdef class person:
cdef readonly int utility
cdef public int customer
def __init__(self, util):
self.utility = util
self.customer = 0
class population(object):
def __init__(self, numpeople, util):
self.people = []
self.cus = []
self.noncus = []
for u in util:
per = person(u)
self.people.append(per)
cdef int f_w_append(popn):
'''Function with append'''
cdef int P = 75
cdef person per
cus = []
noncus = []
# Help CPython a bit
# cus_append, noncus_append = cus.append, noncus.append
for per in popn.people:
if per.utility >= P:
per.customer = 1
cus.append(per)
else:
per.customer = 0
noncus.append(per)
cdef int lcus = len(cus)
return lcus
cdef int f_wo_append(popn):
'''Function without append'''
cdef int P = 75
cdef person per
for per in popn.people:
if per.utility >= P:
per.customer = 1
else:
per.customer = 0
cdef int numcustomers = 0
for per in popn.people:
if per.customer == 1:
numcustomers += 1
return numcustomers
def main():
cdef int i, r, numpeople
cdef double _0, _300
_0 = 0.0
_300 = 300.0
try:
numpeople = int(sys.argv[1])
except:
numpeople = 300000
print "Running for %s people, 100 times." % numpeople
begin = time()
random.seed(1)
# Help CPython a bit
uniform = random.uniform
util = [uniform(_0, _300) for i in xrange(numpeople)]
# util = [random.uniform(0.0, 300.0) for _ in xrange(numpeople)]
popn1 = population(numpeople, util)
start = time()
for i in xrange(100):
r = f_wo_append(popn1)
print r
print "Without append: %s" % (time() - start)
popn2 = population(numpeople, util)
start = time()
for i in xrange(100):
r = f_w_append(popn2)
print r
print "With append: %s" % (time() - start)
print "\n\nTotal time: %s" % (time() - begin)
if __name__ == "__main__":
main()
For building it, it's nice to have a setup.py like this one:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
ext_modules = [Extension("cymakefaster", ["makefaster.pyx"])]
setup(
name = 'Python code to speed up',
cmdclass = {'build_ext': build_ext},
ext_modules = ext_modules
)
You build it with:
python setupfaster.py build_ext --inplace
Then test:
python -c "import cymakefaster; print cymakefaster.file; cymakefaster.main()"
Timings were run five times for each version, with Cython being the fastest and easiest of the code generators to use (Shed Skin aims to be simpler, but cryptic error messages and implicit static typing made it harder here). As for best value, PyPy gives impressive speedup in the counter version with no code changes.
#Results (time in seconds for 30000 people, 100 calls for each function):
Mean Min Times
CPython 2.5.2
Without append: 35.037 34.518 35.124, 36.363, 34.518, 34.620, 34.559
With append: 29.251 29.126 29.339, 29.257, 29.259, 29.126, 29.272
Total time: 69.288 68.739 69.519, 70.614, 68.746, 68.739, 68.823
PyPy 1.4.1
Without append: 2.672 2.655 2.655, 2.670, 2.676, 2.690, 2.668
With append: 13.030 12.672 12.680, 12.725, 14.319, 12.755, 12.672
Total time: 16.551 16.194 16.196, 16.229, 17.840, 16.295, 16.194
Shed Skin 0.7 (gcc -O2)
Without append: 1.601 1.599 1.599, 1.605, 1.600, 1.602, 1.599
With append: 3.811 3.786 3.839, 3.795, 3.798, 3.786, 3.839
Total time: 5.704 5.677 5.715, 5.705, 5.699, 5.677, 5.726
Cython 0.14 (gcc -O2)
Without append: 1.692 1.673 1.673, 1.710, 1.678, 1.688, 1.711
With append: 3.087 3.067 3.079, 3.080, 3.119, 3.090, 3.067
Total time: 5.565 5.561 5.562, 5.561, 5.567, 5.562, 5.572
Edit: Aaaand more meaningful timings, for 80000 calls with 300 people each:
Results (time in seconds for 300 people, 80000 calls for each function):
Mean Min Times
CPython 2.5.2
Without append: 27.790 25.827 25.827, 27.315, 27.985, 28.211, 29.612
With append: 26.449 24.721 24.721, 27.017, 27.653, 25.576, 27.277
Total time: 54.243 50.550 50.550, 54.334, 55.652, 53.789, 56.892
Cython 0.14 (gcc -O2)
Without append: 1.819 1.760 1.760, 1.794, 1.843, 1.827, 1.871
With append: 2.089 2.063 2.100, 2.063, 2.098, 2.104, 2.078
Total time: 3.910 3.859 3.865, 3.859, 3.944, 3.934, 3.951
PyPy 1.4.1
Without append: 0.889 0.887 0.894, 0.888, 0.890, 0.888, 0.887
With append: 1.671 1.665 1.665, 1.666, 1.671, 1.673, 1.681
Total time: 2.561 2.555 2.560, 2.555, 2.561, 2.561, 2.569
Shed Skin 0.7 (g++ -O2)
Without append: 0.310 0.301 0.301, 0.308, 0.317, 0.320, 0.303
With append: 1.712 1.690 1.733, 1.700, 1.735, 1.690, 1.702
Total time: 2.027 2.008 2.035, 2.008, 2.052, 2.011, 2.029
Shed Skin becomes fastest, PyPy surpasses Cython. All three speed things up a lot compared to CPython.
Please consider trimming down your f_wo_append function:
def f_wo_append():
'''Function without append'''
P = 75
numcustomers = 0
for person in popn.people:
person.customer = iscust = person.utility >= P
numcustomers += iscust
return numcustomers
Edit in response to OP's comment """This made it a lot worse! The trimmed version takes 4 times more time than the version I have posted above. """
There is no way that that could take "4 times more" (5 times?) ... here is my code, which demonstrates a significant reduction in the "without append" case, as I suggested, and also introduces a significant improvement in the "with append" case.
import random # instead of numpy
import time
timer_func = time.clock # better on Windows, use time.time on *x platform
class Person(object):
def __init__(self, util):
self.utility = util
self.customer = 0
class Population(object):
def __init__(self, numpeople):
random.seed(1)
self.people = [Person(random.uniform(0, 300)) for i in xrange(numpeople)]
self.cus = []
self.noncus = []
def f_w_append(popn):
'''Function with append'''
P = 75
cus = []
noncus = []
for per in popn.people:
if per.utility >= P:
per.customer = 1
cus.append(per)
else:
per.customer = 0
noncus.append(per)
popn.cus = cus # omitted from OP's code
popn.noncus = noncus # omitted from OP's code
return len(cus)
def f_w_append2(popn):
'''Function with append'''
P = 75
popn.cus = []
popn.noncus = []
cusapp = popn.cus.append
noncusapp = popn.noncus.append
for per in popn.people:
if per.utility >= P:
per.customer = 1
cusapp(per)
else:
per.customer = 0
noncusapp(per)
return len(popn.cus)
def f_wo_append(popn):
'''Function without append'''
P = 75
for per in popn.people:
if per.utility >= P:
per.customer = 1
else:
per.customer = 0
numcustomers = 0
for per in popn.people:
if per.customer == 1:
numcustomers += 1
return numcustomers
def f_wo_append2(popn):
'''Function without append'''
P = 75
numcustomers = 0
for person in popn.people:
person.customer = iscust = person.utility >= P
numcustomers += iscust
return numcustomers
if __name__ == "__main__":
import sys
popsize, which, niter = map(int, sys.argv[1:4])
pop = Population(popsize)
func = (f_w_append, f_w_append2, f_wo_append, f_wo_append2)[which]
t0 = timer_func()
for _unused in xrange(niter):
nc = func(pop)
t1 = timer_func()
print "popsize=%d func=%s niter=%d nc=%d seconds=%.2f" % (
popsize, func.__name__, niter, nc, t1 - t0)
and here are the results of running it (Python 2.7.1, Windows 7 Pro, "Intel Core i3 CPU 540 # 3.07 GHz"):
C:\junk>\python27\python ncust.py 300 0 80000
popsize=300 func=f_w_append niter=80000 nc=218 seconds=5.48
C:\junk>\python27\python ncust.py 300 1 80000
popsize=300 func=f_w_append2 niter=80000 nc=218 seconds=4.62
C:\junk>\python27\python ncust.py 300 2 80000
popsize=300 func=f_wo_append niter=80000 nc=218 seconds=5.55
C:\junk>\python27\python ncust.py 300 3 80000
popsize=300 func=f_wo_append2 niter=80000 nc=218 seconds=4.29
Edit 3 Why numpy takes longer:
>>> import numpy
>>> utils = numpy.random.uniform(0, 300, 10)
>>> print repr(utils[0])
42.777972538362874
>>> type(utils[0])
<type 'numpy.float64'>
and here's why my f_wo_append2 function took 4 times longer:
>>> x = utils[0]
>>> type(x)
<type 'numpy.float64'>
>>> type(x >= 75)
<type 'numpy.bool_'> # iscust refers to a numpy.bool_
>>> type(0 + (x >= 75))
<type 'numpy.int32'> # numcustomers ends up referring to a numpy.int32
>>>
The empirical evidence is that these custom types aren't so fast when used as scalars ... perhaps because they need to reset the floating-point hardware each time they are used. OK for big arrays, not for scalars.
Are you using any other numpy functionality? If not, just use the random module. If you have other uses for numpy, you may wish to coerce the numpy.float64 to float during the population setup.
You can eliminate some lookups by using local function aliases:
def qtyDemanded(self, timePd, priceVector):
'''Returns quantity demanded in period timePd. In addition,
also updates the list of customers and non-customers.
Inputs: timePd and priceVector
Output: count of people for whom priceVector[-1] < utility
'''
price = priceVector[-1]
self.customers = []
self.nonCustomers = []
# local function aliases
addCust = self.customers.append
addNonCust = self.nonCustomers.append
for person in self.people:
if person.utility >= price:
person.customer = 1
addCust(person)
else:
person.customer = 0
addNonCust(person)
return len(self.customers)
This comment rings alarm bells:
'''Returns quantity demanded in period timePd. In addition,
also updates the list of customers and non-customers.
Aside from the fact that timePd is not used in the function, if you really want just to return the quantity, do just that in the function. Do the "in addition" stuff in a separate function.
Then profile again and see which of these two functions you are spending most of your time in.
I like to apply SRP to methods as well as classes: it makes them easier to test.
Depending on how often you add new elements to self.people or change person.utility, you could consider sorting self.people by the utility field.
Then you could use a bisect function to find the lower index i_pivot where the person[i_pivot].utility >= price condition is met. This would have a lower complexity ( O(log N) ) than your exhaustive loop ( O(N) )
With this information, you could then update your people list if needed :
Do you really need to update the utility field each time ? In the sorted case, you could easily deduce this value while iterating : for example, considering your list sorted in incresing order, utility = (index >= i_pivot)
Same question with customers and nonCustomers lists. Why do you need them? They could be replaced by slices of the original sorted list : for example, customers = self.people[0:i_pivot]
All this would allow you to reduce the complexity of your algorithm, and use more built-in (fast) Python functions, this could speedup your implementation.
Some curious things I noted:
timePd is passed as a parameter but never used
price is an array but you only ever use the last entry - why not pass the value there instead of passing the list?
count is initialized and never used
self.people contains multiple person objects which are then copied to either self.customers or self.noncustomers as well as having their customer flag set. Why not skip the copy operation and, on return, just iterate over the list, looking at the customer flag? This would save the expensive append.
Alternatively, try using psyco which can speed up pure Python, sometimes considerably.
It's surprising that the function shown is such a bottleneck because it's so relatively simple. For that reason, I'd double check my profiling procedure and results. However, if they're correct, the most time consuming part of your function has to be the for loop it contains, of course, so it makes sense to focus on speeding that up. One way to do this is by replacing the if/else with straight-line code. You can also reduce the attribute lookup for the append list method slightly. Here's how both of those things could be accomplished:
def qtyDemanded(self, timePd, priceVector):
'''Returns quantity demanded in period timePd. In addition,
also updates the list of customers and non-customers.
Inputs: timePd and priceVector
Output: count of people for whom priceVector[-1] < utility
'''
price = priceVector[-1] # last price
kinds = [[], []] # initialize sublists of noncustomers and customers
kindsAppend = [kinds[b].append for b in (False, True)] # append methods
for person in self.people:
person.customer = person.utility >= price # customer test
kindsAppend[person.customer](person) # add to proper list
self.nonCustomers = kinds[False]
self.customers = kinds[True]
return len(self.customers)
That said, I must add that it seems a little redundant to have both a customer flag in each person object and also put each of them into a separate list depending on that attribute. Not creating these two lists would of course speed the loop up further.
You're asking for guesses, and mostly you're getting guesses.
There's no need to guess. Here's an example.