Develop Reference

Adjusting nested apply() functions for large datasets

I have two dataframes that I'm trying to compare, and am facing a volume issue.
I am passing one row of a new item description through a 4.5 million row inventory list and calculating similarity. I only need the top x recommendations and am realizing my current approach quickly gets overwhelmed with the volume of data and is crashing the kernel.
I have not dealt with this data size before, so I am unsure how to adjust my code.
Any advice is greatly appreciated. The current approach was to put the data into the dataframe first(holding_df) and then groupby to collect the top recommendations, but once this process is scaled to the full size of the data, it crashes.
> df.head()
item_desc
0 paintbrush
1 mop #2
2 red bucket
3 o-light flashlight
> df_inventory.head()
item_desc
0 broom
1 mop
2 bucket
3 flashlight
import pandas as pd
from fuzzywuzzy import fuzz
def calculate_similarity(x, y):
sample_list.append(
{
"New Item": x,
"Inventory Item": y,
"Similarity": fuzz.ratio(str(x).lower(), str(y).lower()),
}
)
return
sample_list = []
df = pd.DataFrame(
{"ITEM_DESC": ["paintbrush", "mop #2", "red bucket", "o-light flashlight"]}
)
df_inventory = pd.DataFrame({"ITEM_DESC": ["broom", "mop", "bucket", "flashlight"]})
temp = df["ITEM_DESC"].apply(
lambda x: df_inventory["ITEM_DESC"].apply(lambda y: calculate_similarity(x, y))
)
holding_df = pd.DataFrame(sample_list)

I implemented something in plain Python that won't break your kernel, but it won't be super fast.
It takes about 6-7 seconds to compare a single new product with the whole inventory. That will probably be too slow for 3.5k items (about 6h and 20 min if I'd run it on my machine). With some work, it can be parallelized though.
6.5s per new item
3500 * 6.5 / 3600 (s/h) -> 6h 20min
The main memory-saver is the FixedSizeLeaderboard class that I implemented to keep track of the top n most similar items for a new product. As the task is now CPU-Bound and not really memory-bound, you can benefit from rewriting it a bit to use the multiprocessing module.
I decided to just generate some test data that may or may not represent actual performance. I added a few comments where you'd plug in your data.
import bisect
import collections
import contextlib
import itertools
import time
import typing
import uuid
from fuzzywuzzy import fuzz
#contextlib.contextmanager
def log_runtime(task: str):
"""Contextmanager that logs the runtime of a piece of code."""
start = time.perf_counter()
yield
runtime = time.perf_counter() - start
print("Task '%s' took %.4f seconds" % (task, runtime))
def inventory_generator() -> typing.Iterable[str]:
"""Returns an iterable that yields product names."""
def string_generator() -> typing.Iterable[str]:
while True:
yield str(uuid.uuid4())
yield from ("aaa", "aba", "def", "dse", "asd")
yield from string_generator()
class FixedSizeLeaderboard:
size: int
_min_score: int
_items: typing.List[typing.Tuple[int, object]]
def __init__(self, size) -> None:
self.size = size
self._items = []
self._min_score = None
def add(self, score: int, item: object) -> None:
if len(self._items) < self.size or score > self._min_score:
self._eject_element_with_lowest_score()
bisect.insort(self._items, (score, item))
self._min_score = self._items[0][0]
def _eject_element_with_lowest_score(self) -> None:
if len(self._items) == self.size:
# The list is sorted, so we can pop the first one
self._items.pop(0)
def get(self) -> typing.List[typing.Tuple[int, object]]:
return sorted(self._items, reverse=True)
def main():
num_new_products = 2
num_products_in_inventory = 4_500_000
top_n_similarities = 3
with log_runtime("Generate dummy-products"):
# Convert everything to lowercase once.
# This is not really required for uuids, but it should happen ONCE
# Instead of the inventory_generator, you'd pass the content of your dataframe here.
new_products = list(
map(str.lower, itertools.islice(inventory_generator(), num_new_products))
)
inventoried_products = list(
map(
str.lower,
itertools.islice(inventory_generator(), num_products_in_inventory),
)
)
task_desc = (
f"{num_new_products} x {num_products_in_inventory}"
f" = {num_new_products * num_products_in_inventory} similarity computations"
)
product_to_leaderboard: typing.Dict[
str, FixedSizeLeaderboard
] = collections.defaultdict(lambda: FixedSizeLeaderboard(top_n_similarities))
with log_runtime(task_desc):
for new_product, existing_product in itertools.product(
new_products, inventoried_products
):
similarity = fuzz.ratio(new_product, existing_product)
product_to_leaderboard[new_product].add(similarity, existing_product)
# Sort of pretty output formatting
for product, similarities in product_to_leaderboard.items():
print("=" * 3, "New Product", product, "=" * 3)
for position, (score, product) in enumerate(similarities.get()):
print(f"{position + 1:02}. score: {score} product: {product}")
if __name__ == "__main__":
main()
If we execute it, we get something like this:
$ python apply_thingy.py
Task 'Generate dummy-products' took 1.6449 seconds
Task '2 x 4500000 = 9000000 similarity computations' took 12.0887 seconds
=== New Product 2d10f990-355e-42f6-b518-0a21a7fb8d5c ===
01. score: 56 product: f2100878-3c3e-4f86-b410-3c362184d195
02. score: 56 product: 5fc9b30c-35ed-4167-b997-1bf0a2af5b68
03. score: 56 product: 523210b2-e5e0-496a-b0b1-a1b2af49b0d5
=== New Product aaa ===
01. score: 100 product: aaa
02. score: 100 product: aaa
03. score: 100 product: aaa

Multiprocessing makes the code slower than expected - using Cython and using vanilla Python

Got to give credit to NeuralNine video about Cython - from here I got the code.
Here I have a .pyx that contains two functions that are doing the same thing, which is given an integer amount returns a list of all the prime numbers below that number.
prime_finder_vanilla - Pure python
prime_finder_optimized - Cython
def prime_finder_vanilla(amount):
primes = []
found = 0
number = 2
while found < amount:
for x in primes:
if number % x == 0:
break
else:
primes.append(number)
found += 1
number += 1
return primes
def prime_finder_optimized(int amount):
cdef int number, x, found
cdef int primes[100_000]
amount = min(amount, 100_000)
found = 0
number = 2
while found < amount:
for x in primes[:found]:
if number % x == 0:
break
else:
primes[found] = number
found += 1
number += 1
return_list = [p for p in primes[:found]] # <--- ?! why not just return primes[:found] lol
return return_list
I've compiled the file with the command line:
python setup.py build_ext --inplace
setup.py
from setuptools import setup
from Cython.Build import cythonize
setup (
ext_modules=cythonize('testing_cython.pyx')
)
Now, I did run both functions, and it really seems that the Cython implementation is faster! but what I always like to do, is build a third part method that runs these functions for a given range, and calculates the average time it took to run these functions and get more accurate statistics of how optimized it really is.
So this is my "third party optimization analysis" file:
import time
from multiprocessing import Pool
import testing_cython
def check_performance(fn, tests_amount, tests_amount_start=1_000):
times = []
for amount in range(tests_amount_start, tests_amount_start + tests_amount + 1):
start = time.perf_counter()
fn(amount)
times.append(time.perf_counter() - start)
return sum(times) / len(times)
if __name__ == '__main__':
starting_time = time.perf_counter()
pool = Pool(processes=2)
res1 = pool.apply_async(check_performance, args=(testing_cython.prime_finder_vanilla, 300))
# res1 = res1.get()
res2 = pool.apply_async(check_performance, args=(testing_cython.prime_finder_optimized, 300))
# res2 = res2.get()
pool.close()
pool.join()
print(f"With multiprocessing: {time.perf_counter() - starting_time} seconds")
# print(f"{res1=}\n{res2=}\n{res1/res2=}")
print('-' * 30)
starting_time = time.perf_counter()
check_performance(testing_cython.prime_finder_vanilla, 300)
print(f"After vanilla sequentially: {time.perf_counter() - starting_time}")
check_performance(testing_cython.prime_finder_optimized, 300)
print(f"Without multiprocessing: {time.perf_counter() - starting_time} seconds")
Basically what it does is it takes any function fn, the amount of tests we want to do tests_amount, and the starting point to check the primes tests_amount_start, so if we give it tests_amount=5 and leave tests_amount_start as it is, it will return the average runtime of: fn(1000), f(1001), f(1002), f(1003), f(1004), f(1005)
Getting this result:
With multiprocessing: 5.528125900003943 seconds
------------------------------
After vanilla sequentially: 4.921810699997877
Without multiprocessing: 5.678506500000367 seconds
What I don't get is that simple math:
If sequentially the average running time of the vanilla is 4.9218 and the whole process sequentially took 5.678, this means that on average, the optimized version should take 5.678 - 4.921 = 0.757.
So, why is it with multiprocessing it takes 5.528 and not 4.921 - which is the running time of the heavy vanilla code? because the optimized version was running in parallel to the vanilla, it should be "included" in the total runtime... no?
What am I missing?

How is it possible python to be faster than pypy [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
I run a piece of code with python2.7 and the cProfile says 35s while cProfile on pypy says 73s ! How is that possible assuming pypy is faster interpreter. The code implements the BWT transformation on input a bitstream. I have two files: bwt.py which is called in fm.py. I called the function as:
pypy -m cProfle fm.py inputfile
and then
python -m cProfle fm.py inputfile
The code from bwt.py is the following:
def rotations(t):
''' Return list of rotations of input string t '''
tt = t * 2
return [ tt[i:i+len(t)] for i in xrange(0, len(t)) ]
def bwm(t):
return sorted(rotations(t))
def bwtViaBwm(t):
''' Given T, returns BWT(T) by way of the BWM '''
return ''.join(map(lambda x: x[-1], bwm(t)))
def rankBwt(bw):
''' Given BWT string bw, return parallel list of B-ranks. Also
returns tots: map from character to # times it appears. '''
tots = dict()
ranks = []
for c in bw:
if c not in tots: tots[c] = 0
ranks.append(tots[c])
tots[c] += 1
return ranks, tots
def firstCol(tots):
''' Return map from character to the range of rows prefixed by
the character. '''
first = {}
totc = 0
for c, count in sorted(tots.iteritems()):
first[c] = (totc, totc + count)
totc += count
return first
def reverseBwt(bw):
''' Make T from BWT(T) '''
ranks, tots = rankBwt(bw)
first = firstCol(tots)
rowi = 0 # start in first row
t = '$' # start with rightmost character
while bw[rowi] != '$':
c = bw[rowi]
t = c + t # prepend to answer
# jump to row that starts with c of same rank
rowi = first[c][0] + ranks[rowi]
return t
def suffixArray(s):
satups = sorted([(s[i:], i) for i in xrange(0, len(s))])
print satups
return map(lambda x: x[1], satups)
def bwtViaSa(t):
# Given T, returns BWT(T) by way of the suffix array
bw = []
for si in suffixArray(t):
if si == 0:
bw.append('$')
else:
bw.append(t[si-1])
return ''.join(bw) # return string-ized version of list bw
def readfile(sd):
s=""
with open(sd,'r') as myfile:
s =myfile.read()
return s.rstrip('\n')
def writefile(sd,N):
with open(sd, "wb") as sink:
sink.write(''.join(random.choice(string.ascii_uppercase + string.digits) for _ in xrange(N)))
sink.write('$')
return
def main():
data=readfile('inp')
b=bwtViaBwm(data)
ranks,tots = rankBwt(b)
print "Input stream = "+ data
print "BWT = " + bwtViaSa(data)
print '\n'.join(bwm(data))
print ("Lc ranking:")
print zip(b,ranks)
fc=[x[0] for x in bwm(data)]
fc= ''.join(fc)
print ("First column="+ fc)
ranks,tots = rankBwt(fc)
print("Fc ranking:")
print zip(fc,ranks)
print reverseBwt(bwtViaSa(data))
if __name__=='__main__':
main()
And this is the code form the fm.py which i call it through pypy:
import bwt
import sys
from collections import Counter
def build_FM(fname):
stream=bwt.readfile(fname)
#print bwt.suffixArray(stream)
b=bwt.bwtViaBwm(stream)
ranks,tots = bwt.rankBwt(b)
lc=zip(b,ranks)
fc=[x[0] for x in bwt.bwm(stream)]
fc= ''.join(fc)
fc= zip(fc,ranks)
#print lc,fc
def main():
fname= sys.argv[1]
build_FM(fname)
return
if __name__=='__main__':
main()

Pypy does not guarantee faster execution of a program.
Firstly, the optimizations it implements take time (sometimes a significant amount of time) to run. Secondly, not all code will run faster under pypy, though most does.
Further, the relative speed of profiled code may well differ hugely between them - pypy code is low-level, and so introducing profiling is likely to slow it down (relatively speaking) more than CPython. What are the results without profiling active?
We'd need to see your program in order to provide much more information that that.

Your script allocates an insane amount of memory in rotations() (O(N**2) where N is the size of the input file). As can be seen in cProfile and vmprof, most of the time is spent there.
What you're seeing is therefore differences in memory handling between PyPy and CPython. My guess is that you're swapping and that PyPy has a higher peak memory use.

Python not in dict condition sentence performance

Does anybody know about what is better to use thinking about speed and resources? Link to some trusted sources would be much appreciated.
if key not in dictionary.keys():
or
if not dictionary.get(key):

Firstly, you'd do
if key not in dictionary:
since dicts are iterated over by keys.
Secondly, the two statements are not equivalent - the second condition would be true if the corresponding values is falsy (0, "", [] etc.), not only if the key doesn't exist.
Lastly, the first method is definitely faster and more pythonic. Function/method calls are expensive. If you're unsure, timeit.

In my experience, using in is faster than using get, although the speed of get can be improved by caching the get method so it doesn't have to be looked up each time. Here are some timeit tests:
''' in vs get speed test
Comparing the speed of cache retrieval / update using `get` vs using `in`
http://stackoverflow.com/a/35451912/4014959
Written by PM 2Ring 2015.12.01
Updated for Python 3 2017.08.08
'''
from __future__ import print_function
from timeit import Timer
from random import randint
import dis
cache = {}
def get_cache(x):
''' retrieve / update cache using `get` '''
res = cache.get(x)
if res is None:
res = cache[x] = x
return res
def get_cache_defarg(x, get=cache.get):
''' retrieve / update cache using defarg `get` '''
res = get(x)
if res is None:
res = cache[x] = x
return res
def in_cache(x):
''' retrieve / update cache using `in` '''
if x in cache:
return cache[x]
else:
res = cache[x] = x
return res
#slow to fast.
funcs = (
get_cache,
get_cache_defarg,
in_cache,
)
def show_bytecode():
for func in funcs:
fname = func.__name__
print('\n%s' % fname)
dis.dis(func)
def time_test(reps, loops):
''' Print timing stats for all the functions '''
for func in funcs:
fname = func.__name__
print('\n%s: %s' % (fname, func.__doc__))
setup = 'from __main__ import data, ' + fname
cmd = 'for v in data: %s(v)' % (fname,)
times = []
t = Timer(cmd, setup)
for i in range(reps):
r = 0
for j in range(loops):
r += t.timeit(1)
cache.clear()
times.append(r)
times.sort()
print(times)
datasize = 1024
maxdata = 32
data = [randint(1, maxdata) for i in range(datasize)]
#show_bytecode()
time_test(3, 500)
typical output on my 2Ghz machine running Python 2.6.6:
get_cache: retrieve / update cache using `get`
[0.65624237060546875, 0.68499755859375, 0.76354193687438965]
get_cache_defarg: retrieve / update cache using defarg `get`
[0.54204297065734863, 0.55032730102539062, 0.56702113151550293]
in_cache: retrieve / update cache using `in`
[0.48754477500915527, 0.49125504493713379, 0.50087881088256836]

TLDR: Use if key not in dictionary. This is idiomatic, robust and fast.
There are four versions of relevance to this question: the 2 posed in the question, and the optimal variant of them:
key not in dictionary.keys() # inA
key not in dictionary # inB
not dictionary.get(key) # getA
sentinel = object()
dictionary.get(key, sentinel) is not sentinel # getB
Both A variants have shortcomings that mean you should not use them. inA needlessly creates a dict view on the keys - this adds an indirection step. getA looks at the truth of the value - this leads to incorrect results for values such as '' or 0.
As for using inB over getB: both do the same thing, namely looking at whether there is a value for key. However, getB also returns that value or default and has to compare it against the sentinel. Consequently, using get is considerably slower:
$ PREPARE="
> import random
> data = {a: True for a in range(0, 512, 2)}
> sentinel=object()"
$ python3 -m perf timeit -s "$PREPARE" '27 in data'
.....................
Mean +- std dev: 33.9 ns +- 0.8 ns
$ python3 -m perf timeit -s "$PREPARE" 'data.get(27, sentinel) is not sentinel'
.....................
Mean +- std dev: 105 ns +- 5 ns
Note that pypy3 has practically the same performance for both variants once the JIT has warmed up.

Ok, I've tested it on python 3.4.3 and all three ways give the same result around 0.00001 second.
import random
a = {}
for i in range(0, 1000000):
a[str(random.random())] = random.random()
import time
t1 = time.time(); 1 in a.keys(); t2 = time.time(); print("Time=%s" % (t2 - t1))
t1 = time.time(); 1 in a; t2 = time.time(); print("Time=%s" % (t2 - t1))
t1 = time.time(); not a.get(1); t2 = time.time(); print("Time=%s" % (t2 - t1))

Solution works for sample data but online judge gives errors?

This is the problem I am trying to solve:
B: The Foxen's Treasure
There are N (1 ≤ N ≤ 4) Foxen guarding a certain valuable treasure,
which you'd love to get your hands on. The problem is, the Foxen
certainly aren't about to allow that - at least, not while they're
awake.
Fortunately, through careful observation, you've seen that each Fox
has a regular sleep cycle. In particular, the ith Fox stays awake for
Ai (1 ≤ Ai ≤ 23) hours, then sleeps for Si (1 ≤ Si ≤ 23) hours,
repeating this pattern indefinitely (2 ≤ Ai + Si ≤ 24). At the start
of your treasure-nabbing attempt, the ith Fox is
exactly Oi (0 ≤ Oi < Ai + Si) hours into its cycle.
There are T (1 ≤ T ≤ 20) scenarios as described above. For each one,
you'd like to determine how soon all of the Foxen will be
simultaneously asleep, allowing you to grab their treasure, or if this
will simply never happen.
Input
Line 1: 1 integer, T
For each scenario:
Line 1: 1 integer, N
Next N lines: 3 integers, Ai, Si, and Oi, for i = 1..N
Output
For each scenario:
Line 1: 1 integer, the minimum number of hours after the start to
wait until all of the Foxen are asleep during the same hour. If this
will never happen, output the string "Foxen are too powerful" (without
quotes) instead.
Sample Input
2
2
2 1 2
2 2 1
3
1 1 0
1 1 0
1 1 1
Sample Output
6
Foxen are too powerful
My Solution works as expected when I input the given sample case and get expected output. But when I submit the code to online judge it gives clipped error. Now there is no detail of the error which makes it difficult to find what the problem is.
Here is the solution which I have worked so far:
# ai is awake hours
# si is sleep hours.
# ai + si <= 24.
# False == sleep. True == awake.
datasets = int(raw_input());
foxen = [];
number_of_foxen = 0;
foxes = [];
class fox:
def __init__(self, a, s, i):
self.awake = a;
self.sleep = s;
self.current = i;
awake = 0;
sleep = 0;
current = 0;
def next(self):
if ( self.sleep + self.awake-1 > self.current ) :
self.current = self.current+1;
else:
self.current = 0;
return self.current;
def check(self):
if(self.current>=self.awake):
return False;
return True;
def printdata(self):
print "awake="+str(self.awake)+" sleep="+str(self.sleep)+" current="+str(self.current);
#return "awake="+str(self.awake)+" sleep="+str(self.sleep)+" current="+str(self.current);
for i in range(0, datasets):
number_of_foxen = int(raw_input());
for j in range(0, number_of_foxen):
foxen.append(raw_input());
x = foxen[j].split();
a = fox(int(x[0]), int(x[1]), int(x[2]));
foxes.append(a);
solution = False;
for j in range(0, 48):
#print "hour number = " + str(j);
#for k in range(0, len(foxes)):
#print "fox number="+ str(k)+" "+ foxes[k].printdata()+str(foxes[k].check());
count = 0 ;
for k in range(0, len(foxes)):
if(foxes[k].check()==False):
count+=1;
#print "count = "+str(count);
#print len(foxes);
if( (int(count) == int(len(foxes))) and (solution == False) ):
#print "this runs now *************";
solution = True;
number = j;
for k in range(0, len(foxes)):
foxes[k].next();
if(solution==True):
print number;
else:
print "Foxen are too powerful";
#print "Foxen are too powerful";
foxen = [];
number_of_foxen = 0;
foxes = [];

The biggest problem with your code is that it is unreadable. Indeed, it looks like it was written with little concept of Python's strengths. Here is my suggestion:
#!/usr/bin/env python3
"""
The Foxen's Treasure puzzle from http://wcipeg.com/problem/acmtryouts1b
"""
from sys import stdin
from itertools import cycle
from euclid import lcm
debug = True # set to False before submission to mechanical judge
class Fox:
"""A Fox cointains its defining integers and other derived
bindings such as its cycle and schedule."""
def __init__(self, trio):
(self.awake_count, self.sleep_count, self.skip_count) = trio
self.cycle = 'a' * self.awake_count + 's' * self.sleep_count
self.schedule = cycle(self.cycle)
if debug: print('<Fox: {}> cycle {}'.format(trio, self.cycle))
# handle skips by discarding the first elements
for _ in range(self.skip_count):
next(self.schedule)
def find_all_sleeping(foxes):
"""Return an hour number if all foxes are sleeping at that hour."""
# only examine the LCM of all fox periods. If not there it will never be.
lcm_period = 1
for fox in foxes:
lcm_period = lcm(lcm_period, len(fox.cycle))
for hour in range(lcm_period):
states = [next(fox.schedule) for fox in foxes]
if debug: print('{:2d} {}'.format(hour, ' '.join(states)))
if 'a' not in states:
return hour
return None
def read_trials(fp):
"""Reads the entire input at once. Returns a list of trials.
Each trial is a list of Fox."""
trials = list()
trial_count = int(fp.readline())
for trial in range(trial_count):
if debug: print('--Read trial {}'.format(trial))
foxes = list()
fox_count = int(fp.readline())
for _ in range(fox_count):
fox = Fox([int(x) for x in fp.readline().split()])
foxes.append(fox)
trials.append(foxes)
return trials
for trial, foxes in enumerate(read_trials(stdin)):
if debug: print('--Run trial {}'.format(trial))
hour = find_all_sleeping(foxes)
if hour is None:
print('Foxen are too powerful')
else:
print(hour)
I suspect that the first concern is that it looks much longer than the OP; that is true, but if you take out the debugging code which shows how things are happening, and the docstrings that explain why it is doing things, it's actually a few line shorter than the OP.
The main loop of the OP is too long to understand without significant study, and a whole bunch of bad variable names makes that even harder. In contrast, there are places here where a value is given a name only to make the code more explicit about what an input line means. You'll find a number of
for _ in range(trial)
to show that the loop value is not used. This is a frequent idiom when dealing with fixed format input.
The Fox representation keeps the inner workings in the problem space. As noted in the exercise page, it makes more sense to look at things as a concurrence between sequences:
--Read trial 0
<Fox: [2, 1, 2]> cycle aas
<Fox: [2, 2, 1]> cycle aass
the offsets skip_count are not shown here, but they are clear in the trial run.
The input from the datafile is all kept inside read_trials() instead of scattered through the code. This confines the mess to one place rather than distributing it through the code. We know from the puzzle instructions that the datafile will not be large enough to care about. read_trials(fp) also takes a file-like object which allows it to read from an actual file, a StringIO buffer, or the standard input.
Once the Fox schedule generator is initialized, itertools.cycle will give an unending supply of the next letter in the sequence; it does the wrap-around for you.
It is worth noting that the primary data structure trials is a plain old list because it doesn't need anything more than that.
I've gotten a little weary of bad code being answered with worse code. Sure, this could be considered way more than the needs of an electronic judge where only the output matters. Contrariwise, I'm still puzzled by bits like (solution == False), a main loop that is 42 lines long and split between the top and bottom of the file, variables like i and j which convey no intent, the memory burden of False == awake (or did I mix them up?), dead code, no-op code, `range(0, n) and a whole bunch of magic numbers throughout.
Sure, you can code like the code doesn't matter, but if you are teaching yourself to code it is good to practice good practice. Yeah, you might never look at this piece of code again, but if you ain't gonna learn it now, then when?
In case you feel it a cheat to have imported lcm() there's no reason to write it a second time, so I referenced a homebrew package of which the relevant lines are:
def gcd(a, b):
"""Return the Greatest Common Divisor of a and b."""
while b:
a, b = b, a % b
return a
def lcm(a, b):
"""Return the Least Common Multiple of a and b."""
return abs(a * b) // gcd(a, b)

Jorge was correct in his comment, there doesn't appear to be any problem with your algorithm other than the arbitrary 48 hour cuttoff.
However:
1) your print statements do not use the correct syntax for Python 3+. For example, your final print statement print "Foxen are too powerful"; must be changed to work in Python 3, try print ('Foxen are too powerful') instead.
2) I'm seeing some odd C/MatLab-like syntax as well, lines being ended by a semicolon, and double brackets surrounding conditions in your if statements. This probably isn't a problem, but depending on how picky the system you are submitting the answer to is, you may want to clean it up a little.
3) Definitely increase the cutoff time for your search. I'd recommend a reasonably large value, on the order of 10,000 hours, just to be sure that it won't be a factor.
I've taken the liberty of making all of the above changes so I'm posting the resultant code now:
# ai is awake hours
# si is sleep hours.
# ai + si <= 24.
# False == sleep. True == awake.
datasets = int(raw_input())
foxen = []
number_of_foxen = 0
foxes = []
class fox:
def __init__(self, a, s, i):
self.awake = a
self.sleep = s
self.current = i
awake = 0
sleep = 0
current = 0
def next(self):
if ( self.sleep + self.awake-1 > self.current ):
self.current = self.current+1
else:
self.current = 0
return self.current
def check(self):
if(self.current>=self.awake):
return False
return True
def printdata(self):
print ("awake="+str(self.awake)+" sleep="+str(self.sleep)+" current="+str(self.current))
#return ("awake="+str(self.awake)+" sleep="+str(self.sleep)+" current="+str(self.current))
for i in range(0, datasets):
number_of_foxen = int(raw_input())
for j in range(0, number_of_foxen):
foxen.append(raw_input())
x = foxen[j].split()
a = fox(int(x[0]), int(x[1]), int(x[2]))
foxes.append(a)
solution = False
for j in range(0, 10000):
#print ("hour number = " + str(j))
#for k in range(0, len(foxes)):
#print ("fox number="+ str(k)+" "+ foxes[k].printdata()+str(foxes[k].check()))
count = 0
for k in range(0, len(foxes)):
if(foxes[k].check()==False):
count+=1
#print ("count = "+str(count))
#print (len(foxes))
if (int(count) == int(len(foxes)) and (solution == False)):
#print ("this runs now *************")
solution = True
number = j
for k in range(0, len(foxes)):
foxes[k].next()
if(solution == True):
print (number)
else:
print ("Foxen are too powerful")
#print ("Foxen are too powerful")
foxen = []
number_of_foxen = 0
foxes = []
Enjoy and Good Luck!

Interesting problem, here's my code:
import sys
# Globals
debugLevel = 0
fileMode = True # True if loading data from a file.
# Constants
AWAKE = 0
ASLEEP = -1
def gcd(a, b):
"""Return greatest common divisor using Euclid's Algorithm."""
while b:
a, b = b, a % b
return a
def lcm(a, b):
"""Return lowest common multiple."""
return a * b // gcd(a, b)
def readData(f):
''' Read in the problem data and store in data structures
'''
numTrials = int(f.readline().strip())
if debugLevel >= 4:
print("Num trials: ", numTrials)
trialData = []
for _ in range(numTrials):
numFoxen = int(f.readline().strip())
allFoxenHoursInfo = []
for _ in range(numFoxen):
aFoxHoursInfo = f.readline().split()
aFoxHoursInfo = list(map(int, aFoxHoursInfo))
allFoxenHoursInfo.append(aFoxHoursInfo)
trialData.append((numFoxen, allFoxenHoursInfo))
if debugLevel >= 8:
print("Trial data\n", trialData)
return numTrials, trialData
def runTrials(trialData):
'''
Go through each lot of foxen, and their sleep/awake schedules and
See if there's a time that all of them will be asleep.
'''
global debugLevel
for trial in trialData:
numFoxen, allFoxenHoursInfo = trial
# Create a table of the status of each fox in each hour
row = [AWAKE] * (numFoxen+1)
row[0] = 0
hoursTable = [row]
# Cycle length for each fox is the number of hours they spend awake then asleep.
cycleLength = [0] * (numFoxen)
# This is the number of hours into the cycle each fox is at the start
startingPosInCycle= [0] * (numFoxen)
# Initialise the first row
for fox in range(numFoxen):
cycleLength[fox] = allFoxenHoursInfo[fox][0] + allFoxenHoursInfo[fox][1] # Time awake plus time asleep
startingPosInCycle[fox] = allFoxenHoursInfo[fox][2] # % cycleLength[fox]
if startingPosInCycle[fox] >= allFoxenHoursInfo[fox][0]:
hoursTable[0][fox+1] = ASLEEP
if debugLevel >= 4:
print("Initial table: ", hoursTable)
# lcm = lowest common multiple and it's implemented above.
# For this problem, we only need to look at the lcm of all the cycle lengths for the foxen.
numIterations = 1
for fox in range(numFoxen):
numIterations = lcm(numIterations, cycleLength[fox])
# Go around a loop adding a new row to the table for each new hour containing the updated
# statuses of each fox.
for hourNum in range(1, numIterations):
allFoxesSleeping = False
# Update our hours table by creating a new row and calculating the status of each fox
newRow = [AWAKE] * (numFoxen+1)
newRow[0] = hourNum
for fox in range(numFoxen):
currentPosInCycle = (startingPosInCycle[fox] + hourNum) % cycleLength[fox]
if currentPosInCycle >= allFoxenHoursInfo[fox][0]:
newRow[fox+1] = ASLEEP
hoursTable.append(newRow)
if debugLevel >= 4:
print("Hours table\n", hoursTable)
# See if all foxen are sleeping, if they are, success
numFoxesSleeping = hoursTable[hourNum].count(ASLEEP)
if numFoxesSleeping == numFoxen:
allFoxesSleeping = True
print(hourNum)
break
if not allFoxesSleeping:
print('Foxen are too powerful')
def main():
'''Reads, runs, and outputs problem specific data.'''
# Initialisation
#strDir = ".\\"
# if fileMode:
# dataSource = open(strDir + "DataFile.txt", 'r')
# else:
dataSource = sys.stdin
# Read in the input data.
numTrials, trialData = readData(dataSource)
# Run each trial, outputting the result of that trial
runTrials(trialData)
sys.stdout.flush()
# Cleanup
# if fileMode:
# dataSource.close()
if __name__ == '__main__':
main()
Unfortunately, it does not pass the judge either. I have no idea why. I get this output:
Test case #1: WA [0.178s, 3628K] (0/1) (Details)
Your Output (clipped)
6
Foxen are too powe
Final score: 0/1
Interesting problem. I have emailed the author, because there is an inconsistency in the problem definition and the sample input data. He says this: Oi (0 ≤ Oi < Ai + Si) but then gives 1 1 1 on the last line of sample input data. And 1 is not strictly less than 1+1.
So who knows what other data the judge might be using...
The bit at the bottom that is commented out re files, lets me work with files and an IPython console rather than a Python console and pasting the data in, which I find slow and annoying.
Also, it's a little strange I reckon to not be able to see the data the judge is using. Surely seeing the data you are working against would enable the problem to be run and debugged offline, then when it's going, a new online submit could be done.