Develop Reference

HackerRank bitwiseAnd challenge - algorithm is too slow?

I am working on this code challenge on HackerRank: Day 29: Bitwise AND:
Task
Given set 𝑆={1,2,3,...,𝑁}. Find two integers, 𝐴 and 𝐵 (where 𝐴 < 𝐵), from set 𝑆 such that the value of 𝐴&𝐵 is the
maximum possible and also less than a given integer, 𝐾. In this case,
& represents the bitwise AND operator.
Function Description
Complete the bitwiseAnd function in the editor below.
bitwiseAnd has the following parameter(s):
int N: the maximum integer to consider
int K: the limit of the result, inclusive
Returns
int: the maximum value of 𝐴&𝐵 within the limit.
Input Format
The first line contains an integer, 𝑇, the number of test cases. Each
of the 𝑇 subsequent lines defines a test case as 2 space-separated
integers, 𝑁 and 𝐾, respectively.
Constraints
1 ≤ 𝑇 ≤ 103
2 ≤ 𝑁 ≤ 103
2 ≤ 𝐾 ≤ 𝑁
Sample Input
STDIN Function
----- --------
3 T = 3
5 2 N = 5, K = 2
8 5 N = 8, K = 5
2 2 N = 2, K = 2*
Sample Output
1
4
0
*At the time of writing the original question has an error here. Corrected in this copy.
I was not able to solve it using Python, as the time limit was exceeded every time, with a message telling me to optimise my code (some test cases had like 1000 requests).
I then tried writing the same code in C#, and it worked perfectly, executing like 10 times faster, even without any effort in optimizing the code.
Is it possible to further optimise the code below, for example with some magic that I don't know about?
Code:
def bitwiseAnd(N, K):
result = 0
for x in range(1, N):
for y in range(x+1, N+1):
if result < x&y < K:
result = x&y
return result
for x in range(int(input())):
print(bitwiseAnd(*[int(x) for x in input().split(' ')]))

Your algorithm has a brute force approach, but it can be done more efficiently.
First, observe some properties of this problem:
𝐴 & 𝐵 will never be greater than 𝐴 nor than 𝐵
If we think we have a solution 𝐶, then both 𝐴 and 𝐵 should have the same 1-bits as 𝐶 has, including possibly a few more.
We want 𝐴 and 𝐵 to not be greater than needed, since they need to be not greater than 𝑁, so given the previous point, we should let 𝐴 be equal to 𝐶, and let 𝐵 just have one 1-bit more than 𝐴 (since it should be a different number).
The least possible value for 𝐵 is then to set a 1-bit at the least bit in 𝐴 that is still 0.
If this 𝐵 is still not greater than 𝑁, then we can conclude that 𝐶 is a solution.
With the above steps in mind, it makes sense to first try with the greatest 𝐶 possible, i.e. 𝐶=𝐾−1, and then reduce 𝐶 until the above routine finds a 𝐵 that is not greater than 𝑁.
Here is the code for that:
def bitwiseAnd(N, K):
for A in range(K - 1, 0, -1):
# Find the least bit that has a zero in this number A:
# Use some "magic" to get the number with just one 1-bit in that position
bit = (A + 1) & -(A + 1)
B = A + bit
if B <= N:
# We know that A & B == B here, so just return A
return A
return 0

Efficient random generator for very large range (in python)

I am trying to create a generator that returns numbers in a given range that pass a particular test given by a function foo. However I would like the numbers to be tested in a random order. The following code will achieve this:
from random import shuffle
def MyGenerator(foo, num):
order = list(range(num))
shuffle(order)
for i in order:
if foo(i):
yield i
The Problem
The problem with this solution is that sometimes the range will be quite large (num might be of the order 10**8 and upwards). This function can become slow, having such a large list in memory. I have tried to avoid this problem, with the following code:
from random import randint
def MyGenerator(foo, num):
tried = set()
while len(tried) <= num - 1:
i = randint(0, num-1)
if i in tried:
continue
tried.add(i)
if foo(i):
yield i
This works well most of the time, since in most cases num will be quite large, foo will pass a reasonable number of numbers and the total number of times the __next__ method will be called will be relatively small (say, a maximum of 200 often much smaller). Therefore its reasonable likely we stumble upon a value that passes the foo test and the size of tried never gets large. (Even if it only passes 10% of the time, we wouldn't expect tried to get larger than about 2000 roughly.)
However, when num is small (close to the number of times that the __next__ method is called, or foo fails most of the time, the above solution becomes very inefficient - randomly guessing numbers until it guesses one that isn't in tried.
My attempted solution...
I was hoping to use some kind of function that maps the numbers 0,1,2,..., n onto themselves in a roughly random way. (This isn't being used for any security purposes and so doesn't matter if it isn't the most 'random' function in the world). The function here (Create a random bijective function which has same domain and range) maps signed 32-bit integers onto themselves, but I am not sure how to adapt the mapping to a smaller range. Given num I don't even need a bijection on 0,1,..num just a value of n larger than and 'close' to num (using whatever definition of close you see fit). Then I can do the following:
def mix_function_factory(num):
# something here???
def foo(index):
# something else here??
return foo
def MyGenerator(foo, num):
mix_function = mix_function_factory(num):
for i in range(num):
index = mix_function(i)
if index <= num:
if foo(index):
yield index
(so long as the bijection isn't on a set of numbers massively larger than num the number of times index <= num isn't True will be small).
My Question
Can you think of one of the following:
A potential solution for mix_function_factory or even a few other potential functions for mix_function that I could attempt to generalise for different values of num?
A better way of solving the original problem?
Many thanks in advance....

The problem is basically generating a random permutation of the integers in the range 0..n-1.
Luckily for us, these numbers have a very useful property: they all have a distinct value modulo n. If we can apply some mathemical operations to these numbers while taking care to keep each number distinct modulo n, it's easy to generate a permutation that appears random. And the best part is that we don't need any memory to keep track of numbers we've already generated, because each number is calculated with a simple formula.
Examples of operations we can perform on every number x in the range include:
Addition: We can add any integer c to x.
Multiplication: We can multiply x with any number m that shares no prime factors with n.
Applying just these two operations on the range 0..n-1 already gives quite satisfactory results:
>>> n = 7
>>> c = 1
>>> m = 3
>>> [((x+c) * m) % n for x in range(n)]
[3, 6, 2, 5, 1, 4, 0]
Looks random, doesn't it?
If we generate c and m from a random number, it'll actually be random, too. But keep in mind that there is no guarantee that this algorithm will generate all possible permutations, or that each permutation has the same probability of being generated.
Implementation
The difficult part about the implementation is really just generating a suitable random m. I used the prime factorization code from this answer to do so.
import random
# credit for prime factorization code goes
# to https://stackoverflow.com/a/17000452/1222951
def prime_factors(n):
gaps = [1,2,2,4,2,4,2,4,6,2,6]
length, cycle = 11, 3
f, fs, next_ = 2, [], 0
while f * f <= n:
while n % f == 0:
fs.append(f)
n /= f
f += gaps[next_]
next_ += 1
if next_ == length:
next_ = cycle
if n > 1: fs.append(n)
return fs
def generate_c_and_m(n, seed=None):
# we need to know n's prime factors to find a suitable multiplier m
p_factors = set(prime_factors(n))
def is_valid_multiplier(m):
# m must not share any prime factors with n
factors = prime_factors(m)
return not p_factors.intersection(factors)
# if no seed was given, generate random values for c and m
if seed is None:
c = random.randint(n)
m = random.randint(1, 2*n)
else:
c = seed
m = seed
# make sure m is valid
while not is_valid_multiplier(m):
m += 1
return c, m
Now that we can generate suitable values for c and m, creating the permutation is trivial:
def random_range(n, seed=None):
c, m = generate_c_and_m(n, seed)
for x in range(n):
yield ((x + c) * m) % n
And your generator function can be implemented as
def MyGenerator(foo, num):
for x in random_range(num):
if foo(x):
yield x

That may be a case where the best algorithm depends on the value of num, so why not using 2 selectable algorithms wrapped in one generator ?
you could mix your shuffle and set solutions with a threshold on the value of num. That's basically assembling your 2 first solutions in one generator:
from random import shuffle,randint
def MyGenerator(foo, num):
if num < 100000 # has to be adjusted by experiments
order = list(range(num))
shuffle(order)
for i in order:
if foo(i):
yield i
else: # big values, few collisions with random generator
tried = set()
while len(tried) < num:
i = randint(0, num-1)
if i in tried:
continue
tried.add(i)
if foo(i):
yield i
The randint solution (for big values of num) works well because there aren't so many repeats in the random generator.

Getting the best performance in Python is much trickier than in lower-level languages. For example, in C, you can often save a little bit in hot inner loops by replacing a multiplication by a shift. The overhead of python bytecode-orientation erases this. Of course, this changes again when you consider which variant of "python" you're targetting (pypy? numpy? cython?)- you really have to write your code based on which one you're using.
But even more important is arranging operations to avoid serialized dependencies, since all CPUs are superscalar these days. Of course, real compilers know about this, but it still matters when choosing an algorithm.
One of the easiest ways to gain a little bit over existing answers would be by by generating numbers in chunks using numpy.arange() and applying the ((x + c) * m) % n to the numpy ndarray directly. Every python-level loop that can be avoided helps.
If the function can be applied directly to numpy ndarrays, that might even better. Of course, a sufficiently-small function in python will be dominated by function-call overhead anyway.
The best fast random-number-generator today is PCG. I wrote a pure-python port here but concentrated on flexibility and ease-of-understanding rather than speed.
Xoroshiro128+ is second-best-quality and faster, but less informative to study.
Python's (and many others') default choice of Mersenne Twister is among the worst.
(there's also something called splitmix64 which I don't know enough about to place - some people say it's better than xoroshiro128+, but it has a period problem - of course, you might want that here)
Both default-PCG and xoroshiro128+ use a 2N-bit state to generate N-bit numbers. This is generally desirable, but means numbers will be repeated. PCG has alternate modes that avoid this, however.
Of course, much of this depends on whether num is (close to) a power of 2. In theory, PCG variants can be created for any bit width, but currently only various word sizes are implemented since you'd need explicit masking. I'm not sure exactly how to generate the parameters for new bit sizes (perhaps it's in the paper?), but they can be tested simply by doing a period/2 jump and verifying that the value is different.
Of course, if you're only making 200 calls to the RNG, you probably don't actually need to avoid duplicates on the math side.
Alternatively, you could use an LFSR, which does exist for every bit size (though note that it never generates the all-zeros value (or equivalently, the all-ones value)). LFSRs are serial and (AFAIK) not jumpable, and thus can't be easily split across multiple tasks. Edit: I figured out that this is untrue, simply represent the advance step as a matrix, and exponentiate it to jump.
Note that LFSRs do have the same obvious biases as simply generating numbers in sequential order based on a random start point - for example, if rng_outputs[a:b] all fail your foo function, then rng_outputs[b] will be much more likely as a first output regardless of starting point. PCG's "stream" parameter avoids this by not generating numbers in the same order.
Edit2: I have completed what I thought was a "brief project" implementing LFSRs in python, including jumping, fully tested.

Capturing all data in non-whole train, test, and validate splits

just wondering if a better solution exists for this sort of problem.
We know that for a X/Y percentage split of an even number we can get an exact split of the data - for example for data size 10:
10 * .6 = 6
10 * .4 = 4
10
Splitting data this way is easy, and we can guarantee we have all of the data and nothing is lost. However where I am struggling is on less friendly numbers - take 11
11 * .6 = 6.6
11 * .4 = 4.4
11
However we can't index into an array at i = 6.6 for example. So we have to decide how to to do this. If we take JUST the integer portion we lose 1 data point -
First set = 0..6
Second set = 6..10
This would be the same case if we floored the numbers.
However, if we take the ceiling of the numbers:
First set = 0..7
Second set = 7..12
And we've read past the end of our array.
This gets even worse when we throw in a 3rd or 4th split (30,30,20,20 for example).
Is there a standard splitting procedure for these kinds of problems? Is data loss accepted? It seems like data loss would be unacceptable for dependent data, such as time series.
Thanks!
EDIT: The values .6 and .4 are chosen by me. They could be any two numbers that sum to 1.

First of all, notice that your problem is not limited to odd-sized arrays as you claim, but any-sized arrays. How would you make the 56%-44% split of a 10 element array? Or a 60%-40% split of a 4 element array?
There is no standard procedure. In many cases, programmers do not care that much about an exact split and they either do it by flooring or rounding one quantity (the size of the first set), while taking the complementary (array length - rounded size) for the other (the size of the second).
This might be ok in most cases when this is an one-off calculation and accuracy is not required. You have to ask yourself what your requirements are. For example: are you taking thousands of 10-sized arrays and each time you are splitting them 56%-44% doing some calculations and returning a result? You have to ask yourself what accuracy do you want. Do you care if your result ends up being
the 60%-40% split or the 50%-50% split?
As another example imagine that you are doing a 4-way equal split of 25%-25%-25%-25%. If you have 10 elements and you apply the rounding technique you end up with 3,3,3,1 elements. Surely this will mess up your results.
If you do care about all these inaccuracies then the first step is consider whether you can to adjust either the array size and/or the split ratio(s).
If these are set in stone then the only way to have an accurate split of any ratios of any sized array is to make it probabilistic. You have to split multiple arrays for this to work (meaning you have to apply the same split ratio to same-sized arrays multiple times). The more arrays the better (or you can use the same array multiple times).
So imagine that you have to make a 56%-44% split of a 10 sized array. This means that you need to split it in 5.6 elements and 4.4 elements on the average.
There are many ways you can achieve a 5.6 element average. The easiest one (and the one with the smallest variance in the sequence of tries) is to have 60% of the time a set with 6 elements and 40% of the time a set that has 5 elements.
0.6*6 + 0.4*5 = 5.6
In terms of code this is what you can do to decide on the size of the set each time:
import random
array_size = 10
first_split = 0.56
avg_split_size = array_size * first_split
floored_split_size = int(avg_split_size)
if avg_split_size > floored_split_size:
if random.uniform(0,1) > avg_split_size - floored_split_size:
this_split_size = floored_split_size
else:
this_split_size = floored_split_size + 1
else:
this_split_size = avg_split_size
You could make the code more compact, I just made an outline here so you get the idea. I hope this helps.

Instead of using ciel() or floor() use round() instead. For example:
>>> round(6.6)
7.0
The value returned will be of float type. For getting the integer value, type-cast it to int as:
>>> int(round(6.6))
7
This will be the value of your first split. For getting the second split, calculate it using len(data) - split1_val. This will be applicable in case of 2 split problem.
In case of 3 split, take round value of two split and take the value of 3rd split as the value of len(my_list) - val_split_1 - val_split2
In a Generic way, For N split:
Take the round() value of N-1 split. And for the last value, do len(data) - "value of N round() values".
where len() gives the length of the list.

Let's first consider just splitting the set into two pieces.
Let n be the number of elements we are splitting, and p and q be the proportions, so that
p+q == 1
I assert that the parts after the decimal point will always sum to either 1 or 0, so we should use floor on one and ceil on the other, and we will always be right.
Here is a function that does that, along with a test. I left the print statements in but they are commented out.
def simpleSplitN(n, p, q):
"split n into proportions p and q and return indices"
np = math.ceil(n*p)
nq = math.floor(n*q)
#print n, sum([np, nq]) #np and nq are the proportions
return [0, np] #these are the indices we would use
#test for simpleSplitN
for i in range(1, 10):
p = i/10.0;
q = 1-p
simpleSplitN(37, p, q);
For the mathematically inclined, here is the proof that the decimal proportions will sum to 1
-----------------------
We can express p*n as n/(1/p), and so by the division algorithm we get integers k and r
n == k*(1/p) + r with 0 <= r < (1/p)
Thus r/(1/p) == p*r < 1
We can do exactly the same for q, getting
q*r < 1 (this is a different r)
It is important to note that q*r and p*r are the part after the decimal when we divide our n.
Now we can add them together (we've added subscripts now)
0 <= p*(r_1) < 1
0 <= q*(r_2) < 1
=> 0 < p*r + q*r == p*n + q*n + k_1 + k_2 == n + k_1 + k_2 < 2
But by closure of the integers, n + k_1 + k_2 is an integer and so
0 < n + k_1 + k_2 < 2
means that p*r + q*r must be either 0 or 1. It will only be 0 in the case that our n is divided evenly.
Otherwise we can now see that our fractional parts will always sum to 1.
-----------------------
We can do a very similar (but slightly more complicated) proof for splitting n into an arbitrary number (say N) parts, but instead of them summing to 1, they will sum to an integer less than N.
Here is the general function, it has uncommented print statements for verification purposes.
import math
import random
def splitN(n, c):
"""Compute indices that can be used to split
a dataset of n items into a list of proportions c
by first dividing them naively and then distributing
the decimal parts of said division randomly
"""
nc = [n*i for i in c];
nr = [n*i - int(n*i) for i in c] #the decimal parts
N = int(round(sum(nr))) #sum of all decimal parts
print N, nc
for i in range(0, len(nc)):
nc[i] = math.floor(nc[i])
for i in range(N): #randomly distribute leftovers
nc[random.randint(1, len(nc)) - 1] += 1
print n,sum(nc); #nc now contains the proportions
out = [0] #compute a cumulative sum
for i in range(0, len(nc) - 1):
out.append(out[-1] + nc[i])
print out
return out
#test for splitN with various proportions
c = [.1,.2,.3,.4]
c = [.2,.2,.2,.2,.2]
c = [.3, .2, .2, .3]
for n in range( 10, 40 ):
print splitN(n, c)
If we have leftovers, we will never get an even split, so we distribute them randomly, like #Thanassis said. If you don't like the dependency on random, then you could just add them all at the beginning or at even intervals.
Both of my functions output indices but they compute proportions and thus could be slightly modified to output those instead per user preference.

Unlucky number 13

I came across this problem Unlucky number 13! recently but could not think of efficient solution this.
Problem statement :
N is taken as input.
N can be very large 0<= N <= 1000000009
Find total number of such strings that are made of exactly N characters which don't include "13". The strings may contain any integer from 0-9, repeated any number of times.
# Example:
# N = 2 :
# output : 99 (0-99 without 13 number)
# N =1 :
# output : 10 (0-9 without 13 number)
My solution:
N = int(raw_input())
if N < 2:
print 10
else:
without_13 = 10
for i in range(10, int('9' * N)+1):
string = str(i)
if string.count("13") >= 1:
continue
without_13 += 1
print without_13
Output
The output file should contain answer to each query in a new line modulo 1000000009.
Any other efficient way to solve this ? My solution gives time limit exceeded on coding site.

I think this can be solved via recursion:
ans(n) = { ans([n/2])^2 - ans([n/2]-1)^2 }, if n is even
ans(n) = { ans([n/2]+1)*ans([n/2]) - ans([n/2])*ans([n/2]-1) }, if n is odd
Base Cases:
ans(0) = 1
ans(1) = 10
It's implementation is running quite fast even for larger inputs like 10^9 ( which is expected as its complexity is O(log[n]) instead of O(n) like the other answers ):
cache = {}
mod = 1000000009
def ans(n):
if cache.has_key(n):
return cache[n]
if n == 0:
cache[n] = 1
return cache[n]
if n == 1:
cache[n] = 10
return cache[n]
temp1 = ans(n/2)
temp2 = ans(n/2-1)
if (n & 1) == 0:
cache[n] = (temp1*temp1 - temp2*temp2) % mod
else:
temp3 = ans(n/2 + 1)
cache[n] = (temp1 * (temp3 - temp2)) % mod
return cache[n]
print ans(1000000000)
Online Demo
Explanation:
Let a string s have even number of digits 'n'.
Let ans(n) be the answer for the input n, i.e. the number of strings without the substring 13 in them.
Therefore, the answer for string s having length n can be written as the multiplication of the answer for the first half of the string (ans([n/2])) and the answer for the second half of the string (ans([n/2])), minus the number of cases where the string 13 appears in the middle of the number n, i.e. when the last digit of the first half is 1 and the first digit of the second half is 3.
This can expressed mathematically as:
ans(n) = ans([n/2])^2 - ans([n/2]-1)*2
Similarly for the cases where the input number n is odd, we can derive the following equation:
ans(n) = ans([n/2]+1)*ans([n/2]) - ans([n/2])*ans([n/2]-1)

I get the feeling that this question is designed with the expectation that you would initially instinctively do it the way you have. However, I believe there's a slightly different approach that would be faster.
You can produce all the numbers that contain the number 13 yourself, without having to loop through all the numbers in between. For example:
2 digits:
13
3 digits position 1:
113
213
313 etc.
3 digits position 2: 131
132
133 etc.
Therefore, you don't have to check all the number from 0 to n*9. You simply count all the numbers with 13 in them until the length is larger than N.
This may not be the fastest solution (in fact I'd be surprised if this couldn't be solved efficiently by using some mathematics trickery) but I believe it will be more efficient than the approach you have currently taken.

This a P&C problem. I'm going to assume 0 is valid string and so is 00, 000 and so on, each being treated distinct from the other.
The total number of strings not containing 13, of length N, is unsurprisingly given by:
(Total Number of strings of length N) - (Total number of strings of length N that have 13 in them)
Now, the Total number of strings of length N is easy, you have 10 digits and N slots to put them in: 10^N.
The number of strings of length N with 13 in them is a little trickier.
You'd think you can do something like this:
=> (N-1)C1 * 10^(N-2)
=> (N-1) * 10^(N-2)
But you'd be wrong, or more accurately, you'd be over counting certain strings. For example, you'd be over counting the set of string that have two or more 13s in them.
What you really need to do is apply the inclusion-exclusion principle to count the number of strings with 13 in them, so that they're all included once.
If you look at this problem as a set counting problem, you have quite a few sets:
S(0,N): Set of all strings of Length N.
S(1,N): Set of all strings of Length N, with at least one '13' in it.
S(2,N): Set of all strings of Length N, with at least two '13's in it.
...
S(N/2,N): Set of all strings of Length N, with at least floor(N/2) '13's in it.
You want the set of all strings with 13 in them, but counted at most once. You can use the inclusion-exclusion principle for computing that set.

Let f(n) be the number of sequences of length n that have no "13" in them, and g(n) be the number of sequences of length n that have "13" in them.
Then f(n) = 10^n - g(n) (in mathematical notation), because it's the number of possible sequences (10^n) minus the ones that contain "13".
Base cases:
f(0) = 1
g(0) = 0
f(1) = 10
g(1) = 0
When looking for the sequences with "13", a sequence can have a "13" at the beginning. That will account for 10^(n-2) possible sequences with "13" in them. It could also have a "13" in the second position, again accounting for 10^(n-2) possible sequences. But if it has a "13" in the third position, and we'd assume there would also be 10^(n-2) possible sequences, we could those twice that already had a "13" in the first position. So we have to substract them. Instead, we count 10^(n-4) times f(2) (because those are exactly the combinations in the first two positions that don't have "13" in them).
E.g. for g(5):
g(5) = 10^(n-2) + 10^(n-2) + f(2)*10^(n-4) + f(3)*10^(n-5)
We can rewrite that to look the same everywhere:
g(5) = f(0)*10^(n-2) + f(1)*10^(n-3) + f(2)*10^(n-4) + f(3)*10^(n-5)
Or simply the sum of f(i)*10^(n-(i+2)) with i ranging from 0 to n-2.
In Python:
from functools import lru_cache
#lru_cache(maxsize=1024)
def f(n):
return 10**n - g(n)
#lru_cache(maxsize=1024)
def g(n):
return sum(f(i)*10**(n-(i+2)) for i in range(n-1)) # range is exclusive
The lru_cache is optional, but often a good idea when working with recursion.
>>> [f(n) for n in range(10)]
[1, 10, 99, 980, 9701, 96030, 950599, 9409960, 93149001, 922080050]
The results are instant and it works for very large numbers.

In fact this question is more about math than about python.
For N figures there is 10^N possible unique strings. To get the answer to the problem we need to subtract the number of string containing "13".
If string starts from "13" we have 10^(N-2) possible unique strings. If we have 13 at the second possition (e.i. a string like x13...), we again have 10^(N-2) possibilities. But we can't continue this logic further as this will lead us to double calculation of string which have 13 at different possitions. For example for N=4 there will be a string "1313" which we will calculate twice. To avoid this we should calculate only those strings which we haven't calculated before. So for "13" on possition p (counting from 0) we should find the number of unique string which don't have "13" on the left side from p, that is for each p
number_of_strings_for_13_at_p = total_number_of_strings_without_13(N=p-1) * 10^(N-p-2)
So we recursevily define the total_number_of_strings_without_13 function.
Here is the idea in the code:
def number_of_strings_without_13(N):
sum_numbers_with_13 = 0
for p in range(N-1):
if p < 2:
sum_numbers_with_13 += 10**(N-2)
else:
sum_numbers_with_13 += number_of_strings_without_13(p) * 10**(N-p-2)
return 10**N - sum_numbers_with_13
I should say that 10**N means 10 in the power of N. All the other is described above. The functions also has a surprisingly pleasent ability to give correct answers for N=1 and N=2.
To test this works correct I've rewritten your code into function and refactored a little bit:
def number_of_strings_without_13_bruteforce(N):
without_13 = 0
for i in range(10**N):
if str(i).count("13"):
continue
without_13 += 1
return without_13
for N in range(1, 7):
print(number_of_strings_without_13(N),
number_of_strings_without_13_bruteforce(N))
They gave the same answers. With bigger N bruteforce is very slow. But for very large N recursive function also gets mush slower. There is a well known solution for that: as we use the value of number_of_strings_without_13 with parameters smaller than N multiple times, we should remember the answers and not recalculate them each time. It's quite simple to do like this:
def number_of_strings_without_13(N, answers=dict()):
if N in answers:
return answers[N]
sum_numbers_with_13 = 0
for p in range(N-1):
if p < 2:
sum_numbers_with_13 += 10**(N-2)
else:
sum_numbers_with_13 += number_of_strings_without_13(p) * 10**(N-p-2)
result = 10**N - sum_numbers_with_13
answers[N] = result
return result

Thanks to L3viathan's comment now it is clear. The logic is beautiful.
Let's assume a(n) is a number of strings of n digits without "13" in it. If we know all the good strings for n-1, we can add one more digit to the left of each string and calculate a(n). As we can combine previous digits with any of 10 new, we will get 10*a(n-1) different strings. But we must subtract the number of strings, which now starts with "13" which we wrongly summed like OK at the previous step. There is a(n-2) of such wrongly added strings. So a(n) = 10*a(n-1) - a(n-2). That is it. Such simple.
What is even more interesting is that this sequence can be calculated without iterations with a formula https://oeis.org/A004189 But practically that doesn't helps much, as the formula requires floating point calculations which will lead to rounding and would not work for big n (will give answer with some mistake).
Nevertheless the original sequence is quite easy to calculate and it doesn't need to store all the previous values, just the last two. So here is the code
def number_of_strings(n):
result = 0
result1 = 99
result2 = 10
if n == 1:
return result2
if n == 2:
return result1
for i in range(3, n+1):
result = 10*result1 - result2
result2 = result1
result1 = result
return result
This one is several orders faster than my previous suggestion. And memory consumption is now just O(n)
P.S. If you run this with Python2, you'd better change range to xrange

This python3 solution meets time and memory requirement of HackerEarth
from functools import lru_cache
mod = 1000000009
#lru_cache(1024)
def ans(n):
if n == 0:
return 1
if n == 1:
return 10
temp1 = ans(n//2)
temp2 = ans(n//2-1)
if (n & 1) == 0:
return (temp1*temp1 - temp2*temp2) % mod
else:
temp3 = ans(n//2 + 1)
return (temp1 * (temp3 - temp2)) % mod
for t in range(int(input())):
n = int(input())
print(ans(n))

I came across this problem on
https://www.hackerearth.com/problem/algorithm/the-unlucky-13-d7aea1ff/
I haven't been able to get the judge to accept my solution(s) in Python but (2) in ANSI C worked just fine.
Straightforward recursive counting of a(n) = 10*a(n-1) - a(n-2) is pretty slow when getting to large numbers but there are several options (one which is not mentioned here yet):
1.) using generating functions:
https://www.wolframalpha.com/input/?i=g%28n%2B1%29%3D10g%28n%29+-g%28n-1%29%2C+g%280%29%3D1%2C+g%281%29%3D10
the powers should be counted using squaring and modulo needs to be inserted cleverly into that and the numbers must be rounded but Python solution was slow for the judge anyway (it took 7s on my laptop and judge needs this to be counted under 1.5s)
2.) using matrices:
the idea is that we can get vector [a(n), a(n-1)] by multiplying vector [a(n-1), a(n-2)] by specific matrix constructed from equation a(n) = 10*a(n-1) - a(n-2)
| a(n) | = | 10 -1 | * | a(n-1) |
| a(n-1) | | 1 0 | | a(n-2) |
and by induction:
| a(n) | = | 10 -1 |^(n-1) * | a(1) |
| a(n-1) | | 1 0 | | a(0) |
the matrix multiplication in 2D should be done via squaring using modulo. It should be hardcoded rather counted via for cycles as it is much faster.
Again this was slow for Python (8s on my laptop) but fast for ANSI C (0.3s)
3.) the solution proposed by Anmol Singh Jaggi above which is the fastest in Python (3s) but the memory consumption for cache is big enough to break memory limits of the judge. Removing cache or limiting it makes the computation very slow.

You are given a string S of length N. The string S consists of digits from 1-9, Consider the string indexing to be 1-based.
You need to divide the string into blocks such that the i block contains the elements from the index((i 1) • X +1) to min(N, (i + X)) (both inclusive). A number is valid if it is formed by choosing exactly one digit from each block and placing the digits in the order of their block
number

sum of even fibonacci numbers up to 4million

The method I've used to try and solve this works but I don't think it's very efficient because as soon as I enter a number that is too large it doesn't work.
def fib_even(n):
fib_even = []
a, b = 0, 1
for i in range(0,n):
c = a+b
if c%2 == 0:
fib_even.append(c)
a, b = b, a+b
return fib_even
def sum_fib_even(n):
fib_evens = fib_even(n)
s = 0
for i in fib_evens:
s = s+i
return s
n = 4000000
answer = sum_fib_even(n)
print answer
This for example doesn't work for 4000000 but will work for 400. Is there a more efficient way of doing this?

It is not necessary to compute all the Fibonacci numbers.
Note: I use in what follows the more standard initial values F[0]=0, F[1]=1 for the Fibonacci sequence. Project Euler #2 starts its sequence with F[2]=1,F[3]=2,F[4]=3,.... For this problem the result is the same for either choice.
Summation of all Fibonacci numbers (as a warm-up)
The recursion equation
F[n+1] = F[n] + F[n-1]
can also be read as
F[n-1] = F[n+1] - F[n]
or
F[n] = F[n+2] - F[n+1]
Summing this up for n from 1 to N (remember F[0]=0, F[1]=1) gives on the left the sum of Fibonacci numbers, and on the right a telescoping sum where all of the inner terms cancel
sum(n=1 to N) F[n] = (F[3]-F[2]) + (F[4]-F[3]) + (F[5]-F[4])
+ ... + (F[N+2]-F[N+1])
= F[N+2] - F[2]
So for the sum using the number N=4,000,000 of the question one would have just to compute
F[4,000,002] - 1
with one of the superfast methods for the computation of single Fibonacci numbers. Either halving-and-squaring, equivalent to exponentiation of the iteration matrix, or the exponential formula based on the golden ratio (computed in the necessary precision).
Since about every 20 Fibonacci numbers you gain 4 additional digits, the final result will consist of about 800000 digits. Better use a data type that can contain all of them.
Summation of the even Fibonacci numbers
Just inspecting the first 10 or 20 Fibonacci numbers reveals that all even members have an index of 3*k. Check by subtracting two successive recursions to get
F[n+3]=2*F[n+2]-F[n]
so F[n+3] always has the same parity as F[n]. Investing more computation one finds a recursion for members three indices apart as
F[n+3] = 4*F[n] + F[n-3]
Setting
S = sum(k=1 to K) F[3*k]
and summing the recursion over n=3*k gives
F[3*K+3]+S-F[3] = 4*S + (-F[3*K]+S+F[0])
or
4*S = (F[3*K]+F[3*K]) - (F[3]+F[0]) = 2*F[3*K+2]-2*F[2]
So the desired sum has the formula
S = (F[3*K+2]-1)/2
A quick calculation with the golden ration formula reveals what N should be so that F[N] is just below the boundary, and thus what K=N div 3 should be,
N = Floor( log( sqrt(5)*Max )/log( 0.5*(1+sqrt(5)) ) )
Reduction of the Euler problem to a simple formula
In the original problem, one finds that N=33 and thus the sum is
S = (F[35]-1)/2;
Reduction of the problem in the question and consequences
Taken the mis-represented problem in the question, N=4,000,000, so K=1,333,333 and the sum is
(F[1,333,335]-1)/2
which still has about 533,400 digits. And yes, biginteger types can handle such numbers, it just takes time to compute with them.
If printed in the format of 60 lines a 80 digits, this number fills 112 sheets of paper, just to get the idea what the output would look like.

It should not be necessary to store all intermediate Fibonacci numbers, perhaps the storage causes a performance problem.