Develop Reference

Searchlight Inefficient Compression Scheme

Write a compression and decompression algorithm for SICS which works as follows: Find the most popular characters, and in order of popularity assign them a hex value 0 to E, F0 to FE, FF0 to FFE, etc. Note, an F indicates that there are more nibbles to follow, anything else is the terminal nibble.
Compress the message by replacing characters with their assigned value. Below are the sample text, but the code should work universally for any given text.
Test case:
text = "Marley was dead: to begin with. There is no doubt whatever about that. The register of his burial was signed by the clergyman, the clerk, the undertaker, and the chief mourner. Scrooge signed it: and Scrooge’s name was good upon ’Change, for anything he chose to put his hand to. Old Marley was as dead as a door-nail. Mind! I don’t mean to say that I know, of my own knowledge, what there is particularly dead about a door-nail. I might have been inclined, myself, to regard a coffin-nail as the deadest piece of ironmongery in the trade. But the wisdom of our ancestors is in the simile; and my unhallowed hands shall not disturb it, or the Country’s done for. You will therefore permit me to repeat, emphatically, that Marley was as dead as a door-nail."
solution = "f826b1d0e2a08128fd0340f61f0750e739f50fe916107a054084cf630e9231ff01602f64c303923f5 0fe91061f07a31604f3097a0f6c672b0e2a0a7f05180f6d03910f2b16f0df125f403910f2b16f9f4039 10c581632f916f4025803910f2971f30f14c6516f50ff1f2644f010a7f0518073fd02580ff1f2644f01fa a052f110e2a0f04480cf7450faff2925f01f40f346025d3975f00910f294a10340f7c3097a09258034f 50ff3b80f826b1d0e2a02a0812802a0208446fb527bf50f8758ff40fc0845fa30f11250340a2d03923 0fc0f954ef404f30f1d04e50f954eb18f01f40e92303916107a0f72637f2cb26bd0812802f64c30208 446fb527bf50fc0f17f093092ff010f6115075f2b7518f40f1da1bf3f4034061f0268020f24f3f375fb52 7b02a0391081281a30f771f2104f307645f145f016d0750391036281f50ff5c303910e7a84f104f30 4c6025f21a346a07a07503910a7f17b1ff602580f1d0c592bb4e1809258a0a92bb0543087a3c6f6 073f404603910ff24c536dfaa084510f346f50ff74c0e7bb039161f34610f716f1730f11034061f7123 f401f1f79237f22bbdf4039230f826b1d0e2a02a0812802a0208446fb527bf5

I am adding a code that can help to solve your problem. It will ask for raw input and you can modify it as per your need.
from collections import Counter
key = []
def initCompute(data): # here i am finding the unique chars and its occurrence to find the popular char
uChar = list(set(data))
xCount = dict(Counter(data))
xCount = dict(sorted(xCount.items(), key=lambda item: item[1], reverse=True))
return len(uChar), xCount
def compute(data, keyList, dataDict): # assigning the hex value(key) to the each characters
j = 0
comDict = {}
decdict = {}
sol = ""
for k in dataDict:
comDict[k] = [keyList[j], dataDict[k]]
decdict[keyList[j]] = k
j += 1
for c in data:
sol += comDict[c][0]
return sol, decdict
def decompression(keyDict,
cData): # this is to find the decompressed data by having the dict of
# hex value assigned to char and compressed data as inputs
sol = ""
fNib = ""
for s in cData:
if s == 'f':
fNib += s
else:
fNib += s
sol += keyDict[fNib]
fNib = ""
# print(sol)
return sol
def compression(): # find the key(hex value) and framing the compressed data
i = 0
fac = 16
pwr = 1
keyLen, dDict = initCompute(text)
while len(key) <= keyLen:
if i <= (fac - 2):
key.append(str(hex(i))[2:]) # finding the hex value and storing in key
i += 1
else:
pwr += 1
fac = pow(16, pwr)
i = fac - 16
sol, newDict = compute(text, key, dDict)
print("Assigned hex values for each character: ")
print(newDict)
return sol, newDict, keyLen
if __name__ == '__main__':
text = input("Input your Data here: ") # input
compressed_data, new_dict, key_len = compression()
print("Compressed data: ",compressed_data)
print(compressed_data)
print("Decompressed data:")
print(decompression(new_dict, compressed_data))

How Does The Base64 Work and How To Encode/Decode in it?

I have a problem that asks me to encode a string to base64 format! I think I got it based on my code! The string: "Man" works and other short strings work. But the string: "this is a string!!" doesn't work! And also I want to use the non-padding version. The questions asks me to use the non-padding version. Can you please explain the process of how to encode this string: "this is a string!!"! I have to turn the letters to ascii, and then turn them into binary and divide them into 6 bytes and then turn them to decimal and refer to a chart of ascii and then use them. This is all I know! But, please don't give me the code. I want to try out the coding on my own. But please explain the process. There are no good videos explaining this topic! And by the way, I am using python Thank you
Here is the code I have:
def decimal(binary):
binary = str(binary); power = len(binary)-1
values = []
for x in binary:
if x == "1":
values.append((x, 2**power))
power -= 1
return sum([v for b,v in values if b == "1"])
string = "Man"
byte = ""
for x in string:
byte += bin(ord(x))[0] + bin(ord(x))[2:]
values = []
for x in range(0, len(byte), 6):
values.append(byte[x:x+6])
abc = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"
table = {x:abc[x] for x in range(len(abc))}
print("".join(table[decimal(x)] for x in values))
I am using python!

Adjusted parts are explained using in-line comments:
import sys # merely for manipulation with supplied arguments
import math
if len(sys.argv) == 1:
string = "This is a string!!!"
else:
string = ' '.join([sys.argv[i] for i in range(1,len(sys.argv))])
def decimal(binary):
binary = str(binary); power = len(binary)-1
values = []
for x in binary:
if x == "1":
values.append((x, 2**power))
power -= 1
return sum([v for b,v in values if b == "1"])
byte = ""
for x in string.encode('utf-8'): # ASCII is a proper subset of UTF-8
byte += bin(x)[2:].rjust(8,'0') # get binary string of length 8
byte = byte.ljust(math.ceil(len(byte)/6)*6,'0') # length must be divisible by 6
values = []
for x in range(0, len(byte), 6):
values.append(byte[x:x+6])
abc = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"
table = {x:abc[x] for x in range(len(abc))}
print(string) # input
padding = '=' * (((3 - len(string.encode('utf-8'))) % 3) % 3)
ooutput = "".join(table[decimal(x)] for x in values)
print(ooutput)
print(ooutput + padding) # for the sake of completeness
import base64 # merely for comparison/reference output
# ↓↓↓ output from base64 module ↓↓↓
print(base64.b64encode(string.encode('utf-8')).decode('utf-8'))
Output: .\SO\66724448.py ěščř ĚŠČŘ & .\SO\66724448.py
ěščř ĚŠČŘ
xJvFocSNxZkgxJrFoMSMxZg
xJvFocSNxZkgxJrFoMSMxZg=
xJvFocSNxZkgxJrFoMSMxZg=
This is a string!!!
VGhpcyBpcyBhIHN0cmluZyEhIQ
VGhpcyBpcyBhIHN0cmluZyEhIQ==
VGhpcyBpcyBhIHN0cmluZyEhIQ==

RSA algorithm fails for some strings

Exercise 11.7 from Chapter 11 of Think Python, by Allen B. Downey, prompts the following:
Exponentiation of large integers is the basis of common algorithms for public-key encryption. Read the Wikipedia page on the RSA algorithm3 and write functions to encode and decode messages.
I translated the plain English algorithm from the Key Generation section of the Wikipedia article. The function RSA takes a string and encodes it as an ASCII integer, creates public and private keys for decryption, encrypts the encoded integer using the public keys, decrypts it using the private key, and then decodes the integer back into a string:
import math
def encode(s):
m = ''
for c in s:
# each character in the message is converted to a ascii decimal value
m += str(ord(c))
return int(m)
def decode(i):
# convert decrypted int back to string
temp = str(i)
decoded = ''
while len(temp):
if temp[:1] == '1':
letter = chr(int(temp[:3]))
temp = temp[3:]
else:
letter = chr(int(temp[0:2]))
temp = temp[2:]
decoded += letter
return decoded
def generate_keys(m):
# requires an int m that is an int representation (in ascii) of our message
# 1. Choose two distinct prime numbers p and q.
##### TEST WIKIPEDIA EXAMPLE #####
# p, q = 61, 53
# find two prime numbers whose product n will be greater than m
i = math.ceil(math.sqrt(m))
primes = []
while len(primes) < 2:
if isPrime(i):
primes += [i]
i += 1
p, q = primes[0], primes[1]
# 2. Compute n = pq.
n = p * q
in_range = int(m) < n
if not in_range:
print('m must be less than n')
exit()
# 3. Compute λ(n), where λ is Carmichael's totient function. Since n = pq, λ(n) = lcm(λ(p),λ(q)), and since p and q are prime, λ(p) = φ(p) = p − 1 and likewise λ(q) = q − 1. Hence λ(n) = lcm(p − 1, q − 1).
# The lcm may be calculated through the Euclidean algorithm, since lcm(a,b) = |ab|/gcd(a,b).
a, b = (p - 1), (q - 1)
lam = int(abs(a * b) / math.gcd(a, b))
# print('lam:', lam)
# 4. Choose an integer e such that 1 < e < λ(n) and gcd(e, λ(n)) = 1; that is, e and λ(n) are coprime.
##### TEST WIKIPEDIA EXAMPLE #####
# e = 17
# accordng to wikipedia, the smallest (and fastest) possible value for e is 3: https://en.wikipedia.org/wiki/RSA_(cryptosystem)#Key_generation
i = 3
while i < lam:
if math.gcd(i, int(lam)) == 1:
e = i
break
i += 1
# 5. compute d where d * e ≡ 1 (mod lam)
d = int(modInverse(e, lam))
return {'public': [n, e], 'private': d}
def encrypt_message(m, public):
n, e = public[0], public[1]
# generate ciphertext
return pow(m, e, n)
def decrypt_message(cipher, n, d):
# decrypt ciphertext
return pow(cipher, d, n)
def RSA(s):
m = encode(s)
keys = generate_keys(m)
cipher = encrypt_message(m, keys['public'])
message = decrypt_message(cipher, keys['public'][0], keys['private'])
messages_match = message == m
if not messages_match:
print('the decoded integer does not equal the encoded integer')
exit()
return decode(message)
# taken from https://www.geeksforgeeks.org/python-program-to-check-whether-a-number-is-prime-or-not/
def isPrime(n) :
# Corner cases
if (n <= 1) :
return False
if (n <= 3) :
return True
# This is checked so that we can skip
# middle five numbers in below loop
if (n % 2 == 0 or n % 3 == 0) :
return False
i = 5
while(i * i <= n) :
if (n % i == 0 or n % (i + 2) == 0) :
return False
i = i + 6
return True
# modInverse(a, m) taken from https://www.geeksforgeeks.org/multiplicative-inverse-under-modulo-m/
def modInverse(a, m) :
m0 = m
y = 0
x = 1
if (m == 1) :
return 0
while (a > 1) :
# q is quotient
q = a // m
t = m
# m is remainder now, process
# same as Euclid's algo
m = a % m
a = t
t = y
# Update x and y
y = x - q * y
x = t
# Make x positive
if (x < 0) :
x = x + m0
return x
if __name__ == "__main__":
print(RSA('python')) # encrypts and decrypts message 'python'
print(RSA('cat')) # encrypts and decrypts message 'cat'
print(RSA('dog')) # encrypts and decrypts message 'dog'
print(RSA('a 1')) # (to rule out spaces as the culprit) encrypts and decrypts message 'a 1'
print(RSA('pythons')) # FAILS - 7 characters in string seems to be the limit
print(RSA('hello world')) # FAILS - encoded string does not equal decoded string
At the end of the script, I test RSA with several strings. The output equals the input for 'python', 'cat', 'dog', and 'a 1' (the intended behavior). But the encoded and decoded integers are different for 'pythons' and 'hello world'.
I suspect that it is the input string length that is causing the problem, but I'm not sure where that is causing the breakdown. My guess is that pow(cipher, d, n) in the function decrypt_message is returning an unexpected result if cipher is of too great a length.
Why does RSA work for some strings but not others? Is it the length of the input string or something else?

How do you use a string to solve a math equation using python?

I'm trying to make a python program which takes in a user equation, for example: "168/24+8=11*3-16", and tries to make both sides of the equation equal to each other by removing any 2 characters from the user input. This is what I have so far:
def compute(side):
val = int(side[0])
x= val
y=0
z=None
for i in range(1, len(side)-1):
if side[i].isdigit():
x= (x*10)+ int(side[i])
if x == side[i].isdigit():
x= int(side[i])
else:
op = side[i]
if op=="+":
val += x
elif op == "-":
val -= x
elif op == "*":
val *= x
else:
val /= x
return print(val)
I have edited my compute function.
def evaluate(e):
side1 = ""
side2 = ""
equalsign = e.index("=")
side1= e[:equalsign - 1]
side2= e[:equalsign + 1]
if compute (side1) == compute(side2):
return True
else:
return False
def solve():
# use a for loop with in a for loop to compare all possible pairs
pass
def main():
e= input("Enter an equation: ")
evaluate(e)
main()
For the actual solve function I want to test all possible pairs for each side of the equation and with every pair removed check if the equation is equal to the other side. I was thinking of using a for loop that said:
for i in side1:
j= [:x]+[x+1:y]+[y+1:]
if compute(j)==compute(side2):
val= compute(j)
return val
How should I go about doing this? I'm getting a little confused on how to really approach this program.

Let's get to the preliminary issues.
e = raw_input("Enter an equation: ") # input is fine if you are using Python3.x
side1 = e[:equalsign] #note that a[start:end] does not include a[end]
side2 = e[equalsign + 1:] # not e[:equalsign + 1].
val = int(side[0]) # not val = side[0] which will make val a string
In the operations part, you are doing val += side # or -= / *= / /= .. remember side is a string
Edits:
Yeah, I'm still stuck up with Python 2.7 (use input if Python 3)
To solve for the value of each side, you could simply use eval(side1) # or eval(side2). There could be alternatives to using eval. (I am a novice myself). eval will also take care of PEMDAS.
Added edit to side1 expression.
Updated with code written so far.
def compute(side):
return eval(side)
def evaluate(e):
side1, side2 = e.split('=')
if compute(side1) == compute(side2):
return (True, e)
else:
return (False, 'Not Possible')
def solve(e):
for i in range(len(e)): # loop through user input
if e[i] in '=': # you dont want to remove the equal sign
continue
for j in range(i+1, len(e)): # loop from the next index, you dont want
if e[j] in '=': # to remove the same char
continue # you dont want to remove '=' or operators
new_exp = e[:i] + e[i+1:j] + e[j+1:] # e[i] and e[j] are the removed chars
#print e[i], e[j], new_exp # this is the new expression
s1, s2 = new_exp.split('=')
try:
if compute(s1) == compute(s2):
return (True, new_exp)
except:
continue
return (False, 'not possible')
def main():
e= raw_input("Enter an equation: ")
print evaluate(e.replace(' ', ''))
main()
This is what I have come up with so far (works for your example at least).
It assumes that operators are not to be removed
Final edit: Updated code taking into account #Chronical 's suggestions
Removed the try-except block in each loop and instead just use it after calculating each side

Here is code that does exactly what you want:
from itertools import combinations
def calc(term):
try:
return eval(term)
except SyntaxError:
return None
def check(e):
sides = e.split("=")
if len(sides) != 2:
return False
return calc(sides[0]) == calc(sides[1])
equation = "168/24+8 = 11*3-16".replace(" ", "")
for (a, b) in combinations(range(len(equation)), 2):
equ = equation[:a] + equation[a+1:b] + equation[b+1:]
if check(equ):
print equ
Core tricks:
use eval() for evaluation. If you use this for anything, please be aware of the security implications of this trick.
use itertools.combinations to create all possible pairs of characters to remove
Do not try to handle = too specially – just catch it in check()

Encoding a 128-bit integer in Python?

Inspired by the "encoding scheme" of the answer to this question, I implemented my own encoding algorithm in Python.
Here is what it looks like:
import random
from math import pow
from string import ascii_letters, digits
# RFC 2396 unreserved URI characters
unreserved = '-_.!~*\'()'
characters = ascii_letters + digits + unreserved
size = len(characters)
seq = range(0,size)
# Seed random generator with same randomly generated number
random.seed(914576904)
random.shuffle(seq)
dictionary = dict(zip(seq, characters))
reverse_dictionary = dict((v,k) for k,v in dictionary.iteritems())
def encode(n):
d = []
n = n
while n > 0:
qr = divmod(n, size)
n = qr[0]
d.append(qr[1])
chars = ''
for i in d:
chars += dictionary[i]
return chars
def decode(str):
d = []
for c in str:
d.append(reverse_dictionary[c])
value = 0
for i in range(0, len(d)):
value += d[i] * pow(size, i)
return value
The issue I'm running into is encoding and decoding very large integers. For example, this is how a large number is currently encoded and decoded:
s = encode(88291326719355847026813766449910520462)
# print s -> "3_r(AUqqMvPRkf~JXaWj8"
i = decode(s)
# print i -> "8.82913267194e+37"
# print long(i) -> "88291326719355843047833376688611262464"
The highest 16 places match up perfectly, but after those the number deviates from its original.
I assume this is a problem with the precision of extremely large integers when dividing in Python. Is there any way to circumvent this problem? Or is there another issue that I'm not aware of?

The problem lies within this line:
value += d[i] * pow(size, i)
It seems like you're using math.pow here instead of the built-in pow method. It returns a floating point number, so you lose accuracy for your large numbers. You should use the built-in pow or the ** operator or, even better, keep the current power of the base in an integer variable:
def decode(s):
d = [reverse_dictionary[c] for c in s]
result, power = 0, 1
for x in d:
result += x * power
power *= size
return result
It gives me the following result now:
print decode(encode(88291326719355847026813766449910520462))
# => 88291326719355847026813766449910520462