Related
I found this code for printing a program trace and it works fine in Python2.
However, in Python 3 there are issues. I addressed the first one by replacing execfile(file_name) with exec(open(filename).read()), but now there is still an error of KeyError: 'do_setlocale'
I'm out of my depth here - I just want an easy way to trace variables in programs line by line - I like the way this program works and it would be great to get it working with Python 3. I even tried an online conversion program but got the same KeyError: 'do_setlocale'
Can anyone please help me to get it working?
import sys
if len(sys.argv) < 2:
print __doc__
exit()
else:
file_name = sys.argv[1]
past_locals = {}
variable_list = []
table_content = ""
ignored_variables = set([
'file_name',
'trace',
'sys',
'past_locals',
'variable_list',
'table_content',
'getattr',
'name',
'self',
'object',
'consumed',
'data',
'ignored_variables'])
def trace(frame, event, arg_unused):
global past_locals, variable_list, table_content, ignored_variables
relevant_locals = {}
all_locals = frame.f_locals.copy()
for k,v in all_locals.items():
if not k.startswith("__") and k not in ignored_variables:
relevant_locals[k] = v
if len(relevant_locals) > 0 and past_locals != relevant_locals:
for i in relevant_locals:
if i not in past_locals:
variable_list.append(i)
table_content += str(frame.f_lineno) + " || "
for variable in variable_list:
table_content += str(relevant_locals[variable]) + " | "
table_content = table_content[:-2]
table_content += '\n'
past_locals = relevant_locals
return trace
sys.settrace(trace)
execfile(file_name)
table_header = "L || "
for variable in variable_list:
table_header += variable + ' | '
table_header = table_header[:-2]
print table_header
print table_content
# python traceTable.py problem1.py
# problem1.py
a = 1
b = 2
a = a + b
That program has a couple of major flaws – for example, if the program being traced includes any functions with local variables, it will crash, even in Python 2.
Therefore, since I have nothing better to do, I wrote a program to do something like this called pytrace. It's written for Python 3.6, although it probably wouldn't take too long to make it work on lower versions if you need to.
Its output is a little different to your program's, but not massively so – the only thing that's missing is line numbers, which I imagine you could add in fairly easily (print frame.f_lineno at appropriate points). The rest is purely how the data are presented (your program stores all the output until the end so it can work out table headers, whereas mine prints everything as it goes).
Edit: To the people that downvoted: I was perfectly clear that I did not want code and that I had already tried it myself. All I was looking for was an explanation of what mathematical process yielded the sample results.
First question. I have done a lot of research and finally resorted to asking, so if I missed the answer somewhere I apologize. I have a problem I am really struggling with:
Write a Python 3 script that takes three command line arguments: 1. The name of a text file that contains n strings separated
by white spaces. 2. A positive integer k. 3. The name of a
text file that the script will create in order to store all possible
subsequences of k unique strings out of the n strings from the input
file, one subsequence per line. For example, assume the
command line is gen.py input.txt 3 output.txt and the file input.txt
contains the following line: Python Java C++ Java Java Python
Then the program should create the file output.txt containing
the following lines (in any order): Python Java C++ Python
C++ Java Java C++ Python C++ Java Python The
combinations should be generated with your implementation of a
generator function (i.e. using the keyword yield).
From my understanding, based on the sample output this doesn't quite follow the definition of a subsequence; nor are they quite permutations, so I'm at a loss for how to go about this. I know how to do the file IO and command line argument portions, I just can't get the right subsequences. I don't need a direct answer as I am supposed to solve this, but if someone could give me some helpful insight it would be much appreciated.
If you're allowed to use itertools:
import itertools
import sys
def unique_substrings(txt_lst:list, k:int) -> set:
return set([' '.join(combo) for combo in itertools.combinations(txt_lst, 3) \
if len(set(combo))==3])
if __name__ == "__main__":
infile, k, outfile = sys.argv[1:]
with open(infile) as inf:
txt_lst = infile.read().split()
with open(outfile) as outf:
for line in unique_substrings(txt_lst, k):
outf.write(line + "\n")
However from your instructor's comment:
The combinations should be generated with your implementation of a generator function (i.e. using the keyword yield).
It doesn't look like that's actually going to work.
itertools.combinations could be re-implemented with something approximating the following (from the docs):
def combinations(iterable, r):
# combinations('ABCD', 2) --> AB AC AD BC BD CD
# combinations(range(4), 3) --> 012 013 023 123
pool = tuple(iterable)
n = len(pool)
if r > n:
return
indices = list(range(r))
yield tuple(pool[i] for i in indices)
while True:
for i in reversed(range(r)):
if indices[i] != i + n - r:
break
else:
return
indices[i] += 1
for j in range(i+1, r):
indices[j] = indices[j-1] + 1
yield tuple(pool[i] for i in indices)
I am trying to solve MILP in puLP (Python), and I keep getting the following error:
Traceback (most recent call last):
File "main_lp.py", line 63, in <module>
ans = solve_lp(C)
File "/home/ashwin/Documents/Williams/f2014/math317_or/project/solve_lp.py", line 36, in solve_lp
prob.solve()
File "/usr/local/lib/python2.7/dist-packages/PuLP-1.5.6-py2.7.egg/pulp/pulp.py", line 1619, in solve
status = solver.actualSolve(self, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/PuLP-1.5.6-py2.7.egg/pulp/solvers.py", line 1283, in actualSolve
return self.solve_CBC(lp, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/PuLP-1.5.6-py2.7.egg/pulp/solvers.py", line 1346, in solve_CBC
raise PulpSolverError("Pulp: Error while executing "+self.path)
pulp.solvers.PulpSolverError: Pulp: Error while executing /usr/local/lib/python2.7/dist-packages/PuLP-1.5.6-py2.7.egg/pulp/solverdir/cbc-32
For my linear programming problem, I am attempting to take sums of different vectors as the constraint, and I think I must have done that wrong somehow, because a much simpler problem works with no hitches. I have attached the code (C is an N by N numpy array).
def solve_lp(C):
N = len(C)
prob=LpProblem('Scheduling',LpMinimize)
X = [[LpVariable('X' + str(i+1) + str(j+1), 0, C[i,j],LpBinary)
for j in range(N)] for i in range(N)]
X = np.array(X)
X_o = [LpVariable('X0' + str(i), 0, None, LpBinary) for i in range(N)]
X_t = [LpVariable('X' + str(i) + 't', 0, None, LpBinary) for i in range(N)]
# Objective Function
ones_vec = list(np.ones(len(X_o)))
prob += lpDot(ones_vec,X_o), 'Minimize Buses'
# Constraints
for i in range(N):
row = list(X[i,:]) + [X_t[i]]
ones_vec = list(np.ones(len(row)))
prob += lpDot(ones_vec, row) == 1, 'Only one destination for ' + str(i)
for j in range(N):
col = list(X[:,j]) + [X_o[j]]
ones_vec = list(np.ones(len(col)))
prob += lpDot(ones_vec,col) == 1, 'Only one source for ' + str(j)
prob.solve()
return X, value(prob.objective)
Make sure you don't have duplicate LpVariable names, and watch out for LpVariable names with the unsupported characters -+[] ->/ since all of those characters are silently converted to underscores _
Setting LpSolverDefault.msg = 1 before calling prob.solve() may help by printing the solvers output to the console.
I recently had a similar problem due to Nan inputs in the model. I had the data in a DataFrame where some the cells should not bee converted to variables to improve performance. However, upon creating the objective function and the constraints, I noticed the presence of Nan and when I changed them it worked perfectly.
I think that you have duplicate LpVariable names. I just had the same problem and saw it thanks to levis501's answer. Here:
X = [[LpVariable('X' + str(i+1) + str(j+1), 0, C[i,j],LpBinary)
for j in range(N)] for i in range(N)]
X contains some variables with the same name. For example for i = 0 and j = 10 you get 'X111', and for i = 10 and j = 0 you also get 'X111'.
I had a similar problem, and indeed with duplicate LpVariable names just like levis501's answer and aleon answered.
my code was:
var = [[pulp.LpVariable(f'x{i}{j}', lowBound=0, cat=pulp.LpInteger) for j in range(col)] for i in range(row)]
when i=1 j=11, x would be x111, and when i=11 j=1, x also would be x111
I changed to:
var = [[pulp.LpVariable(f'x{i}_{j}', lowBound=0, cat=pulp.LpInteger) for j in range(col)] for i in range(row)]
After changing, it runs successfully.
I recently had the same error raised aswell. The reason this error code was raised in my case was that my data frame was not correctly populated. i incorrectly had NaN on the RHS on some of my constraints
What is had was something like:
Matrix = df.pivot(1st_dimension, 2nd_dimension, value)
This operation automatically puts NaN for the instances that are not in the original dataframe.
in my case the NaN'sare replaced with 0 which is what i was expecting it to be:
Matrix = Matrix.fillna(0)
I experienced the same issue when launching multiple instances of the LPSolver class. As fmars stated, the problem is that the path 'tmpSol' does not exist, which is defined in the following lines of code within the solvers.py file of pulp:
pid = os.getpid()
tmpLp = os.path.join(self.tmpDir, "%d-pulp.lp" % pid)
tmpMps = os.path.join(self.tmpDir, "%d-pulp.mps" % pid)
tmpSol = os.path.join(self.tmpDir, "%d-pulp.sol" % pid)
This bit of code appears in every solver. The problem is that these paths are deleted later on, but may coincide for different instances of the LPSolver class (as the variable pid is not unique).
The solution is to get a unique path for each instance of LPSolver, using, for example, the current time. Replacing the above lines by the following four will do the trick.
currentTime = time()
tmpLp = os.path.join(self.tmpDir, "%f3-pulp.lp" % currentTime)
tmpMps = os.path.join(self.tmpDir, "%f3-pulp.mps" % currentTime)
tmpSol = os.path.join(self.tmpDir, "%f3-pulp.sol" % currentTime)
Don't forget to
from time import time
Cheers,
Tim
I have met some similar problem which because of some PuLP's bug. When some problem is infeasible and the solver failed to solve the problem, PuLP raise an exception rather than return status equals to infeasible. Below is the reason.
(You may first want to check out latest codebase of PuLP because the line number you paste doesn't match the latest one. I'll explain based on latest one but you can look at yours very simply.)
https://github.com/coin-or/pulp/blob/master/src/pulp/solvers.py#L1405-L1406
This is where exception prompt.
if not os.path.exists(tmpSol):
raise PulpSolverError("Pulp: Error while executing "+self.path)
tmpSol is the temporary file where stores the solution. If PuLP cannot find such solution file, it will throw the exception you saw. And the bug I mentioned above is, if the problem itself is infeasible, then PuLP won't be able to generate such temporary file. So it will always throw such exception.
One thing you can do is, send a pull request to PuLP repo and fix it.
And a simple workaround is, rather than directly invoke
prob.solve()
in your code, you should always do
try:
prob.solve()
except Exception:
logger.debug('Problem infeasible')
I had this problem today, and it was because the temporary files for CBC were trying to be written in a location whose path had spaces in it, and the command that pulp passes to subprocess.Popen() to run CBC does not use quotes, and thus the command was misinterpreted and CBC couldn't find the location to create temp files.
For this I found two solutions:
(1) Explicitly set a temporary file directory that doesn't have spaces in it,
pulp.LpSolverDefault.tmpDir = self.tmp_file_dir # tmp_file_dir can't have spaces!
prob.solve()
or
(2) don't use CBC (my problem is small)
prob.solve(pulp.GLPK_CMD())
I have a work-related constraint that a lot of my work is stuck in directories that have spaces.
Edit: I did a first version, which Eike helped me to advance quite a bit on it. I'm now stuck to a more specific problem, which I will describe bellow. You can have a look at the original question in the history
I'm using pyparsing to parse a small language used to request specific data from a database. It features numerous keyword, operators and datatypes as well as boolean logic.
I'm trying to improve the error message sent to the user when he does a syntax error, since the current one is not very useful. I designed a small example, similar to what I'm doing with the language aforementioned but much smaller:
#!/usr/bin/env python
from pyparsing import *
def validate_number(s, loc, tokens):
if int(tokens[0]) != 0:
raise ParseFatalException(s, loc, "number musth be 0")
def fail(s, loc, tokens):
raise ParseFatalException(s, loc, "Unknown token %s" % tokens[0])
def fail_value(s, loc, expr, err):
raise ParseFatalException(s, loc, "Wrong value")
number = Word(nums).setParseAction(validate_number).setFailAction(fail_value)
operator = Literal("=")
error = Word(alphas).setParseAction(fail)
rules = MatchFirst([
Literal('x') + operator + number,
])
rules = operatorPrecedence(rules | error , [
(Literal("and"), 2, opAssoc.RIGHT),
])
def try_parse(expression):
try:
rules.parseString(expression, parseAll=True)
except Exception as e:
msg = str(e)
print("%s: %s" % (msg, expression))
print(" " * (len("%s: " % msg) + (e.loc)) + "^^^")
So basically, the only things which we can do with this language, is writing series of x = 0, joined together with and and parenthesis.
Now, there are cases, when and and parenthesis are used, where the error reporting is not very good. Consider the following examples:
>>> try_parse("x = a and x = 0") # This one is actually good!
Wrong value (at char 4), (line:1, col:5): x = a and x = 0
^^^
>>> try_parse("x = 0 and x = a")
Expected end of text (at char 6), (line:1, col:1): x = 0 and x = a
^^^
>>> try_parse("x = 0 and (x = 0 and (x = 0 and (x = a)))")
Expected end of text (at char 6), (line:1, col:1): x = 0 and (x = 0 and (x = 0 and (x = a)))
^^^
>>> try_parse("x = 0 and (x = 0 and (x = 0 and (x = 0)))")
Expected end of text (at char 6), (line:1, col:1): x = 0 and (x = 0 and (x = 0 and (xxxxxxxx = 0)))
^^^
Actually, it seems that if the parser can't parse (and parse here is important) something after a and, it doesn't produce good error messages anymore :(
And I mean parse, since if it can parse 5 but the "validation" fails in the parse action, it still produces a good error message. But, if it can't parse a valid number (like a) or a valid keyword (like xxxxxx), it stops producing the right error messages.
Any idea?
Pyparsing will always have somewhat bad error messages, because it backtracks. The error message is generated in the last rule that the parser tries. The parser can't know where the error really is, it only knows that there is no matching rule.
For good error messages you need a parser that gives up early. These parsers are less flexible than Pyparsing, but most conventional programming languages can be parsed with such parsers. (C++ and Scala IMHO can't.)
To improve error messages in Pyparsing use the - operator, it works like the + operator, but it does not backtrack. You would use it like this:
assignment = Literal("let") - varname - "=" - expression
Here is a small article on improving error reporting, by Pyparsing's author.
Edit
You could also generate good error messages for the invalid numbers in the parse actions that do the validation. If the number is invalid you raise an exception that is not caught by Pyparsing. This exception can contain a good error message.
Parse actions can have three arguments [1]:
s = the original string being parsed (see note below)
loc = the location of the matching substring
toks = a list of the matched tokens, packaged as a ParseResults object
There are also three useful helper methods for creating good error messages [2]:
lineno(loc, string) - function to give the line number of the location within the string; the first line is line 1, newlines start new rows.
col(loc, string) - function to give the column number of the location within the string; the first column is column 1, newlines reset the column number to 1.
line(loc, string) - function to retrieve the line of text representing lineno(loc, string). Useful when printing out diagnostic messages for exceptions.
Your validating parse action would then be like this:
def validate_odd_number(s, loc, toks):
value = toks[0]
value = int(value)
if value % 2 == 0:
raise MyFatalParseException(
"not an odd number. Line {l}, column {c}.".format(l=lineno(loc, s),
c=col(loc, s)))
[1] http://pythonhosted.org/pyparsing/pyparsing.pyparsing.ParserElement-class.html#setParseAction
[2] HowToUsePyparsing
Edit
Here [3] is an improved version of the question's current (2013-4-10) script. It gets the example errors right, but other error are indicated at the wrong position. I believe there are bugs in my version of Pyparsing ('1.5.7'), but maybe I just don't understand how Pyparsing works. The issues are:
ParseFatalException seems not to be always fatal. The script works as expected when I use my own exception.
The - operator seems not to work.
[3] http://pastebin.com/7E4kSnkm
Hi I found in book:Numerical Methods in engineering with Python the module run_kut5, but for that module I need module printSoln, all provided in the book. Now I cp the code, made necessary line adjustments and so. The code looks like:
# -*- coding: cp1250 -*-
## module printSoln
''' printSoln(X,Y,freq).
Prints X and Y returned from the differential
equation solvers using printput frequency ’freq’.
freq = n prints every nth step.
freq = 0 prints initial and final values only.
'''
def printSoln(X,Y,freq):
def printHead(n):
print "\n x ",
for i in range (n):
print " y[",i,"] ",
print
def printLine(x,y,n):
print "%13.4e"% x,f
for i in range (n):
print "%13.4e"% y[i],
print
m = len(Y)
try: n = len(Y[0])
except TypeError: n = 1
if freq == 0: freq = m
printHead(n)
for i in range(0,m,freq):
printLine(X[i],Y[i],n)
if i != m - 1: printLine(X[m - 1],Y[m - 1],n)
Now, when I run the program it says:
line 24, in <module>
m = len(Y)
NameError: name 'Y' is not defined
But I cp'd from the book :\ So now when I call the run_kut module I get the same error, no Y defined in printSoln...
I'm trying to figure this out but I suck :(
Help, please...
I would guess it's a tabs/spaces problem - check that you don't have mixed tabs and spaces in the indentation.
EDIT: If it's not indentation, and since your error message contains "<module>" and not a filename, I'm guessing you're pasting this into an interactive interpreter.
Instead, you should paste the code into a file called printsoln.py, and then run this in the interactive interpreter:
from printsoln import printSoln
If you still want to paste it all in the interpreter then you'll probably need to remove the blank lines - those after def printSoln and after each internal function. The interactive interpreter uses blank lines to figure out when you're done with a multi-line definition, and tries to evaluate m = len(Y) outside the context of function printSoln. In this context, variable Y doesn't exist.