Develop Reference

Python-How to execute code and store into variable?

So I have been struggling with this issue for what seems like forever now (I'm pretty new to Python). I am using Python 3.7 (need it to be 3.7 due to variations in the versions of packages I am using for the project) to develop an AI chatbot system that can converse with you based on your text input. The program reads the contents of a series of .yml files when it starts. In one of the .yml files I am developing a syntax for when the first 5 characters match a ^###^ pattern, it will instead execute the code and return the result of that execution rather than just output text back to the user. For example:
Normal Conversation:
- - What is AI?
- Artificial Intelligence is the branch of engineering and science devoted to constructing machines that think.
Service/Code-based conversation:
- - Say hello to me
- ^###^print("HELLO")
The idea is that when you ask it to say hello to you, the ^##^print("HELLO") string will be retrieved from the .yml file, the first 5 characters of the response will be removed, the response will be sent to a separate function in the python code where it will run the code and store the result into a variable which will be returned from the function into a variable that will give the nice, clean result of HELLO to the user. I realize that this may be a bit hard to follow, but I will straighten up my code and condense everything once I have this whole error resolved. As a side note: Oracle is just what I am calling the project. I'm not trying to weave Java into this whole mess.
THE PROBLEM is that it does not store the result of the code being run/executed/evaluated into the variable like it should.
My code:
def executecode(input):
print("The code to be executed is: ",input)
#note: the input may occasionally have single quotes and/or double quotes in the input string
result = eval("{}".format(input))
print ("The result of the code eval: ", result)
test = eval("2+2")
test
print(test)
return result
#app.route("/get")
def get_bot_response():
userText = request.args.get('msg')
print("Oracle INTERPRETED input: ", userText)
ChatbotResponse = str(english_bot.get_response(userText))
print("CHATBOT RESPONSE VARIABLE: ", ChatbotResponse)
#The interpreted string was a request due to the ^###^ pattern in front of the response in the custom .yml file
if ChatbotResponse[:5] == '^###^':
print("---SERVICE REQUEST---")
print(executecode(ChatbotResponse[5:]))
interpreter_response = executecode(ChatbotResponse[5:])
print("Oracle RESPONDED with: ", interpreter_response)
else:
print("Oracle RESPONDED with: ", ChatbotResponse)
return ChatbotResponse
When I run this code, this is the output:
Oracle INTERPRETED input: How much RAM do you have?
CHATBOT RESPONSE VARIABLE: ^###^print("HELLO")
---SERVICE REQUEST---
The code to be executed is: print("HELLO")
HELLO
The result of the code eval: None
4
None
The code to be executed is: print("HELLO")
HELLO
The result of the code eval: None
4
Oracle RESPONDED with: None
Output on the website interface
Essentially, need it to say HELLO for the "The result of the code eval:" output. This should get it to where the chatbot responds with HELLO in the web interface, which is the end goal here. It seems as if it IS executing the code due to the HELLO's after the "The code to be executed is:" output text. It's just not storing it into a variable like I need it to.
I have tried eval, exec, ast.literal_eval(), converting the input to string with str(), changing up the single and double quotes, putting \ before pairs of quotes, and a few other things. Whenever I get it to where the program interprets "print("HELLO")" when it executes the code, it complains about the syntax. Also, from several days of looking online I have figured out that exec and eval aren't generally favored due to a bunch of issues, however I genuinely do not care about that at the moment because I am trying to make something that works before I make something that is good and works. I have a feeling the problem is something small and stupid like it always is, but I have no idea what it could be. :(
I used these 2 resources as the foundation for the whole chatbot project:
Text Guide
Youtube Guide
Also, I am sorry for the rather lengthy and descriptive question. It's rare that I have to ask a question of my own on stackoverflow because if I have a question, it usually already has a good answer. It feels like I've tried everything at this point. If you have a better suggestion of how to do this whole system or you think I should try approaching this another way, I'm open to ideas.
Thank you for any/all help. It is very much appreciated! :)

The issue is that python's print() doesn't have a return value, meaning it will always return None. eval simply evaluates some expression, and returns back the return value from that expression. Since print() returns None, an eval of some print statement will also return None.
>>> from_print = print('Hello')
Hello
>>> from_eval = eval("print('Hello')")
Hello
>>> from_print is from_eval is None
True
What you need is a io stream manager! Here is a possible solution that captures any io output and returns that if the expression evaluates to None.
from contextlib import redirect_stout, redirect_stderr
from io import StringIO
# NOTE: I use the arg name `code` since `input` is a python builtin
def executecodehelper(code):
# Capture all potential output from the code
stdout_io = StringIO()
stderr_io = StringIO()
with redirect_stdout(stdout_io), redirect_stderr(stderr_io):
# If `code` is already a string, this should work just fine without the need for formatting.
result = eval(code)
return result, stdout_io.getvalue(), stderr_io.getvalue()
def executecode(code):
result, std_out, std_err = executecodehelper(code)
if result is None:
# This code didn't return anything. Maybe it printed something?
if std_out:
return std_out.rstrip() # Deal with trailing whitespace
elif std_err:
return std_err.rstrip()
else:
# Nothing was printed AND the return value is None!
return None
else:
return result
As a final note, this approach is heavily linked to eval since eval can only evaluate a single statement. If you want to extend your bot to multiple line statements, you will need to use exec, which changes the logic. Here's a great resource detailing the differences between eval and exec: What's the difference between eval, exec, and compile?

It is easy just convert try to create a new list and add the the updated values of that variable to it, for example:
if you've a variable name myVar store the values or even the questions no matter.
1- First declare a new list in your code as below:
myList = []
2- If you've need to answer or display the value through myVar then you can do like below:
myList.append(myVar)
and this if you have like a generator for the values instead if you need the opposite which means the values are already stored then you will just update the second step to be like the following:
myList[0]='The first answer of the first question'
myList[1]='The second answer of the second question'
ans here all the values will be stored in your list and you can also do this in other way, for example using loops is will be much better if you have multiple values or answers.

How to use python Transaction without database ?

I have two line in my code which first one is os.unlink and second one is os.symlink. like :
os.unlink(path)
os.symlink(new_path)
The sequence should not be change, The problem is, some times it unlink a file (in other word it remove it's shortcut) but second line could not create symbolic link (do to some addressing issue).
My question is: Is there any all or non transaction tool like the one we have in database, to do both line or non ?

you could try this:
import os
linkname = '/tmp/test.lnk'
orig_target = os.path.realpath(linkname)
os.unlink(linkname)
try:
os.symlink(new_target, linkname)
except:
os.symlink(orig_target, linkname)
maybe check what exceptions can occur and only catch the ones that are relevant.

Strictly speaking it is not possible unless you use Transactional filesystem like TxF (https://en.wikipedia.org/wiki/Transactional_NTFS) because nothing prevents your machine from poweroff between two commands.
I can see 2 ways here:
1) Switch to Database
2) Check all conditions before unlinking. What prevents you from symlinking?

Using Python to split a Unicode file object into dictionary Keys and values

Hi and thanks for reading. I’ll admit that this is a progression on from a previous question I asked earlier, after I partially solved the issue. I am trying to process a block of text (file_object) in an earlier working function. The text or file_object happens to be in Unicode, but I have managed to convert to ascii text and split on a line by line basis. I am hoping to then further split the text on the ‘=’ symbol so that I can drop the text into a dictionary. For example Key: Value as ‘GPS Time’:’ 14:18:43’ so removing the trailing '.000' from the time (though this is a second issue).
Here’s the file_object format…
2015 Jan 01 20:07:16.047 GPS Info #Log packet ID
GPS Time = 14:18:43.000
Longitude = 000.65341
Latitude = +41.25385
Altitude = +111.400
This is my partially working function…
def process_data(file_object):
file_object = file_object.encode('ascii','ignore')
split = file_object.split('\n')
for i in range(len(split)):
while '=' in split[i]:
processed_data = (split[i].split('=', 1) for _ in xrange(len(split)))
return {k.strip(): v.strip() for k, v in processed_data}
This is the initial section of the main script that prompts the above function, and then sets GPS Time as the Dictionary key…
while (mypkt.Next()): #mypkt.Next is an API function in the log processor app I am using – essentially it grabs the whole GPS Info packet shown above
data = process_data(mypkt.Text, 1)
packets[data['GPS Time']] = data
The code above has no problem splitting the first instance ‘GPS Time’, but it ignores Lonitude, Latitude etc, To make matters worse, there is sometimes a blank line between each packet item too. I guess I need to store previous dictionary related splits before the ‘return’, but I am having difficulty trying to find out how to do this.
The dict output I am currently getting is…
'14:19:09.000': {'GPS Time': '14:19:09.000'},
But What I am hoping for is…
'14:19:09': {'GPS Time': '14:19:09',
‘Longitude’:’000.65341’,
‘Latitude’:’+41.25385’,
‘Altitude’:’+111.400’},
Thanks in advance for any help.
MikG

All this use of range(len(whatever)) is nonsense. You almost never need to do that in Python. Just iterate through the thing.
Your problem however is more fundamental: you return from inside the while loop. That means you only ever get one element, because as soon as that first line is processed, you return and the function ends.
Also, you have a while loop which means that processing will end as soon as the program encounters a line without an equals; but you have blank lines between each data line, so again execution would never proceed past the first one.
So all you need is:
split_data = file_object.split('\n')
result = {}
for line in split_data:
if '=' in line:
key, value = line.split('=', 1)
result[key.strip()] = value.strip()
return result

Python - How to parse smartctl program output?

I am writing a wrapper for smartctl in python 2.7.3...
I am having a hell of a time trying to wrap my head around how to parse the output from the smartctl program in Linux (Ubuntu x64 to be specific)
I am running smartctl -l selftest /dev/sdx via subprocess and grabbing the output into a variable
This variable is broken up into a list, then I drop the useless header data and blank lines from the output.
Now, I am left with a list of strings, which is great!
The data is sort-of tabular, and I want to parse it into a dict() full of lists (I think this is the correct way to represent tabular data in Python from reading the docs)
Here's a sample of the data:
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 44796 -
# 2 Short offline Completed without error 00% 44796 -
# 3 Short offline Completed without error 00% 44796 -
# 4 Short offline Completed without error 00% 44796 -
# 5 Short offline Completed without error 00% 44796 -
# 6 Extended offline Completed without error 00% 44771 -
# 7 Short offline Completed without error 00% 44771 -
# 8 Short offline Completed without error 00% 44741 -
# 9 Short offline Completed without error 00% 1 -
#10 Short offline Self-test routine in progress 70% 44813 -
I can see some issues with trying to parse this, and am open to solutions, but i may also just be doing this all wrong ;-):
The Status Text Self-test routine in progress flows past the first character of the text Remaining
In the Num column, the numbers after 9 are not separated from the # character by a space
I might be way off-base here, but this is my first time trying to parse something this eccentric.
Thank everyone who even bothers to read this wall of text in advance!!!
Here's my code so far, if anyone feels it necessary or finds it useful:
#testStatus.py
#This module provides an interface for retrieving
#test status and results for ongoing and completed
#drive tests
import subprocess
#this function takes a list of strings and removes
#strings which do not have pertinent information
def cleanOutput(data):
cleanedOutput = []
del data[0:3] #This deletes records 0-3 (lines 1-4) from the list
for item in data:
if item == '': #This removes blank items from remaining list
pass
else:
cleanedOutput.append(item)
return cleanedOutput
def resultsOutput(data):
headerLines = []
resultsLines = []
resultsTable = {}
for line in data:
if "START OF READ" in line or "log structure revision" in line:
headerLines.append(line)
else:
resultsLines.append(line)
nameLine = resultsLines[0].split()
print nameLine
def getStatus(sdxPath):
try:
output = subprocess.check_output(["smartctl", "-l", "selftest", sdxPath])
except subprocess.CalledProcessError:
print ("smartctl command failed...")
except Exception as e:
print (e)
splitOutput = output.split('\n')
cleanedOutput = cleanOutput(splitOutput)
resultsOutput(cleanedOutput)
#For Testing
getStatus("/dev/sdb")

For what it's worth (this is an old question): smartctl has a --json flag which you can use and then parse the output like normal JSON since version 7.0
release notes

The main parsing problem seems to be the first three columns; the remaining data is more straight forward. Assuming the output uses blanks between fields (instead of tab characters, which would be much easier to parse), I'd go for fixed length parsing, something like:
num = line[1:2]
desc = line[5:25]
status = line[25:54]
remain = line[54:58]
lifetime = line[60:68]
lba = line[77:99]
The header line would be handled differently. What structure you put the data into depends on what you want to do with it. A dictionary keyed by "num" might be appropriate if you mainly wanted to randomly access data by that "num" identifier. Otherwise a list might be better. Each entry (per line) could be a tuple, a list, a dictionary, a class instance, or maybe other things. If you want to access fields by name, then a dictionary or class instance per entry might be appropriate.

Which of the following datastructure is the best regarding frequent searching?

I have a text file with some content. I need to search this content frequently. I have the following two options, which one is the best (by means of faster execution) ?
METHOD 1:
def search_list(search_string):
if search_word in li:
print "found at line ",li.indexOf(search_word)+1
if __name__="__main__":
f=open("input.txt","r")
li=[]
for i in f.readlines():
li.append(i.rstrip("\n"))
search_list("appendix")
METHOD 2:
def search_dict(search_string):
if d.has_key(search_word):
print "found at line ",d[search_word]
if __name__="__main__":
f=open("input.txt","r")
d={}
for i,j in zip(range(1,len(f.readlines())),f.readlines()):
d[j.rstrip("\n")]=i
search_dict("appendix")

For frequent searching, a dictionary is definitely better (provided you have enough memory to store the line numbers also) since the keys are hashed and looked up in O(1) operations. However, your implementation won't work. The first f.readlines() will exhaust the file object and you won't read anytihng with the second f.readlines().
What you're looking for is enumerate:
with open('data') as f:
d = dict((j[:-1],i) for i,j in enumerate(f,1))
It should also be pointed out that in both cases, the function which does the searching will be faster if you use try/except provided that the index you're looking for is typically found. (In the first case, it might be faster anyway since in is an order N operation and so is .index for a list).
e.g.:
def search_dict(d, search_string):
try:
print "found at line {0}".format(d[search_string])
except KeyError:
print "string not found"
or for the list:
def search_list(search_string):
try:
print "found at line {0}".format(li.indexOf(search_word)+1)
except ValueError:
print "string not found"

If you do it really frequently, then the second method will be faster (you've built something like an index).
Just adapt it a little bit:
def search_dict(d, search_string):
line = d.get(search_string)
if line:
print "found at line {}".format(line)
else:
print "string not found"
d = {}
with open("input.txt", "r") as f:
for i, word in enumerate(f.readlines(), 1):
d[word.rstrip()] = i
search_dict(d, "appendix")

I'm posting this after reading the answers of eumiro and mgilson.
If you compare your two methods on the command line, I think you'll find that the first one is faster. The other answers that say the second method is faster, but they are based on the premise that you'll do several searches on the file after you've built your index. If you use them as-is from the command line, you will not.
The building of the index is slower than just searching for the string directly, but once you've built an index, searches can be done very quickly, making up for the time spent building it. This extra time is wasted if you just use it once, because when the program is complete, the index is discarded and has to be rebuilt the next run. You need to keep the created index in memory between queries for this to pay off.
There are several ways of doing this, one is making a daemon to hold the index and use a front-end script to query it. Searching for something like python daemon client communication on google will give you pointers on implementing this -- here's one method.

First one is O(n); second one is O(1), but it requires searching on the key. I'd pick the second one.
Neither one will work if you're ad hoc searches in the document. For that you'll need to parse and index using something like Lucene.

Another option to throw in is using the FTS provided by SQLite3... (untested and making the assumption you're looking for wholewords, not substrings of words or other such things)
import sqlite3
# create db and table
db = sqlite3.connect(':memory:') # replace with file on-disk?
db.execute('create virtual table somedata using fts4(line)')
# insert the data
with open('yourfile.txt') as fin:
for lineno, line in enumerate(fin):
# You could put in a check here I guess...
if somestring in line:
print lineo # or whatever....
# put row into FTS table
db.execute('insert into somedata (line) values (?)', (line,))
# or possibly more efficient
db.executemany('insert into somedata (line) values (?)', fin)
db.commit()
look_for = 'somestring'
matches = db.execute('select rowid from somedata where line match ?', (look_for,) )
print '{} is on lines: {}'.format(look_for, ', '.join(match[0] for match in matches))
If you only wanted the first line, then add limit 1 to the end of the query.
You could also look at using mmap to map the file, then use the .find method to get the earliest offset of the string, then assuming it's not -1 (ie, not found - let's say 123456), then do mapped_file[:123456].count('\n') + 1 to get the line number.