Develop Reference

Why doesn't the value of variable from another module get propagated to other functions even after it's set?

The variable SCRIPT_ENV gets set correctly in main block but that value does not propagate to other functions.
Here's my full working code:
import argparse
import settings
from multiprocessing import Pool
def set_brokers_and_cert_path():
brokers = None
cert_full_path = None
print("I am here man\n\n {0}".format(settings.SCRIPT_ENV))
if settings.SCRIPT_ENV == "some value":
brokers = # use these brokers
cert_full_path = settings.BASE_CERT_PATH + "test_env/"
if settings.SCRIPT_ENV == "some other value":
brokers = # use those brokers
cert_full_path = settings.BASE_CERT_PATH + "new_env/"
return brokers, cert_full_path
def func_b(partition):
kafka_brokers, cert_full_path = set_brokers_and_cert_path()
producer = KafkaProducer(bootstrap_servers=kafka_brokers,
security_protocol='SSL',
ssl_check_hostname=True,
ssl_cafile=cert_full_path +'cacert.pem',
ssl_certfile=cert_full_path + 'certificate.pem',
ssl_keyfile=cert_full_path + 'key.pem',
max_block_ms=1200000,
value_serializer=lambda v: json.dumps(v).encode('utf-8'),
key_serializer=str.encode
)
try:
producer.send(settings.KAFKA_TOPIC,
value="some val",
key="some key",
timestamp_ms=int(time.time()),
headers=[some headers],
partition=partition)
producer.flush()
except AssertionError as e:
print("Error in partition: {0}, {1}".format(partition, e))
def main():
with Pool(settings.NUM_PROCESSES) as p:
p.map(func_b, [i for i in range(0, 24)])
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("--env", help="Environment against which the script needs to run")
args = parser.parse_args()
if args.env:
settings.SCRIPT_ENV = args.env
main()
else:
raise Exception("Please pass env argument. Ex: --env test/accept")
In the line "I am here man", it prints None as the value of SCRIPT_ENV.
Here, SCRIPT_ENV gets set perfectly in the if __name__ == "__main__" block, but in func_a, it comes as None.
contents of settings.py:
KAFKA_TOPIC = "some topic"
NUM_PROCESSES = 8
NUM_MESSAGES = 1000
SCRIPT_ENV = None
NUM_PARTITIONS = 24
TEST_BROKERS = [some brokers]
ACCEPT_BROKERS = [some brokers]
BASE_CERT_PATH = "base path"
I run it like this:
python <script.py> --env <value>

In your code example a few references are missing, without which it is not executable. First, we should find out the program parts that lead to the behavior you describe.
I do not suppose the argparse to be crucial. Basically you change an attribute in the module instance settings and call main(), so you can reduce the last part of the program to a few lines:
settings.SCRIPT_ENV = "test"
main()
Function main() propagates function func_b() over multiple processes. Function func_b() itself performs following steps:
Call of function set_brokers_and_cert_path() (you earlier named func_a())
Create an instance of KafkaProducer (producer)
Send some data per producer.
I cannot see that the created instance has any effect on the content of settings.SCRIPT_ENV, so I reduce function func_b() as well and leave only the first function call (for simplicity I take the original name func_a()).
Function func_a() (aka set_brokers_and_cert_path()) likewise has no apparent influence on settings.SCRIPT_ENV, since only a few strings are generated.
For debugging purposes I left the output of the text variables in the function.
All together, I come up with the following executable minimal example, which does not recreate for me the problem you describe (see output).
You can take this example and transfer it piece by piece back to your code and then see which part of the program leads to the problems.
import settings
from multiprocessing import Pool
def func_a():
print("I am here man\n{0}\n".format(settings.SCRIPT_ENV))
def func_b(partition):
func_a()
def main():
with Pool(settings.NUM_PROCESSES) as p:
p.map(func_b, [i for i in range(0, 24)])
settings.SCRIPT_ENV = "Test"
main()
Output
I am here man
Test
I am here man
Test
I am here man
Test
I am here man
Test
...
#RajatBhardwaj, at this point, it is up to you to decide whether you want to participate in finding a solution.
All others who are not interested in a solution would do well not to criticize solutions without having understood its intention.

Python 3 function import error

I am new to python please suggest me the solution
I have two python files first is imtest.py as follows:
d = dict()
if __name__ == '__main__':
def salry(a):
d["f"] = a
print (d)
second is testim.py as follows:
import imtest
a= 90
b = 200
imtest.salry(a)
imtest.salry(b)
When I am trying to run testim.py, it's giving error as :
AttributeError: module 'imtest' has no attribute 'salry'
Then I have modified second file testim.py as follows:
from imtest import salry
a= 90
b = 200
salry(a)
salry(b)
Now I am getting error as
ImportError: cannot import name 'salry'
What could be the error why I am not able to import that function?

The __name__ magic variable has different value depending on whether the module is executed as a script (python imtest.py) or imported from another module. In the first case, __name__ is set to "__main__", in the second case it gets the module's name ("imtest" in your case). The result is that everything in the if __name__ == "__main__": block is only executed when using imtest.py as a script. When you import it from testim.py this part is not executed, so the def salry statement is not executed, so the function is not defined, so you cannot import it.
The solution is quite obvious: put the function definition outside this block:
d = dict()
def salry(a):
d["f"] = a
if __name__ == '__main__':
print (d)

If you want your salary function to be available to other modules, it needs to be in outer scope, so:
d = dict()
def salry(a):
d["f"] = a
print (d)

Instead of this you have to this
d = dict()
def salry(a):
d["f"] = a
print(d)
now you can use it as
from imtest import salry
Now it works fine

Your imtest.py file is imported in your testim.py file, it's not the file that you're running. So it's not your main file, therefore the condition (name == 'main') will be false, and the salry function won't be defined.
You must remove the if statement:
d = dict()
def salry(a):
d["f"] = a
print (d)

Accessible variables at the root of a python script

I've declared a number of variables at the start of my script, as I'm using them in a number of different methods ("Functions" in python?). When I try to access them, I can't seem to get their value = or set them to another value for that matter. For example:
baseFile = open('C:/Users/<redacted>/Documents/python dev/ATM/Data.ICSF', 'a+')
secFile = open('C:/Users/<redacted>/Documents/python dev/ATM/security.ICSF', 'a+')
def usrInput(raw_input):
if raw_input == "99999":
self.close(True)
else:
identity = raw_input
def splitValues(source, string):
if source == "ident":
usrTitle = string.split('>')[1]
usrFN = string.split('>')[2]
usrLN = string.split('>')[3]
x = string.split('>')[4]
usrBal = Decimal(x)
usrBalDisplay = str(locale.currency(usrBal))
elif source == "sec":
usrPIN = string.split('>')[1]
pinAttempts = string.split('>')[2]
def openAccount(identity):
#read all the file first. it's f***ing heavy but it'll do here.
plString = baseFile.read()
xList = plString.split('|')
parm = str(identity)
for i in xList:
substr = i[0:4]
if parm == substr:
print "success"
usrString = str(i)
else:
lNumFunds = lNumFunds + 1
splitValues("ident", usrString)
When I place baseFile and secFile in the openAccount method, I can access the respective files as normal. However, when I place them at the root of the script, as in the example above, I can no longer access the file - although I can still "see" the variable.
Is there a reason to this? For reference, I am using Python 2.7.

methods ("Functions" in python?)
"function" when they "stand free"; "methods" when they are members of a class. So, functions in your case.
What you describe does definitely work in python. Hence, my diagnosis is that you already read something from the file elsewhere before you call openAccount, so that the read pointer is not at the beginning of the file.

How detect code change?

Let's suppose some code like this:
class X:
def myfunc(self):
print "orig"
def new_myfunc(self):
print "new"
X().myfunc()
X.myfunc = new_myfunc
X().myfunc()
where the new function is injected by a cheater.
Some functions can be altered,others not.
I would like to know how i can detect this code change.
For example i could make an dict that contain original function codes( with "func_code" ) and then check if they are changed
but how i can run the check at every "import"? there is a way to edit the autoloader in python?
edit: this is what i would like to do, but automatically for every import,how?
protection = {'X':'myfunc'}
f = {}
class X:
def myfunc(self):
print "orig"
def new_myfunc(self):
print "new"
#system check
for key,value in protection.iteritems():
protectedFunc = getattr(eval(key), value)
f[key] = { value : protectedFunc.func_code}
#cheater code
X.myfunc = new_myfunc
#system check
for key,value in protection.iteritems():
protectedFunc = getattr(eval(key), value)
if f[key][value] != protectedFunc.func_code:
print 'detected'
#call by my app
X().myfunc()

I know several ways for your task. For example you can check md5 or other. As variant you can use diff. Python has support of it one, two.

Using var from from function A to function B

On this sample code i want to use the variables on the function db_properties at the function connect_and_query. To accomplish that I choose the return. So, using that strategy the code works perfectly. But, in this example the db.properties files only has 4 variables. That said, if the properties file had 20+ variables, should I continue using return? Or is there a most elegant/cleaner/correct way to do that?
import psycopg2
import sys
from ConfigParser import SafeConfigParser
class Main:
def db_properties(self):
cfgFile='c:\test\db.properties'
parser = SafeConfigParser()
parser.read(cfgFile)
dbHost = parser.get('database','db_host')
dbName = parser.get('database','db_name')
dbUser = parser.get('database','db_login')
dbPass = parser.get('database','db_pass')
return dbHost,dbName,dbUser,dbPass
def connect_and_query(self):
try:
con = None
dbHost=self.db_properties()[0]
dbName=self.db_properties()[1]
dbUser=self.db_properties()[2]
dbPass=self.db_properties()[3]
con = None
qry=("select star from galaxy")
con = psycopg2.connect(host=dbHost,database=dbName, user=dbUser,
password=dbPass)
cur = con.cursor()
cur.execute(qry)
data = cur.fetchall()
for result in data:
qryResult = result[0]
print "the test result is : " +qryResult
except psycopg2.DatabaseError, e:
print 'Error %s' % e
sys.exit(1)
finally:
if con:
con.close()
operation=Main()
operation.connect_and_query()
Im using python 2.7
Regards

If there are a lot of variables, or if you want to easily change the variables being read, return a dictionary.
def db_properties(self, *variables):
cfgFile='c:\test\db.properties'
parser = SafeConfigParser()
parser.read(cfgFile)
return {
variable: parser.get('database', variable) for variable in variables
}
def connect_and_query(self):
try:
con = None
config = self.db_properties(
'db_host',
'db_name',
'db_login',
'db_pass',
)
#or you can use:
# variables = ['db_host','db_name','db_login','db_pass','db_whatever','db_whatever2',...]
# config = self.db_properties(*variables)
#now you can use any variable like: config['db_host']
# ---rest of the function here---
Edit: I refactored the code so you can specify the variables you want to load in the calling function itself.

You certainly don't want to call db_properties() 4 times; just call it once and store the result.
It's also almost certainly better to return a dict rather than a tuple, since as it is the caller needs to know what the method returns in order, rather than just having access to the values by their names. As the number of values getting passed around grows, this gets even harder to maintain.
e.g.:
class Main:
def db_properties(self):
cfgFile='c:\test\db.properties'
parser = SafeConfigParser()
parser.read(cfgFile)
configDict= dict()
configDict['dbHost'] = parser.get('database','db_host')
configDict['dbName'] = parser.get('database','db_name')
configDict['dbUser'] = parser.get('database','db_login')
configDict['dbPass'] = parser.get('database','db_pass')
return configDict
def connect_and_query(self):
try:
con = None
conf = self.db_properties()
con = None
qry=("select star from galaxy")
con = psycopg2.connect(host=conf['dbHost'],database=conf['dbName'],
user=conf['dbUser'],
password=conf['dbPass'])

NB: untested
You could change your db_properties to return a dict:
from functools import partial
# call as db_properties('db_host', 'db_name'...)
def db_properties(self, *args):
parser = SafeConfigParser()
parser.read('config file')
getter = partial(parser.get, 'database')
return dict(zip(args, map(getter, args)))
But otherwise it's probably best to keep the parser as an attribute of the instance, and provide a convenience method...
class whatever(object):
def init(self, *args, **kwargs):
# blah blah blah
cfgFile='c:\test\db.properties'
self._parser = SafeConfigParser()
self._parser.read(cfgFile)
#property
def db_config(self, key):
return self._parser.get('database', key)
Then use con = psycopg2.connect(host=self.db_config('db_host')...)

I'd suggest returning a namedtuple:
from collections import namedtuple
# in db_properties()
return namedtuple("dbconfig", "host name user password")(
parser.get('database','db_host'),
parser.get('database','db_name'),
parser.get('database','db_login'),
parser.get('database','db_pass'),
)
Now you have an object that you can access either by index or by attribute.
config = self.db_properties()
print config[0] # db_host
print config.host # same