I am creating a trading bot. I have 2 files a settings.json file and a main.py file.
my settings.json file :
`{
"username": "51410030",
"password": "s5p3GI1zY",
"server": "Alpari-MT5-Demo",
"mt5Pathway": "C://Program Files/Alpari MT5/terminal64.exe",
"symbols": ["USDJPY.a"],
"timeframe": "M30"
}
and my main.py file :
import json
import os
import mt5_interface
import strategy
# Function to import settings from settings.json
def get_project_settings(importFilepath):
# Test the filepath to sure it exists
if os.path.exists(importFilepath):
# Open the file
f = open(importFilepath, "r")
# Get the information from file
project_settings = json.load(f)
# Close the file
f.close()
project_settings = list(project_settings)
# Return project settings to program
return project_settings
else:
return ImportError
# Main function
if __name__ == '__main__':
# Set up the import filepath
import_filepath = "C:/Users/james/PycharmProjects/how_to_build_a_metatrader5_trading_bot_expert_advisor/settings.json"
# Import project settings
project_settings = get_project_settings(import_filepath)
# Start MT5
mt5_interface.start_mt5(project_settings["username"], project_settings["password"], project_settings["server"],
project_settings["mt5Pathway"])
# Initialize symbols
mt5_interface.initialize_symbols(project_settings["symbols"])
# Select symbol to run strategy on
symbol_for_strategy = project_settings['symbols'][0]
# Start strategy one on selected symbol
strategy.strategy_one(symbol=symbol_for_strategy, timeframe=project_settings['timeframe'],
pip_size=project_settings['pip_size'])
my prblem is when i run my main.py file it gives me this error:
Traceback (most recent call last):
File "i:\Traiding Bot\code\main.py", line 32, in <module>
mt5_interface.start_mt5(project_settings["username"], project_settings["password"], project_settings["server"],
TypeError: 'type' object is not subscriptable
please help me.
I couldn't find a solution please help me.
There are a few issues with your code:
You're trying to access fields in a list. That's not possible, you should keep your list a dictionary if you want access its fields.
You're returning an ImportError, if you want to raise an error, use raise ImportError("Your error message"). Or if you want to catch the error, use a try: <your code> except: return None and then check if you're returning None or not.
I am trying to collect my main functions in a separate .py module, that I am importing in my Juptyer Notebook. So I can call a function when I need it. I would like to keep this simple, as I am a beginner in Python.
I have 2 main problems:
When I import the db.py module in my Jupyter notebook, all def functions are executed.
I read this is normal in Python and it can be prevented by placing if __name__ == "__main__": before the functions I want to execute manually. And it seems to work.
However, if I add the code above, when I run the functions manually, I get this other error
module 'ch_db' has no attribute 'get_data'
Any suggestion?
db.py - this is the file where i would like to collect my functions
def main():
print('Credentials required')
print('Enter User:')
pass_user = getpass.getpass()
print('Enter db pssw:')
pass_clickhouse = getpass.getpass()
if __name__ == "__main__":
def get_data():
# ... some other code to get data from database ...
df_ch = pd.DataFrame(result)
return df_ch
df_ch = get_data()
def seg_data():
seg_startdate = input('Enter start date (yyyy-mm-dd): ')
seg_finishdate = input('Enter end date (yyyy-mm-dd): ')
df_ch_seg = df_ch[(df_ch['event_datetime'] > seg_startdate)
& (df_ch['event_datetime'] < seg_finishdate)]
return df_ch_seg
df_ch_seg = seg_data()
In Jupyter Notebook:
In[1]: import ch_db
In[2]: db.get_data()
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-2-78c2ac644f6e> in <module>
----> 1 ch_db.get_data()
AttributeError: module 'db' has no attribute 'get_data'
I have read many article over the last 6 hours and i still don't understand mocking and unit-testing. I want to unit test a open function, how can i do this correctly?
i am also concerned as the bulk of my code is using external files for data import and manipulation. I understand that i need to mock them for testing, but I am struggling to understand how to move forward.
Some advice please. Thank you in advance
prototype5.py
import os
import sys
import io
import pandas
pandas.set_option('display.width', None)
def openSetupConfig (a):
"""
SUMMARY
Read setup file
setup file will ONLY hold the file path of the working directory
:param a: str
:return: contents of the file stored as str
"""
try:
setupConfig = open(a, "r")
return setupConfig.read()
except Exception as ve:
ve = (str(ve) + "\n\nPlease ensure setup file " + str(a) + " is available")
sys.exit(ve)
dirPath = openSetupConfig("Setup.dat")
test_prototype5.py
import prototype5
import unittest
class TEST_openSetupConfig (unittest.TestCase):
"""
Test the openSetupConfig function from the prototype 5 library
"""
def test_open_correct_file(self):
result = prototype5.openSetupConfig("Setup.dat")
self.assertTrue(result)
if __name__ == '__main__':
unittest.main()
So the rule of thumb is to mock, stub or fake all external dependencies to the method/function under test. The point is to test the logic in isolation. So in your case you want to test that it can open a file or log an error message if it can't be opened.
import unittest
from mock import patch
from prototype5 import openSetupConfig # you don't want to run the whole file
import __builtin__ # needed to mock open
def test_openSetupConfig_with_valid_file(self):
"""
It should return file contents when passed a valid file.
"""
expect = 'fake_contents'
with patch('__builtin__.open', return_value=expect) as mock_open:
actual = openSetupConfig("Setup.dat")
self.assertEqual(expect, actual)
mock_open.assert_called()
#patch('prototype5.sys.exit')
def test_openSetupConfig_with_invalid_file(self, mock_exit):
"""
It should log an error and exit when passed an invalid file.
"""
with patch('__builtin__.open', side_effect=FileNotFoundError) as mock_open:
openSetupConfig('foo')
mock_exit.assert_called()
I'm trying this for almost two hours now, without any luck.
I have a module that looks like this:
try:
from zope.component import queryUtility # and things like this
except ImportError:
# do some fallback operations <-- how to test this?
Later in the code:
try:
queryUtility(foo)
except NameError:
# do some fallback actions <-- this one is easy with mocking
# zope.component.queryUtility to raise a NameError
Any ideas?
EDIT:
Alex's suggestion doesn't seem to work:
>>> import __builtin__
>>> realimport = __builtin__.__import__
>>> def fakeimport(name, *args, **kw):
... if name == 'zope.component':
... raise ImportError
... realimport(name, *args, **kw)
...
>>> __builtin__.__import__ = fakeimport
When running the tests:
aatiis#aiur ~/work/ao.shorturl $ ./bin/test --coverage .
Running zope.testing.testrunner.layer.UnitTests tests:
Set up zope.testing.testrunner.layer.UnitTests in 0.000 seconds.
Error in test /home/aatiis/work/ao.shorturl/src/ao/shorturl/shorturl.txt
Traceback (most recent call last):
File "/usr/lib64/python2.5/unittest.py", line 260, in run
testMethod()
File "/usr/lib64/python2.5/doctest.py", line 2123, in runTest
test, out=new.write, clear_globs=False)
File "/usr/lib64/python2.5/doctest.py", line 1361, in run
return self.__run(test, compileflags, out)
File "/usr/lib64/python2.5/doctest.py", line 1282, in __run
exc_info)
File "/usr/lib64/python2.5/doctest.py", line 1148, in report_unexpected_exception
'Exception raised:\n' + _indent(_exception_traceback(exc_info)))
File "/usr/lib64/python2.5/doctest.py", line 1163, in _failure_header
out.append(_indent(source))
File "/usr/lib64/python2.5/doctest.py", line 224, in _indent
return re.sub('(?m)^(?!$)', indent*' ', s)
File "/usr/lib64/python2.5/re.py", line 150, in sub
return _compile(pattern, 0).sub(repl, string, count)
File "/usr/lib64/python2.5/re.py", line 239, in _compile
p = sre_compile.compile(pattern, flags)
File "/usr/lib64/python2.5/sre_compile.py", line 507, in compile
p = sre_parse.parse(p, flags)
AttributeError: 'NoneType' object has no attribute 'parse'
Error in test BaseShortUrlHandler (ao.shorturl)
Traceback (most recent call last):
File "/usr/lib64/python2.5/unittest.py", line 260, in run
testMethod()
File "/usr/lib64/python2.5/doctest.py", line 2123, in runTest
test, out=new.write, clear_globs=False)
File "/usr/lib64/python2.5/doctest.py", line 1351, in run
self.debugger = _OutputRedirectingPdb(save_stdout)
File "/usr/lib64/python2.5/doctest.py", line 324, in __init__
pdb.Pdb.__init__(self, stdout=out)
File "/usr/lib64/python2.5/pdb.py", line 57, in __init__
cmd.Cmd.__init__(self, completekey, stdin, stdout)
File "/usr/lib64/python2.5/cmd.py", line 90, in __init__
import sys
File "<doctest shorturl.txt[10]>", line 4, in fakeimport
NameError: global name 'realimport' is not defined
However, it does work when I run the same code from the python interactive console.
MORE EDIT:
I'm using zope.testing and a test file, shorturl.txt that has all the tests specific to this part of my module. First I'm importing the module with zope.component available, to demonstrate & test the usual usage. The absence of zope.* packages is considered an edge-case, so I'm testing it later. Thus, I have to reload() my module, after making zope.* unavailable, somehow.
So far I've even tried using tempfile.mktempdir() and empty zope/__init__.py and zope/component/__init__.py files in the tempdir, then inserting tempdir to sys.path[0], and removing the old zope.* packages from sys.modules.
Didn't work either.
EVEN MORE EDIT:
In the meantime, I've tried this:
>>> class NoZope(object):
... def find_module(self, fullname, path):
... if fullname.startswith('zope'):
... raise ImportError
...
>>> import sys
>>> sys.path.insert(0, NoZope())
And it works well for the namespace of the test suite (= for all imports in shorturl.txt), but it is not executed in my main module, ao.shorturl. Not even when I reload() it. Any idea why?
>>> import zope # ok, this raises an ImportError
>>> reload(ao.shorturl) <module ...>
Importing zope.interfaces raises an ImportError, so it doesn't get to the part where I import zope.component, and it remains in the ao.shorturl namespace. Why?!
>>> ao.shorturl.zope.component # why?!
<module ...>
Just monkeypatch into the builtins your own version of __import__ -- it can raise whatever you wish when it recognizes it's being called on the specific modules for which you want to mock up errors. See the docs for copious detail. Roughly:
try:
import builtins
except ImportError:
import __builtin__ as builtins
realimport = builtins.__import__
def myimport(name, globals, locals, fromlist, level):
if ...:
raise ImportError
return realimport(name, globals, locals, fromlist, level)
builtins.__import__ = myimport
In lieu of the ..., you can hardcode name == 'zope.component', or arrange things more flexibly with a callback of your own that can make imports raise on demand in different cases, depending on your specific testing needs, without requiring you to code multiple __import__-alike functions;-).
Note also that if what you use, instead of import zope.component or from zope.component import something, is from zope import component, the name will then be 'zope', and 'component' will then be the only item in the fromlist.
Edit: the docs for the __import__ function say that the name to import is builtin (like in Python 3), but in fact you need __builtins__ -- I've edited the code above so that it works either way.
This is what I justed in my unittests.
It uses PEP-302 "New Import Hooks". (Warning: the PEP-302 document and the more concise release notes I linked aren't exactly accurate.)
I use meta_path because it's as early as possible in the import sequence.
If the module has already been imported (as in my case, because earlier unittests mock against it), then it's necessary to remove it from sys.modules before doing the reload on the dependent module.
# Ensure we fallback to using ~/.pif if XDG doesn't exist.
>>> import sys
>>> class _():
... def __init__(self, modules):
... self.modules = modules
...
... def find_module(self, fullname, path=None):
... if fullname in self.modules:
... raise ImportError('Debug import failure for %s' % fullname)
>>> fail_loader = _(['xdg.BaseDirectory'])
>>> sys.meta_path.append(fail_loader)
>>> del sys.modules['xdg.BaseDirectory']
>>> reload(pif.index) #doctest: +ELLIPSIS
<module 'pif.index' from '...'>
>>> pif.index.CONFIG_DIR == os.path.expanduser('~/.pif')
True
>>> sys.meta_path.remove(fail_loader)
Where the code inside pif.index looks like:
try:
import xdg.BaseDirectory
CONFIG_DIR = os.path.join(xdg.BaseDirectory.xdg_data_home, 'pif')
except ImportError:
CONFIG_DIR = os.path.expanduser('~/.pif')
To answer the question about why the newly reloaded module has properties of the old and new loads, here are two example files.
The first is a module y with an import failure case.
# y.py
try:
import sys
_loaded_with = 'sys'
except ImportError:
import os
_loaded_with = 'os'
The second is x which demonstrates how leaving handles about for a module can affect its properties when being reloaded.
# x.py
import sys
import y
assert y._loaded_with == 'sys'
assert y.sys
class _():
def __init__(self, modules):
self.modules = modules
def find_module(self, fullname, path=None):
if fullname in self.modules:
raise ImportError('Debug import failure for %s' % fullname)
# Importing sys will not raise an ImportError.
fail_loader = _(['sys'])
sys.meta_path.append(fail_loader)
# Demonstrate that reloading doesn't work if the module is already in the
# cache.
reload(y)
assert y._loaded_with == 'sys'
assert y.sys
# Now we remove sys from the modules cache, and try again.
del sys.modules['sys']
reload(y)
assert y._loaded_with == 'os'
assert y.sys
assert y.os
# Now we remove the handles to the old y so it can get garbage-collected.
del sys.modules['y']
del y
import y
assert y._loaded_with == 'os'
try:
assert y.sys
except AttributeError:
pass
assert y.os
If you don't mind changing your program itself, you could also put the import call in a function and patch that in your tests.
I'm working on writing a test for a module for docker-py, but I can't seem to get the test to work properly.
The function I'm testing looks as follows:
def parse_env_file(env_file):
"""
Reads a line-separated environment file.
The format of each line should be "key=value".
"""
environment = []
if os.path.isfile(env_file):
with open(env_file, 'r') as f:
# We can't use f.readlines() here as it's not implemented in Mock
for line in f.read().split('\n'):
parse_line = line.strip().split('=')
if len(parse_line) == 2 and line[0] != '#':
k = parse_line[0]
v = parse_line[1]
environment.append('{0}={1}'.format(k, v))
return environment
The test then looks like this:
def test_parse_env_file_proper(self):
with mock.patch('os.path.isfile', return_value=True):
mock_env_file = 'USER=jdoe\nPASS=secret'
with mock.patch('{}.open'.format(__name__), mock.mock_open(read_data=mock_env_file)):
get_parse_env_file = parse_env_file('foobar')
self.assertEqual(get_parse_env_file, ['USER=jdoe', 'PASS=secret'])
When I run the test however, I get the following error:
======================================================================
ERROR: test_parse_env_file_proper (__main__.UtilsTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "tests/utils_test.py", line 102, in test_parse_env_file_proper
with mock.patch('{}.open'.format(__name__), mock.mock_open(read_data=mock_env_file)):
File "/Users/mvip/code/private/github/docker-py/.tox/py27/lib/python2.7/site-packages/mock.py", line 1268, in __enter__
original, local = self.get_original()
File "/Users/mvip/code/private/github/docker-py/.tox/py27/lib/python2.7/site-packages/mock.py", line 1242, in get_original
"%s does not have the attribute %r" % (target, name)
AttributeError: <module '__main__' from 'tests/utils_test.py'> does not have the attribute 'open'
Any pointers here would be helpful.
The problem is that, as a builtin, open isn't directly found in the module, but rather as fallback in the builtins module.
To get around this you should include create=True when patching.
from unittest import mock
with mock.patch(__name__+".open", mock.mock_open(read_data="data"), create=True):
with open("somefile") as f:
assert f.read() == "data"
However, this only patches open in the current module (the module running the test, not the module under test).
So you'd be better off doing:
import unittest
from unittest.mock import mock_open, patch
import module_under_test
def func():
with open("somefile") as f:
return f.read()
class MyTestCase(unittest.TestCase):
def test_open(self):
data = "some data"
with patch.object(module_under_test, "open", mock_open(read_data=data), create=True):
result = module_under_test.func()
self.assertEqual(result, data)
if __name__ == "__main__":
unittest.main()