Develop Reference

Python: Project Package / Module structure dependency Problem

I was hoping someone could help me figure out an odd "dependency" problem. I have a fairly large python project, with a slimmed down structure that looks like:
Sitka
│ DataTickers.py
│ example.csv
│ FinDates.py
│ SitkaMongo.py
│ tickers_csv.csv
│ __init__.py
│
├───Fin
│ │ main.py
│ │ md_provider_control.py
│ │ Tofino.py
│ │ __init__.py
│ │
│ │
│ ├───Instruments
│ │ │ market_standard_instruments.py
│ │ └ __init__.py
│ │
│ ├───Env
│ │ │ CurveClass.py
│ │
│ ├───Utils
│ │ charting.py
│ │ exchange_identifier_mapper.py
│ │ fin_mapper.py
│ │ md_provider_simulation.py
│ └ __init__.py
Tofino.py:
from .Env.CurveClass import CurveData as _CurveData
class Tofino():
def __init__(self, mdp, VAL_ENV = None):
mdp.tofino = self # link Tofino
# Public VE Refernce
self.val_env = VAL_ENV
self.ir_config = VAL_ENV.market
market_standard_instruments.py:
# Standard Imports
import Sitka.FinDates as fdate
import datetime as dt
import re
from itertools import product
# bunch of functions after this.
CurveClass.py:
import pandas as pd
import datetime as dt
from dateutil.relativedelta import relativedelta
class CurveData():
def __init__(self):
self.do_stuff= self._stuff()
main.py
from Sitka.FinDates import getMainDates
# Sitka- Custom Imports
from .md_provider_control import MD_ProviderV3
from .Tofino import Tofino
import Sitka.Fin.Instruments.market_standard_instruments as mkt_std
def main() -> Tofino:
# < ---- do a bunch of stuff ---- >
return Tofino(mdp = mdp, VAL_ENV=ve.GLOBAL_VALN_ENV)
And lastly, Sitka.Fin.__ init __.py:
import logging
import traceback
# Run Valuation Environment Startup
from .main import main
# Global Variables:
from .Tofino import Tofino as _Tofino
tofino : _Tofino
tofino = None
try:
tofino = main() # I was trying some stuff out here, hence the weird traceback in try
except:
print(traceback.format_exc())
My issue is, after all that, is when I run import Sitka.Fin as fin, this line in main.py
import Sitka.Fin.Instruments.market_standard_instruments as mkt_std
fires off the Sitka.Fin__init__ process again before we even get to the try block (so init basically runs 2x).
Any help is appreciated!
P.S. Basically I'm just including subfolder init's because its the only way I know how to get Intellsense/autocomplete in the IDE to work nicely... I would love to know how to make my code 'cleaner' from that sense.
Edit:
A simpler way to look at the problem. Lets say I open a new IPython console, and only do:
import Sitka.Fin.Instruments.market_standard_instruments as mkt_std
Simply doing this kicks off the entire Sitka.Fin.__init__ procedure [which I wouldn't have expected]

It seems you only want some code of the main.py to run when the file itself is running. Try using:
if __name__ in "__main__": # All sikta imports
from Sitka.FinDates import getMainDates
from .md_provider_control import MD_ProviderV3
from .Tofino import Tofino
import Sitka.Fin.Instruments.market_standard_instruments as mkt_std

Building Hierarchy Graph from Strings

I am trying to build a hierarchy graph from a list of strings I have. Each string just consists of its absolute hierarchy seperated by dots. Example Strings:
memberA.memberB.memberC
memberA.memberE.memberG
memberA.memberE
memberA.memberB
memberA.memberF.memberX
memberA.memberF
memberA.memberF.memberG #in this case this should be treated as a seperate leaf node and not the same as in memberA.memberE.memberG
I tried using Anytree and Treelib to achieve this but I could not come up with a working solution. Although this problem looks simple (might not be) I just can not figure it out.

You'd need to keep track of which node objects correspond to a certain path. For that you can use a dictionary, that maps a given path to a node object.
With AnyTree it could look like this:
from anytree import Node, RenderTree
strings = [
"memberA.memberB.memberC",
"memberA.memberE.memberG",
"memberA.memberE",
"memberA.memberB",
"memberA.memberF.memberX",
"memberA.memberF",
"memberA.memberF.memberG"
]
d = {}
root = Node("root")
for s in strings:
path = "root"
parent = root
for name in s.split("."):
path += "." + name
if path not in d:
d[path] = Node(name, parent=parent)
parent = d[path]
print(RenderTree(root))
Output:
Node('/root')
└── Node('/root/memberA')
├── Node('/root/memberA/memberB')
│ └── Node('/root/memberA/memberB/memberC')
├── Node('/root/memberA/memberE')
│ └── Node('/root/memberA/memberE/memberG')
└── Node('/root/memberA/memberF')
├── Node('/root/memberA/memberF/memberX')
└── Node('/root/memberA/memberF/memberG')
In case you want "memberA" to be the root, then you need to make sure your input data only has strings that start with "memberA". And then at the end of the above script do:
root = root.children[0]
root.parent = None
print(RenderTree(root))
Output:
Node('/memberA')
├── Node('/memberA/memberB')
│ └── Node('/memberA/memberB/memberC')
├── Node('/memberA/memberE')
│ └── Node('/memberA/memberE/memberG')
└── Node('/memberA/memberF')
├── Node('/memberA/memberF/memberX')
└── Node('/memberA/memberF/memberG')

How do I pass in a self argument to python cProfile

I am trying to use cProfiling with python.
My python project has the following directory structure:
my-project
├── src
│ ├── lib
│ └── app
│ └── data
│ └── car_sim.py
│
│
│
│
├── ptests
│ ├── src
│ └── lib
│ └── app
│ └── data
│ └── cprofile_test.py
I have a function inside car_sim.py that I want to cprofile and it is called "sim_text". It contains a function called:
#car_sim.py
import os
class RootSimulator:
def sim_text(self, text):
return text
I use the following code inside cprofile_test.py:
#cprofile_test.py
import cProfile
import pstats
import io
import src.lib.app.data.car_sim as car_sim_functions
pr = cProfile.Profile()
pr.enable()
text = 'my blabla sentence' #i can pass in this text below i guess...
#how do i pass to the below????!!
my_result = car_sim_functions.RootSimulator.sim_text()
pr.disable()
s = io.StringIO()
ps = pstats.Stats(pr, stream=s).sort_stats('tottime')
ps.print_stats()
with open('test.txt', 'w+') as f:
f.write(s.getvalue())
Now... when I run it using the command
python -m cProfile ptests/src/lib/app/data/cprofile_test.py
I get the following error:
TypeError: sim_text() missing 2 required positional arguments: 'self' and 'text'
My question is... It expects 2 args, so how do I pass in the "self" arg. For the 2nd arg, "text" I can pass in a value no problem.

class RootSimulator:
def sim_text(self, text):
return text
Defines an instance method on instances of RootSimulator. You are trying to call sim_text from the class itself. You need to create an instance:
simulator = car_sim_functions.RootSimulator()
my_result = simulator.sim_text()
If sim_text() does not actually need to be attached to an instance of the simulator, perhaps you don't need a class at all (just make it a plain function), or you could make it a static method:
class RootSimulator:
#staticmethod
def sim_text(text):
return text
Note that it doesn't need self anymore.

How to get relative path for logging.ini and log file in python over multiple OS

I have created one wrapper logger class which actually wraps the functionality of logging module. I have created it so that every class in my application using logging doesn't need to add logging.config.fileConfig() and logging.getLogger() at the top every-time.
My wrapper class looks like this -
class MyFileLogger():
# The config file for logging formatter.
logging.config.fileConfig('logging.ini', disable_existing_loggers=False)
def __init__(self, name):
self.logger=logging.getLogger(name)
def log_debug(self, message):
self.logger.debug(message)
logging.ini file -
[loggers]
keys=root
[handlers]
keys=fileHandler
[formatters]
keys=myFormatter
[logger_root]
level=NOTSET
handlers=fileHandler
[handler_fileHandler]
class=handlers.TimedRotatingFileHandler
level=DEBUG
formatter=myFormatter
args=('testlog.log','midnight',)
[formatter_myFormatter]
format=%(asctime)s |%(name)-12s |%(levelname)-8s |%(message)s
datefmt=
class=logging.Formatter
MyFileLogger.py and logging.ini file resides on the same path. My intention to create testlog.log is also in same path as logging.ini file.
Implementation of the MyFileLogger.py would be like -
imp.py -
fileLogger = MyFileLogger(__name__)
message = "hey!"
fileLogger.log_debug(message)
Now this imp.py doesn't reside on the same path as MyFileLogger.py and logging.ini. That's why when I run it, this gives me below error cause the python is running at imp.py path.
File "C:\Python34\lib\logging\config.py", line 76, in fileConfig
formatters = _create_formatters(cp)
File "C:\Python34\lib\logging\config.py", line 109, in _create_formatters
flist = cp["formatters"]["keys"]
File "C:\Python34\lib\configparser.py", line 937, in __getitem__
raise KeyError(key)
KeyError: 'formatters'
I know if I just give an absolute path to logging.ini and testlog.log it will work. But I don't want to go there, since this application can be ran in Windows or in Linux. So there would be path problem.
Now even if I don't care about the Windows-Linux thing and only concentrate on the Linux. There is another problem. So if we assume the current directory structure as this ("ProjectDir/src" is added in PYTHONPATH and assuming there is __init__.py in every module folders)-
ProjectDir
├── src
│ └── sales
│ ├── common
│ │ ├── logger
│ │ │ ├── logging.ini
│ │ │ ├── MyFileLogger.py
│ │ │ └── testlog.log
│ │ └── implmnts
│ │ └── imp.py
│ └── unitTest
│ └── test_MyFileLogger.py
└── bar.txt
Let's assume:
imp.py -
fileLogger = MyFileLogger(__name__)
message = "From imp.py"
fileLogger.log_debug(message)
test_MyFileLogger.py -
fileLogger = MyFileLogger(__name__)
message = "from test_MyFileLogger.py"
fileLogger.log_debug(message)
Then if I run imp.py I need to change the logging.config.fileConfig('logging.ini', disable_existing_loggers=False) to logging.config.fileConfig('../logger/logging.ini', disable_existing_loggers=False)
After that it will run but when I try to run test_MyFileLogger.py I need to change again to logging.config.fileConfig('../common/logger/logging.ini', disable_existing_loggers=False)
So my code is dependent upon which file I am running which I want to get rid of. Is there any way I can dynamically get the path of logging.ini and testlog.log independent of which file(from other path) is invoking this? I don't want to hardcore the absolute path, since this needs to run in both Windows and Linux, any solution?

Multiprocessing error : can't import module

I'm getting an error message that i'm unable to tackle. I don't get what's the issue with the multiprocessing library and i don't understand why it says that it is impossible to import the build_database module but in the same time it executes perfectly a function from that module.
Could somebody tell me is he sees something. Thank you.
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Python27\lib\multiprocessing\forking.py", line 380, in main
Traceback (most recent call last):
File "<string>", line 1, in <module>
prepare(preparation_data)
File "C:\Python27\lib\multiprocessing\forking.py", line 380, in main
File "C:\Python27\lib\multiprocessing\forking.py", line 495, in prepare
prepare(preparation_data)
'__parents_main__', file, path_name, etc
File "C:\Python27\lib\multiprocessing\forking.py", line 495, in prepare
File "C:\Users\Comp3\Desktop\User\Data\main.py", line 4, in <module>
'__parents_main__', file, path_name, etc
import database.build_database
File "C:\Users\Comp3\Desktop\User\Data\main.py", line 4, in <module>
ImportError : import database.build_database
NImportErroro module named build_database:
No module named build_database
This is what i have in my load_bigquery.py file:
# Send CSV to Cloud Storage
def load_send_csv(table):
job = multiprocessing.current_process().name
print '[' + table + '] : job starting (' + job + ')'
bigquery.send_csv(table)
#timer.print_timing
def send_csv(tables):
jobs = []
build_csv(tables)
for t in tables:
if t not in csv_targets:
continue
print ">>>> Starting " + t
# Load CSV in BigQuery, as parallel jobs
j = multiprocessing.Process(target=load_send_csv, args=(t,))
jobs.append(j)
j.start()
# Wait for jobs to complete
for j in jobs:
j.join()
And i call it like this from my main.py :
bigquery.load_bigquery.send_csv(tables)
My folder is like this:
src
| main.py
|
├───bigquery
│ │ bigquery.py
│ │ bigquery2.dat
│ │ client_secrets.json
│ │ herokudb.py
│ │ herokudb.pyc
│ │ distimo.py
│ │ flurry.py
│ │ load_bigquery.py
│ │ load_bigquery.pyc
│ │ timer.py
│ │ __init__.py
│ │ __init__.pyc
│ │
│ │
├───database
│ │ build_database.py
│ │ build_database.pyc
│ │ build_database2.py
│ │ postgresql.py
│ │ timer.py
│ │ __init__.py
│ │ __init__.pyc
That function works perfectly if i execute load_bigquery.py alone but if i import it into main.py it fails with the errors given above.
UPDATE :
Here are my import, maybe it might help:
main.py
import database.build_database
import bigquery.load_bigquery
import views.build_analytics
import argparse
import getopt
import sys
import os
load_bigquery.py
import sys
import os
import subprocess
import time
import timer
import distimo
import flurry
import herokudb
import bigquery
import multiprocessing
import httplib2
bigquery.py
import sys
import os
import subprocess
import json
import time
import timer
import httplib2
from pprint import pprint
from apiclient.discovery import build
from oauth2client.file import Storage
from oauth2client.client import AccessTokenRefreshError
from oauth2client.client import OAuth2WebServerFlow
from oauth2client.client import flow_from_clientsecrets
from oauth2client.tools import run
from apiclient.errors import HttpError
Maybe the issue is with the fact that load_bigquery.py imports multiprocessing and then main.py imports load_bigquery.py ?

You are probably missing the __init__.py inside src/bigquery/. So your source folders should be:
> src/main.py
> src/bigquery/__init__.py
> src/bigquery/load_bigquery.py
> src/bigquery/bigquery.py
The __init__.py just needs to be empty and is only there so that Python knows that bigquery is a Python package.
UPDATED: Apparently the __init__.py file is present. The actual error message talks about a different error, which is it cannot import database.build_database.
My suggestion is to look into that. It is not mentioned as being in the src folder...
UPDATE 2: I think you have a clash with your imports. Python 2 has a slightly fuzzy relative import, which sometimes catches people out. You have both a package at the same level of main.py called database and one inside bigquery called database. I think somehow you are ending up with the one inside bigquery, which doesn't have build_database. Try renaming one of them.