I am trying to use cProfiling with python.
My python project has the following directory structure:
my-project
├── src
│ ├── lib
│ └── app
│ └── data
│ └── car_sim.py
│
│
│
│
├── ptests
│ ├── src
│ └── lib
│ └── app
│ └── data
│ └── cprofile_test.py
I have a function inside car_sim.py that I want to cprofile and it is called "sim_text". It contains a function called:
#car_sim.py
import os
class RootSimulator:
def sim_text(self, text):
return text
I use the following code inside cprofile_test.py:
#cprofile_test.py
import cProfile
import pstats
import io
import src.lib.app.data.car_sim as car_sim_functions
pr = cProfile.Profile()
pr.enable()
text = 'my blabla sentence' #i can pass in this text below i guess...
#how do i pass to the below????!!
my_result = car_sim_functions.RootSimulator.sim_text()
pr.disable()
s = io.StringIO()
ps = pstats.Stats(pr, stream=s).sort_stats('tottime')
ps.print_stats()
with open('test.txt', 'w+') as f:
f.write(s.getvalue())
Now... when I run it using the command
python -m cProfile ptests/src/lib/app/data/cprofile_test.py
I get the following error:
TypeError: sim_text() missing 2 required positional arguments: 'self' and 'text'
My question is... It expects 2 args, so how do I pass in the "self" arg. For the 2nd arg, "text" I can pass in a value no problem.
class RootSimulator:
def sim_text(self, text):
return text
Defines an instance method on instances of RootSimulator. You are trying to call sim_text from the class itself. You need to create an instance:
simulator = car_sim_functions.RootSimulator()
my_result = simulator.sim_text()
If sim_text() does not actually need to be attached to an instance of the simulator, perhaps you don't need a class at all (just make it a plain function), or you could make it a static method:
class RootSimulator:
#staticmethod
def sim_text(text):
return text
Note that it doesn't need self anymore.
Related
I was hoping someone could help me figure out an odd "dependency" problem. I have a fairly large python project, with a slimmed down structure that looks like:
Sitka
│ DataTickers.py
│ example.csv
│ FinDates.py
│ SitkaMongo.py
│ tickers_csv.csv
│ __init__.py
│
├───Fin
│ │ main.py
│ │ md_provider_control.py
│ │ Tofino.py
│ │ __init__.py
│ │
│ │
│ ├───Instruments
│ │ │ market_standard_instruments.py
│ │ └ __init__.py
│ │
│ ├───Env
│ │ │ CurveClass.py
│ │
│ ├───Utils
│ │ charting.py
│ │ exchange_identifier_mapper.py
│ │ fin_mapper.py
│ │ md_provider_simulation.py
│ └ __init__.py
Tofino.py:
from .Env.CurveClass import CurveData as _CurveData
class Tofino():
def __init__(self, mdp, VAL_ENV = None):
mdp.tofino = self # link Tofino
# Public VE Refernce
self.val_env = VAL_ENV
self.ir_config = VAL_ENV.market
market_standard_instruments.py:
# Standard Imports
import Sitka.FinDates as fdate
import datetime as dt
import re
from itertools import product
# bunch of functions after this.
CurveClass.py:
import pandas as pd
import datetime as dt
from dateutil.relativedelta import relativedelta
class CurveData():
def __init__(self):
self.do_stuff= self._stuff()
main.py
from Sitka.FinDates import getMainDates
# Sitka- Custom Imports
from .md_provider_control import MD_ProviderV3
from .Tofino import Tofino
import Sitka.Fin.Instruments.market_standard_instruments as mkt_std
def main() -> Tofino:
# < ---- do a bunch of stuff ---- >
return Tofino(mdp = mdp, VAL_ENV=ve.GLOBAL_VALN_ENV)
And lastly, Sitka.Fin.__ init __.py:
import logging
import traceback
# Run Valuation Environment Startup
from .main import main
# Global Variables:
from .Tofino import Tofino as _Tofino
tofino : _Tofino
tofino = None
try:
tofino = main() # I was trying some stuff out here, hence the weird traceback in try
except:
print(traceback.format_exc())
My issue is, after all that, is when I run import Sitka.Fin as fin, this line in main.py
import Sitka.Fin.Instruments.market_standard_instruments as mkt_std
fires off the Sitka.Fin__init__ process again before we even get to the try block (so init basically runs 2x).
Any help is appreciated!
P.S. Basically I'm just including subfolder init's because its the only way I know how to get Intellsense/autocomplete in the IDE to work nicely... I would love to know how to make my code 'cleaner' from that sense.
Edit:
A simpler way to look at the problem. Lets say I open a new IPython console, and only do:
import Sitka.Fin.Instruments.market_standard_instruments as mkt_std
Simply doing this kicks off the entire Sitka.Fin.__init__ procedure [which I wouldn't have expected]
It seems you only want some code of the main.py to run when the file itself is running. Try using:
if __name__ in "__main__": # All sikta imports
from Sitka.FinDates import getMainDates
from .md_provider_control import MD_ProviderV3
from .Tofino import Tofino
import Sitka.Fin.Instruments.market_standard_instruments as mkt_std
So I'm working on a rather big Python 3 project where I'm writing unit tests for some of the files using the unittest library. In one unit test, the tested file imports a function from another python package whose __init__ file itself imports from the tested file. This leads to an ImportError during the unit test.
It is desired for the __init__.py to import from the tested file periphery\foo.py, so I would like to know if there is a possibility to make the unit test work without removing the import from __init__.py
Since the project contains a lot of files, I created a minimal example that illustrates the structure and in which the error can be reproduced. The project structure looks like this
├───core
│ bar.py
│ __init__.py
│
├───periphery
│ foo.py
│ __init__.py
│
└───tests
│ __init__.py
│
├───core
│ __init__.py
│
└───periphery
test_foo.py
__init__.py
The init file in core core/__init__.py contains the code
# --- periphery ---
from periphery.foo import Foo
while periphery/foo.py, which is the file to be tested, looks like
from core.bar import Bar
def Foo():
bar = Bar()
return bar
Finally, the unit test has the following structure
from unittest import TestCase
from periphery.foo import Foo
class Test(TestCase):
def test_foo(self):
""" Test that Foo() returns "bar" """
self.assertEqual(Foo(), "bar")
Running the unit test yields the following error:
ImportError: cannot import name 'Foo' from partially initialized module 'periphery.foo' (most likely due to a circular import)
I generate the documentation of my Python code from the docstrings via Doxygen (1.9.1) in addition with doxypypy (git version from today).
My project is called Project and the packages name in it (which should be imported) is mypackage. When I look into the tree sidebar of the generated html it looks like this:
└── Project
└── Packages
└── Packages
└──mypackages
The two packages are linking to the same target: ../html/namespaces.html.
The files and folders are structured like this
Project
├── LICENSE
├── README.md
├── docs
└── ...
└── src
├── mypackage
│ ├── a.py
│ ├── __init__.py
│ └── _mypackage.py
├── setup.cfg
└── setup.py
The Doxyfile is located in Project/docs, doxygen is run in there and use ../src/mypackage as INPUT directory.
More details
__init__.py
__version__ = '0.0.1a'
from ._mypackage import *
_mypackage.py
# -*- coding: utf-8 -*-
"""Example __init__.py short.
Now some longer with multiple lines. Here
comes the scond line.
"""
def foo(bar):
"""
This is foo() in mypackage.
Args:
bar (str): A paramenter.
Returns:
(int): Fixed seven.
"""
print(bar)
return 7
a.py
# -*- coding: utf-8 -*-
"""This is mypackage.a
"""
import mypackage
def bar(bar):
"""
This is the function named bar.
The function calls `mypackage.foo()` and returns an 'a'.
Paramters:
bar (str): Just a parameter.
Returns:
str: Just an 'a'.
"""
mypackage.foo(bar)
return('a')
Some (maybe) related Doxyfile settings
BRIEF_MEMBER_DESC = YES
REPEAT_BRIEF = YES
ALWAYS_DETAILED_SEC = NO
FULL_PATH_NAMES = YES
JAVADOC_AUTOBRIEF = NO
PYTHON_DOCSTRING = YES
OPTIMIZE_OUTPUT_FOR_C = NO
OPTIMIZE_OUTPUT_JAVA = YES
OPTIMIZE_FOR_FORTRAN = NO
OPTIMIZE_OUTPUT_VHDL = NO
OPTIMIZE_OUTPUT_SLICE = NO
MARKDOWN_SUPPORT = YES
EXTRACT_ALL = YES
EXTRACT_PRIVATE = YES
EXTRACT_PACKAGE = YES
EXTRACT_STATIC = YES
EXTRACT_LOCAL_CLASSES = YES
EXTRACT_LOCAL_METHODS = YES
EXTRACT_ANON_NSPACES = NO
RESOLVE_UNNAMED_PARAMS = YES
HIDE_UNDOC_MEMBERS = NO
HIDE_UNDOC_CLASSES = NO
HIDE_FRIEND_COMPOUNDS = NO
HIDE_IN_BODY_DOCS = NO
INPUT = ../src/mypackage
FILE_PATTERNS =
RECURSIVE = YES
FILTER_PATTERNS = *.py=./py_filter
GENERATE_HTML = YES
GENERATE_TREEVIEW = YES
I opened an Issue about that.
In this tutorial one line of the code reads
from schemas.tokens import Token
Which package do I need to install? I cannot find it out by Google.
Further down the tutorial we read:
We need a schema to verify that we are returning an access_token and token_type as defined in our response_model. Let's put this code in schemas > tokens.py
So it's a package created in the tutorial itself, i.e. a custom package, not from some library.
yeah. thats the problem.
if you've read the entire tutorial, you would see this tree structure
backend/
├─.env
├─apis/
│ └─general_pages/
│ └─route_homepage.py
├─core/
│ └─config.py
├─db/
│ ├─base.py
│ ├─base_class.py
│ ├─models/
│ │ ├─jobs.py
│ │ └─users.py
│ └─session.py
├─main.py
├─requirements.txt
├─schemas/ # <---------------- HERE
│ ├─jobs.py
│ └─users.py
├─static/
│ └─images/
│ └─logo.png
└─templates/
├─components/
│ └─navbar.html
├─general_pages/
│ └─homepage.html
└─shared/
└─base.html
where schemas is package inside root project
This question is exactly like this stackoverflow question. And I am hoping I get some different answer with this one. I have been trying to achieve this for over a year. but can't seem to achieve it.
Stripped down version of my ansible directory looks like this:
[root#python-test ansible]# tree
.
├── files
├── inventory
│ ├── prod
│ │ ├── group_vars
│ │ │ └── all
│ │ └── hosts -> ../../scripts/inventory.py
│ └── staging
│ ├── group_vars
│ │ └── all
│ └── hosts -> ../../scripts/inventory.py
├── roles
│ ├── ant
│ ├── build
│ ├── jdk
│ └── python
├── scripts
│ └── inventory.py
├── templates
└── vars
└── all.yaml
I would like to use some of the variables declared in group_vars/all file. It has some endpoint details that I can use during the script execution. A striped down version of this group_vars/all looks like this:
inv_cloudprovider: aws
inv_environment: prod
inv_environment_type: production
inv_vpc_cidr: 10.0.0.0/16
inv_build_url: 'https://build.{{inv_environment}}.local'
inv_cloud_vpc_name: '{{inv_environment}}-vpc'
inv_vpc_id: '{{inv_environment_type}}-{{inv_cloud_vpc_name}}'
inv_glassfish_version: 4
At this point I am loading this file using yaml.safe_loads() and then using them in script execution. but problem with that is there are variables which are recursive jinja templates. and I am having hard time getting to make those variables work. So I was wondering if I can use ansible to do this for me.
I am using ansible==2.2.3.0 and python 2.7.
The closest I have been to achieving this in python by doing this:
from ansible.parsing.dataloader import DataLoader
from ansible.vars import VariableManager
from ansible.inventory import Inventory
inventory_path = '/root/ansible/inventory/prod/hosts'
inventory = Inventory(DataLoader(), VariableManager(), inventory_path)
group = inventory.get_group('all')
group_vars = group.get_vars()
Which is probably not the correct way because my script tries to execute itself and that goes on recursively.
Is it possible to parse group vars variable using ansible? If no, how do I best get final value of variables from that file?
The stripped down version of code that can do what this question asked is:
#!/usr/bin/env python
import json
import os
import sys
import yaml
from jinja2 import Environment
class VarLoader(yaml.SafeLoader):
#staticmethod
def construct_python_string(text):
return str(text.value)
def yaml_remove_parse(self, text):
self.construct_python_string(text)
env_name = os.environ.get('ENV_NAME')
script_path = os.path.dirname(os.path.abspath(__file__))
ansible_dir_path = os.path.abspath(script_path + '/../')
global_vars_path = ansible_dir_path + '/vars/all.yaml'
inventory_vars_path = ansible_dir_path + '/inventory/' + env_name + '/group_vars/all'
with open(global_vars_path) as f1, open(inventory_vars_path) as f2:
global_vars_data = f1.read()
inventory_vars_data = f2.read()
all_vars_data = global_vars_data + '\n' + inventory_vars_data
VarLoader.add_constructor('tag:yaml.org,2002:float', VarLoader.yaml_remove_parse)
data = yaml.load(all_vars_data, Loader=VarLoader)
jinja_env = Environment()
template_file = jinja_env.from_string(all_vars_data)
for key in data:
val = data[key]
if isinstance(val, str) and '{{' in val:
template = jinja_env.from_string(val)
data[key] = template.render(**data)
yaml_data = yaml.load(template_file.render(data), Loader=VarLoader)
print(json.dumps(yaml_data))
This works with both python3 and python2. Output:
[root#python-test ansible]# python scripts/inventory.py | jq .
{
"inv_environment": "prod",
"inv_environment_type": "production",
"inv_cloudprovider": "aws",
"gv_redis_port": 6379,
"inv_glassfish_version": 4,
"inv_vpc_id": "production-prod-vpc",
"gv_redis_version": "4.0.11",
"inv_build_url": "https://build.prod.local",
"gv_glassfish_admin_user": "admin",
"inv_vpc_cidr": "10.0.0.0/16",
"gv_glassfish_asadmin_path": "/usr/local/glassfish/bin/asadmin",
"gv_java_home": "/usr/local/java/default",
"inv_cloud_vpc_name": "prod-vpc",
"gv_tomcat_home": "/usr/local/tomcat",
"gv_java_path": "/usr/local/java/default/bin/java"
}