A way I usually use to understand a project architecture in C++ is to set a break point in a particular function using GDB, and then using the traceback feature, I can easily understand the function calls between the different classes. I am totally new to Python and I want to ask what are the best Technics used to understand the Python project,
I took a look at traceback but the problem is that it is only tracing the functions inside the same module, so if the caller is in a different module it won't be traced. Besides, the size of the stack is also limited, correct me if I am wrong.
Could you please share the Technics you are using based on your own experience.
I don't know where you got the idea that python's tracebacks would be limited to a single module or limited in size - when an exception occurs, you of course have the full stack trace available.
This being said, Python has a full step debugger in it's stdlib, which lets you inspect and navigate the whole call stack. And there are of course third-part step debuggers in various IDEs and custom shells or environments (ie IPython etc).
NB the inspect module might also be of interest to you - and not only for stack inspection ;-)
Related
In a program, I am using multiple libraries that include boost bindings to the C++ eigen library. (In my case, OMPL & Pinocchio, but I guess the problem is not related to the specific libraries)
Both of them implement a binding for the same C++ class (an STL vector of integers). My problem is, the two bindings differ from each other - in my specific case,
pinocchio.pinocchio_pywrap.StdVec_Int has a tolist() method, while ompl.util._util.vectorInt doesn't.
It seems that, whatever library is imported first, is the deciding factor for which binding is used. This is dangerous because it leads to errors that are hard to trace, especially for externals that are using the code - all of a sudden, methods fail that worked before, just because of a new import.
My preferred solution to this problem would be forcing both libraries to use their own bindings only - but I don't know if that is possible.
Alternatively - is there a way to globally define which C++ bindings to prefer in case two libraries collide?
Example for the "hard-to-explain-bug": Imagine running several unittests in one session - and the one on ompl-related methods runs first. It fails later on when the tolist() method is used on an object which is expected to be of type pinocchio.pinocchio_pywrap.StdVec_Int. You start to debug and run the failing unittest only - and it runs smoothly because ompl was not imported before...
So you've got some legacy code lying around in a fairly hefty project. How can you find and delete dead functions?
I've seen these two references: Find unused code and Tool to find unused functions in php project, but they seem specific to C# and PHP, respectively.
Is there a Python tool that'll help you find functions that aren't referenced anywhere else in the source code (notwithstanding reflection/etc.)?
In Python you can find unused code by using dynamic or static code analyzers. Two examples for dynamic analyzers are coverage and figleaf. They have the drawback that you have to run all possible branches of your code in order to find unused parts, but they also have the advantage that you get very reliable results.
Alternatively, you can use static code analyzers that just look at your code, but don't actually run it. They run much faster, but due to Python's dynamic nature the results may contain false positives.
Two tools in this category are pyflakes and vulture. Pyflakes finds unused imports and unused local variables. Vulture finds all kinds of unused and unreachable code. (Full disclosure: I'm the maintainer of Vulture.)
The tools are available in the Python Package Index https://pypi.org/.
I'm not sure if this is helpful, but you might try using the coverage, figleaf or other similar modules, which record which parts of your source code is used as you actually run your scripts/application.
Because of the fairly strict way python code is presented, would it be that hard to build a list of functions based on a regex looking for def function_name(..) ?
And then search for each name and tot up how many times it features in the code. It wouldn't naturally take comments into account but as long as you're having a look at functions with less than two or three instances...
It's a bit Spartan but it sounds like a nice sleepy-weekend task =)
unless you know that your code uses reflection, as you said, I would go for a trivial grep. Do not underestimate the power of the asterisk in vim as well (performs a search of the word you have under your cursor in the file), albeit this is limited only to the file you are currently editing.
Another solution you could implement is to have a very good testsuite (seldomly happens, unfortunately) and then wrap the routine with a deprecation routine. if you get the deprecation output, it means that the routine was called, so it's still used somewhere. This works even for reflection behavior, but of course you can never be sure if you don't trigger the situation when your routine call is performed.
its not only searching function names, but also all the imported packages not in use.
you need to search the code for all the imported packages (including aliases) and search used functions, then create a list of the specific imports from each package (example instead of import os, replace with from os import listdir, getcwd,......)
I want to use zipkin to profile the internals of a traditional program.
I use the term "traditional", since AFAIK zipkin is for tracing in a microservice environment where one request gets computed by N sub-requests.
I would like to analyse the performance of my python program.
I would like to trace all python method calls and all linux syscalls which gets done.
How to trace the python method calls and linux syscalls to get the spans into zipkin?
Even if it is not feasible, I am interesting how this could be done. I would like to learn how zipkin works.
In zipkin lingo, what you are asking about is often called "local spans" or "local tracing", basically an operation that neither originated, nor resulted in a remote call.
I'm not aware of anything at the syscall level, but many tracers support explicit instrumentation of function calls.
For example, using py_zipkin
#zipkin_span(service_name='my_service', span_name='some_function')
def some_function(a, b):
return do_stuff(a, b)
Besides explicit instrumentation like this, one could also export data to zipkin. For example, one could convert trace data that is made in another tool to zipkin's json format.
This probably doesn't answer your question, but I hope it helps.
So you've got some legacy code lying around in a fairly hefty project. How can you find and delete dead functions?
I've seen these two references: Find unused code and Tool to find unused functions in php project, but they seem specific to C# and PHP, respectively.
Is there a Python tool that'll help you find functions that aren't referenced anywhere else in the source code (notwithstanding reflection/etc.)?
In Python you can find unused code by using dynamic or static code analyzers. Two examples for dynamic analyzers are coverage and figleaf. They have the drawback that you have to run all possible branches of your code in order to find unused parts, but they also have the advantage that you get very reliable results.
Alternatively, you can use static code analyzers that just look at your code, but don't actually run it. They run much faster, but due to Python's dynamic nature the results may contain false positives.
Two tools in this category are pyflakes and vulture. Pyflakes finds unused imports and unused local variables. Vulture finds all kinds of unused and unreachable code. (Full disclosure: I'm the maintainer of Vulture.)
The tools are available in the Python Package Index https://pypi.org/.
I'm not sure if this is helpful, but you might try using the coverage, figleaf or other similar modules, which record which parts of your source code is used as you actually run your scripts/application.
Because of the fairly strict way python code is presented, would it be that hard to build a list of functions based on a regex looking for def function_name(..) ?
And then search for each name and tot up how many times it features in the code. It wouldn't naturally take comments into account but as long as you're having a look at functions with less than two or three instances...
It's a bit Spartan but it sounds like a nice sleepy-weekend task =)
unless you know that your code uses reflection, as you said, I would go for a trivial grep. Do not underestimate the power of the asterisk in vim as well (performs a search of the word you have under your cursor in the file), albeit this is limited only to the file you are currently editing.
Another solution you could implement is to have a very good testsuite (seldomly happens, unfortunately) and then wrap the routine with a deprecation routine. if you get the deprecation output, it means that the routine was called, so it's still used somewhere. This works even for reflection behavior, but of course you can never be sure if you don't trigger the situation when your routine call is performed.
its not only searching function names, but also all the imported packages not in use.
you need to search the code for all the imported packages (including aliases) and search used functions, then create a list of the specific imports from each package (example instead of import os, replace with from os import listdir, getcwd,......)
Sampling with Activity Monitor/Instruments/Shark will show stack traces full of C functions for the Python interpreter. I would be helpful to see the corresponding Python symbol names. Is there some DTrace magic that can do that? Python's cProfile module can be useful for profiling individual subtrees of Python calls, but not so much for getting a picture of what's going on with the whole application in response to user events.
The answer is "dtrace", but it won't work on sufficiently old macs.
http://tech.marshallfamily.com.au/archives/python-dtrace-on-os-x-leopard-part-1/
http://tech.marshallfamily.com.au/archives/python-dtrace-on-os-x-leopard-part-2/