Below is the program that defines a function within another function.
1) When we say python program.py Does every line of python source directly gets converted to set of machine instructions that get executed on processor?
2) Above diagram has GlobalFrame and LocalFrame and Objects. In the above program, Where does Frames Objects and code reside in runtime? Is there a separate memory space given to this program within python interpreter's virtual memory address space?
"Does every line of python source directly gets converted to set of machine instructions that get executed on processor?"
No. Python code (not necessarily by line) typically gets converted to an intermediate code which is then interpreted by what some call a "virtual machine" (confusingly, as VM means something completely different in other contexts, but ah well). CPython, the most popular implementation (which everybody thinks of as "python":-), uses its own bytecode and interpreter thereof. Jython uses Java bytecode and a JVM to run it. And so on. PyPy, perhaps the most interesting implementation, can emit almost any sort of resulting code, including machine code -- but it's far from a line by line process!-)
"Where does Frames Objects and code reside in runtime"
On the "heap", as defined by the malloc, or equivalent, in the C programming language in the CPython implementation (or Java for Jython, etc, etc).
That is, whenever a new PyObject is made (in CPython's internals), a malloc or equivalent happens and that object is forevermore referred via a pointer (a PyObject*, in C syntax). Functions, frames, code objects, and so forth, almost everything is an object in Python -- no special treatment, "everything is first-class"!-)
Related
I am looking into Cpython implementation and got to learn about how python tackles operator overloading (for example comparison operators) using something like richcmpfunc tp_richcompare; field in _typeobject struct. Where the type is defined as typedef PyObject *(*richcmpfunc) (PyObject *, PyObject *, int);. And so whenever there is need for PyObject being operated by these operators it tries to call tp_richcompare function.
My doubt is that in python we use magic functions like __gt__ etc. to override these operators. So how does python code gets converted into C code as a tp_richcompare and is being used everywhere where we interpret any comparison operator for PyObject.
My second doubt is kind of general version of this: How code in a particular language (here Python) to override things (operators, hash etc.) which are interpreted in another language (C in case of CPython) calls the function defined in first language (Python). As far as I know, when bytecode is generated it's a low-level instruction based representation (which is essentially array of uint8_t).
Another example of this is __hash__ which would be defined in python but is needed in the C-based implementation of the dictionary while lookdict. Again they use C function typedef Py_hash_t (*hashfunc)(PyObject *); everywhere hash is needed for a PyObject but translation of __hash__ to this C function is mysterious.
Python code is not transformed into C code. It is interpreted by C code (in CPython), but that's a completely different concept.
There are many ways to interpret a Python program, and the language reference does not specify any particular mechanism. CPython does it by transforming the each Python function into a list of virtual machine instructions, which can then be interpreted with a virtual machine emulator. That's one approach. Another one would be to just build the AST and then define a (recursive) evaluate method on each AST node.
Of course, it would also be possible to transform the program into C code and compile the C code for future execution. (Here, "C" is not important. It could be any compiled language which seems convenient.) However, there's not much benefit to doing that, and lots of disadvantages. One problem, which I guess is the one behind your question, is that Python types don't correspond to any C primitive type. The only way to represent a Python object in C is to use a structure, such as CPython PyObject, which is effectively a low-level mechanism for defining classes (a concept foreign to C) by including a pointer to a type object which contains a virtual method table, which contains pointers to the functions used to implement the various operations on objects of that type. In effect, that will end up calling the same functions as the interpreter would call to implement each operation; the only purpose of the compiled C code is to sequence the calls without having to walk through an interpretable structure (VM list or AST or whatever). That might be slightly faster, since it avoids a switch statement on each AST node or VM operation, but it's also a lot bulkier, because a function call occupies a lot more space in memory than a single opcode byte.
An intermediate possibility, in common use these days, is to dynamically compile descriptions of programs (ASTs or VM lists or whatever) into actual machine code at runtime, taking into account what can be discovered about the actual dynamic types and values of the referenced variables and functions. That's called "just-in-time (JIT) compilation", and it can produce huge speedups at runtime, if it's implemented well. On the other hand, it's very hard to get it right, and discussing how to do it is well beyond the scope of a SO answer.
As a postscript, I understand from a different question that you are reading Robert Nystrom's book, Crafting Interpreters. That's probably a good way of learning these concepts, although I'm personally partial to a much older but still very current textbook, also freely available on the internet, The Structure and Interpretation of Computer Programs, by Gerald Sussman, Hal Abelson, and Julie Sussman. The books are not really comparable, but both attempt to explain what it means to "interpret a program", and that's an extremely important concept, which probably cannot be communicated in four paragraphs (the size of this answer).
Whichever textbook you use, it's important to not just read the words. You must do the exercises, which is the only way to actually understand the underlying concepts. That's a lot more time-consuming, but it's also a lot more rewarding. One of the weaknesses of Nystrom's book (although I would still recommend it) is that it lays out a complete implementation for you. That's great if you understand the concepts and are looking for something which you can tweak into a rapid prototype, but it leaves open the temptation of skipping over the didactic material, which the is most important part for someone interested in learning how computer languages work.
I am an (old) engineer not a programmer so forgive me for asking a naïve question.
My understanding is that to get really fast execution times for a program, it needs to be compiled to native machine code. And there are a relatively small number of languages still in use that do this (e.g. C and C++).
But I much prefer the syntax of Python over that of the C-derived compiled languages. However my understanding is that interpreted Python (and pseudo-compiled Python run on a virtual machine) cannot match the execution speed of a truly compiled language.
Is there some reason that a true native-code Python compiler cannot be developed?
[I am interested specifically in Python but I am not aware of any language that can be interpreted and also compiled to native machine code.]
The key distinction is a clean separation between compile time and run time. In Python, for instance, import happens at runtime, and can happen conditionally. And per the halting problem, that means a compiler cannot determine up front if a given import will happen. Yet, this affects the code that would need to be generated.
As Bill the Lizard notes, if the language does have a clean distinction, an interpreter can still choose to ignore it. C's #include can happen before main, but that does not mean an interpreter must do so.
Outside the syntax, Python is also virtually uncompilable due to weak typing. In C, + has a very limited set of meanings - integer, floating point or pointer, and the compiler will know the static type of the arguments. C++ has far more extensive overloading, but the same basic principle applies. virtual functions allow some run-time flexibility, but from an finite set of options, all compiled before main starts.
Also not syntax is the memory model - both C and C++ have a memory model that's an improved derivative of Java's memory model, which makes threading quite efficient. (Unlike Java, not every object can be synchronized, you need special members). As CPU's gain more and more cores, the advantages only continue to grow. Compilers can see pretty well where memory and CPU registers need to be brought in sync.
I am experiencing a difficulty using boost python facilities to extend my C++ code to Python. I've written the boost.python wrappers successfully. I also have access to my C++ objects from Python without any error, in addition called a Python file (module) method from C++ using boost attr("")() function without any problem.
My problem is the execution time of the Python method. Referencing to the wrapped objects are about microseconds in Python code as I've printed. Although the time calling the Python method takes is about milliseconds and it increases with respect to the number of references I've made in the Python to my wrapped C++ objects (and only referencing/assigning not any further use). Thus I've made some search and my assumptions about this increasing time is:
some reference policies (default policies) causes this problem by doing some unnecessary operation(s) when returning from the Python code. So probably I'm doing something wrong in the wrappers.
Boost.Python call method has some overhead, which there might be some options I'm not aware of.
It worth mentioning that the Python method called in each execution cycle of my program and each time I get a very same (not exact) time.
I hope my description were enough. Below is also a part of my sample code:
One of my Wrappers:
class_<Vertex<> >("Vertex")
.def(init<float, float>())
.def_readwrite("x", &Vertex<>::x)
.def_readwrite("y", &Vertex<>::y)
.def("abs", &Vertex<>::abs)
.def("angle", &Vertex<>::angle)
.def(self - self)
.def(self -= self)
;
Calling a Python module method (which is "run"):
pyFile = import(fileName.c_str());
scope scope1(pyFile);
object pyNameSpace = scope1.attr("__dict__");
return extract<int>(pyFile.attr("run")());
In compiled languages, the source code is turned into object code by the compiler and the different object files (if there are multiple files) are linked by the linker and loaded into the memory by the loader for execution.
If I have an application written using an interpreted language (for eg., ruby or python) and if the source code is split across files, when exactly are the files brought together. To put it other words when is the linking done? Do interpreted languages have Linkers and Loaders in the first place or the interpreter does everything?
I am really confused about this and not able to get my head around it!! Can anyone shine some light on this?!
An interpreted language is more or less a large configuration for an executable that is called interpreter. That executable (e. g. /usr/bin/python) is the program which actually runs. It then reads the script it shall execute (e. g. /home/alfe/bin/factorial.py) and executes it, in the simplest form line-by-line.
During that process it can encounter references to other files (other modules, e. g. /usr/python/lib/math.py) and then it will read and interpret those.
Many such languages have mechanisms built in to reduce the overhead of this process by creating byte-code versions of the scripts they interpreted. So there might well be a file /usr/python/lib/math.pyc for instance, which the interpreter put there after first processing and which it can faster read and interpret than the original /usr/python/lib/math.py. But this is not really part of the concept of interpreted languages¹.
Sometimes, a binary library is part of an interpreted language; depending on the sophistication of the interpreter it can link that library at runtime and then use it. This is most typical for the system modules and stuff which needs to be highly optimized.
But in general one can say that no binary machine code gets generated at all. And nothing is linked at the compile time. Actually, there is no real compile time, even though one could call that first processing of the input scripts a compile step.
Footnotes:
¹) The concept of interpreting scripts does encompass neither that "compiling" (pre-translating of the source into a faster-to-interpret form) nor that "caching" of this form by storing files like the .pyc files. WRT to your question concerning linking and splitting programs into several files or modules, these aspects of precompiling and caching are just technical details to speed up things. The concept itself is: read one line of the input script & execute it. Then read the next line and so on.
Well, in Python, modules are loaded and executed or parsed when the interpreter finds some method or indication to do so. There's no linking but there is loading of course (when the file is requested in the code).
Python do something clever to improve its performance. It compiles to bytecode (.pyc files) the first time it executes a file. This improves substantially the execution of the code next time the module is imported or executed.
So the behavior is more or less:
A file is executed
Inside the file, the interpreter finds a reference to another file
It parses it and potentially execute it. This means that every class, variable or method definition will become available in the runtime.
And this is how the process is done (very general). Of course, there are optimizations and caches to improve the performance.
Hope this helps!
After viewing lots of posts about the difference between compilers and interpreters, I'm still not able to figure out the difference in their construction and internal mechanism.
The most common difference I read was that a compiler produces a target program which is executable { means machine code as its output } which can run on a system and than be fed with input.
Whereas an interpreter simply runs the input line by line { what exactly is happening here ?} and produces the output.
My main doubts are :
1) A compiler consists of a lexical analyzer, parser, intermediate code generator and code generator but what are the parts of an interpreter?
2) Who gives the run-time support to interpreted languages, I mean who manages the heap and stacks for recursive functions?
3) This is specific to the Python language:
Python comprises of a compiler stage and than interpreter stage as well
compiler produces some byte-code and and than this byte-code is interpreted by its Virtual Machine.
if I were to design only the compiler for Python (Python -> bytecode)
a) will I have to manage memory { write code to manage stack and heap } for it?
b) how will this compiler differ from the traditional compiler or say interpreter?
I know this is a whole lot to ask here but I really want to understand these minute details.
I'm referring the compiler book by Alfred V. Aho
Based on the feedback and some further study I think I should modify my question
A compiler need not produce only machine code as its output
But one question is still bugging me
Let say I want to design a ( Python->bytecode ) compiler and then bytecode will be interpreted by the virtual machine.. (correct me if I'm wrong ).
Then I'll have to write a lexical analyzer for Python and then a parser which will generate some sort of abstract syntax tree.. after this do I have to generate some intermediate code (3 address code as mentioned in the dragon book) or direct bytecode instructions ( which I suppose will be given in the VM's documentation ) ?
Will I have to write code for handling stack as well to provide support for recursion and scope ?
First off, "compiler" does not imply "outputs machine code". You can compile from any language to any other, be it a high-level programming language, some intermediate format, code for a virtual machine (bytecode) or code for a physical machine (machine code).
Like a compiler, an interpreter needs to read and understand the language it implements. Thus you have the same front-end code (though today's interpreters usually implement far simpler language - the bytecode only; therefore these interpreters only need a very simple frontend). Unlike a compiler, an interpreter's backend doesn't generate code, but executes it. Obviously, this is a different problem entirely and hence an interpreter looks quite difference from a compiler. It emulates a computer (often one that's far more high-level than real life machines) instead of producing a representation of an equivalent program.
Assuming today's somewhat-high-level virtual machines, this is the job of the interpreter - there are dedicated instructions for, among other things, calling functions and creating objects, and garbage collection is baked into the VM. When you target lower-level machines (such as the x86 instruction set), many such details need to be baked into the generated code, be it directly (system calls or whatever) or via calling into the C standard library implementation.
3.
a) Probably not, as a VM dedicated to Python won't require it. It would be very easy to screw up, unnecessary and arguably incompatible to Python semantics as it allows manual memory management. On the other hand, if you were to target something low-level like LLVM, you'd have to take special care - the details depend on the target language. That's part of why nobody does it.
b) It would be a perfectly fine compiler, and obviously not an interpreter. You'd probably have a simpler backend than a compiler targeting machine code, and due to the nature of the input language you wouldn't have quite as much analysis and optimization to do, but there's no fundamental difference.