Develop Reference

Generating Python and Go code from the same .proto file - problem with imports

I'm having hard time generating the code for Python and Go using shared .proto file. The problematic part is that I am utilising the timestamp.proto (by google) which needs to be imported differently depending on what language the generated code should be in.
The Python code generator requires this form:
import "google/protobuf/timestamp.proto";
While Go code generator requires this:
import "github.com/golang/protobuf/ptypes/timestamp/timestamp.proto";
Is it possible to make this import work for both languages? How?

This proto path is wrong:
import "github.com/golang/protobuf/ptypes/timestamp/timestamp.proto"; // WRONG path
This is the correct import path no matter what language you are using - Go or Python etc:
import "google/protobuf/timestamp.proto"; // correct path for any language (go, python etc)
This timestamp.proto file is located by the protoc-gen-go tool (when generating Go code) using its default INCLUDE_PATH.
For example, on my Mac for instance, the default INCLUDE_PATH is:
/usr/local/Cellar/protobuf/3.7.1/include
with the full proto file path being:
/usr/local/Cellar/protobuf/3.7.1/include/google/protobuf/timestamp.proto
You can see other standard proto definitions that come with your gRPC installation like duration.proto:
$ pwd # my default gRPC include path
/usr/local/Cellar/protobuf/3.7.1/include
$ find . -name "*.proto"
./google/protobuf/timestamp.proto
./google/protobuf/field_mask.proto
./google/protobuf/api.proto
./google/protobuf/duration.proto
./google/protobuf/struct.proto
./google/protobuf/wrappers.proto
./google/protobuf/source_context.proto
./google/protobuf/any.proto
./google/protobuf/type.proto
./google/protobuf/empty.proto
./google/protobuf/compiler/plugin.proto
./google/protobuf/descriptor.proto
Provided you have installed the gRPC toolkit (and its headers) in the correct location, as per the install docs, then the above directory hierarchy should match for any OS build.
P.S. this question explains how to set an explicit INCLUDE_PATH when using the proto-compiler.

Cannot import Python function in Jython

I wrote a Python script which is executed via Jython 2.7. I need SQLite, so I decided to use sqlite3 for Jython (link) which is under /usr/local/lib/jython/Lib.
ghidra_batch.py
import sys
sys.path.append("/usr/local/lib/jython/Lib")
sys.path.append("/path/to/my/project/directory")
import sqlite3
I created another file where I define some functions for my database:
db.py
import platform
import sys
if platform.python_implementation == 'Jython':
sys.path.append("/usr/local/lib/jython/Lib")
sys.path.append("/path/to/my/project/directory")
import sqlite3
def open_db():
[some code]
def create_schema():
[some code]
Note: I check Python implementation because this script is run also via CPython. I append the path only when run via Jython to make it find its sqlite3 module, in case of CPython standard sqlite3 module is used.
Now my problem happens when I import open_db() in ghidra_batch.py:
from db import open_db
The result is the following:
ImportError: cannot import name open_db
Thanks for your help.

As a general rule: when working with Python, when something isn't not what you're expecting, simply print it.
Your from db import open_db line which was triggering that exception "told" me that:
The db module (package) was found
It's not the one that you're expecting to be (your db.py)
That's why I suggested in my comment to print information about it (obviously, before the error is hit):
import db
print(db)
print(dir(db))
The output confirmed it. So, there is another db module which is imported before yours. I tried replicating your environment (installed Jython, but I wasn't able to install jython-sqlite3).
After a bit of research, I think it's [BitBucket]: Taro L. Saito/sqlite-jdbc/Source - sqlite-jdbc/src/main/java/org/sqlite/DB.java (sqlite-jdbc is a jython-sqlite3 dependency).
The reasonable way is to modify your module name to something else (e.g.: sqlite_db_wrapper.py), and also update the import statement(s).
As a(n other) general rule, don't give your modules (common) names that might conflict with modules from Python's library.

Import python module directly from a remote git repo? [duplicate]

Is it possible to import a Python module from over the internet using the http(s), ftp, smb or any other protocol? If so, how? If not, why?
I guess it's about making Python use more the one protocol(reading the filesystem) and enabling it to use others as well. Yes I agree it would be many folds slower, but some optimization and larger future bandwidths would certainly balance it out.
E.g.:
import site
site.addsitedir("https://bitbucket.org/zzzeek/sqlalchemy/src/e8167548429b9d4937caaa09740ffe9bdab1ef61/lib")
import sqlalchemy
import sqlalchemy.engine

Another version,
I like this answer. when applied it, i simplified it a bit - similar to the look and feel of javascript includes over HTTP.
This is the result:
import os
import imp
import requests
def import_cdn(uri, name=None):
if not name:
name = os.path.basename(uri).lower().rstrip('.py')
r = requests.get(uri)
r.raise_for_status()
codeobj = compile(r.content, uri, 'exec')
module = imp.new_module(name)
exec (codeobj, module.__dict__)
return module
Usage:
redisdl = import_cdn("https://raw.githubusercontent.com/p/redis-dump-load/master/redisdl.py")
# Regular usage of the dynamic included library
json_text = redisdl.dumps(host='127.0.0.1')
Tip - place the import_cdn function in a common library, this way you could re-use this small function
Bear in mind It will fail when no connectivity to that file over http

In principle, yes, but all of the tools built-in which kinda support this go through the filesystem.
To do this, you're going to have to load the source from wherever, compile it with compile, and exec it with the __dict__ of a new module. See below.
I have left the actually grabbing text from the internet, and parsing uris etc as an exercise for the reader (for beginners: I suggest using requests)
In pep 302 terms, this would be the implementation behind a loader.load_module function (the parameters are different). See that document for details on how to integrate this with the import statement.
import imp
modulesource = 'a=1;b=2' #load from internet or wherever
def makemodule(modulesource,sourcestr='http://some/url/or/whatever',modname=None):
#if loading from the internet, you'd probably want to parse the uri,
# and use the last part as the modulename. It'll come up in tracebacks
# and the like.
if not modname: modname = 'newmodulename'
#must be exec mode
# every module needs a source to be identified, can be any value
# but if loading from the internet, you'd use the URI
codeobj = compile(modulesource, sourcestr, 'exec')
newmodule = imp.new_module(modname)
exec(codeobj,newmodule.__dict__)
return newmodule
newmodule = makemodule(modulesource)
print(newmodule.a)
At this point newmodule is already a module object in scope, so you don't need to import it or anything.
modulesource = '''
a = 'foo'
def myfun(astr):
return a + astr
'''
newmod = makemodule(modulesource)
print(newmod.myfun('bat'))
Ideone here: http://ideone.com/dXGziO
Tested with python 2, should work with python 3 (textually compatible print used;function-like exec syntax used).

This seems to be a use case for a self-written import hook. Look up in PEP 302 how exactly they work.
Essentially, you'll have to provide a finder object which, in turn, provides a loader object. I don't understand the process at the very first glance (otherwise I'd be more explicit), but the PEP contains all needed details for implementing the stuff.

As glglgl's has it this import hook has been implemented for Python2 and Python3 in a module called httpimport.
It uses a custom finder/loader object to locate resources using HTTP/S.
Additionally, the import_cdn function in Jossef Harush's answer is almost identically implemented in httpimport's github_repo, and bitbucket_repo functions.
#Marcin's answer contains a good portion of the code of the httpimport's loader class.

Need to re-load libraries in Python, with Canopy

I am working on a python project with Canopy, using my own library, which I modify from time to time to change or add functions inside.
At the beginning of myfile.py I have from my_library import * but if I change a function in this library and compute again myfile.py it keep using the previous version of my function.
I tried the reload function :
import my_library
reload(my_library)
from other_python_file import *
from my_library import *
and it uses my recently changed library.
But if it is :
import my_library
reload(my_library)
from my_library import *
from other_python_file import *
It gives me the result due to the version loaded the first time I launched myfile.py.
Why is there a different outcome inverting the 3rd and 4th line ?

Without seeing the source code, it's hard to be certain. (For future reference, it is most useful to post a minimal example, which I suspect would be about 10 lines of code in this case.)
However from your description of the problem, my guess is that your other_python_file also imports my_library. So when you do from other_python_file import *, you are also importing everything that it has already imported from my_library, which in your second example, will override the imports directly from my_library (Since you didn't reload other_python_file, it will still be using the previous version of my_library.)
This is one out of approximately a zillionteen reasons why you should almost never use the form from xxx import * except on the fly in interactive mode (and even there it can be dangerous but can be worth the tradeoff for the convenience). In a python source file, there's no comparable justification for this practice. See the final point in the Imports section of PEP-8.

python importing file on virtual environment

I am writing a web app using python3, venv and c9.io PAAS. I have the following structure of my code:
batch_runner.py
logic/
__init__.py
parsers/
__init__.py
time_parser.py
abstract_parser.py
here batch_runner imports abstract_parser, which, in it turn, import from time_parser. everything was installed and runs with venv activated.
To be specific, batch_runner.py contains:
from logic.parsers import abstract
from sys import argv
url = argv[1]
a = abstract(url)
logic/__init__.py is empty. logic/parsers/__init__.py contains:
from abstract_parser import abstract
from time_parser import _timeInfo
If I go to logic and run python abstract_parser.py directly, everything works as expected. However, if I go one level up, and run python batch_runner.py, it is able to import abstract_parser, but it can't find time_parser which is called from abstract_parser, throwing ImportError: No module named 'abstract'

Change this:
from abstract_parser import abstract
To
from logic.parsers.abstract_parser import abstract

Do read about importing from the python documentation on modules.
In this case, one possible solution is to use relative imports inside your package:
That is, in logic/parsers/__init__.py, use:
from .abstract_parser import abstract
from .time_parser import _timeInfo
and in abstract_parser.py:
from .time_parser import _timeInfo
This should let parsers/__init__.py find the abstract_parser module and the time_parser module.
The python import system has a surprising number of traps that you can fall into. This blog post by Nick Coghlan describes many of them, and I personally consider it a must-read if you're planning to develop a package.