Develop Reference

Fixed python pip error spiraled into rabbit hole of solving errors for scripts within scripts within scripts

So I have a hobby I like to do where I program automated systems that upload things to social media with no human input (and make a little side cash). But anyways, I've been trying to crack YouTube recently. YouTube has their own little way to upload videos automatically, but they watch over it INTENSELY. So, I found a work around using youtube_uploader_selenium, a cool little github project that allows you to upload as many videos as you want to YouTube through selenium firefox.
I know if I can get it to work, I can have a fully automated YouTube channel, however one single, important line of code doesn't work properly.
So the module is accessed via an import command in my main script
from youtube_uploader_selenium import YouTubeUploader
I was able to get that working, however when I did I got an error within the module from this line of code saying that selenium_firefox wasn't a valid pip/module.
from selenium_firefox.firefox import Firefox, By, Keys
So, I did some digging and found a PipInstaller script I could use.
import subprocess
import sys
py_exec = sys.executable
# ensure pip is installed & update
subprocess.call([str(py_exec), "-m", "ensurepip", "--user"])
# install dependencies using pip
subprocess.call([str(py_exec),"-m", "pip", "install", "selenium_firefox"])
That had some odd error involving Rust and the Cryptography pip so I decided to take a different approach. I tracked down the selenium_firefox github then downloaded the module from there. In the script I added a line which made it look for the module selenium_firefox in a certain folder where it was downloaded.
from typing import DefaultDict, Optional
from selenium_firefox.firefox import Firefox, By, Keys # problem code
from collections import defaultdict
import json
import time
from .Constant import *
from pathlib import Path
import logging
import platform
selenium_firefox.path.append(
"C:\\Program Files\\Blender Foundation\\Blender2.92\\2.92\\python\\PythonModules\\modules\\")
That <kinda???> worked. This is the error I currently have,
Traceback (most recent call last):
File "C:\Users\CLASSIFIED\Downloads\ASAS\zxxxs.blend\CLASSIFIEDSCRIPTNAME", line 5, in <module>
File "C:\Program Files\Blender Foundation\Blender 2.92\2.92\python\PythonModules\modules\youtube_uploader_selenium\__init__.py", line 2, in <module>
from selenium_firefox.firefox import Firefox, By, Keys
File "C:\Program Files\Blender Foundation\Blender 2.92\2.92\python\PythonModules\modules\selenium_firefox\__init__.py", line 1, in <module>
from .firefox import Firefox
ModuleNotFoundError: No module named 'selenium_firefox.firefox'
Error: Python script failed, check the message in the system console
I don't know where to go from here. The module within the module within the module has errored. This is what the script for the script that's currently producing this error.
from .firefox import Firefox
from .models import *
from .firefox_addons import *
from selenium_browser import *

Import error even though file is clearly in path

I am running a script and I am getting the error:
Traceback (most recent call last):
File "common/tensorflow/run_tf_benchmark.py", line 30, in <module>
from common.base_benchmark_util import BaseBenchmarkUtil
ModuleNotFoundError: No module named 'common'
I am running on an ec2 instance with the following AMI: Deep Learning AMI (Ubuntu 16.04) Version 25.0 (ami-025c308193ac1a136) and I am also working in the tensorflow_p36 anaconda environment that comes from the AMI. My Ubuntu directory is structured as so:
home
--ubuntu
--benchmark_models
--benchmark
--common
--__init__.py
--base_benchmark_util.py
--tensorflow
--run_tf_benchmark.py
--__init__.py
Note, there are other files in the directory, but these are the relevant python files.
Currently, the imports at the top of run_tf_benchmark.py are
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
import sys
sys.path.append('/home/ubuntu/benchmark_models/benchmarks/common')
print(sys.path)
from argparse import ArgumentParser
from common.base_benchmark_util import BaseBenchmarkUtil
and the printed sys.path is
['/home/ubuntu/benchmark_models/benchmarks/common/tensorflow', '/home/ubuntu/src/cntk/bindings/python',
'/home/ubuntu/benchmark_models/models/image_recognition/tensorflow/mobilenet_v2', '/home/ubuntu/models',
'/home/ubuntu/models/research', '/home/ubuntu/models/research/slim',
'/home/ubuntu/anaconda3/lib/python36.zip', '/home/ubuntu/anaconda3/lib/python3.6',
'/home/ubuntu/anaconda3/lib/python3.6/lib-dynload', '/home/ubuntu/anaconda3/lib/python3.6/site-packages',
'/home/ubuntu/benchmark_models/benchmarks/common']
I can clearly see common in the python path, but the import error still happens. What am I doing wrong?

Just remove 'common' while importing because you are already in 'common' directory
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
import sys
sys.path.append('/home/ubuntu/benchmark_models/benchmarks/common')
print(sys.path)
from argparse import ArgumentParser
from base_benchmark_util import BaseBenchmarkUtil

Alright, I solved the issue. The issue was that I added /home/ubuntu/benchmark_models/benchmarks/common, not /home/ubuntu/benchmark_models/benchmarks/. By import benchmarks, it also allowed all of my other code to import common as well.

python shell windows - import error

I'm trying to import module from local path in Python2.7.10 Shell on Windows
I add local path to sys.path by:
import sys
sys.path.append('C:\download')
next I try to import by:
from download.program01 import *
but I've got this error:
Traceback (most recent call last):
File "<pyshell#5>", line 1, in <module>
from download.program01 import *
ImportError: No module named download.program01
On Linux this code works fine.
Does someone know what is wrong?

If download is in your pythonpath, then you should import program01 directly.
Also, please don't import *; it makes things very hard to debug. Just do import program01.

put a file __init__.py in your download folder so that python knows it is a module and do sys.path.append('C:') instead.
If you want to keep just using path and not create a module file (the __init___.py) then just keep your code like that but import doing
import program01

Import pexpect in child module - 'no module named pexpect'

I'm seeing weird behavior that I don't understand, so I'm turning to the experts here on SO for help. I have looked at similar questions, without finding anything that looked helpful.
I'm writing a program that imports two custom modules.
If I run the main program using the cli "./ld_config_automation.py" I get the following error:
laptop$ ./ld_config_automation.py
Traceback (most recent call last):
File "./ld_config_automation.py", line 34, in <module>
from modules.networkcomponent import NetworkComponent
File "/path/to/pprograms/ssh_telnet_automation/modules/networkcomponent.py", line 7, in <module>
import pexpect
ImportError: No module named pexpect
However, if I run the main program using the cli "python ld_config_automation.py" everything works fine and the program runs.
My main program uses the following import statements.
import sys
import getopt
from modules.networkcomponent import NetworkComponent
from modules.recordclass import Record
The module I'm having trouble with "NetworkComponent" uses the following import statements:
import re
import pexpect
Also, if I remove import pexpect from the NetworkComponent module it works as expected without python in front (up to when it has to use pexpect. of course). So it doesn't seem to be a problem with importing modules, as re works ok, just not pexpect.
I'm developing this on mac os X but will implement on CentOS.

How/where does Python look for modules?

I am completely new to Python and wanted to use py2neo and tornado module.
In order to do that I ran setup.py for both modules and placed them into folders
C:\Python32\modules\py2neo
and
C:\Python32\modules\tornado
In the main program I guess these lines tell the interpreter where to look for files:
import sys
sys.path.append(r'C:\Python32\modules')
# Import Neo4j modules
from py2neo import neo4j, cypher
Reading the book I also added environmental variable (in Windows 7)
PYTHONPATH = C:\Python32\modules;C:\Python32\modules\tornado;C:\Python32\modules\py2neo
Edit
Now I figured out that Python Shell has to be restarted in order to load modified PYTHONPATH variable
In case the variable value is PYTHONPATH = C:\Python32\modules
and the program contains the line
from py2neo import neo4j, cypher
then the following lines are useless:
import sys
sys.path.append(r'C:\Python32\modules')
When I run the program however I get the following error:
Traceback (most recent call last):
File "C:\...\Python Projects\HelloPython\HelloPython\Hellopy2neo.py", line 15, in <module>
from py2neo import neo4j, cypher
File "C:\Python32\modules\py2neo\neo4j.py", line 38, in <module>
import rest, batch, cypher
ImportError: No module named rest
In the file neo4j.py there are the following lines:
try:
import json
except ImportError:
import simplejson as json
try:
from urllib.parse import quote
except ImportError:
from urllib import quote
try:
from . import rest, batch, cypher
except ImportError:
import rest, batch, cypher #line38
and rest.py file is located in folder C:\Python32\modules\py2neo so I don't know why I get the error
ImportError: No module named rest
Edit2:
Trying to import the py2neo directoy in Python Shell and list modules I get:
>>> import py2neo
>>> [name for name in dir(py2neo) if name[0] != '_']
['rest']
I guess there are some unneccesary imports as well and would be very thankful if anyone explained, which imports should be added and excluded (in PYTHONPATH and scripts) in order the program to run without errors.

I suspect the problem is that the import syntax for relative imports has changed in transition from Python 2 to Python 3:
The only acceptable syntax for relative imports is from .[module]
import name. All import forms not starting with . are interpreted as
absolute imports.
The modules you installed use the syntax that would work in Python 2. You could either install them for Python 2, or look for a version of py2neo that supports Python 3, or try to port it manually (the import line should look like from . import rest, but you'll probably face other problems later) or with 2to3 tool.
Update: I tried installing py2neo with pip. It failed for Python3 and finished successfully for Python 2. The version is 1.2.14.