I currently trying to learn mrjob and how to implement it in AWS EMR so please forgive me if I am asking already asked question [searched many places but did not find the answer] and sorry if it is a silly question
This is my python script :
from mrjob.job import MRJob
class MRWordFrequencyCount(MRJob):
def mapper(self, _, line):
yield "chars", len(line)
yield "words", len(line.split())
yield "lines", 1
def reducer(self, key, values):
yield key, sum(values)
if __name__ == '__main__':
MRWordFrequencyCount.run()
When I run it in local mode I got the result
cmd:
python sample.py input.txt
So I tried to run this in EMR
by creating a mrjob.conf file
which looks like this :
runners:
emr:
aws_access_key_id:
aws_secret_access_key:
aws_region: us-west-2a
ec2_key_pair: emr
ec2_key_pair_file: ~/Desktop/emr.pem
ec2_instance_type: m1.small
num_ec2_instances: 5
local:
base_tmp_dir: /tmp
First attempt
Trying it locally on my windows system
python check.py -r emr --conf-path ./mrjob.conf word.txt
Note :
Same error came when I kept the input in s3 location and gave it as an argument
I got this traceback:
Traceback (most recent call last):
File "check.py", line 16, in <module>
MRWordFrequencyCount.run()
File "C:\Users\MOB140003207\AppData\Local\Enthought\Canopy32\User\lib\site-pac
kages\mrjob\job.py", line 461, in run
mr_job.execute()
File "C:\Users\MOB140003207\AppData\Local\Enthought\Canopy32\User\lib\site-pac
kages\mrjob\job.py", line 479, in execute
super(MRJob, self).execute()
File "C:\Users\MOB140003207\AppData\Local\Enthought\Canopy32\User\lib\site-pac
kages\mrjob\launch.py", line 153, in execute
self.run_job()
File "C:\Users\MOB140003207\AppData\Local\Enthought\Canopy32\User\lib\site-pac
kages\mrjob\launch.py", line 215, in run_job
with self.make_runner() as runner:
File "C:\Users\MOB140003207\AppData\Local\Enthought\Canopy32\User\lib\site-pac
kages\mrjob\job.py", line 502, in make_runner
return super(MRJob, self).make_runner()
File "C:\Users\MOB140003207\AppData\Local\Enthought\Canopy32\User\lib\site-pac
kages\mrjob\launch.py", line 168, in make_runner
return EMRJobRunner(**self.emr_job_runner_kwargs())
File "C:\Users\MOB140003207\AppData\Local\Enthought\Canopy32\User\lib\site-pac
kages\mrjob\emr.py", line 643, in __init__
self._fix_s3_scratch_and_log_uri_opts()
File "C:\Users\MOB140003207\AppData\Local\Enthought\Canopy32\User\lib\site-pac
kages\mrjob\emr.py", line 760, in _fix_s3_scratch_and_log_uri_opts
self._set_s3_scratch_uri(s3_conn)
File "C:\Users\MOB140003207\AppData\Local\Enthought\Canopy32\User\lib\site-pac
kages\mrjob\emr.py", line 776, in _set_s3_scratch_uri
buckets = s3_conn.get_all_buckets()
File "C:\Users\MOB140003207\AppData\Local\Enthought\Canopy32\User\lib\site-pac
kages\mrjob\retry.py", line 149, in call_and_maybe_retry
return f(*args, **kwargs)
File "C:\Users\MOB140003207\AppData\Local\Enthought\Canopy32\User\lib\site-pac
kages\boto\s3\connection.py", line 436, in get_all_buckets
response = self.make_request('GET', headers=headers)
File "C:\Users\MOB140003207\AppData\Local\Enthought\Canopy32\User\lib\site-pac
kages\boto\s3\connection.py", line 664, in make_request
retry_handler=retry_handler
File "C:\Users\MOB140003207\AppData\Local\Enthought\Canopy32\User\lib\site-pac
kages\boto\connection.py", line 1070, in make_request
retry_handler=retry_handler)
File "C:\Users\MOB140003207\AppData\Local\Enthought\Canopy32\User\lib\site-pac
kages\boto\connection.py", line 1029, in _mexe
raise ex
socket.gaierror: [Errno 11004] getaddrinfo failed
When I tried to run it in aws EC2 instance
I got this error
Traceback (most recent call last):
File "check.py", line 16, in <module>
MRWordFrequencyCount.run()
File "/usr/local/lib/python2.7/dist-packages/mrjob/job.py", line 461, in run
mr_job.execute()
File "/usr/local/lib/python2.7/dist-packages/mrjob/job.py", line 479, in execute
super(MRJob, self).execute()
File "/usr/local/lib/python2.7/dist-packages/mrjob/launch.py", line 153, in execute
self.run_job()
File "/usr/local/lib/python2.7/dist-packages/mrjob/launch.py", line 215, in run_job
with self.make_runner() as runner:
File "/usr/local/lib/python2.7/dist-packages/mrjob/job.py", line 502, in make_runner
return super(MRJob, self).make_runner()
File "/usr/local/lib/python2.7/dist-packages/mrjob/launch.py", line 168, in make_runner
return EMRJobRunner(**self.emr_job_runner_kwargs())
File "/usr/local/lib/python2.7/dist-packages/mrjob/emr.py", line 643, in __init__
self._fix_s3_scratch_and_log_uri_opts()
File "/usr/local/lib/python2.7/dist-packages/mrjob/emr.py", line 760, in _fix_s3_scratch_and_log_uri_opts
self._set_s3_scratch_uri(s3_conn)
File "/usr/local/lib/python2.7/dist-packages/mrjob/emr.py", line 776, in _set_s3_scratch_uri
buckets = s3_conn.get_all_buckets()
File "/usr/local/lib/python2.7/dist-packages/mrjob/retry.py", line 149, in call_and_maybe_retry
return f(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 436, in get_all_buckets
response = self.make_request('GET', headers=headers)
File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 664, in make_request
retry_handler=retry_handler
File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1071, in make_request
retry_handler=retry_handler)
File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1030, in _mexe
raise ex
socket.gaierror: [Errno -2] Name or service not known
I don't know what I am doing wrong
python version 2.7 mrjob version '0.4.5'
After hours of searching and investigating I found the problem
It was in this line
aws_region: us-west-2a
Where as it should have been
aws_region: us-west-2
I just want to keep the question alive since it may save other people time
Related
This question already has answers here:
pipenv Failing to Install Packages
(3 answers)
Closed 1 year ago.
I have already installed python's latest version and pipenv on Windows Powershell. When I try to install requests it is showing this error:
FileNotFoundError: [WinError 2] The system cannot find the file specified
ValueError: Not a valid python path: 'C:/ProgramData/Anaconda3/python.exe'
Full error code:
Traceback (most recent call last):
File "C:\Python39\Lib\site-packages\pipenv\vendor\pythonfinder\models\python.py", line 618, in parse_executable
result_version = get_python_version(path)
File "C:\Python39\Lib\site-packages\pipenv\vendor\pythonfinder\utils.py", line 105, in get_python_version
c = subprocess.Popen(version_cmd, **subprocess_kwargs)
File "c:\python39\lib\subprocess.py", line 951, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "c:\python39\lib\subprocess.py", line 1420, in _execute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\python39\lib\runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "c:\python39\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\Python39\Scripts\pipenv.exe\__main__.py", line 7, in <module>
File "C:\Python39\Lib\site-packages\pipenv\vendor\click\core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "C:\Python39\Lib\site-packages\pipenv\vendor\click\core.py", line 782, in main
rv = self.invoke(ctx)
File "C:\Python39\Lib\site-packages\pipenv\vendor\click\core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "C:\Python39\Lib\site-packages\pipenv\vendor\click\core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "C:\Python39\Lib\site-packages\pipenv\vendor\click\core.py", line 610, in invoke
return callback(*args, **kwargs)
File "C:\Python39\Lib\site-packages\pipenv\vendor\click\decorators.py", line 73, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
File "C:\Python39\Lib\site-packages\pipenv\vendor\click\core.py", line 610, in invoke
return callback(*args, **kwargs)
File "C:\Python39\Lib\site-packages\pipenv\vendor\click\decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File "c:\python39\lib\site-packages\pipenv\cli\command.py", line 233, in install
retcode = do_install(
File "c:\python39\lib\site-packages\pipenv\core.py", line 1920, in do_install
ensure_project(
File "c:\python39\lib\site-packages\pipenv\core.py", line 576, in ensure_project
ensure_virtualenv(
File "c:\python39\lib\site-packages\pipenv\core.py", line 498, in ensure_virtualenv
python = ensure_python(three=three, python=python)
File "c:\python39\lib\site-packages\pipenv\core.py", line 388, in ensure_python
path_to_python = find_a_system_python(python)
File "c:\python39\lib\site-packages\pipenv\core.py", line 350, in find_a_system_python
return next(iter(finder.find_all_python_versions()), None)
File "c:\python39\lib\site-packages\pipenv\vendor\pythonfinder\pythonfinder.py", line 312, in find_all_python_versions
python_version_dict = getattr(self.system_path, "python_version_dict", {})
File "c:\python39\lib\site-packages\pipenv\vendor\pythonfinder\pythonfinder.py", line 120, in system_path
self._system_path = self.create_system_path()
File "c:\python39\lib\site-packages\pipenv\vendor\pythonfinder\pythonfinder.py", line 82, in create_system_path
return pyfinder_path.SystemPath.create(
File "C:\Python39\Lib\site-packages\pipenv\vendor\pythonfinder\models\path.py", line 682, in create
instance = instance._run_setup()
File "C:\Python39\Lib\site-packages\pipenv\vendor\pythonfinder\models\path.py", line 233, in _run_setup
new_instance = new_instance._setup_windows()
File "C:\Python39\Lib\site-packages\pipenv\vendor\pythonfinder\models\path.py", line 411, in _setup_windows
windows_finder = WindowsFinder.create()
File "C:\Python39\Lib\site-packages\pipenv\vendor\pythonfinder\models\windows.py", line 146, in create
return cls()
File "<attrs generated init pythonfinder.models.windows.WindowsFinder>", line 13, in __init__
File "C:\Python39\Lib\site-packages\pipenv\vendor\pythonfinder\models\windows.py", line 113, in get_versions
versions[py_version.version_tuple[:5]] = base_dir
File "C:\Python39\Lib\site-packages\pipenv\vendor\pythonfinder\models\python.py", line 365, in __getattribute__
result = super(PythonVersion, self).__getattribute__(key)
File "C:\Python39\Lib\site-packages\pipenv\vendor\pythonfinder\models\python.py", line 430, in version_tuple
self.patch,
File "C:\Python39\Lib\site-packages\pipenv\vendor\pythonfinder\models\python.py", line 375, in __getattribute__
instance_dict = self.parse_executable(executable)
File "C:\Python39\Lib\site-packages\pipenv\vendor\pythonfinder\models\python.py", line 620, in parse_executable
raise ValueError("Not a valid python path: %r" % path)
ValueError: Not a valid python path: 'C:/ProgramData/Anaconda3/python.exe'
It seems like you have two different Python distributions installed on your system: the Python 3.9 in C:\Python39\ and the Anaconda Python in C:/ProgramData/Anaconda3/. There is a conflict between them.
I recommend that you use one of them only - the Anaconda or the official Python, and uninstall another.
I tried installing tensorflow but I got this KeyError:
>py -3.7-64 -m pip install tensorflow
Collecting tensorflow
Downloading tensorflow-2.4.1-cp37-cp37m-win_amd64.whl (370.7 MB)
|██ | 20.2 MB 3.3 MB/s eta 0:01:47ERROR: Exception:
Traceback (most recent call last):
File "C:\...\Python37\lib\site-packages\pip\_vendor\resolvelib\resolvers.py", line 171, in _merge_into_criterion
crit = self.state.criteria[name]
KeyError: 'tensorflow'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\...\Python37\lib\site-packages\pip\_vendor\urllib3\response.py", line 438, in _error_catcher
yield
File "C:\...\Python37\lib\site-packages\pip\_vendor\urllib3\response.py", line 519, in read
data = self._fp.read(amt) if not fp_closed else b""
File "C:\...\Python37\lib\site-packages\pip\_vendor\cachecontrol\filewrapper.py", line 62, in read
data = self.__fp.read(amt)
File "C:\...\Python37\lib\http\client.py", line 461, in read
n = self.readinto(b)
File "C:\...\Python37\lib\http\client.py", line 505, in readinto
n = self.fp.readinto(b)
File "C:\...\Python37\lib\socket.py", line 589, in readinto
return self._sock.recv_into(b)
File "C:\...\Python37\lib\ssl.py", line 1071, in recv_into
return self.read(nbytes, buffer)
File "C:\...\Python37\lib\ssl.py", line 929, in read
return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\...\Python37\lib\site-packages\pip\_internal\cli\base_command.py", line 189, in _main
status = self.run(options, args)
File "C:\...\Python37\lib\site-packages\pip\_internal\cli\req_command.py", line 178, in wrapper
return func(self, options, args)
File "C:\...\Python37\lib\site-packages\pip\_internal\commands\install.py", line 317, in run
reqs, check_supported_wheels=not options.target_dir
File "C:\...\Python37\lib\site-packages\pip\_internal\resolution\resolvelib\resolver.py", line 122, in resolve
requirements, max_rounds=try_to_avoid_resolution_too_deep,
File "C:\...\Python37\lib\site-packages\pip\_vendor\resolvelib\resolvers.py", line 453, in resolve
state = resolution.resolve(requirements, max_rounds=max_rounds)
File "C:\...\Python37\lib\site-packages\pip\_vendor\resolvelib\resolvers.py", line 318, in resolve
name, crit = self._merge_into_criterion(r, parent=None)
File "C:\...\Python37\lib\site-packages\pip\_vendor\resolvelib\resolvers.py", line 173, in _merge_into_criterion
crit = Criterion.from_requirement(self._p, requirement, parent)
File "C:\...\Python37\lib\site-packages\pip\_vendor\resolvelib\resolvers.py", line 82, in from_requirement
if not cands:
File "C:\...\Python37\lib\site-packages\pip\_vendor\resolvelib\structs.py", line 124, in __bool__
return bool(self._sequence)
File "C:\...\Python37\lib\site-packages\pip\_internal\resolution\resolvelib\found_candidates.py", line 143, in __bool__
return any(self)
File "C:\...\Python37\lib\site-packages\pip\_internal\resolution\resolvelib\found_candidates.py", line 38, in _iter_built
candidate = func()
File "C:\...\Python37\lib\site-packages\pip\_internal\resolution\resolvelib\factory.py", line 169, in _make_candidate_from_link
name=name, version=version,
File "C:\...\Python37\lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 306, in __init__
version=version,
File "C:\...\Python37\lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 144, in __init__
self.dist = self._prepare()
File "C:\...\Python37\lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 226, in _prepare
dist = self._prepare_distribution()
File "C:\...\Python37\lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 312, in _prepare_distribution
self._ireq, parallel_builds=True,
File "C:\...\Python37\lib\site-packages\pip\_internal\operations\prepare.py", line 457, in prepare_linked_requirement
return self._prepare_linked_requirement(req, parallel_builds)
File "C:\...\Python37\lib\site-packages\pip\_internal\operations\prepare.py", line 482, in _prepare_linked_requirement
self.download_dir, hashes,
File "C:\...\Python37\lib\site-packages\pip\_internal\operations\prepare.py", line 234, in unpack_url
hashes=hashes,
File "C:\...\Python37\lib\site-packages\pip\_internal\operations\prepare.py", line 108, in get_http_url
from_path, content_type = download(link, temp_dir.path)
File "C:\...\Python37\lib\site-packages\pip\_internal\network\download.py", line 163, in __call__
for chunk in chunks:
File "C:\...\Python37\lib\site-packages\pip\_internal\cli\progress_bars.py", line 159, in iter
for x in it:
File "C:\...\Python37\lib\site-packages\pip\_internal\network\utils.py", line 88, in response_chunks
decode_content=False,
File "C:\...\Python37\lib\site-packages\pip\_vendor\urllib3\response.py", line 576, in stream
data = self.read(amt=amt, decode_content=decode_content)
File "C:\...\Python37\lib\site-packages\pip\_vendor\urllib3\response.py", line 541, in read
raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
File "C:\...\Python37\lib\contextlib.py", line 130, in __exit__
self.gen.throw(type, value, traceback)
File "C:\...\Python37\lib\site-packages\pip\_vendor\urllib3\response.py", line 443, in _error_catcher
raise ReadTimeoutError(self._pool, None, "Read timed out.")
pip._vendor.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out.
Then I tried installing it using a virtual environment (venv), and the progress bar went a bit further this time, but it still threw pretty much the same error.
Any ideas how to fix this?
Thanks.
Just use --default-timeout=100 parameter with the install. e.g
py -3.7-64 -m pip install --default-timeout=100 tensorflow
I hope, this will help you.
I've tried a lot of methods but useless. Finally I have to reinstalled my python and it works.
I am currently trying to figure out how to use netmiko to automate some of my routine work.Such as getting configuration backup, creating vlans etc.I've managed to use it on Aruba and Huawei Switches without problem.But on alcatel switch I'm facing this issue:
Unknown exception: module 'cryptography.utils' has no attribute 'bit_length'
Traceback (most recent call last):
File "C:\Users\melih.celik\AppData\Local\Programs\Python\Python39\lib\site-packages\paramiko\transport.py", line 2075, in run
self.kex_engine.parse_next(ptype, m)
File "C:\Users\melih.celik\AppData\Local\Programs\Python\Python39\lib\site-packages\paramiko\kex_gex.py", line 101, in parse_next
return self._parse_kexdh_gex_reply(m)
File "C:\Users\melih.celik\AppData\Local\Programs\Python\Python39\lib\site-packages\paramiko\kex_gex.py", line 281, in _parse_kexdh_gex_reply
self.transport._verify_key(host_key, sig)
File "C:\Users\melih.celik\AppData\Local\Programs\Python\Python39\lib\site-packages\paramiko\transport.py", line 1886, in _verify_key
if not key.verify_ssh_sig(self.H, Message(sig)):
File "C:\Users\melih.celik\AppData\Local\Programs\Python\Python39\lib\site-packages\paramiko\dsskey.py", line 148, in verify_ssh_sig
key = dsa.DSAPublicNumbers(
File "C:\Users\melih.celik\AppData\Local\Programs\Python\Python39\lib\site-packages\cryptography\hazmat\primitives\asymmetric\dsa.py", line 212, in public_key
return backend.load_dsa_public_numbers(self)
File "C:\Users\melih.celik\AppData\Local\Programs\Python\Python39\lib\site-packages\cryptography\hazmat\backends\openssl\backend.py", line 871, in load_dsa_public_numbers
dsa._check_dsa_parameters(numbers.parameter_numbers)
File "C:\Users\melih.celik\AppData\Local\Programs\Python\Python39\lib\site-packages\netmiko\tplink\tplink_jetstream.py", line 145, in _override_check_dsa_parameters
if crypto_utils.bit_length(parameters.q) not in [160, 256]:
AttributeError: module 'cryptography.utils' has no attribute 'bit_length'
Traceback (most recent call last):
File "C:\Users\melih.celik\Desktop\New_Backup\Yedek\Coding\Rand stuff\ssh_deneme(toplu).py", line 75, in
config_backup(cihaz_secim,ip_address,username,password)
File "C:\Users\melih.celik\Desktop\New_Backup\Yedek\Coding\Rand stuff\ssh_deneme(toplu).py", line 12, in config_backup
net_connect=ConnectHandler(**switch) #Baglanti kuruldu.
File "C:\Users\melih.celik\AppData\Local\Programs\Python\Python39\lib\site-packages\netmiko\ssh_dispatcher.py", line 312, in ConnectHandler
return ConnectionClass(*args, **kwargs)
File "C:\Users\melih.celik\AppData\Local\Programs\Python\Python39\lib\site-packages\netmiko\base_connection.py", line 346, in init
self._open()
File "C:\Users\melih.celik\AppData\Local\Programs\Python\Python39\lib\site-packages\netmiko\base_connection.py", line 351, in _open
self.establish_connection()
File "C:\Users\melih.celik\AppData\Local\Programs\Python\Python39\lib\site-packages\netmiko\base_connection.py", line 920, in establish_connection
self.remote_conn_pre.connect(**ssh_connect_params)
File "C:\Users\melih.celik\AppData\Local\Programs\Python\Python39\lib\site-packages\paramiko\client.py", line 406, in connect
t.start_client(timeout=timeout)
File "C:\Users\melih.celik\AppData\Local\Programs\Python\Python39\lib\site-packages\paramiko\transport.py", line 660, in start_client
raise e
File "C:\Users\melih.celik\AppData\Local\Programs\Python\Python39\lib\site-packages\paramiko\transport.py", line 2075, in run
self.kex_engine.parse_next(ptype, m)
File "C:\Users\melih.celik\AppData\Local\Programs\Python\Python39\lib\site-packages\paramiko\kex_gex.py", line 101, in parse_next
return self._parse_kexdh_gex_reply(m)
File "C:\Users\melih.celik\AppData\Local\Programs\Python\Python39\lib\site-packages\paramiko\kex_gex.py", line 281, in _parse_kexdh_gex_reply
self.transport._verify_key(host_key, sig)
File "C:\Users\melih.celik\AppData\Local\Programs\Python\Python39\lib\site-packages\paramiko\transport.py", line 1886, in _verify_key
if not key.verify_ssh_sig(self.H, Message(sig)):
File "C:\Users\melih.celik\AppData\Local\Programs\Python\Python39\lib\site-packages\paramiko\dsskey.py", line 148, in verify_ssh_sig
key = dsa.DSAPublicNumbers(
File "C:\Users\melih.celik\AppData\Local\Programs\Python\Python39\lib\site-packages\cryptography\hazmat\primitives\asymmetric\dsa.py", line 212, in public_key
return backend.load_dsa_public_numbers(self)
File "C:\Users\melih.celik\AppData\Local\Programs\Python\Python39\lib\site-packages\cryptography\hazmat\backends\openssl\backend.py", line 871, in load_dsa_public_numbers
dsa._check_dsa_parameters(numbers.parameter_numbers)
File "C:\Users\melih.celik\AppData\Local\Programs\Python\Python39\lib\site-packages\netmiko\tplink\tplink_jetstream.py", line 145, in _override_check_dsa_parameters
if crypto_utils.bit_length(parameters.q) not in [160, 256]:
AttributeError: module 'cryptography.utils' has no attribute 'bit_length'
Thanks for your help in advance.
It looks like tplink_jetstream.py assumes that a (now removed) helper function is available to it. The simplest fix here would be to go into that file and modify the line containing crypto_utils.bit_length(parameters.q) to read parameters.q.bit_length() instead.
I'm having issues while trying to run $pipenv --python 3.7, in this case I am running it while inside my virtual environment. As I see, this could be having trouble with another virtual environment called venvAmazon, thar I don't need or use anymore.
I noticed that, when I created my actual virtual environment, 1. it didn't created a pipfile, 2. It was created in python 2.7. (I do have 3.7 in my pc), 3. It also throws me an error if I try to run outside the venv
Traceback (most recent call last):
File "/home/daniel/.local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/models/python.py", line 112, in get_versions
version = PythonVersion.parse(p.name)
File "/home/daniel/.local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/models/python.py", line 359, in parse
version_dict = parse_python_version(str(version))
File "/home/daniel/.local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/utils.py", line 86, in parse_python_version
raise InvalidPythonVersion("%s is not a python version" % version_str)
pipenv.vendor.pythonfinder.exceptions.InvalidPythonVersion: venvAmazon is not a python version
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/daniel/.local/bin/pipenv", line 11, in <module>
sys.exit(cli())
File "/home/daniel/.local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/home/daniel/.local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/home/daniel/.local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 1114, in invoke
return Command.invoke(self, ctx)
File "/home/daniel/.local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/daniel/.local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/home/daniel/.local/lib/python3.6/site-packages/pipenv/vendor/click/decorators.py", line 64, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
File "/home/daniel/.local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/home/daniel/.local/lib/python3.6/site-packages/pipenv/vendor/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/daniel/.local/lib/python3.6/site-packages/pipenv/cli/command.py", line 208, in cli
clear=state.clear,
File "/home/daniel/.local/lib/python3.6/site-packages/pipenv/core.py", line 574, in ensure_project
pypi_mirror=pypi_mirror,
File "/home/daniel/.local/lib/python3.6/site-packages/pipenv/core.py", line 516, in ensure_virtualenv
ensure_python(three=three, python=python)
File "/home/daniel/.local/lib/python3.6/site-packages/pipenv/core.py", line 397, in ensure_python
path_to_python = find_a_system_python(python)
File "/home/daniel/.local/lib/python3.6/site-packages/pipenv/core.py", line 360, in find_a_system_python
python_entry = finder.find_python_version(line)
File "/home/daniel/.local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/pythonfinder.py", line 113, in find_python_version
return self.system_path.find_python_version(
File "/home/daniel/.local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/pythonfinder.py", line 54, in system_path
ignore_unsupported=self.ignore_unsupported,
File "/home/daniel/.local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/models/path.py", line 451, in create
ignore_unsupported=ignore_unsupported,
File "<attrs generated init 75e45f144e3d5510d54dd5fca6730b98fa0220a9>", line 38, in __init__
File "/home/daniel/.local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/models/path.py", line 116, in __attrs_post_init__
self._setup_pyenv()
File "/home/daniel/.local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/models/path.py", line 196, in _setup_pyenv
version_glob_path="versions/*", ignore_unsupported=self.ignore_unsupported
File "/home/daniel/.local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/models/python.py", line 156, in create
sort_function=sort_function, version_glob_path=version_glob_path)
File "<attrs generated init 3d06d7bc6a0a13af152279598b774c410bafffe4>", line 17, in __init__
File "/home/daniel/.local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/models/python.py", line 114, in get_versions
entry = next(iter(version_path.find_all_python_versions()), None)
AttributeError: 'NoneType' object has no attribute 'find_all_python_versions'
After I tried pip install --upgrade --pre pipenv It gave me another error:
File "/home/daniel/.local/lib/python2.7/site-packages/pipenv/core.py", line 541, in ensure_virtualenv
pypi_mirror=pypi_mirror,
File "/home/daniel/.local/lib/python2.7/site-packages/pipenv/core.py", line 541, in ensure_virtualenv
pypi_mirror=pypi_mirror,
File "/home/daniel/.local/lib/python2.7/site-packages/pipenv/core.py", line 541, in ensure_virtualenv
pypi_mirror=pypi_mirror,
File "/home/daniel/.local/lib/python2.7/site-packages/pipenv/core.py", line 535, in ensure_virtualenv
cleanup_virtualenv(bare=True)
File "/home/daniel/.local/lib/python2.7/site-packages/pipenv/core.py", line 181, in cleanup_virtualenv
vistir.path.rmtree(project.virtualenv_location)
File "/home/daniel/.local/lib/python2.7/site-packages/pipenv/vendor/vistir/path.py", line 433, in rmtree
shutil.rmtree(directory, ignore_errors=ignore_errors, onerror=onerror)
File "/usr/lib/python2.7/shutil.py", line 270, in rmtree
rmtree(fullname, ignore_errors, onerror)
File "/usr/lib/python2.7/shutil.py", line 275, in rmtree
onerror(os.remove, fullname, sys.exc_info())
File "/home/daniel/.local/lib/python2.7/site-packages/pipenv/vendor/vistir/path.py", line 494, in handle_remove_readonly
if is_readonly_path(path):
File "/home/daniel/.local/lib/python2.7/site-packages/pipenv/vendor/vistir/path.py", line 263, in is_readonly_path
fn = fs_decode(fs_encode(fn))
File "/home/daniel/.local/lib/python2.7/site-packages/pipenv/vendor/vistir/compat.py", line 382, in fs_encode
path = _get_path(path)
File "/home/daniel/.local/lib/python2.7/site-packages/pipenv/vendor/vistir/compat.py", line 297, in _get_path
if isinstance(path, (six.string_types, bytes)):
RuntimeError: maximum recursion depth exceeded while calling a Python object
It seems this is a currently open issue with pipenv. It might worth trying what they suggest there:
pip install --upgrade --pre pipenv
I have two python scripts, currentdataupload.py and productioncount.py on my Ubuntu server. currentdataupload.py uploads dummy current data and productioncount.py accesses that data to do some calculations. The code runs perfectly in other scenarios without disrupting the MongoDB, but when I run both of these codes simultaneously using nohup after a few seconds it exits the mongoDB service and is no longer able to upload or retrieve data. I would like to know if I am doing the entire process the right way or I am messing up somewhere big time.
Error log
Traceback (most recent call last):
File "/python_codes/currentdataupload.py", line 38, in <module>
result = posts.insert_one(post_data)
File "/root/anaconda3/lib/python3.5/site-packages/pymongo/collection.py", line 630, in insert_one
bypass_doc_val=bypass_document_validation),
File "/root/anaconda3/lib/python3.5/site-packages/pymongo/collection.py", line 535, in _insert
check_keys, manipulate, write_concern, op_id, bypass_doc_val)
File "/root/anaconda3/lib/python3.5/site-packages/pymongo/collection.py", line 516, in _insert_one
check_keys=check_keys)
File "/root/anaconda3/lib/python3.5/site-packages/pymongo/pool.py", line 244, in command
self._raise_connection_failure(error)
File "/root/anaconda3/lib/python3.5/site-packages/pymongo/pool.py", line 372, in _raise_connection_failure
raise error
File "/root/anaconda3/lib/python3.5/site-packages/pymongo/pool.py", line 239, in command
read_concern)
File "/root/anaconda3/lib/python3.5/site-packages/pymongo/network.py", line 96, in command
response = receive_message(sock, 1, request_id)
File "/root/anaconda3/lib/python3.5/site-packages/pymongo/network.py", line 123, in receive_message
header = _receive_data_on_socket(sock, 16)
File "/root/anaconda3/lib/python3.5/site-packages/pymongo/network.py", line 161, in _receive_data_on_socket
raise AutoReconnect("connection closed")
pymongo.errors.AutoReconnect: connection closed
Traceback (most recent call last):
File "/root/anaconda3/lib/python3.5/site-packages/pymongo/pool.py", line 543, in connect
sock = _configured_socket(self.address, self.opts)
File "/root/anaconda3/lib/python3.5/site-packages/pymongo/pool.py", line 452, in _configured_socket
sock = _create_connection(address, options)
File "/root/anaconda3/lib/python3.5/site-packages/pymongo/pool.py", line 436, in _create_connection
raise err
File "/root/anaconda3/lib/python3.5/site-packages/pymongo/pool.py", line 429, in _create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/python_codes/productioncount.py", line 267, in <module>
result = col2.find_one({"date": date, "machine": machine})
File "/root/anaconda3/lib/python3.5/site-packages/pymongo/collection.py", line 1014, in find_one
for result in cursor.limit(-1):
File "/root/anaconda3/lib/python3.5/site-packages/pymongo/cursor.py", line 1090, in next
if len(self.__data) or self._refresh():
File "/root/anaconda3/lib/python3.5/site-packages/pymongo/cursor.py", line 1012, in _refresh
self.__read_concern))
File "/root/anaconda3/lib/python3.5/site-packages/pymongo/cursor.py", line 850, in __send_message
**kwargs)
File "/root/anaconda3/lib/python3.5/site-packages/pymongo/mongo_client.py", line 844, in _send_message_with_response
exhaust)
File "/root/anaconda3/lib/python3.5/site-packages/pymongo/mongo_client.py", line 855, in _reset_on_error
return func(*args, **kwargs)
File "/root/anaconda3/lib/python3.5/site-packages/pymongo/server.py", line 99, in send_message_with_response
with self.get_socket(all_credentials, exhaust) as sock_info:
File "/root/anaconda3/lib/python3.5/contextlib.py", line 59, in __enter__
return next(self.gen)
File "/root/anaconda3/lib/python3.5/site-packages/pymongo/server.py", line 163, in get_socket
with self.pool.get_socket(all_credentials, checkout) as sock_info:
File "/root/anaconda3/lib/python3.5/contextlib.py", line 59, in __enter__
return next(self.gen)
File "/root/anaconda3/lib/python3.5/site-packages/pymongo/pool.py", line 582, in get_socket
sock_info = self._get_socket_no_auth()
File "/root/anaconda3/lib/python3.5/site-packages/pymongo/pool.py", line 628, in _get_socket_no_auth
sock_info = self._check(sock_info)
File "/root/anaconda3/lib/python3.5/site-packages/pymongo/pool.py", line 682, in _check
return self.connect()
File "/root/anaconda3/lib/python3.5/site-packages/pymongo/pool.py", line 555, in connect
_raise_connection_failure(self.address, error)
File "/root/anaconda3/lib/python3.5/site-packages/pymongo/pool.py", line 65, in _raise_connection_failure
raise AutoReconnect(msg)
pymongo.errors.AutoReconnect: localhost:27017: [Errno 111] Connection refused
MongoDB shouldn't crash. Please file a bug report, including the Python script you run and a complete MongoDB logfile, at jira.mongodb.org.