Develop Reference

Ansible playbooks stop running on new Ubuntu 20.04 box (Ansible 2.10 / Python 3.8.5)

One of my playbook ran ok on the following setup
ansible 2.9.11 config file = None configured module search path = ['/home/alice/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules'] ansible python module location = /usr/local/lib/python3.6/dist-packages/ansible executable location = /usr/local/bin/ansible python version = 3.6.5 (default, Apr 1 2018, 05:46:30) [GCC 7.3.0]
Things broke, however, when I moved the same playbook to the following setup
ansible 2.10.2
config file = /home/bob/ansible/ansible.cfg
configured module search path = ['/home/bob/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/local/lib/python3.8/dist-packages/ansible
executable location = /usr/local/bin/ansible
python version = 3.8.5 (default, Jul 28 2020, 12:59:40) [GCC 9.3.0]
I got the following error and am trying to figure out if it is the new python version, the Ansible version, or maybe the Ansible Galaxy module version (upgraded from Fortinet.Fortimanager 1.0.5 to 2.0.0, but when I force install 1.0.5, I get another error, which makes me think this isn't the main issue)
TASK [Add model device] ******************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ansible_collections.fortinet.fortimanager.plugins.module_utils.common.FMGBaseException: An attempt was made at communicating with a FMG with no valid session and an unexpected error was discovered.
fatal: [fmg01]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):\n File \"/tmp/ansible_fmgr_dvm_cmd_add_device_payload_ekiptyph/ansible_fmgr_dvm_cmd_add_device_payload.zip/ansible_collections/fortinet/fortimanager/plugins/modules/fmgr_dvm_cmd_add_device.py\", line 351, in main\n File \"/tmp/ansible_fmgr_dvm_cmd_add_device_payload_ekiptyph/ansible_fmgr_dvm_cmd_add_device_payload.zip/ansible/module_utils/connection.py\", line 195, in __rpc__\nansible.module_utils.connection.ConnectionError: An attempt was made at communicating with a FMG with no valid session and an unexpected error was discovered.\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/home/bob/.ansible/tmp/ansible-local-12942ssko667z/ansible-tmp-1603819794.3367019-13026-201727037698625/AnsiballZ_fmgr_dvm_cmd_add_device.py\", line 102, in <module>\n _ansiballz_main()\n File \"/home/bob/.ansible/tmp/ansible-local-12942ssko667z/ansible-tmp-1603819794.3367019-13026-201727037698625/AnsiballZ_fmgr_dvm_cmd_add_device.py\", line 94, in _ansiballz_main\n invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n File \"/home/bob/.ansible/tmp/ansible-local-12942ssko667z/ansible-tmp-1603819794.3367019-13026-201727037698625/AnsiballZ_fmgr_dvm_cmd_add_device.py\", line 40, in invoke_module\n runpy.run_module(mod_name='ansible_collections.fortinet.fortimanager.plugins.modules.fmgr_dvm_cmd_add_device', init_globals=None, run_name='__main__', alter_sys=True)\n File \"/usr/lib/python3.8/runpy.py\", line 207, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/usr/lib/python3.8/runpy.py\", line 97, in _run_module_code\n _run_code(code, mod_globals, init_globals,\n File \"/usr/lib/python3.8/runpy.py\", line 87, in _run_code\n exec(code, run_globals)\n File \"/tmp/ansible_fmgr_dvm_cmd_add_device_payload_ekiptyph/ansible_fmgr_dvm_cmd_add_device_payload.zip/ansible_collections/fortinet/fortimanager/plugins/modules/fmgr_dvm_cmd_add_device.py\", line 362, in <module>\n File \"/tmp/ansible_fmgr_dvm_cmd_add_device_payload_ekiptyph/ansible_fmgr_dvm_cmd_add_device_payload.zip/ansible_collections/fortinet/fortimanager/plugins/modules/fmgr_dvm_cmd_add_device.py\", line 356, in main\nansible_collections.fortinet.fortimanager.plugins.module_utils.common.FMGBaseException: An attempt was made at communicating with a FMG with no valid session and an unexpected error was discovered.\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
Any suggestions on how I isolate this given the output error?
Here is the playbook in question (super simple)
---
- name: Add model device to FMG and install Policy Package
hosts: fmg01
# gather_facts: no
connection: httpapi
collections:
- fortinet.fortimanager
tasks:
- name: Add model device
fmgr_dvm_cmd_add_device:
loose_validation: true
method: exec
params:
- data:
adom: root
device:
# device action: add_model
mgmt_mode: 'fmg'
#os_ver: 6
#mr: 4
sn: FGVM01TMxxxxxxxx
adm_pass: 'password'
adm_usr: 'admin'
ip: '192.168.0.100'

Whoops, the problem seems to be that the following item was missing from Fortinet Fortimanager:
config system admin user
edit admin
set rpc-permit read-write
end

.pyw program crashes upon execution, works as normal .py

I have a python program that I want to run without the console popping up. Naturally I went for a .pyw file to hide the console. However I noticed that after converting the file to a .pyw file. It would crash immediately upon executing.
I have tried uninstalling and reinstalling python and Flask. I ran the program within the Python IDLE and was finally able to get an error readout. I then looked it up and the answer ended up being "you have to run it in py". However the file runs completely fine on my desktop but not on my laptop or any other device that I have tried when set to a .pyw. This is the error traceback:
Python 3.7.4 (tags/v3.7.4:e09359112e, Jul 8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license()" for more information.
>>>
=========== RESTART: C:\Users\emidd\OneDrive\Desktop\FlaskApp.pyw ===========
Traceback (most recent call last):
File "C:\Users\emidd\OneDrive\Desktop\FlaskApp.pyw", line 16, in <module>
app.run()
File "C:\Users\emidd\AppData\Local\Programs\Python\Python37\lib\site-packages\flask\app.py", line 985, in run
cli.show_server_banner(self.env, self.debug, self.name, False)
File "C:\Users\emidd\AppData\Local\Programs\Python\Python37\lib\site-packages\flask\cli.py", line 670, in show_server_banner
click.echo(message)
File "C:\Users\emidd\AppData\Local\Programs\Python\Python37\lib\site-packages\click\utils.py", line 218, in echo
file = _default_text_stdout()
File "C:\Users\emidd\AppData\Local\Programs\Python\Python37\lib\site-packages\click\_compat.py", line 675, in func
rv = wrapper_func()
File "C:\Users\emidd\AppData\Local\Programs\Python\Python37\lib\site-packages\click\_compat.py", line 436, in get_text_stdout
rv = _get_windows_console_stream(sys.stdout, encoding, errors)
File "C:\Users\emidd\AppData\Local\Programs\Python\Python37\lib\site-packages\click\_winconsole.py", line 295, in _get_windows_console_stream
func = _stream_factories.get(f.fileno())
io.UnsupportedOperation: fileno
I know that it can work because I have it running on my desktop, however I am not sure why it works on it but not my laptop. When it works, the readout on the IDLE is "running on http://127.0.0.1:5000/" if anyone could help me get this working with Pythonw that would be great. However it does technically work with normal python.

Try running it on the commandline, IDLE has different IO objects which can mess with it.

Python 2.7 & exported 1 Selenium IDE script = Blank Firefox Page - Why?

I'm attempting to execute 1 test script (exported from Selenium IDE 3.4 where it runs fine) from Python 2.7 with Firefox 53.03 (64 bit) and Geckodriver 0.16. on my local machine - Windows 7 Enterprise.
Steps to Reproduce:
From Selenium IDE: Export Test Case As 'Python 2/unittest /WebDriver'.
Saved as 'Search by Vol & Folio - Unsuccessful.py'
Launched Python 2.7.12 Shell
File / Open / 'Search by Vol & Folio - Unsuccessful.py'
Run / Run Module
Results in:
cmd sessions opens with Geckodriver.exe + Blank Firefox page
Error message from Python console is:
Python 2.7.12 (v2.7.12:d33e0cf91556, Jun 27 2016, 15:24:40) [MSC v.1500 64 bit (AMD64)] on win32
Type "copyright", "credits" or "license()" for more information.
RESTART: C:\Temp\Selenium\selenium-java-3.4.0\Search by Vol & Folio - Unsuccessful.py
====================================================================== ERROR: test_search_by_vol_folio_unsuccessful_cancel
(main.SearchByVolFolioUnsuccessfulCancel)
---------------------------------------------------------------------- Traceback (most recent call last): File
"C:\Temp\Selenium\selenium-java-3.4.0\Search by Vol & Folio -
Unsuccessful.py", line 13, in setUp
self.driver = webdriver.Firefox() File "C:\Program Files\Common Files\Python\2.7\lib\site-packages\selenium\webdriver\firefox\webdriver.py",
line 155, in init
keep_alive=True) File "C:\Program Files\Common Files\Python\2.7\lib\site-packages\selenium\webdriver\remote\webdriver.py",
line 92, in init
self.start_session(desired_capabilities, browser_profile) File "C:\Program Files\Common
Files\Python\2.7\lib\site-packages\selenium\webdriver\remote\webdriver.py",
line 183, in start_session
self.capabilities = response['value'] KeyError: 'value'
---------------------------------------------------------------------- Ran 1 test in 5.250s
FAILED (errors=1)

It's not clear there problem came from.
self.capabilities = response['value']
but response not have key value

PySpark: [Errno 8] nodename nor servname provided, or not known

I am using pre-built 'spark-2.0.1-bin-hadoop2.7’ and when I try to start pyspark, I get following message.
Any ideas what could be wrong? I tried using python3, setting SPARK_LOCAL_IP to 127.0.0.1 but same error.
~ -> cd /Applications/spark-2.0.1-bin-hadoop2.7/bin/
/Applications/spark-2.0.1-bin-hadoop2.7/bin -> pyspark
Python 2.7.12 (default, Oct 11 2016, 05:24:00)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.38)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel).
16/12/19 14:50:47 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/12/19 14:50:47 WARN Utils: Your hostname, XXXXXX.com resolves to a loopback address: 127.0.0.1; using XX.XX.XX.XXX instead (on interface en0)
16/12/19 14:50:47 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
Traceback (most recent call last):
File "/Applications/spark-2.0.1-bin-hadoop2.7/python/pyspark/shell.py", line 43, in <module>
spark = SparkSession.builder\
File "/Applications/spark-2.0.1-bin-hadoop2.7/python/pyspark/sql/session.py", line 169, in getOrCreate
sc = SparkContext.getOrCreate(sparkConf)
File "/Applications/spark-2.0.1-bin-hadoop2.7/python/pyspark/context.py", line 294, in getOrCreate
SparkContext(conf=conf or SparkConf())
File "/Applications/spark-2.0.1-bin-hadoop2.7/python/pyspark/context.py", line 115, in __init__
conf, jsc, profiler_cls)
File "/Applications/spark-2.0.1-bin-hadoop2.7/python/pyspark/context.py", line 174, in _do_init
self._accumulatorServer = accumulators._start_update_server()
File "/Applications/spark-2.0.1-bin-hadoop2.7/python/pyspark/accumulators.py", line 259, in _start_update_server
server = AccumulatorServer(("localhost", 0), _UpdateRequestHandler)
File "/usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 417, in __init__
self.server_bind()
File "/usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 431, in server_bind
self.socket.bind(self.server_address)
File "/usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 228, in meth
return getattr(self._sock,name)(*args)
socket.gaierror: [Errno 8] nodename nor servname provided, or not known
Thanks

Found it. Some how my host mapping was messing it up. Changing it to point to localhost worked.:
/etc/host
#127.0.0.1 XXXXXX.com
127.0.0.1 localhost

In cases when you cannot cleanup /etc/hosts (such as it's being tempered with by some VPN solution), here is a workaround:
from pyspark.sql import SparkSession
def patch_pyspark_accumulators():
from inspect import getsource
import pyspark.accumulators as pa
exec(getsource(pa._start_update_server).replace("localhost", "127.0.0.1"), pa.__dict__)
patch_pyspark_accumulators()
spark = SparkSession.builder.getOrCreate()

Cannot connect to FTP server

I'm not able to connect to FTP server getting below error :-
vmware#localhost ~]$ python try_ftp.py
Traceback (most recent call last):
File "try_ftp.py", line 5, in <module>
f = ftplib.FTP('ftp.python.org')
File "/usr/lib/python2.6/ftplib.py", line 116, in __init__
self.connect(host)
File "/usr/lib/python2.6/ftplib.py", line 131, in connect
self.sock = socket.create_connection((self.host, self.port), self.timeout)
File "/usr/lib/python2.6/socket.py", line 567, in create_connection
raise error, msg
socket.error: [Errno 101] Network is unreachable
I'm writing a very simple code
import ftplib
f = ftplib.FTP('ftp.python.org')
f.login('anonymous','sausaxen#xyz.com')
f.dir()
f.retrlines('RETR motd')
f.quit()
I checked my proxy settings , but it is set to "System proxy setttings"
Please suggest what should I do.
Thanks,
Sam

[torxed#archie ~]$ telnet ftp.python.org 21
Trying 82.94.164.162...
Connection failed: Connection refused
Trying 2001:888:2000:d::a2...
telnet: Unable to connect to remote host: Network is unreachable
It's not as much the hostname that's bad (ping works you mentioned) but the default port of 21 is bad. Or they're not running a standard FTP server on that host at all but rather they're using HTTP as a transport: https://www.python.org/ftp/python/
Try against ftp.acc.umu.se instead.
[torxed#archie ~]$ python
Python 3.3.5 (default, Mar 10 2014, 03:21:31)
[GCC 4.8.2 20140206 (prerelease)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ftplib
>>> f = ftplib.FTP('ftp.acc.umu.se')
>>>

The address ftp.python.org seems bad
EDIT:
the f = ftplib.FTP('ftp.python.org') gives the error message but ping works.

Try pinging the "ftp.python.org" address.
If you need to pass through a proxy, check the you have ftp_proxy set as an environment variable. Normally, what I do is to set the proxy explicitly.
Also, as an alternative, try using httplib or requests