Develop Reference

Using Python 3.6.2 to connect to MongoDB on IBM Bluemix

So I'm attempting connect to a MongoDB on Bluemix (Compose for MongoDB service) through Python 3.6.2 on my local machine using the following code:
import json
import urllib.request
import pymongo
import ssl
#uri is string taken from Service Credentials section of MongoDB Bluemix page
uri_string = "mongodb://admin:<password>#server1:port,server2:port/compose?ssl=true&authSource=admin"
client = pymongo.MongoClient(uri_string)
db = client.handle
#reading in json data from api_url
data = urllib.request.urlopen("api_url")
parsed = json.loads(data)
for item in parsed['resultItemList']:
db.insert_one(item)
The goal is to insert the JSON data into my MongoDB but after running the for loop I am getting the following error message:
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
File "/usr/local/lib/python3.6/site-packages/pymongo/collection.py", line 667, in insert_one
with self._socket_for_writes() as sock_info:
File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/usr/local/lib/python3.6/site-packages/pymongo/mongo_client.py", line 868, in _get_socket
server = self._get_topology().select_server(selector)
File "/usr/local/lib/python3.6/site-packages/pymongo/topology.py", line 214, in select_server
address))
File "/usr/local/lib/python3.6/site-packages/pymongo/topology.py", line 189, in select_servers
self._error_message(selector))
pymongo.errors.ServerSelectionTimeoutError: SSL handshake failed: [SSL:
CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:748),SSL
handshake failed: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify
failed (_ssl.c:748)
I've searched extensively online and haven't been able to figure it out. I understand I somehow need to retrieve the SSL Certificate from Bluemix but do not understand the proper way to do so within python and certain options in MongoClient(). I've already installed the certifi python package:
python3 -m pip install certifi
which was unsuccessful and also tried turning off the SSL requirement with
ssl_cert_reqs=ssl.CERT_NONE

Try this. Mine works.
import os
import pymongo
import ssl
MONGODB_URL = os.environ.get('MONGODB_URL')
client = pymongo.MongoClient(MONGODB_URL,ssl_cert_reqs=ssl.CERT_NONE)
db = client.get_default_database()
print db.collection_names()

pip install not working windows 7

I have downloaded pip frpm https://sites.google.com/site/pydatalog/python/pip-for-windows
Now when I type any package name or upgrade in the command section I get the following error
Downloading https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py
Traceback (most recent call last):
File "C:\Python33\lib\urllib\request.py", line 1248, in do_open
h.request(req.get_method(), req.selector, req.data, headers)
File "C:\Python33\lib\http\client.py", line 1065, in request
self._send_request(method, url, body, headers)
File "C:\Python33\lib\http\client.py", line 1103, in _send_request
self.endheaders(body)
File "C:\Python33\lib\http\client.py", line 1061, in endheaders
self._send_output(message_body)
File "C:\Python33\lib\http\client.py", line 906, in _send_output
self.send(msg)
File "C:\Python33\lib\http\client.py", line 844, in send
self.connect()
File "C:\Python33\lib\http\client.py", line 1198, in connect
self.timeout, self.source_address)
File "C:\Python33\lib\socket.py", line 435, in create_connection
raise err
File "C:\Python33\lib\socket.py", line 426, in create_connection
sock.connect(sa)
TimeoutError: [WinError 10060] A connection attempt failed because the connected
party did not properly respond after a period of time, or established connectio
n failed because connected host has failed to respond
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\Admin~1\AppData\Local\Temp\rmv_setup.py", line 60, in <module>
download(url, "ez_setup.py")
File "C:\Users\Admin~1\AppData\Local\Temp\rmv_setup.py", line 30, in download
src = urlopen(url)
File "C:\Python33\lib\urllib\request.py", line 156, in urlopen
return opener.open(url, data, timeout)
urllib.error.URLError: <urlopen error [WinError 10060] A connection attempt fail
ed because the connected party did not properly respond after a period of time,
or established connection failed because connected host has failed to respond>
I am behind a proxy. but I can access bitbucket.org through browser. How can I fix this issue?

Considering that you successfully installed pip and now you are trying to install another module using pip.
pip has "proxy" option.Please try to use it and check whether it is helpful.
C:\Users\Administrator\Desktop>pip --help
Usage:
pip <command> [options]
Commands:
install Install packages.
uninstall Uninstall packages.
freeze Output installed packages in requirements format.
list List installed packages.
show Show information about installed packages.
search Search PyPI for packages.
wheel Build wheels from your requirements.
zip DEPRECATED. Zip individual packages.
unzip DEPRECATED. Unzip individual packages.
bundle DEPRECATED. Create pybundles.
help Show help for commands.
General Options:
-h, --help Show help.
-v, --verbose Give more output. Option is additive, and can be
used up to 3 times.
-V, --version Show version and exit.
-q, --quiet Give less output.
--log-file <path> Path to a verbose non-appending log, that only
logs failures. This log is active by default at
C:\Users\Administrator\pip\pip.log.
--log <path> Path to a verbose appending log. This log is
inactive by default.
--proxy <proxy> Specify a proxy in the form
[user:passwd#]proxy.server:port.
--timeout <sec> Set the socket timeout (default 15 seconds).
--exists-action <action> Default action when a path already exists:
(s)witch, (i)gnore, (w)ipe, (b)ackup.
--cert <path> Path to alternate CA bundle.
C:\Users\Administrator\Desktop>

This is more like a comment than an answer.
If your access to internet is through the proxy, i.e. you have no access to internet except establishing the proxy, then pip/pip3 won't be able to download the files. Try ping google.com if you see no response then try installing an application like proxyfire to enforce the proxy settings across the entire system.
Hope it helps.

Error 10060 means it cannot connect to the remote peer.
You probably want to check if ping and telnet work through port 80. If only ping works and telnet doesn't, then HTTP port 80 is closed on your machine.
You can try to connect through another tool such as ncat
I know there exists a function in the urllib2 library that will allow you to handle proxy support which is something like:
import urllib2
proxy_handle = urllib2.ProxyHandler({"http":"http://123.124.125.126:80"})
opener = urllib2.build_opener(proxy_handle)
urllib2.install_opener(opener)

SSL: CERTIFICATE_VERIFY_FAILED with Python3

I apologize if this is a silly question, but I have been trying to teach myself how to use BeautifulSoup so that I can create a few projects.
I was following this link as a tutorial: https://www.youtube.com/watch?v=5GzVNi0oTxQ
After following the exact same code as him, this is the error that I get:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 1240, in do_open
h.request(req.get_method(), req.selector, req.data, headers)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1083, in request
self._send_request(method, url, body, headers)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1128, in _send_request
self.endheaders(body)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1079, in endheaders
self._send_output(message_body)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 911, in _send_output
self.send(msg)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 854, in send
self.connect()
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1237, in connect
server_hostname=server_hostname)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 376, in wrap_socket
_context=self)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 747, in __init__
self.do_handshake()
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 983, in do_handshake
self._sslobj.do_handshake()
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 628, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "WorldCup.py", line 3, in <module>
x = urllib.request.urlopen('https://www.google.com')
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 162, in urlopen
return opener.open(url, data, timeout)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 465, in open
response = self._open(req, data)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 483, in _open
'_open', req)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 443, in _call_chain
result = func(*args)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 1283, in https_open
context=self._context, check_hostname=self._check_hostname)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 1242, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)>
Can someone help me figure out how to fix this?

In my case, I used the ssl module to "workaround" the certification like so:
import ssl
ssl._create_default_https_context = ssl._create_unverified_context
Then to read your link content, you can use:
urllib.request.urlopen(urllink)

Go to the folder where Python is installed, e.g., in my case (Mac OS) it is installed in the Applications folder with the folder name 'Python 3.6'. Now double click on 'Install Certificates.command'. You will no longer face this error.
For those not running a mac, or having a different setup and can't find this file, the file merely runs:
pip install --upgrade certifi

On Debian 9 I had to:
$ sudo update-ca-certificates --fresh
$ export SSL_CERT_DIR=/etc/ssl/certs
I'm not sure why, but this enviroment variable was never set.

This has changed in recent versions of the ssl library. The SSLContext was moved to it's own property. This is the equivalent of Jia's answer in Python 3.8
import ssl
ssl.SSLContext.verify_mode = ssl.VerifyMode.CERT_OPTIONAL

Building on the update to Jia's 2018 answer in deltree's late 2021 one I was able to achieve equivalent functionality with:
import urllib.request
import ssl
def urllib_get_2018():
# Using a protected member like this is not any more fragile
# than extending the class and using it. I would use it.
url = 'https://localhost:6667/my-endpoint'
ssl._create_default_https_context = ssl._create_unverified_context
with urllib.request.urlopen(url = url) as f:
print(f.read().decode('utf-8'))
def urllib_get_2022():
# Finally! Able to use the publice API. Happy happy!
url = 'https://localhost:6667/my-endpoint'
scontext = ssl.SSLContext(ssl.PROTOCOL_TLS)
scontext.verify_mode = ssl.VerifyMode.CERT_NONE
with urllib.request.urlopen(url = url, context=scontext) as f:
print(f.read().decode('utf-8'))
I needed to use CERT_NONE instead of CERT_OPTIONAL as well as creating a ssl.SSLContext(ssl.PROTOCOL_TLS) to pass to urlopen.
It's important to keep using urllib as it makes sense when working with small container images where pip might not be installed, yet.

When you are using a self signed cert urllib3 version 1.25.3 refuses to ignore the SSL cert
To fix remove urllib3-1.25.3 and install urllib3-1.24.3
pip3 uninstall urllib3
pip3 install urllib3==1.24.3
Tested on Linux MacOS and Window$

As a workaround (not secure), you can turn certificate verification off by setting PYTHONHTTPSVERIFY environment variable to 0:
export PYTHONHTTPSVERIFY=0

I have a lib what use https://requests.readthedocs.io/en/master/ what use https://pypi.org/project/certifi/ but I have a custom CA included in my /etc/ssl/certs.
So I solved my problem like this:
# Your TLS certificates directory (Debian like)
export SSL_CERT_DIR=/etc/ssl/certs
# CA bundle PATH (Debian like again)
export CA_BUNDLE_PATH="${SSL_CERT_DIR}/ca-certificates.crt"
# If you have a virtualenv:
. ./.venv/bin/activate
# Get the current certifi CA bundle
CERTFI_PATH=`python -c 'import certifi; print(certifi.where())'`
test -L $CERTFI_PATH || rm $CERTFI_PATH
test -L $CERTFI_PATH || ln -s $CA_BUNDLE_PATH $CERTFI_PATH
Et voilà !

I faced the same issue with Ubuntu 20.4 and have tried many solutions but nothing worked out. Finally I just checked openssl version. Even after update and upgrade, the openssl version showed OpenSSL 1.1.1h [22 Sep 2020]. But in my windows system, where the code works without any issue, openssl version is OpenSSL 1.1.1k 25 Mar 2021.
I decided to update the openssl manually and it worked! Thank God!!!
Steps are as follows(Ubuntu 20.4):
*To check openssl version
openssl version -a
*To update openssl:
sudo apt install build-essential checkinstall zlib1g-dev
cd /usr/local/src/
sudo wget https://www.openssl.org/source/openssl-1.1.1k.tar.gz
sudo tar -xf openssl-1.1.1k.tar.gz
cd openssl-1.1.1k
sudo ./config --prefix=/usr/local/ssl --openssldir=/usr/local/ssl shared zlib
sudo make
sudo make test
sudo make install
cd /etc/ld.so.conf.d/
sudo nano openssl-1.1.1k.conf
*Type /usr/local/ssl/lib and save
sudo ldconfig -v
sudo nano /etc/environment
*Add ':/usr/local/ssl/bin' to the path
source /etc/environment
echo $PATH
*Now check openssl version
openssl version -a

you might exec command: pip install --upgrade certifi
or you might opened charles/fiddler, just close it

I had this problem in MacOS, and I solved it by linking the brew installed python 3 version, with
brew link python3
After that, it worked without a problem.

imapClient error (diff version python and windows)

This question was asked by paul_h, but I'm too new to comment with (hopefully) relevant information. I'm having the same problem with imapClient, and it's obviously not Python/Windows version specific. Paul H is running Python 2.7.11 on win 10. imapclient error on Windows
I was running 3.4.3, and even upgraded to 3.5.1. on both win7, and even tried an old Vista machine.
import imapclient
imapObj = imapclient.IMAPClient('imap.gmail.com', ssl=True)
It returns:
Traceback (most recent call last):
File "<pyshell#8>", line 1, in <module>
imapObj = imapclient.IMAPClient('imap.gmail.com', ssl=True)
File "C:\Python\lib\site-packages\imapclient\imapclient.py", line 152, in __init__
self._imap = self._create_IMAP4()
File "C:\Python\lib\site-packages\imapclient\imapclient.py", line 164, in _create_IMAP4
self._timeout)
File "C:\Python\lib\site-packages\imapclient\tls.py", line 153, in __init__
imaplib.IMAP4.__init__(self, host, port)
File "C:\Python\lib\imaplib.py", line 189, in __init__
self.open(host, port)
File "C:\Python\lib\site-packages\imapclient\tls.py", line 159, in open
self.sock = wrap_socket(sock, self.ssl_context, host)
File "C:\Python\lib\site-packages\imapclient\tls.py", line 126, in wrap_socket
ssl_context = create_default_context()
File "C:\Python\lib\site-packages\imapclient\tls.py", line 109, in create_default_context
context.load_verify_locations(cadata=certs)
File "C:\Python\lib\site-packages\backports\ssl\core.py", line 654, in load_verify_locations
self._ctx.load_verify_locations(cafile, capath)
File "C:\Python\lib\site-packages\OpenSSL\SSL.py", line 528, in load_verify_locations
_raise_current_error()
File "C:\Python\lib\site-packages\OpenSSL\_util.py", line 48, in exception_from_error_queue
raise exception_type(errors)
OpenSSL.SSL.Error: []
Meanwhile the stock imaplib connects with ssl fine.
M = imaplib.IMAP4_SSL('imap.gmail.com')
And sorry about starting a new question, but I didn't want to post this as an answer.

I had this problem as well , I was waiting for someone to answer this but took to long.
We were using IMAPclient 1.0.1 and got this error , so workaround is to install imapclient 0.13
pip uninstall imapclient
pip install imapclient==0.13

Here's a workaround that worked for me (Python 3.5, Windows 10):
from backports import ssl
from imapclient import IMAPClient
context = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2)
imapObj = IMAPClient('imap.gmail.com', ssl=True, ssl_context=context)
The above code was derived from the developer's workaround here, but I found I only needed the one line defining context to make it work. Specifying other SSL/TLS protocols also worked.

Can't use fabric put - Is there any server configuration needed?

I'm using fabric to do a remote deployment of my app on a rackspace server. I've tried my scripts on virtual machines using the same OS (Ubuntu Server 10.04) on my home computer and they all seem to work.
Strangely, all put fabric commands fail on the real server. All other commands (run, cd, sudo, etc) seem to work ok.
This only happens when targeting this specific server, this is the command I execute:
fab test --host remote-server
remote-server is an alias on my .ssh/config. My fabfile:
#task
def test():
sudo("echo testing")
put("/tmp/file.txt", "/tmp/")
tmp/test_file.txt is just a text file I'm using for my tests
This is the output
[remote-server] Executing task 'test'
[remote-server] sudo: echo testing
[remote-server] out: testing
Traceback (most recent call last):
File "/home/user/env/lib/python2.6/site-packages/fabric/main.py", line 712, in main
*args, **kwargs
File "/home/user/env/lib/python2.6/site-packages/fabric/tasks.py", line 298, in execute
multiprocessing
File "/home/user/env/lib/python2.6/site-packages/fabric/tasks.py", line 197, in _execute
return task.run(*args, **kwargs)
File "/home/user/env/lib/python2.6/site-packages/fabric/tasks.py", line 112, in run
return self.wrapped(*args, **kwargs)
File "/home/user/project/fabfile/__init__.py", line 33, in test
put("/tmp/file.txt", "/tmp/")
File "/home/user/env/lib/python2.6/site-packages/fabric/network.py", line 457, in host_prompting_wrapper
return func(*args, **kwargs)
File "/home/user/env/lib/python2.6/site-packages/fabric/operations.py", line 338, in put
ftp = SFTP(env.host_string)
File "/home/user/env/lib/python2.6/site-packages/fabric/sftp.py", line 20, in __init__
self.ftp = connections[host_string].open_sftp()
File "/home/user/env/lib/python2.6/site-packages/ssh/client.py", line 399, in open_sftp
return self._transport.open_sftp_client()
File "/home/user/env/lib/python2.6/site-packages/ssh/transport.py", line 844, in open_sftp_client
return SFTPClient.from_transport(self)
File "/home/user/env/lib/python2.6/site-packages/ssh/sftp_client.py", line 105, in from_transport
chan.invoke_subsystem('sftp')
File "/home/user/env/lib/python2.6/site-packages/ssh/channel.py", line 240, in invoke_subsystem
self._wait_for_event()
File "/home/user/env/lib/python2.6/site-packages/ssh/channel.py", line 1114, in _wait_for_event
raise e
ssh.SSHException: Channel closed.
Disconnecting from root#server.com... done.
Is there anything I need to configure on the remote server to be able to send files using put?

Thanks to #Drake I found out that there was an issue with the sftp server on the remote machine.
To test for this:
$ sftp remote-server
subsystem request failed on channel 0
Couldn't read packet: Connection reset by peer
I read that in order to enable sftp I needed to add the line
Subsystem sftp /usr/lib/openssh/sftp-server
to /etc/ssh/sshd_config and restart (/etc/init.d/ssh restart) the ssh service. But the line was already there and it wasn't working.
Then, after reading http://forums.debian.net/viewtopic.php?f=5&t=42818, I changed that line for
Subsystem sftp internal-sftp
restarted the ssh service, and it is now working:
$ sftp remote-server
Connected to remote-server
sftp>

I had same issue.
I found sftp was not installed on my server.
I installed openssh and restarted sshd service.
yum -y install openssh
service sshd restart
Then also i had same issue.
I checked the system log /var/log/messages. I found following error
Jul 3 04:23:20 <ip> sshd[13996]: subsystem request for sftp
Jul 3 04:23:20 <ip> sshd[13996]: error: subsystem: cannot stat /usr/libexec/sftp- server: No such file or directory
Jul 3 04:23:20 <ip> sshd[13996]: subsystem request for sftp failed, subsystem not found
I locate my sftp-server location which was in "/usr/libexec/openssh/sftp-server" and script was looking at "/usr/libexec/sftp-server" location
I created symbolic link and my issue got resolved.
root#<ip> fabric]# locate sftp-server
/usr/libexec/openssh/sftp-server
/usr/share/man/man8/sftp-server.8.gz
ln -s /usr/libexec/openssh/sftp-server /usr/libexec/sftp-server