Develop Reference

How to properly connect to SQL Server from a python script when python packages are based on github?

Suppose that due to an HTTP 403 Error it's not possible to download the packages from the PyPi repo (nor pip install <package> commands) which causes me to install the pyodbc by cloning the repo from Github (https://github.com/mkleehammer/pyodbc) and by running the next .cmd windows file:
cd "root_folder"
git activate
git clone https://github.com/mkleehammer/pyodbc.git --depth 1
Note that this package is downloaded to the same root folder where my python script is, after this I try to set a connection to Microsoft SQL Server:
import pyodbc as pyodbc
# set connection settings
server="servername"
database="DB1"
user="user1"
password="123"
# establishing connection to db
conn = pyodbc.connect("DRIVER={SQL Server};SERVER="+server+";DATABASE="+database+";UID="+user+";PWD="+password)
cursor=conn.cursor()
print("Succesful connection to sql server")
However, when I run the above code the next traceback error arises:
Traceback (most recent call last):
File "/dcleaner.py", line 47, in
conn = pyodbc.connect("DRIVER={SQL Server};SERVER="+server+";DATABASE="+database+";UID="+user+";PWD="+password)
AttributeError: module 'pyodbc' has no attribute 'connect'
Do you know how can I properly connect from a py script to a sql-server based database?

After you have cloned PYODBC
cd "root_folder"
git activate
git clone https://github.com/mkleehammer/pyodbc.git --depth 1
On your Local Machine, Go into the cloned directory and open terminal and run below command
python setup.py build
if it errors then try to install appropriate C++ compiler (the error might reveal this detail/ on VSCode it gave the URL to open and download, which I have shared below) install it from here link - choose this one
reboot machine and run this again
python setup.py build #if success then continue with below one
python setup.py install
after that you should be able to import and run the below from your local machine
import pyodbc as pyodbc

Using pyhive with kerberos ticket to connect to kerberized hadoop cluster

I would like to connect to Hive on our kerberized Hadoop cluster and then run some hql queries (obviously haha :)) from machine, which already has its own Kerberose Client and it works, keytab has been passed and tested.
Our Hadoop runs HWS 3.1 and CentOS7, my machine als runs CentOS7
I'm using Python 3.7.3 and PyHive (0.6.1).
I have installed bunch of libraries (and I also tried to uninstall them), as I was going through different forums (HWS, Cloudera, here SO...)
I installed through pip sasl libraries
pure-sasl (0.6.1)
pysasl (0.4.1)
sasl (0.2.1)
thrift-sasl (0.3.0)
I installed through yum
cyrus-sasl-2.1.26-23.el7.x86_64
cyrus-sasl-lib-2.1.26-23.el7.x86_64
cyrus-sasl-plain-2.1.26-23.el7.x86_64
saslwrapper-devel-0.16-5.el7.x86_64
saslwrapper-0.16-5.el7.x86_64
cyrus-sasl-lib-2.1.26-23.el7.i686
cyrus-sasl-devel-2.1.26-23.el7.x86_64
Below lies my connection to the hive
return hive.Connection(host=self.host, port=self.port,
database=self.database, auth=self.__auth,
kerberos_service_name=self.__kerberos_service_name)
This is part of my yaml
hive_interni_hdp:
db_type: hive
host: domain.xx.lan
database: database_name
user: user_name
port: 10000
auth: KERBEROS
kerberos_service_name: hive
When I try to run the code, I'm getting following error.
File "/opt/Python3.7.3/lib/python3.7/site-packages/dfpy/location.py", line 1647, in conn
self.__conn = self._create_connection()
File "/opt/Python3.7.3/lib/python3.7/site-packages/dfpy/location.py", line 1633, in _create_connection
kerberos_service_name=self.__kerberos_service_name)
File "/opt/Python3.7.3/lib/python3.7/site-packages/pyhive/hive.py", line 192, in __init__
self._transport.open()
File "/opt/Python3.7.3/lib/python3.7/site-packages/thrift_sasl/__init__.py", line 79, in open
message=("Could not start SASL: %s" % self.sasl.getError()))
thrift.transport.TTransport.TTransportException: Could not start SASL: b'Error in sasl_client_start (-4) SASL(-4): no mechanism available: No worthy mechs found'
Did anyone had luck? Where is the obstacle? Pyhive libs, wrong Kerberos connection settings?

I found an solution, I checked out this documentation https://www.cyrusimap.org/sasl/sasl/sysadmin.html
where is GSSAPI mentioned (with Kerberos 5, which I'm using) and I have checked, that I have no support for gssapi on my machine using
sasl2-shared-mechlist
It stated
GSS-SPNEGO,LOGIN,PLAIN,ANONYMOUS
but after installing gssapi library
yum install cyrus-sasl-gssapi
mechlist states
GSS-SPNEGO,GSSAPI,LOGIN,PLAIN,ANONYMOUS
Than I run the code again and Hooray!
P.S. Don't forget to autentificate and verify your keytab is valid
kinit -kt /root/user.keytab user#domain.com
klist

redis.exceptions.ConnectionError: Error 97 connecting to localhost:6379. Address family not supported by protocol

when ever i try to run my program following error will will raise.
redis.exceptions.ConnectionError: Error 97 connecting to localhost:6379. Address family not supported by protocol.
Previously the program runs normally now this error will be raised.
Traceback (most recent call last):
File "securit.py", line 26, in <module>
bank = red.get('bank')
File "/usr/local/lib/python2.7/dist-packages/redis/client.py", line 880, in get
return self.execute_command('GET', name)
File "/usr/local/lib/python2.7/dist-packages/redis/client.py", line 578, in execute_command
connection.send_command(*args)
File "/usr/local/lib/python2.7/dist-packages/redis/connection.py", line 563, in send_command
self.send_packed_command(self.pack_command(*args))
File "/usr/local/lib/python2.7/dist-packages/redis/connection.py", line 538, in send_packed_command
self.connect()
File "/usr/local/lib/python2.7/dist-packages/redis/connection.py", line 442, in connect
raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 97 connecting to localhost:6379. Address family not supported by protocol.

Finally i got answer for above qustion.Step by step the following done
Setup
Before you install redis, there are a couple of prerequisites that need to be downloaded to make the installation as easy as possible.
Start off by updating all of the apt-get packages:
**sudo apt-get update**
Once the process finishes, download a compiler with build essential which will help us install Redis from source:
**sudo apt-get install build-essential**
Finally, we need to download tcl:
**sudo apt-get install tcl8.5**
Installing Redis
With all of the prerequisites and dependencies downloaded to the server, we can go ahead and begin to install redis from source:
Download the latest stable release tarball from Redis.io.
**wget http://download.redis.io/releases/redis-stable.tar.gz**
Untar it and switch into that directory:
**tar xzf redis-stable.tar.gz**
**cd redis-stable**
Proceed to with the make command:
**make**
Run the recommended make test:
**make test**
Finish up by running make install, which installs the program system-wide.
**sudo make install**
Once the program has been installed, Redis comes with a built in script that sets up Redis to run as a background daemon.
To access the script move into the utils directory:
**cd utils**
From there, run the Ubuntu/Debian install script:
**sudo ./install_server.sh**
As the script runs, you can choose the default options by pressing enter. Once the script completes, the redis-server will be running in the background.

Had the same problem. Deleting IPv6 addresses from /etc/hosts file helped for me

Python module PyMySQL not taking arguments

I wrote a python script that communicate with MySQL server on OSX 10.10, it runs fine Mac until I put it on an VPS running Ubuntu 14.04
The problem mainly lies on the pyMySQL module. :I can't even run their example script on the git page here
for running the following code:
import pymysql.cursors
# Connect to the database
connection = pymysql.connect(host='localhost',
user='user',
password='passwd',
db='db',
charset='utf8mb4',
cursorclass=pymysql.cursors.DictCursor)
try:
with connection.cursor() as cursor:
# Create a new record
sql = "INSERT INTO `users` (`email`, `password`) VALUES (%s, %s)"
cursor.execute(sql, ('webmaster#python.org', 'very-secret'))
# connection is not autocommit by default. So you must commit to save
# your changes.
connection.commit()
with connection.cursor() as cursor:
# Read a single record
sql = "SELECT `id`, `password` FROM `users` WHERE `email`=%s"
cursor.execute(sql, ('webmaster#python.org',))
result = cursor.fetchone()
print(result)
finally:
connection.close()
The result is
Traceback (most recent call last):
File "1.py", line 9, in <module>
cursorclass=pymysql.cursors.DictCursor)
File "build/bdist.linux-x86_64/egg/pymysql/__init__.py", line 93, in Connect
TypeError: __init__() got an unexpected keyword argument 'password'
Environment:
Ubuntu 14.04 x86_64
Python 2.7.10
PyMySQL 0.6.7
Mac is running the above script fine but not the ubuntu.
Thanks in advance.

I didn't do much checking on the error message. Instead I uninstall all the packages on the unbuntu server and follow the package list on the Mac to install them again one by one to the same version. Amazingly, the problem is solved. Unfortunately I can't tell which module is causing the problem
You might want to give it a try if you don't want to spend much time on updating modules one by one.

It seems that this is a ubuntu problem, which is solved if you install the latest pypi version.
I removed the package from ubuntu in dist-packages and installed it in site-packages:
sudo apt-get purge python3-pymysql
sudo pip3 install --upgrade pymysql
You don't (really) want to do this, especially on a server. Alternatively you use a virtualenv:
virtualenv -p python3.4 venv3
source venv3/bin/activate
pip3 install --update pip
pip3 install --update pymysql
You need to learn a bit more about virtualenv's to use them, but the idea is that you have a complete and separate python3.4 environment installed in venv3 which you activate with source venv3/bin/activate. This way it is harder to break your ubuntu (which heavily depends on a working python). Moreover, you can install different versions of packages in different virtualenv's. Also, when debugging, you can add print-statements to python-code in the site-packages folder of the virtualenv, without risking breaking your system severely. This is great, as many good python libraries have poor error reporting.

Faced this error with Python:3.6, PyMySQL3:0.5.
solution: change 'password' parameter to 'passwd'
Below is the parameters list with default values
host="localhost", user=None, passwd="", db=None, port=3306, unix_socket=None, charset='', sql_mode=None, read_default_file=None, conv=decoders, use_unicode=None, client_flag=0, cursorclass=Cursor, init_command=None, connect_timeout=None, ssl=None, read_default_group=None, compress=None, named_pipe=None

stratum-mining-proxy error - Can't decode message

I'm attempting to run stratum-mining-proxy with minerd. Proxy starts and runs with the following command:
python ./mining_proxy.py -o ltc-stratum.kattare.com -p 3333 -pa scrypt
Proxy starts fine. Run Minerd (U/P removed):
minerd -a scrypt -r 1 -s 6 -o http://127.0.0.1:3333 -O USERNAME.1:PASSWORD
Following errors are received. This one from the proxy:
2013-07-18 01:33:59,981 ERROR protocol protocol.dataReceived # Processing of message failed
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/stratum-0.2.12-py2.7.egg/stratum/protocol.py", line 185, in dataReceived
self.lineReceived(line, request_counter)
File "/usr/local/lib/python2.7/dist-packages/stratum-0.2.12-py2.7.egg/stratum/protocol.py", line 216, in lineReceived
raise custom_exceptions.ProtocolException("Cannot decode message '%s'" % line)
'rotocolException: Cannot decode message 'POST / HTTP/1.1
And this from minerd. What am I doing wrong? Any help is appreciated!
[2013-07-18 01:33:59] HTTP request failed: Empty reply from server
[2013-07-18 01:33:59] json_rpc_call failed, retry after 30 seconds

I am a little curious, I don't know as a fact but I was under the impression that the mining proxy was for BTC not LTC.
But anyways I believe I got a similar message when I first installed it as well. To fix, or rather to actually get it running I had to use the Git installation method instead of installing manually.
Installation on Linux using Git
This is advanced option for experienced users, but give you the easiest way for updating the proxy.
1.git clone git://github.com/slush0/stratum-mining-proxy.git
2.cd stratum-mining-proxy
3.sudo apt-get install python-dev # Development package of Python are necessary
4.sudo python distribute_setup.py # This will upgrade setuptools package
5.sudo python setup.py develop # This will install required dependencies (namely Twisted and Stratum libraries), but don't install the package into the system.
6.You can start the proxy by typing "./mining_proxy.py" in the terminal window. Using default settings, proxy connects to Slush's pool interface.
7.If you want to connect to another pool or change other proxy settings, type "./mining_proxy.py --help".
8.If you want to update the proxy, type "git pull" in the package directory.