I would like to get started with working with python and PSQL databases.
The script I have in mind will (at least not at the beginning) run on hosts with an installed PSQL Database, but I want the script to connect remotely to the database.
In fact (for the start):
user hosts: run the script (read xls, convert, manipulate, etc) and write into remote DB
DB Host: this will host the db and gets connections from the user hosts and the "Sync Host"
Sync Host: a cloud service which will connect to the db server to read the databases and do some "magic" with it.
from what I have read, the best python module for PSQL connection is psycopg, but this seems to require an installed PSQL Database, which is something I do not have (and don't want to install) on the user hosts.
At a later stage I will remove the "user hosts" and provide a webinterface for uploading the xls and do the conversion, etc on the db host, but for the beginning I wanted to start as mentioned above.
My questions:
Is my thinking totally wrong? should I start with a central approach (webinterface, etc) right awa
If not, how can I get a PSQL connection method implemented in python without installing a PSQL Database?
All User hosts are Mac OS X, so Python is already installed.
thanks a lot in advance
Andre
Related
I have a problem that I've been trying to solve for a very long time and it seems like I'm missing something very basic.
I use a Linux server with Anaconda, Oracle client, Pycharm and jupyter-notebook installed.
I use python scripts in which I write and read data to Oracle DB, and I use the Cx_oracle extension.
The server has several users with a personal username for each and each of them has sudo privileges.
I performed all the installations on the server with sudo privileges.
When I try to connect from the server to Oracle DB I connect properly.
When I connect to Python on the server itself, I connect properly to Oracle DB.
When I connect using Pycharm and I define ORACLE_HOME=/OracleTools/19.0.0/ in the environment variables, I connect properly to the Oracle DB.
My problem starts when I want to use jupyter-notebook
When I try to connect to the DB I get the error -
DatabaseError: Error while trying to retrieve text for error ORA-12715
I noticed when I execute os.environ I see that it is defined for me:
ORACLE_HOME: /OracleTools/19.0.0/bin
and should be
/OracleTools/19.0.0
So I changed using command os.environ['ORACLE_HOME'] = '/OracleTools/19.0.0'
Then I get an error:
DatabaseError: ORA-12715: invalid character set specified
And of course this change is not permanently saved ...
If on the server itself I execute the env command both in the private user and in sudo I see ORACLE_HOME: /OracleTools/19.0.0 and not ORACLE_HOME: /OracleTools/19.0.0/bin
My questions:
Where does the data I get in the os.environ command come from?
How can I edit them permanently ?
Is this even the problem I'm having or should I check something else?
I manage to import cx_oracle, which means that there is no problem of expansion
Thanks!
I need to use a table of a database on a university server for test data for a project.
I'm pretty new to databases and MySQL. My professor has send me username and password for the server. And an extra username&password for the MySQL server.
It took me a while but in the end I was able to connect to the server over ssh and then managed to navigate to $cd / $cd usr/bin/MySQL then logged in and found the data/sentences in a table in one of the databases.
Now there is the question: How do I get the data on my computer? I thought about a python script. But I cannot write a script what is logging in on a different server and then navigates to the MySQL folder to log in there to copy somehow the sentences in the table to a txt file I can use?
You could use the mysqldump program (if it is available on your machine) to get a dump of the table you need.
if you would like to have the dump as SQL so you could insert it on your own database, you would call it with
mysqldump -u "USERNAME" -p "PASSWORD" -h "HOSTNAME" "DATABASE" "TABLE" > dump.sql
Omitting the password is strongly preferred so your password isn't saved in the shell history. You'll be prompted for it during execution.
and you would get a file with all SQL-Queries to recreate this table.
You can as well export data from that table as CSV or XML, depending on the parameters used at call time.
if the database server would only be accessible from the second server you mentioned, but you can access this one via ssh, then you could try calling
ssh USER#HOST 'mysqldump --version'
which hopefully presents you with an output similar to this:
mysqldump Ver xx.yy Distrib x.y.z, for debian-linux-gnu (x86_64)
if so, you would know there is this program installed and you could use the following line
ssh USER#HOST 'mysqldump -u "USERNAME" -p "PASSWORD" -h "HOSTNAME" "DATABASE" "TABLE"' | cat > dump.sql
which should present you the dump on your local machine.
You might not need to ssh into the remote server; depending how their server and database are set up you may be able to connect a mysql client on your local machine to the database server. While there are security advantages to limiting where connections are permitted from, accessing a database at localhost is actually just a special case.
You might not even need a python script. You can export directly from mysql to a text file, or your client may have a feature to copy data directly from the remote server into a local database.
I would guess that something like this would work for you, although getting the output into the format you want can be tricky:
mysql -h "host address" -u "username" -p -e "SELECT * FROM `table`" > localFile.txt
If you wanted to do it with a python script running on the server as you're describing, you'll want to use the ssh credentials to do FTP over SSH to get the files back and forth. Your FTP client will certainly support that.
I have a mydb.fdb file, how do I can load it into docker container and then connect to it from python. I do the following in my docker-compose:
version: '2'
services:
firebird:
image: jacobalberty/firebird
environment:
ISC_PASSWORD: pass
volumes:
- ./database:/databases
Then I do:
docker exec -it <container-id> bin/bash
And I see my .fdb file inside /databases folder in container, but when I do commands inside container:
cd /usr/local/firebird/bin
./isql
SQL> CONNECT "/databases/mydb.FDB" user sysdba password masterkey;
I received:
Use of database at location /databases/mydb.FDB is not allowed by server configuration
And also I don't understand how to connect to this db via fdb python module. I do:
import fdb
con = fdb.connect(
host='0.0.0.0',
port='3050',
database='mydb.FDB',
user='sysdba',
password='masterkey')
And received:
raise Exception("The location of Firebird Client Library could not be
determined.")
You have two different problems in your question, in the future, please make sure to ask them as separate questions.
As to your first problem, the setup of the Firebird docker image by default expects databases in the location /firebird/data, and explicitly configures Firebird to restrict access to only that location. If you want to use a different location, then you must set the environment variable DBPATH to the correct path. See also issue 12, the Dockerfile and the build.sh of the image. This option doesn't seem to be documented; I have left a comment on that ticket.
As to your second problem, it has two parts. One, you can't use 0.0.0.0 to connect, that is just a shorthand for "all ip addresses of this host", it can only be used when listening for connections, not when connecting. You will need to use 127.0.0.1 or localhost when connecting from the machine running docker.
On top of that, the error suggests that you have no Firebird native client installed. The FDB driver requires fbclient.dll (Windows) or libfbclients.so (Linux). You either need to install the Firebird native client, or switch to pyfirebirdsql, which is a Firebird driver in pure Python (it implements the Firebird protocol in Python).
I'm creating a Python 3 spider that scrapes Tor hidden services for useful data. I'm storing this data in a PostgreSQL database using the psycopg2 library. Currently, the spider script and the database are hosted on the same network, so they have no trouble communicating. However, I plan to migrate the database to a remote server on a VPS so that I can have a team of users running the spider script from a number of remote locations, all contributing to the same database. For example, I could be running the script at my house, my friend could run it from his VPS, and my professor could run the script from a few different systems in the lab at the university, and all of these individual systems could synchronize with the PostgreSQL server runnning on my remote VPS.
This would be easy enough if I simply opened the database VPS to accept connections from anywhere, making the database public. However, I do not want to do this, for security reasons. I know I could tunnel the connection through SSH, but that would require giving each person a username and password that would grant them access to the server itself. I don't wish to do this. I'd prefer simply giving them access to the database without granting access to a shell account.
I'd prefer to limit connections to the local system 127.0.0.1 and create a Tor hidden service .onion address for the database, so that my remote spider clients can connect to the database .onion through Tor.
The problem is, I don't know how to connect to a remote database through a proxy using psycopg2. I can connect to remote databases, but I don't see any option for connecting through a proxy.
Does anyone know how this might be done?
This would be easy enough if I simply opened the database VPS to accept connections from anywhere
Here lies your issue. Just simply lock down your VPS using fail2ban and ufw. Create a ufw role to only allow connection to your Postgres port from the IP address you want to give access from to that VPS ip address.
This way, you don't open your Postgres port to anyone (from *) but only to a specific other server or servers that you control. This is how you do it. Don't run an onion service to connect Postgres content because that will only complicate things and slow down the reads to your Postgres database that I am assuming an API will be consuming eventually to get to the "useful data" you will be scraping.
I hope that at least points you in the right direction. Your question was pretty general, so I am keeping my answer along the same vein.
I have a MariaDB 5.5.57-1ubuntu0.14.04.1 installed on Ubuntu Linux 14.04 and configured with the following to authenticate MySQL users using our OpenLDAP infrastructure via PAM:
# cat /etc/mysql/conf.d/ldap.cnf
[mysqld]
pam-use-cleartext-plugin
plugin-load = auth_pam.so
# cat /etc/mysql/conf.d/ssl.cnf
[mysqld]
ssl-cipher = DHE-RSA-AES256-SHA
ssl-ca=/etc/mysql/ssl/ca-cert.pem
ssl-cert=/etc/mysql/ssl/server-cert.pem
ssl-key=/etc/mysql/ssl/server-key.pem
I've created my user in mysql like this:
MariaDB [(none)]> CREATE user 'ldap1'#'%' IDENTIFIED VIA pam;
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]> GRANT ALL PRIVILEGES ON test.* TO 'ldap1'#'%' IDENTIFIED VIA pam REQUIRE SSL;
I can authenticate from a remote Linux host doing this:
mysql -u ldap1 -p -h my.server.net --ssl --enable-cleartext-plugin
When I connect from MySQL Workbench on Windows 7 I can tick 'Enable Cleartext Authentication Plugin' under the Advanced tab (Require SSL of course) and authenticate as well there too.
When I run my Python script, I get the error message below. Anyone know a fix or is there a DLL I need to install?
NOTE:The code I'm using was sent to me from an outside developer and not so sure I can share the code so just posting the error output. If this is not a common problem with a known workaround (It's been difficult finding all the right pieces to do this from both Linux and Windows platforms) I can certainly email said developer and ask about posting watered down version of the connect routine. The code does import MySQLdb to establish the DB connection if it matters.