I am trying to build a data processing pipline. In a nutshell the pipline includes the follwing tasks: connect to a remote SQL db, extract data, process it and output files. I've built all these steps in python. Since the db is on a server I can only run the pipline only from a private server (with a static IP address) - for security reasons. So I ran all my py scripts from my own server (ubuntu). Now, to stitch the components of the pipilne I want to use apache-airflow. I followed a couple of instructions from different sources and failed to make it work. Everything seems to run but when I run airflow webserver the local host refuses the connection (for some reason). My question is rather general. I want to know if there is a good step by step instruction on how to install and use apache airflow specifically on remote server (ubuntu).
I have tried the following:
https://medium.com/#taufiq_ibrahim/apache-airflow-installation-on-ubuntu-ddc087482c14
https://vujade.co/install-apache-airflow-ubuntu-18-04/
https://towardsdatascience.com/getting-started-with-apache-airflow-df1aa77d7b1b
plus the same with docker.
Here's the log of the error I get: (connection refused)
Here's the the screen shot of when I run airflow webserver:
Related
I'm a student and I'm trying to write some sensor values into a MySQL database.
As IDE I'll be using Inteliji.
First off I started by installing the database Plug-in.
This was done successfully
Next I tried to connect to the data base (see figure below)
Figure of successful connection
Now The next thing I want to do is use a MySQL connector.
Therefore I've installed MySQL onto the r-PI and used following code to implement it.
import mysql.connector
print("Step 1")
cnx = mysql.connector.connect(user='a21ib2a01',
password='secret',
host='mysql.studev.groept.be',
database='a21ib2a01')
Print("Step 2")
When now I run my code the terminal will output:
Step1
For some reason I don't know; the connect function always times my program out with the next occurring errors:
mysql.connector.errors.InterfaceError: 2003: Can't connect to MySQL server on 'mysql.studev.groept.be:3306' (110 Connection timed out)
So does anyone know why my connection is successful but I can't connect to it?
Long story short what am I doing wrong and how do I fix this?
Thanks in advance!
Your timeout means the network on your rPi cannot reach -- cannot find a route to -- your MySQL host mysql.studev.groept.be.
If you do traceroute mysql.studev.groept.be in a shell in the rPi you may see what's wrong.
When in a shell on your rPi, can you ssh to any machine in your uni's network? If so, you might be able to use ssh port-forwarding to get a route to the database server.
Do you run intelliJ on the rPi directly, or on your laptop? If you run it on the laptop, it looks like the laptop can find a route to your server but the rPi cannot.
(If this were my project, I'd install a MySQL server on my laptop to reduce the network-engineering hassles of connecting through multiple hops involving a VPN.)
I am trying to connect to vm using run powershell on target machine task using classic editor to trigger a python code which needs to be downloaded from artifact on the target machine (here, vm).
I am getting an error " The SSL certificate contains a common name (CN) that does not match the hostname. "
I have tried to change settings in certificate snap-in but it still gave me same error.
Thank you #VenkateshDodda-MT & #srbrills Posting your suggestion as an answer to help other community members.
The server certificate on the TARGET system, not the host file or the build agent, is the problem.
The problem happens when you build an Azure VM without a DNS Name Label for your public IP and then add one afterwards (something like example.centralus.cloudapp.azure.com). It can also happen if the DNS name label is changed.
You need to make sure that, what address you want to use to connect to the machine. This is the target machine address.
On the target machine, open PowerShell as an administrator. Enter the following command.
New-SelfSignedCertificate -DnsName WhateverTargetMachineAddressYouNeed -CertStoreLocation Cert:\LocalMachine\My
Enter the following commands one at a time in PowerShell.
winrm delete winrm/config/listener?Address=*+Transport=HTTPS
Then:
winrm create winrm/config/listener?Address=*+Transport=HTTPS '#{Hostname="WhateverTargetMachineAddressYouNeed";CertificateThumbprint="TheThumbprintYouCopied";port="5986"}'
For more information please refer this SO THREAD & Configure WinRM to execute PowerShell Script on a remote Azure machine with ARM
I'm working on a simple Python program to query a Redshift cluster using psycopg2. When I run the code on my local machine it works as expected: it creates the connection, it runs the queries and I get the expected outcome. However, I loaded it on my EC2 instance because I want to schedule several runs a week and the execution fails with the following error:
psycopg2.OperationalError: could not connect to server: Connection timed out
Is the server running on host "xxxx" and accepting
TCP/IP connections on port 5439?
Considering that the code is working without problems on the local machine and the security settings should be the same as EC2, do you have any suggestions and/or workarounds?
Thanks a lot.
When I'm trying to connect to the mongodb from my python script I see this warning many times and error finally:
[Dec 13 11:58:56] WARNING torsocks[8133]: [connect] Connection to a local address are denied since it might be a TCP DNS query to a local DNS server. Rejecting it for safety reasons. (in tsocks_connect() at connect.c:177)
pymongo.errors.ServerSelectionTimeoutError: localhost:27017: [Errno 1] Operation not permitted
I use torify command for starting my command - torify python myscript.py. Without torify it works.
What I'm doing wrong? The same situation is on another machines to.
I found a solution for this case. I have to create a Hidden Service first. To do that I have to add this lines to my tor config file (/etc/tor/torrc):
HiddenServiceDir /tmp/tormongo
HiddenServicePort 27017 127.0.0.1:27017
After that I need to restart my tor service. If everything was done right the folder has to appear in /tmp. In this folder will be the new file hostname with the string like sgwqrpepus3lwcke.onion. This is the host what I can use instead of localhost in mongodb connection settings.
Or I can add a new ENV variable for this setting and set it every script start by using command like that:
export DB_HOST=$(cat /tmp/tormongo/hostname) && torify python /var/www/myproject/myscript.py
I am executing a Python script on my Linux server that uses pysftp to connect to another server in order to read files that are sitting in a directory of that remote server. When I run the script, it fails out while connecting to the remote server and creates a text file with the title: 'This service allows sftp connections only.'
This file is created inside my project directory. Below is the part of my code that is failing:
def sftp_get_file(sftp_host, sftp_username):
with pysftp.Connection(sftp_host, sftp_username) as sftp:
# transfer file from remote to local
sftp.get(remote_file, local_file)
Code is very simple and works when I've tested it using my local server as the remote server. When I tested it in the new environment by actually depending on SFTP, then it failed. Any suggestions? Is pysftp using SSH at some point when it should be using only SFTP?
Turns out the problem was due to me performing sftp.execute('ls') a couple lines down in the script. The server I was remoting onto only supported sftp commands and that command was forbidden.