I hope this doesn't cross into superuser territory.
So I have an embedded linux, where system processes are naturally quite stripped. I'm not quite sure which system process monitors to physical layer and starts a dhcp client when network cable is plugged in, but i made one myself.
¨
The problem is, that if i have a python script, using http connections, running before i have an IP address, it will never get a connection. Even after i have a valid IP, the python still has
"Temporary error in name resolution"
So how can I get the python to realize the new connection available, without restarting the script?
Alternatively , am I missing some normal procedure Linux runs normally at network cable connect.
The dhcp client I am using is udhcpc and python version is 2.6. Using httplib for connections.
Sounds like httplib caches /etc/resolv.conf which is updated after your DHCP client obtains an IP.
You might consider writing a wrapper script that waits until an IP address is obtained, then calls your python script.
Alternatively, you could try to, within your python script, not open any socket connections until you have acquired an IP address and /etc/resolv.conf updated.
Edit
httplib uses socket.create_connection() which, after some searching, I found does cache /etc/resolv.conf. (Well, it seems that the embedded version of libc is actually doing the caching).
You could try SugarLabs' solution, although I can't speak to it's effectiveness.
it might be that one of the python modules is caching the state of the network, and so it isn't using the latest settings. Try and reload all the network related modules. A simple example of this is:
import sys, socket, urllib
for i in [sys, socket, urllib]:
reload(i)
if that doesn't work look around in sys.modules to see what else got imported, and try more modules there. A more extreme version of the reloading would be the code below, that you could try as a last ditch effort.
code
import sys
print 'reloading %s modules ' % len(sys.modules)
for name, module in sys.modules.items():
try:
reload(module)
except:
print 'failed to import %s' % name
output
reloading 42 modules
failed to import __main__
failed to import encodings.encodings
failed to import encodings.codecs
failed to import lazr
failed to import encodings.__builtin__
After alot more research, the glibc problem jedwards suggested, seemed to be the problem. I did not find a solution, but made workaround for my usecase.
Considering I only use one URL, I added my own "resolv.file" .
A small daemon gets the IP address of the URL when PHY reports cable connected. This IP is saved to "my own resolv.conf". From this file the python script retrieves the IP to use for posts.
Not really a good solution, but a solution.
Related
from os import system
system("ping www.twitter.com")
system("ping www.yahoo.com")
system("ping www.facebook.com")
I am in China, and Twitter and Facebook are banned here. I can open them in the browser using Clash for Windows software.
I have to download tweets from Twitter. So I need to ping the websites using Python to get tweets. I cannot ping the websites though.
How do I make my Python code use the Clash for Windows.
Output of the above code:
Pinging www.twitter.com [108.160.169.186] with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Ping statistics for 108.160.169.186:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),
Pinging new-fp-shed.wg1.b.yahoo.com [180.222.102.201] with 32 bytes of data:
Reply from 180.222.102.201: bytes=32 time=258ms TTL=42
Reply from 180.222.102.201: bytes=32 time=229ms TTL=42
Reply from 180.222.102.201: bytes=32 time=230ms TTL=42
Request timed out.
Ping statistics for 180.222.102.201:
Packets: Sent = 4, Received = 3, Lost = 1 (25% loss),
Approximate round trip times in milli-seconds:
Minimum = 229ms, Maximum = 258ms, Average = 239ms
Pinging www.facebook.com [69.63.184.14] with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Ping statistics for 69.63.184.14:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),
OS: Windows 10 (updated to latest edition). Using PyCharm as my IDE.
You said in comment that you are using Clash, however Clash is not a VPN:
Clash - A rule-based tunnel in Go.
Features:
Local HTTP/HTTPS/SOCKS server with authentication support
VMess, Shadowsocks, Trojan, Snell protocol support for remote connections
Built-in DNS server that aims to minimize DNS pollution attack impact, supports DoH/DoT upstream and fake IP.
Rules based off domains, GEOIP, IP CIDR or ports to forward packets to different nodes
Remote groups allow users to implement powerful rules. Supports automatic fallback, load balancing or auto select node based off
latency
Remote providers, allowing users to get node lists remotely instead of hardcoding in config
Netfilter TCP redirecting. Deploy Clash on your Internet gateway with iptables.
Comprehensive HTTP RESTful API controller
source: https://github.com/Dreamacro/clash
I'm not sure exactly how it works, my current understanding is that it allows you to use proxies.
Although it also has TUN mode as a "Premium Feature" (not sure what does it mean, I see no option to buy "Premium") which may work similarly to VPN, but I'm not sure:
Premium Features:
TUN mode on macOS, Linux and Windows. Doc
Match your tunnel by Script
Rule Provider
Clash for Windows which you try to use, is GUI for Clash.
Documentation is available, but it is only in Chinese:
https://github.com/Fndroid/clash-win-docs-new
screenshot:
I tried to use it, but I don't know how to configure it. It didn't work at all.
I see many possible solutions:
You can switch to proper VPN. Paid VPNs are more recommended than free ones. Cost is typically 3-10$/month depending on offer. Alternatively you can try to setup OpenVPN on your own VPS, it may be a bit cheaper, but not necessarily.
You can configure your script to use proxy. Libraries like requests support it: Proxies with Python 'Requests' module
You can try to read Clash for Windows documentation to see if it is possible what you try to achieve. Maybe it is enough to turn on System Proxy as visible on screenshot above?
You can try to configure Clash in TUN mode. In my opinion it may be more difficult than solutions 1 and 2. If you prefer this way, I suggest to read Clash documentation thoroughly: https://github.com/Dreamacro/clash/wiki/premium-core-features#tun-device
If you can afford it, I recommend solution 1 in most use-cases. I understand that you prefer free solution, however servers are not free, somebody needs to incur costs of running them (hardware, electricity etc.)
There is a variety of Python libraries that can help you. openpyn is one of them. Firstly, call this command in your terminal for setup:
sudo openpyn --init
After which all your Internet traffic can be redirected to a VPN server using the following command:
openpyn us
The command above is the default, to transfer all the traffic to the US. Other locations can be chosen, see the link above for more info.
As soon as your traffic is redirected, you are free to see banned sites as you did:
from os import system
system("ping www.twitter.com")
system("ping www.facebook.com")
When you have VPN running and active then it should redirect all traffic via VPN server. If it is not redirecting all traffic, then maybe it was configured to redirect only webrowser traffic. Also it is possible that you aren't using true VPN, but a proxy.
Please share which VPN are you using, it will help us to help you.
You can just start VPN manually and then try executing your Python code.
Alternatively you can control when your VPN is running from Python. It requires different library for each VPN provider. Please tell us which VPN do you use.
This is working example for NordVPN using nordvpn_switcher package:
import time
from nordvpn_switcher import initialize_VPN,rotate_VPN,terminate_VPN
initialize_VPN(save=1,area_input=['complete rotation'])
for i in range(1):
rotate_VPN()
from os import system
system("ping www.twitter.com")
system("ping www.yahoo.com")
system("ping www.facebook.com")
print('\nDo whatever you want here (e.g. pinging). Pausing for 10 seconds...\n')
time.sleep(10)
terminate_VPN()
Tor uses SOCKS 5 proxy server to give anonymity. In case you're using Tor, there are two ways to use SOCKS 5 proxy, one is to configure it at application level(browser, python code etc..), another to configure at Network Interface level. If you're using Tor, simply follow this answer, socks server is running at localhost:9050 by default - How to make python Requests work via socks proxy
Since you haven't done anything and it's already working, I guess you're using tunnel-based VPN. In this case, it should work automatically. In your case, ping could be blocked by the VPN provider.
Ping uses ICMP protocol while HTTP/HTTPS uses TCP protocol in the Transport layer (in OSI model). They are very different things, and one working doesn't guarantee other working. In many cases, ping is blocked, by the server or other middlewares that don't support ICMP protocols. Most cloud providers don't support ICMP protocols in their networking components, for ex. Azure doesn't, for their Load Balancers. So, instead of trying ping, you should try the real http request. Following is sample code
import requests
r = requests.get("www.google.com")
You can use this code:
from ping3 import ping
p=ping("example.com")
print(p)
Here is the solution:
go to https://www.wintun.net and download the latest release, copy the right wintun.dll into Clash home directory
restart Clash for windows
open clash dashboard, switch TUN Mode on
https://github.com/Dreamacro/clash/wiki/premium-core-features#windows
Here is an update for people facing the same problem.
There is an option in settings>System Proxy>Specify Protocol in Clash for Windows , I turned it on.
Before i was not able to run pip commands, after turning this on, i can do that. I hope it will be useful for someone. (This is not exact answer to the question but indeed it is useful for someone, i am sure)
Let's try to isolate this problem if it's Python or network-related.
Does it work if you run ping in the shell directly?
ping twitter.com
It's possible that your VPN has a setting that blocks Pings from the internet. It should be one of the configurations in the Firewall.
I have just started using ngrok, and while using the standard procedure, I can start the tunnel using ./ngrok tcp 22 and see that tunnel open in my dashboard,
But I would like to use pyngrok, and here when I use:
from pyngrok.conf import PyngrokConfig
from pyngrok import ngrok
ngrok.set_auth_token("<NGROK_AUTH_TOKEN>")
pyngrok_config = PyngrokConfig(config_path="/opt/ngrok/ngrok.yml")
ngrok.get_tunnels(pyngrok_config=pyngrok_config)
ssh_url = ngrok.connect()
It connects and generates a tunnel, but I can't see anything open in the dashboard, why?
Maybe because the python script executes and generates URL and then stops and comes out of it, but then how to make it keep running, or how to even start a tunnel using python or even API ? Please suggest the correct script, using python or API?
The thread with the ngrok tunnel will terminate as soon as the Python process terminates. So you are correct, the reason this is happening is because your script is not long lived. The easiest way to accomplish this is by following the example in the documentation.
Another issue is how you're setting the authtoken. Since you're not using the default config_path, you need to set this before setting the authtoken so it gets updated in the correct file (you'd also need to pass it to connect()). There are a couple ways to do this, but the easiest way from the docs is to just update the default config (since that's what will be used if you don't pass a pyngrok_config to any future method calls).
I also see that you're response variable is ssh_url, so you probably want to start a TCP tunnel to a port other than 80 (the default)—perhaps you've configured this in your ngrok.yml, but if not, I've updated the call to connect() to ensure this is the type of tunnel started for you and in case others try to use this same code snippet.
Full disclosure, I am the developer of pyngrok. Here is your code snippet updated with my changes.
import os, time
from pyngrok.conf import PyngrokConfig
from pyngrok import ngrok, conf
conf.get_default().config_path = "/opt/ngrok/ngrok.yml"
ngrok.set_auth_token(os.environ.get("NGROK_AUTH_TOKEN"))
ssh_tunnel = ngrok.connect(22, "tcp")
ngrok_process = ngrok.get_ngrok_process()
try:
# Block until CTRL-C or some other terminating event
ngrok_process.proc.wait()
except KeyboardInterrupt:
print(" Shutting down server.")
ngrok.kill()
I have a troublesome problem socket.error error: [Errno 10048]: Address already in use. Only one usage of each socket address (protocol/IP address/port) is normally permitted during automated tests using Selenium with Python. The problem is so interesting that it runs on one machine (Linux) works correctly, but on another machine (WindowsXP) generates this error.
I would add that the problem arose after the reinstallation of the system and set up all over again - with the previous configuration everything worked properly.
Is there maybe something I forgot? Has anyone come up with such a problem before?
Does anyone have an idea of how to deal with this problem?
The current configuration / libraries:
python 2.7, numpy, selenium.py
If you open/close the socket multiple times, it could be in the TIME_WAIT state. This would explain why it acts differently on separate platforms (different TIME_WAIT settings and TCP stack). If you're controlling the socket object, you can set SO_REUSEADDR before binding to fix the problem.
For example:
sock = socket.socket()
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, server.getsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR) | 1)
You can run netstat -b from the command prompt to give you a list of open sockets with the state and owning process.
I found the answer in the post below:
Python urllib2. URLError: <urlopen error [Errno 10048] Only one usage of each socket address (protocol/network address/port) is normally permitted>
It turned out that this problem is limitation of Windows
There are several possibilities. If none of your tests can listen on some port (you don't say what port) then perhaps your Windows machine is running something on a port that you previously had open; this new service may have appeared during the reinstall. If, on the other hand, it's only a problem for some tests, or it's a little sporadic, then it may be either a programming issue (forgetting to close a socket in an early test which interferes with a later one) or a timing issue (the earlier test's socket isn't quite through closing before the new one tries to open up). Obviously there are different ways to address each of these problems, but I don't think we can help more than this without more details.
Maybe there is a software on your Windows that already use port 4444, can you try set Selenium to another port and try again?
I want to write a Python script that will check the users local network for other instances of the script currently running.
For the purposes of this question, let's say that I'm writing an application that runs solely via the command line, and will just update the screen when another instance of the application is "found" on the local network. Sample output below:
$ python question.py
Thanks for running ThisApp! You are 192.168.1.101.
Found 192.168.1.102 running this application.
Found 192.168.1.104 running this application.
What libraries/projects exist to help facilitate something like this?
One of the ways to do this would be the Application under question is broadcasting UDP packets and your application is receiving that from different nodes and then displaying it. Twisted Networking Framework provides facilities for doing such a job. The documentation provides some simple examples too.
Well, you could write something using the socket module. You would have to have two programs though, a server on the users local computer, and then a client program that would interface with the server. The server would also use the select module to listen for multiple connections. You would then have a client program that sends something to the server when it is run, or whenever you want it to. The server could then print out which connections it is maintaining, including the details such as IP address.
This is documented extremely well at this link, more so than you need but it will explain it to you as it did to me. http://ilab.cs.byu.edu/python/
You can try broadcast UDP, I found some example here: http://vizible.wordpress.com/2009/01/31/python-broadcast-udp/
You can have a server-based solution: a central server where clients register themselves, and query for other clients being registered. A server framework like Twisted can help here.
In a peer-to-peer setting, push technologies like UDP broadcasts can be used, where each client is putting out a heartbeat packet ever so often on the network, for others to receive. Basic modules like socket would help with that.
Alternatively, you could go for a pull approach, where the interesting peer would need to discover the others actively. This is probably the least straight-forward. For one, you need to scan the network, i.e. find out which IPs belong to the local network and go through them. Then you would need to contact each IP in turn. If your program opens a TCP port, you could try to connect to this and find out your program is running there. If you want your program to be completely ignorant of these queries, you might need to open an ssh connection to the remote IP and scan the process list for your program. All this might involve various modules and libraries. One you might want to look at is execnet.
Problem: to get the command working here. My domain is http://cs.edu.com/user/share_dir, but I cannot get the command working by typing it to a browser:
http://cs.edu.com/user/share_dir:8000
Question: How can I get the command working?
Your URL is incorrect. The port number should be specified after the domain name:
http://cs.edu.com:8000/
Some other things you should keep in mind:
If this is a shared host, port 8000 might already be in use by someone else
The host might not be accessible from 'outside' of the network, due to firewall restrictions on non-standard ports
The system you see internally could map to a different system outside, so the domain/hostname could be different from what you expect.