Develop Reference

How to route internet traffic via `Clash for Windows` (Ping from Python code is not working)

from os import system
system("ping www.twitter.com")
system("ping www.yahoo.com")
system("ping www.facebook.com")
I am in China, and Twitter and Facebook are banned here. I can open them in the browser using Clash for Windows software.
I have to download tweets from Twitter. So I need to ping the websites using Python to get tweets. I cannot ping the websites though.
How do I make my Python code use the Clash for Windows.
Output of the above code:
Pinging www.twitter.com [108.160.169.186] with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Ping statistics for 108.160.169.186:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),
Pinging new-fp-shed.wg1.b.yahoo.com [180.222.102.201] with 32 bytes of data:
Reply from 180.222.102.201: bytes=32 time=258ms TTL=42
Reply from 180.222.102.201: bytes=32 time=229ms TTL=42
Reply from 180.222.102.201: bytes=32 time=230ms TTL=42
Request timed out.
Ping statistics for 180.222.102.201:
Packets: Sent = 4, Received = 3, Lost = 1 (25% loss),
Approximate round trip times in milli-seconds:
Minimum = 229ms, Maximum = 258ms, Average = 239ms
Pinging www.facebook.com [69.63.184.14] with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Ping statistics for 69.63.184.14:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),
OS: Windows 10 (updated to latest edition). Using PyCharm as my IDE.

You said in comment that you are using Clash, however Clash is not a VPN:
Clash - A rule-based tunnel in Go.
Features:
Local HTTP/HTTPS/SOCKS server with authentication support
VMess, Shadowsocks, Trojan, Snell protocol support for remote connections
Built-in DNS server that aims to minimize DNS pollution attack impact, supports DoH/DoT upstream and fake IP.
Rules based off domains, GEOIP, IP CIDR or ports to forward packets to different nodes
Remote groups allow users to implement powerful rules. Supports automatic fallback, load balancing or auto select node based off
latency
Remote providers, allowing users to get node lists remotely instead of hardcoding in config
Netfilter TCP redirecting. Deploy Clash on your Internet gateway with iptables.
Comprehensive HTTP RESTful API controller
source: https://github.com/Dreamacro/clash
I'm not sure exactly how it works, my current understanding is that it allows you to use proxies.
Although it also has TUN mode as a "Premium Feature" (not sure what does it mean, I see no option to buy "Premium") which may work similarly to VPN, but I'm not sure:
Premium Features:
TUN mode on macOS, Linux and Windows. Doc
Match your tunnel by Script
Rule Provider
Clash for Windows which you try to use, is GUI for Clash.
Documentation is available, but it is only in Chinese:
https://github.com/Fndroid/clash-win-docs-new
screenshot:
I tried to use it, but I don't know how to configure it. It didn't work at all.
I see many possible solutions:
You can switch to proper VPN. Paid VPNs are more recommended than free ones. Cost is typically 3-10$/month depending on offer. Alternatively you can try to setup OpenVPN on your own VPS, it may be a bit cheaper, but not necessarily.
You can configure your script to use proxy. Libraries like requests support it: Proxies with Python 'Requests' module
You can try to read Clash for Windows documentation to see if it is possible what you try to achieve. Maybe it is enough to turn on System Proxy as visible on screenshot above?
You can try to configure Clash in TUN mode. In my opinion it may be more difficult than solutions 1 and 2. If you prefer this way, I suggest to read Clash documentation thoroughly: https://github.com/Dreamacro/clash/wiki/premium-core-features#tun-device
If you can afford it, I recommend solution 1 in most use-cases. I understand that you prefer free solution, however servers are not free, somebody needs to incur costs of running them (hardware, electricity etc.)

There is a variety of Python libraries that can help you. openpyn is one of them. Firstly, call this command in your terminal for setup:
sudo openpyn --init
After which all your Internet traffic can be redirected to a VPN server using the following command:
openpyn us
The command above is the default, to transfer all the traffic to the US. Other locations can be chosen, see the link above for more info.
As soon as your traffic is redirected, you are free to see banned sites as you did:
from os import system
system("ping www.twitter.com")
system("ping www.facebook.com")

When you have VPN running and active then it should redirect all traffic via VPN server. If it is not redirecting all traffic, then maybe it was configured to redirect only webrowser traffic. Also it is possible that you aren't using true VPN, but a proxy.
Please share which VPN are you using, it will help us to help you.
You can just start VPN manually and then try executing your Python code.
Alternatively you can control when your VPN is running from Python. It requires different library for each VPN provider. Please tell us which VPN do you use.
This is working example for NordVPN using nordvpn_switcher package:
import time
from nordvpn_switcher import initialize_VPN,rotate_VPN,terminate_VPN
initialize_VPN(save=1,area_input=['complete rotation'])
for i in range(1):
rotate_VPN()
from os import system
system("ping www.twitter.com")
system("ping www.yahoo.com")
system("ping www.facebook.com")
print('\nDo whatever you want here (e.g. pinging). Pausing for 10 seconds...\n')
time.sleep(10)
terminate_VPN()

Tor uses SOCKS 5 proxy server to give anonymity. In case you're using Tor, there are two ways to use SOCKS 5 proxy, one is to configure it at application level(browser, python code etc..), another to configure at Network Interface level. If you're using Tor, simply follow this answer, socks server is running at localhost:9050 by default - How to make python Requests work via socks proxy
Since you haven't done anything and it's already working, I guess you're using tunnel-based VPN. In this case, it should work automatically. In your case, ping could be blocked by the VPN provider.
Ping uses ICMP protocol while HTTP/HTTPS uses TCP protocol in the Transport layer (in OSI model). They are very different things, and one working doesn't guarantee other working. In many cases, ping is blocked, by the server or other middlewares that don't support ICMP protocols. Most cloud providers don't support ICMP protocols in their networking components, for ex. Azure doesn't, for their Load Balancers. So, instead of trying ping, you should try the real http request. Following is sample code
import requests
r = requests.get("www.google.com")

You can use this code:
from ping3 import ping
p=ping("example.com")
print(p)

Here is the solution:
go to https://www.wintun.net and download the latest release, copy the right wintun.dll into Clash home directory
restart Clash for windows
open clash dashboard, switch TUN Mode on
https://github.com/Dreamacro/clash/wiki/premium-core-features#windows

Here is an update for people facing the same problem.
There is an option in settings>System Proxy>Specify Protocol in Clash for Windows , I turned it on.
Before i was not able to run pip commands, after turning this on, i can do that. I hope it will be useful for someone. (This is not exact answer to the question but indeed it is useful for someone, i am sure)

Let's try to isolate this problem if it's Python or network-related.
Does it work if you run ping in the shell directly?
ping twitter.com
It's possible that your VPN has a setting that blocks Pings from the internet. It should be one of the configurations in the Firewall.

Ignore a request in Flask

I want to not answer a request handled by Flask. I don't want to return any error code, data, or an answer at all.
What I am trying to accomplish by doing this is that there is an endpoint takes sensor data and do not return any information. The clients POST the data to this endpoint, but they do not wait for an answer and shutdown (I have no control over the clients.) So I'm seeing the following error: "[Errno 10053] An established connection was aborted by the software in your host machine". So I asked myself, why do I even respond to these requests.

I can think of two reasons to do something like this:
You have a "friend" that you want to prevent from accessing your site, or
You have the misguided notion that this will help prevent (D)DoS attacks.
When you say "ignore a request totally" you kind of actually can't do that, generally speaking. Unless you know the IP address that the traffic is coming from, and then you can instruct your OS, Network card, router, switch, load balancer, maybe even ISP to filter out the traffic coming from that IP.
Otherwise, you're kind of out of luck because of how the Internet works.
HTTP works over TCP*. Specifically the client process looks something like this:
Translate DNS (e.g. google.com) to IP address (e.g. 216.58.218.174)
open up a TCP connection to 216.58.218.174:80 (using google for the example)
send the HTTP header over to Google:
GET / HTTP/1.1
read the response
Once that TCP/IP connection has been created to your server, at the very least you're going to have to terminate the connection.
There's really no good way to do this from within Python itself, and certainly not within Flask.
As you've updated your answer, it turns out you really don't have to change anything, Flask is already handling the error behind the scenes. It may be routing the message to a specific logger that you might be able to handle if you really don't want to see the messages, but it's not really important.
The only thing you may want to do, if your return processing is expensive (like tying up the database with a several second long query) is look into streaming your response instead, which will fail much more cheaply.
*Mostly. Sure you can do it over UDP, but you probably aren't

Raw TCP Listen Socket on Cloud or Web Server

I have hardware that connects to raw TCP socket on any given IP and port combination. It then continually sends characters. The following piece of Python code may give you an idea of what the hardware does.
import socket
serverIP = '*server IP or domain*'
serverPort = 60000
Sock = socket(AF_INET, SOCK_STREAM)
Sock.connect((serverIP, serverPort))
while (1):
f = open ("send-data.txt","r")
while 1:
c = f.readline()
if not c:
break
Sock.send(c + '\n')
Sock.shutdown(0)
Sock.close()
When this code is run it exactly behaves like my hardware system. The send-data.txt file contains characters similar to what hardware sends.
I have written a socket server in Python using SocketServer library. It allows connections, receives character stream, and stores it into a local (newly created) file. Currently, I am running this code on my system, as localhost and it works. I would like to serve these files through a webpage.
I want to be able to do the same on remote server. As you can see, my hardware limits me to use only raw TCP sockets. From what I understand, I'll need low-level access to the server machine like IaaS. I tried pythonanywhere, but I guess they don't allow simple python sockets. Heroku also requires you to write a web app, and I don't know how to go about that or whether it'll work with my hardware.
What hosting/Cloud solution out there could act as above-mentioned socket server and also as HTTP server which would later serve these files and webpages.

If I understand your question correctly, you'd like to know which affordable hosting solution would allow you to communicate via arbitrary TCP sockets.
The answer is simple: Pretty much any VPS (Virtual Private Server) company or IaaS provider. Since you tagged your question with Amazon-EC2, yes they do too, but the learning curve to get your first instance running and the security groups (read: firewall rules, which live outside your VM) configured, is rather steep. That said, you do have a so-called "Free Tier" there for one year, which enables you to try out most of their services free-of-charge.
Other providers might be more suitable. (I'm not sure if it's allowed to suggest providers here, but you could for example look at Linode or Rackspace Cloud; they offer much less flexibility than EC2, but it's a whole lot easier to get started.)
As with any IaaS option, it would be beneficial to know Linux, networking and some security basics (at the very least) as you will be solely responsible for the things you create.
Talking about security...
If that piece of code you posted has a similarly rudimentary receiving end, you're setting yourself up for trouble as soon as it's out there in public, as the communication is done in plain text [*] and doesn't seem to require any kind of authentication. Anybody could probably telnet to the receiving end and just inject some lines of text?
(That's exactly why considerably sane PaaS providers often don't let you communicate over arbitrary ports and sockets :-) )
[*] I am guessing that, because you use readline. If any encryption was involved, you'd likely write/read in chunks of bytes.

Which ports will python use to send html request? with urllib or urllib2

I'm using webpy to make a small site. When I want to use OAuth, i find that the firewall will stop the http request to any site, I even can't use IE to browse the Internet.
So i asked the aministrator to open some ports for me, but i don't know which ports will be used by python or IE to send http request.
Thanks!

I assume you're talking about the remote ports. In that case, just tell the admin to open the standard web ports. Really, if your admin doesn't know how to make IE work through the firewall, he's hopeless. I suggest walking up to random people on the street and say "80 and 443" until someone looks up, then fire your admin and hire that guy; he can't be any worse.
If your admin does know what he's doing, and wants you to use an HTTP proxy instead of connecting directly, ask him to set up the proxy for you in IE, look at the settings he uses, then come back here and search for how to use HTTP proxies in Python (there are lots of answers on that), and ask for help if you get stuck.
If you're talking about the local ports, because you're got an insane firewall, they'll be picked at random from some large range. If you want to cover every common OS, you need all of 1024-65535 to be opened up, although if you only need to deal with a single platform, most use a smaller range than that, and if the machine won't be doing much but running your program, most have a way to restrict it to an even smaller range (e.g., as small as 255 ports on Windows). Google "ephemeral port" for details.
If you need to restrict your local port, the key is to call bind on your socket before calling connect. If you think you're talking about the local ports, you're probably wrong. Go ask your admin (or the new one you just hired) and make sure. But if you are…
If you're using urllib/urllib2, it does not have any way to do what you want, so you can't do that anymore. You can drop down to httplib, which lets you pass a source_address, a (host, port) tuple that it will use to bind the socket before connecting. It's not as simple as what you're using, but it's a lot easier than implementing HTTP from scratch.
You might also want to look at requests, which I know has its own native OAuth support, and probably has a way to specify a source address. (I haven't looked, but I usually find that whenever I want to know if requests can do X, it can, and in the most obvious way I think of…) The API for requests is generally similar to urllib2 when urllib2 is sane, simpler and cleaner when urllib2 is messy, so it's usually pretty easy to port things.
At any rate, however you do this, you will have to consider the fact that only one socket can be bound to the same local port at a time. So, what happens if two programs are running at once, and they both need to make an outbound connections, and your admin has only given you one port? One of them will fail. Is that acceptable?
If not, what you really need to do is open a range of ports, and write code that does a random.shuffle on the range, then tries to bind them until one succeeds. Which means you'll need an HTTP library that lets you feed in a socket factory or a pre-opened socket instead of just specifying a port, which most of them do not, which probably means you'll be hacking up a copy of the httplib source.
If all else fails, you can always set up a local proxy that binds to whatever source port (or port range) you want when proxying outward. Then you can just use your favorite high-level library, as-is, and connect to the local proxy, and there's no way the firewall can tell what's going on behind the proxy.
As you can see, this is not easy. That's mainly because you very actually rarely this.

Generally when a program wants to use a port but doesn't care which number it has, it uses an "ephemeral port." This is typical for client applications, where the remote port is fixed (by the server), but the local port doesn't make any difference.
Often a firewall will allow outgoing connections from any port, simply blocking incoming connections to unknown ports, on the theory that internal machines making outgoing requests should be allowed to decide what is proper, and that bad actors are all on the "public" side.
You may find that your administrator requires you to use an "HTTP proxy." If so, here are the instructions for Ruby which I imagine you can port to Python: Ruby and Rails - oauth and http proxy

Decentralized networking in Python - How?

I want to write a Python script that will check the users local network for other instances of the script currently running.
For the purposes of this question, let's say that I'm writing an application that runs solely via the command line, and will just update the screen when another instance of the application is "found" on the local network. Sample output below:
$ python question.py
Thanks for running ThisApp! You are 192.168.1.101.
Found 192.168.1.102 running this application.
Found 192.168.1.104 running this application.
What libraries/projects exist to help facilitate something like this?

One of the ways to do this would be the Application under question is broadcasting UDP packets and your application is receiving that from different nodes and then displaying it. Twisted Networking Framework provides facilities for doing such a job. The documentation provides some simple examples too.

Well, you could write something using the socket module. You would have to have two programs though, a server on the users local computer, and then a client program that would interface with the server. The server would also use the select module to listen for multiple connections. You would then have a client program that sends something to the server when it is run, or whenever you want it to. The server could then print out which connections it is maintaining, including the details such as IP address.
This is documented extremely well at this link, more so than you need but it will explain it to you as it did to me. http://ilab.cs.byu.edu/python/

You can try broadcast UDP, I found some example here: http://vizible.wordpress.com/2009/01/31/python-broadcast-udp/

You can have a server-based solution: a central server where clients register themselves, and query for other clients being registered. A server framework like Twisted can help here.
In a peer-to-peer setting, push technologies like UDP broadcasts can be used, where each client is putting out a heartbeat packet ever so often on the network, for others to receive. Basic modules like socket would help with that.
Alternatively, you could go for a pull approach, where the interesting peer would need to discover the others actively. This is probably the least straight-forward. For one, you need to scan the network, i.e. find out which IPs belong to the local network and go through them. Then you would need to contact each IP in turn. If your program opens a TCP port, you could try to connect to this and find out your program is running there. If you want your program to be completely ignorant of these queries, you might need to open an ssh connection to the remote IP and scan the process list for your program. All this might involve various modules and libraries. One you might want to look at is execnet.