Develop Reference

Scrapy-splash no results at "Server listening on http://0.0.0.0:8050"

I'm on Windows 10 Home Edition using Docker tool box (my BIOS configuration doesn't allow me to install Linux)
When I run
docker run -it -p 8050:8050 --rm scrapinghub/splash
I receive the log message Server listening on http://0.0.0.0:8050, as it should be in normal cases. But when I go to the address at my Chrome, the only thing that returns is ERR_ADDRESS_INVALID, not possible to access the page.
I have tried the following solutions to this problem:
Changing the SPLASH_URL on my settings.py according to my Docker IP adress;
Adding new ports to my docker run, such as 5023:5023 and 8051:8051.
I imagine that this is the kind of problem which I need to investigate its caused by myself. But I have tried everything that I could find on the Internet.

I'm gonna write the right answer, thanks to the users Selcuk, for explaning some technical terms, and David Maze, for giving me the final solution.
Since we are using the Docker Toolbox, we need to use our docker-machine ip.
Go to your terminal and type:
docker-machine ip
Now that you have the right ip, go to your settings.py and set
SPLASH_URL = http://[docker-machine ip]:8050
And you're good to go!

You don't need to change any settings.
If you launch docker terminal, it shows the correct ip address at the very first, right after the whale drawing.
## .
## ## ## ==
## ## ## ## ## ===
/"""""""""""""""""\___/ ===
~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ / ===- ~~~
\______ o __/
\ \ __/
\____\_______/
docker is configured to use the default machine with IP 192.168.99.102
For help getting started, check out the docs at https://docs.docker.com
So in my case the ip was 192.168.99.102.
If I visit 192.168.99.102:8050 on chrome, splash works.

Old topic but it has been helpfull for me. I add another way to get the IP address, without docker-machine.
With the container ID you can get the IP address with:
sudo docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' CONTAINER ID
To get the container ID use : sudo docker ps
You can also use the container name instead of the ID

Unable to access locust on web-UI using docker

I have installed locust inside docker, i mapped docker port as well but when I run locust command I get below error, I am able to run locust on command line but not on Web-ui, may be i may miss-understood which host or port should use while accessing.
COMMAND:
locust -f locustfile.py
Error:
oserror errno 97 address family not supported by protocol
Command:
locust -f locustfile.py --web-host=localhost
Result:
[2019-12-18 11:24:47,101] ABZ-218/INFO/locust.main: Starting web
monitor at http://localhost:8089
[2019-12-18 11:24:47,102] ABZ-218/INFO/locust.main: Starting Locust 0.13.2
but not able to access it on browser.
I have mapped port 0.0.0.0:8089->80
so which command should i use while hitting locust and which command should i use while accessing it from chrome browser?

--web-host=localhost is not needed, by default locust will listen on all interfaces. Try removing it and seeing if that helps.

You can find the IP Address of where your application is running and pass that as an argument to --host e.g. --host http://127.0.0.1:8000.

Python process blocking socket [Errno 98]

I'm running two raspis connected to each other. They both run a program on startup (which starts fine).
However, I keep getting a error: [Errno 98] Address already in use error in the background, and the communication between both Pis doesn't even start.
I went for tracing down the python processes running via ps -fA ¦ grep python and tadaaa, I can see, that another process starts trying to use the same address.
The output of ps looks like:
root 923 917 0 12:25 pts/1 00:00:00 sudo python /home/pi/Documents/3_multithread.py
root 927 923 10 12:25 pts/1 00:00:00 python /home/pi/Documents/3_multithread.py
pi 932 881 0 12:25 pts/0 00:00:00 grep --color=auto python
From what I understand is, that it looks like the same process (3_multithread.py) is called twice (once as su, once not as su).
EDIT#2: I start the program via the .bashrc (sudo python /home/pi/Documents/3_multithread.py) and a lxsession. Might this be the reason for blocking the address?
The errorlog looks like that:
File "/home/pi/Documents/3_multithread.py", line 276, in set_server bind((HOST, PORT))
File "/usr/lib/python2.7/socket.py", line 228, in meth return getattr(self._sock, name)(*args)
error: [Errno 98] Address already in use
I just also checked the netstat -lptn. I get the following output:
tcp 0 0 0.0.0.0:22 0.0.0.0.* LISTEN -
EDIT#123: I traced the error further down. I imported and printed:
import os
os.getpid()
To get the process PID. Now what happens is: On bootup my script returned the PID 754, while the ps -fA ¦ grep python only stats two python processes with the PIDs 535 and 539. Does .bashrc and lxterminal start the process twice?
This is how i configure my socket inside the python script
def set_server():
global conn_global
global socket_global
global conn_established
HOST = ''
PORT = 22
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((HOST, PORT))
s.listen(1)
conn, addr = s.accept()
socket_global = s
conn_global = conn
conn_established = 1
I'm using global variables to send commands to send the commands to the other Pi, which is working absolutely perfect (once the connection is established).
Thank you guys very much in advance!
With best regards,
Tobi

So I was FINALLY able to solve that issue.
As Hannu suggested first, the problem was the way I booted the system. First of all, what did I want to do?
I need to boot both Raspis into the X environment (Graphical user interface), since both of my Pis are running a Tkinter GUI. That's why I went for the lxsession based autostart solution. Using the .bashrc file to start my python code failed, since it looked like the .bashrc is called twice on boot.
So what solved my issue was, to use a shell script to start the python code on bootup from a lxterminal.
In order to do this:
Create a shell script in your home directory (the tilde ~ directory)
pi#raspberrypi:~ $ nano start.sh
and add the hashbang as well as the command to start your python script
#!/bin/sh
sudo python ~/Your/file/here.py
Edit the user LX autostart file like:
sudo nano ~/.config/lxsession/LXDE-pi/autostart
and add the call for the lxterminal including your shell script like
#lxterminal -e ./start.sh
Finally, for my application I had to grant access rights to everyone
chmod 775 start.sh
and use global file paths for all images, like
/home/Documents/Your/files/here.png
Save everything and reboot your pi. Now Everything should boot up perfectly.
Thank you all for helping me, especially Hannu!

Problems establishing connection with DB

Platform: LINUX.
I am a beginner of MongoDB and pymongo. After installing pymongo, here is a simple test I tried on ipython:
import pymongo
client = pymongo.MongoClient();
# Also tried to specify the local host and port number
db = client['myDB']
collections = db['temptables']
collections.insert({'a':'1'})
At this point, it chokes. And in the end, spits out a "Error 111: connection refused" error. So, I tried invoking MongoDB straight from the terminal and I still got the error below [look at the far end]. So, I searched a bit and tried:
removing the lock ( sudo rm /var/lib/mongodb/mongod.lock ). Turns out there was no lock in the first place.
sudo mongod --repair
I even saw a suggestion to comment out the host and port number from the config file. Tried that too, didn't work.
None of the above worked.
This is the error I see when I try to invoke mongodb from command line.
017-08-17T15:25:30.265-0700 W NETWORK [thread1] Failed to connect to 127.0.0.1:27017, in(checking socket for error after poll), reason: Connection refused
2017-08-17T15:25:30.265-0700 E QUERY [thread1] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed :
connect#src/mongo/shell/mongo.js:237:13
#(connect):1:6
exception: connect failed
Help, please.

Your mongo server isn't running.
You can confirm this by executing sudo ps -ef | grep mongod
If you have mongo installed and in your path, you can execute:
cd && mkdir -p ~/temp_mongo_db && mongod --dbpath=./temp_mongo_db
This will launch mongo and place all database files in your home directory under 'temp_mongo_db'.
Finally, in a new terminal window, execute sudo ps -ef | grep mongod again. You'll now see mongod running.
If you want to run mongo in production, you should configure it to be managed by SystemD or some other init system.

How to run python bottle on port 80?

When trying to run python bottle on port 80 I get the following:
socket.error: [Errno 10013] An attempt was made to access a socket in a way forb
idden by its access permissions
My goal is to run the web server on port 80 so the url's will be nice and tidy without any need to specify the port
for example:
http://localhost/doSomething
instead of
http://localhost:8080/doSomething
Any ideas?
Thanks

Exactly as the error says. You need to have permissions to run something on the 80th port, normal user cannot do it. You can execute the bottle webapp as root (or maybe www-data) and it should be fine as long as the port is free.
But taking security (and stability) into consideration you should look at different ways of deployment, for example nginx together with gunicorn.
Gunicorn Docs
Nginx Docs

Check your system's firewall setting.
Check whether another application already use port 80 using following commands:
On unix: netstat -an | grep :80
On Windows: netstat -an | findstr :80
According to Windows Sockets Error Codes:
WSAEACCES 10013
Permission denied.
An attempt was made to access a socket in a way forbidden by its
access permissions. An example is using a broadcast address for sendto
without broadcast permission being set using setsockopt(SO_BROADCAST).
Another possible reason for the WSAEACCES error is that when the bind
function is called (on Windows NT 4.0 with SP4 and later), another
application, service, or kernel mode driver is bound to the same
address with exclusive access. Such exclusive access is a new feature
of Windows NT 4.0 with SP4 and later, and is implemented by using the
SO_EXCLUSIVEADDRUSE option.

Sometimes dont need to want install nginx, python with gunicorn is a viable alternative with supervisor but you need to make lot of tricks for working
i assume you know install supervisor, and later install again requirements
pip3 install virtualenv
mkdir /home/user/.envpython
virtualenv /home/user/.envpython/bin/activate
source /home/user/.envpython/bin/activate
cd /home/user/python-path/
pip3 install -r requirements
create a supervisor file like that
nano /etc/supervisord.d/python-file.conf
and edit with this example, edit the program that you need, remember the python3 run in other ports > 1024
;example with path python supervisor in background
[program:python]
environment=HOME="/home/user",USER="user"
user=user
directory = /home/user/python-path
command = python3 /home/user/python-path/main.py
priority = 900
autostart = true
autorestart = true
stopsignal = TERM
;redirect_stderr = true
stdout_logfile = /home/user/.main-python-stdout.log
stderr_logfile = /home/user/.main-python-stderr.log
;example with path python gunicorn supervisor and port 80
[program:gunicorn]
;environment=HOME="/home/user",USER="user"
;user=user
directory = /home/user/python-path
command = bash /home/user/.scripts/supervisor-initiate.sh
priority = 900
autostart = true
autorestart = true
stopsignal = TERM
;redirect_stderr = true
stdout_logfile = /home/user/.main-python-stdout.log
stderr_logfile = /home/user/.main-python-stderr.log
and create the file
nano /home/user/.scripts/supervisor-initiate.sh
with the following content
source /home/user/.envpython/bin/activate
cd /home/user/python-path
gunicorn -w 1 -t 120 -b 0.0.0.0:80 main:app
i assume you file in python is called main and you initiate the app with flask or django that called "app"
only restart supervisord process
systemctl restart supervisord
and you have the app with gunicorn in the port 80, i post because i find for a very long time for this solution
Waiting works for anyone