I can't access scrapyd port 6800 from browser - python

I searched a lot on this, it may have a simple solution that I am missing.
I have setup scrapy + scrapyd on both my local machine and my server. They work both ok when I try as "scrapyd".
I can deploy to local without a problem, and I can access to localhost:6800 as well from the browser and I can run spiders on local.
After running scrapyd on remote, I try to deploy to http://remoteip:6800/ with the same as I did deploy locally,
I get,
Packing version 1500333306
Deploying to project "projectX" in http://remoteip:6800/addversion.json
Deploy failed: <urlopen error [Errno 111] Connection refused>
I also can't access http://remoteip:6800/ from my local PC, but I can access from ssh on remote PC (with curl)
I opened inbound and outbound connections on the remote server, what else I am missing?
Thanks

First check if its running or not, run curl localhost:6800 on the server where ScrapyD is running
Check if firewall is enabled
sudo ufw status
Ideally, just allow tcp connections to 6800instead of disabling firewall, to do so
sudo ufw allow 6800/tcp
sudo ufw reload
Check your scrapyd.conf please set
bind_address=0.0.0.0
instead of
bind_address=127.x.x.x
0.0.0.0 will make scrapyD accessible for incoming connections outside the server/instance, not only localhost.
Then stop scrapyD, I do killall scrapyd to stop scrapyd
Then restart scrapyD using command scrapyd
Note:
If you want to keep scrapyd running even after you disconnect from server, do this
nohup scrapyd >& /dev/null &
Also see my answer to set ScrapyD as a System Service

I know this answer may be late, but I hope it can help others like me.
From the official documentation, it will search the config file in these places:
/etc/scrapyd/scrapyd.conf (Unix)
c:\scrapyd\scrapyd.conf (Windows)
/etc/scrapyd/conf.d/* (in alphabetical order, Unix) scrapyd.conf
~/.scrapyd.conf (users home directory)
So you need to create a scrapyd.conf file, and put some configurations in it.
Here is an example configuration file with all the defaults from the the documentation:
[scrapyd]
eggs_dir = eggs
logs_dir = logs
items_dir =
jobs_to_keep = 5
dbs_dir = dbs
max_proc = 0
max_proc_per_cpu = 4
finished_to_keep = 100
poll_interval = 5.0
bind_address = 127.0.0.1
http_port = 6800
debug = off
runner = scrapyd.runner
application = scrapyd.app.application
launcher = scrapyd.launcher.Launcher
webroot = scrapyd.website.Root
[services]
schedule.json = scrapyd.webservice.Schedule
cancel.json = scrapyd.webservice.Cancel
addversion.json = scrapyd.webservice.AddVersion
listprojects.json = scrapyd.webservice.ListProjects
listversions.json = scrapyd.webservice.ListVersions
listspiders.json = scrapyd.webservice.ListSpiders
delproject.json = scrapyd.webservice.DeleteProject
delversion.json = scrapyd.webservice.DeleteVersion
listjobs.json = scrapyd.webservice.ListJobs
daemonstatus.json = scrapyd.webservice.DaemonStatus
And what you need to do is: change the bind_address to 0.0.0.0

Related

Django running on an ECS task does not work. "Connection refused" or "No data response" when requesting the webapp

I have some problems running Django on an ECS task.
I want to have a Django webapp running on an ECS task and accessible to the world.
Here are the symptoms:
When I run an ECS task using Django python manage.py runserver 0.0.0.0:8000 as entry point for my container, I have a connection refused response.
When I run the task using Gunicorn using gunicorn --bind 0.0.0.0:8000 my-project.wsgi I have no data response.
I don't see logs on CloudWatch and I can't find any server's logs when I ssh to the ECS instance.
Here are some of my settings related to that kind of issue:
I have set my ECS instance security groups inbound to All TCP | TCP | 0 - 65535 | 0.0.0.0/0 to be sure it's not a firewall problem. And I can assert that because I can run a ruby on rails server on the same ECS instance perfectly.
In my container task definition I set a port mapping to 80:8000 and an other to 8000:8000.
In my settings.py, I have set ALLOWED_HOSTS = ["*"] and DEBUG = False.
Locally my server run perfectly on the same docker image when doing a docker run -it -p 8000:8000 my-image gunicorn --bind=0.0.0.0:8000 wsgi or same with manage.py runserver.
Here is my docker file for a Gunicorn web server.
FROM python:3.6
WORKDIR /usr/src/my-django-project
COPY my-django-project .
RUN pip install -r requirements.txt
EXPOSE 8000
CMD ["gunicorn","--bind","0.0.0.0:8000","wsgi"]
# CMD ["python","manage.py", "runserver", "0.0.0.0:8000"]
Any help would be grateful!
To help you debugging:
What is the status of the job when you are trying to access your webapp.
Figure out which instance the job is running and try docker ps on that ecs instance for the running job.
If you are able see the container or the job running on the instance, try access your webapp directly on the server with command like curl http://localhost:8000 or wget
If you container is not running. Try docker ps -a and see which one has just stopped and check with docker logs -f
With this approach, you can cut out all AWS firewall settings, so that you can see if your container is configured correctly. I think it will help you tracking down the issue easier.
After you figuring out the container is running fine and you are able to request with localhost, then you can work on security group inbound/outbound filter.

In zulip dev environment, how can i access zulip's first dev web browser using run-dev.py in vagrant env but i can't access from global ip

I have some problem now using zulip dev-env ...
I did success-installation by Dev-env(using run-dev.py in vagrant env) in my Mac PC and then i could access the first-dev page successfully through my mac-pc's browser
But i have been trying to show the first page using Dev-dev in my cloud hosting service (like AWS or google Cloud hosting) for several days
but i could not access and could not show the first page by using global IP and browser
Which part must I change the setting environment?
(that is, I want to access my dev-env[using run-dev.py] by global IP
after that, i want to test... )
Now I am using a cloud hosting. Of course I opened port 9991 and etc...
I use vagrant env and succeed it in cloud host.
In vagrant ssh, I could access zulip by using curl 127.0.0.1:9991 and received a success message:
29/Sep/2016 00:42:04]"GET / HTTP/1.0" 302 0" from running console
but i cannot access it by host console like this
ai2#mrtalk:~/project/zulip$ curl 127.0.0.1:9991
curl: (7) Failed to connect to 127.0.0.1 port 9991: Connection refused
of course , i can't access http://182.162.90.79:9991/ through my pc browser
for this I tried like these
In zproject/dev_setting.py (I changed...these lines)
EXTERNAL_HOST = '0.0.0.0:9991' #orgin localhost but change 0.0.0.0
ALLOWED_HOSTS = ['localhost','182.162.90.79','0.0.0.0'] #orgin localhost but add + '182.162.90.79','0.0.0.0'
In Vagrantfile I changed this line:
config.vm.network "forwarded_port", guest: 9991, host: host_port, host_ip: 127.0.0.1, auto_correct: true
What is wrong with here? Or which part should I change my dev-setting?

Running ubuntu vagrant server won't show anything on debian local browser

I've done this multiple times and never had an issue.
I'm trying to run a website on vagrant (trusty64) through shh on my Debian 8 Jessie x64.
I have used the same Vagrantfile n times before and it always worked perfectly but for I don't know what reason, on my new laptop, when I enter (on vagrant server - tried with sudo and without):
vagrant up
vagrant ssh
cd /opt/site/
python manage.py runserver 0.0.0.0:8000
nothing happens in console. It shows the usual:
System check identified 1 issue (0 silenced).
October 28, 2015 - 10:16:41
Django version 1.8.4, using settings 'settings.development'
Starting development server at http://0.0.0.0:8000/
Quit the server with CONTROL-C.
and when I type the ip:port in my local browser it just say:
This webpage is not available
ERR_CONNECTION_REFUSED
Any workaround for this ?
My vagrant config file:
# -*- mode: ruby -*-
# vi: set ft=ruby :
VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
config.vm.provider "virtualbox" do |v|
v.memory = 512
end
config.vm.box = "trusty64"
config.vm.box_url = "https://cloud-images.ubuntu.com/vagrant/trusty/current/trusty-server-cloudimg-amd64-vagrant-disk1.box"
config.vm.network "private_network", ip: "192.168.100.10"
config.vm.synced_folder ".", "/opt/site"
config.vm.hostname = "alexander"
end
As a mention: I tried multiple ports and still nothing. More, I have the exactly same configuration on another pc and it works flawless.
If changing the port forwarding doesn't work.You might also need to change in the settings of virtual box from NAT to Bridged Adapter.
The firewall will block incoming traffix on port 8000, so you will have to forward that port. Try adding this this to the configuration. you're using:
config.vm.network "forwarded_port", guest: 8000, host: 8080
and then browse to http://192.168.100.10:8080 it should work.
Also dont forget to privision your box again, or even restart just to be sure.
vagrant reload --provision
Why it works on your other machine is a mystery to me. Maybe there is no firewall, or the port is already forwarded?

Can I use the uwsgi protocol to call http?

Here's a data flow:
http <--> nginx <--> uWSGI <--> python webapp
I guess there's http2uwsgi transfer in nginx, and uwsgi2http in uWSGI.
What if I want to directly call uWSGI to test an API in a webapp?
actually i'm using pyramid. just config [uwsgi] in .ini and run uWSGI. but i want to test if uWSGI hold webapp function normally, the uWSGI socket is not directly reachable by http.
Try using uwsgi_curl
$ pip install uwsgi-tools
$ uwsgi_curl 10.0.0.1:3030 /path
or if you need to do some more requests try uwsgi_proxy from the same package
$ uwsgi_proxy 10.0.0.1:3030
Proxying remote uWSGI server 10.0.0.1:3030 "" to local HTTP server 127.0.0.1:3030...
so you can browse it locally at http://127.0.0.1:3030/.
If your application allows only certain Host header, you can specify host name as well
$ uwsgi_curl 10.0.0.1:3030 host.name/path
$ uwsgi_proxy 10.0.0.1:3030 -n host.name
If application has static files, you can redirect such requests to your front server using -s argument. You can also specify different local port if needed.
From your question I'm assuming, you want to directly run your WSGI-compliant app with uWSGI and open an HTTP-Socket. You can do so by configuring your uwsgi.ini (or whatever the filename is) with
http=127.0.0.1:8080
uwsgi will now open an HTTP-socket that listen on port 8080 for incoming connections from localhost (see documentation: http://uwsgi-docs.readthedocs.org/en/latest/HTTP.html)
Alternatively you can directly start your process from the command-line with the http-parameter:
$ uwsgi --http=127.0.0.1:8080 --module=yourapp:wsgi_entry_point
If you use unix-sockets to configure uwsgi nginx is able to communicate with that socket via the uwsgi-protocol (http://uwsgi-docs.readthedocs.org/en/latest/Protocol.html).
Keep in mind, that if you usually serve static content (css, javascript, images) through nginx you will need to set that up, too, if you run uwsgi directly. But if you only want to test a REST-API this should work out for you.
First, consider those questions:
On which port is uWSGI running?
Is uWSGI running on your or on a remote machine?
If it's running on a remote machine, is the port accessible from your computer? (iptables rules might forbid external access)
If you made sure you have access, you can just call http://hostname:port/path/to/uWSGI for direct API access.
I know this is an old question but I just needed this and found out that this docker+nginx solution works for me the best
cat > /tmp/nginx.conf << EOF
events {}
http {
server {
listen 8000;
location / {
include uwsgi_params;
uwsgi_pass 127.0.0.1:3031;
}
}
}
EOF
docker run -it --network=host --rm --name uswgi-proxy -v /tmp/nginx.conf:/etc/nginx/nginx.conf:ro nginx

Properly specifying path for git pull from my local development machine

I'm trying to setup Fabric so that I can automatically deploy my Django app to my web server.
What I want to do is to pull the data from my Development machine (os X) to the server.
How do I correctly specify my path in the git url?
This is the error I'm getting:
$ git pull
fatal: '/Users/Bryan/work/tempReview_app/.': unable to chdir or not a git archive
fatal: The remote end hung up unexpectedly
This is .git/config:
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
ignorecase = true
[remote "origin"]
fetch = +refs/heads/*:refs/remotes/origin/*
url = /Users/Bryan/work/my_app/.
[branch "master"]
remote = origin
merge = refs/heads/master
On your server, create a folder called myapp. Chdir to this folder, and then run
server ~/myapp$ git init
Then, let git know about your server. After this, push to the server's repository from your local machine.
local ~/myapp$ git remote add origin user#server:~/myapp.git
local ~/myapp$ git push origin master
Anytime you want to push changes to your server, just run git push. If you make a mistake, just log in to your server and git co last-known-good-commit or something to that effect.
Git hooks are also very useful in situations such as the one you're facing. I would give you pointers on that but I don't know what your workflow is like, so it probably wouldn't be very helpful.

Categories

Resources