I have install the scrapyd like this
pip install scrapyd
I want to use scrapyd-deploy
when i type scrapyd
i got this exception in cmd:
'scrapyd' is not recognized as an internal or external command, operable program or batch file.
I ran into the same issue, and I also read some opinions that scrapyd isn't available / can't run on windows and nearly gave it up (didn't really need it as I intend on deploying to a linux machine, wanted scrapyd on windows for debug purposes). However, after some research I found a way. As I haven't found any clear instructions on this, I will try to make my answer as detailed as possible, listing all the steps that worked for me.
Assuming you want to run scrapyd on your local machine, you will need two command lines running: The first is used to connect to scrapyd and keeping the connection open, and the second is for deploying and scheduling.
You already pip installed scrapyd.
Create a folder C:\scrapyd and an empty .log file named scrapyd.log in this folder (not sure this step is necessary).
Open your cmd and cd to your Scripts folder inside Python. This is usually something like: C:\Python27\Scripts. Type: python scrapyd
At this point you should see something like that:
2014-03-26 13:57:30+0200 [-] Log opened.
2014-03-26 13:57:30+0200 [-] twistd 13.2.0 (C:\Python27\python.exe 2.7.6) starting up.
2014-03-26 13:57:30+0200 [-] reactor class: twisted.internet.selectreactor.SelectReactor.
2014-03-26 13:57:30+0200 [-] Site starting on 6800
2014-03-26 13:57:30+0200 [-] Starting factory <twisted.web.server.Site instance at 0x0000000003F69208>
2014-03-26 13:57:30+0200 [Launcher] Scrapyd 1.0.1 started: max_proc=16, runner='scrapyd.runner'
If you can open your browser and go to http://localhost:6800 then you're ok. The command line window should stay open in the background as connection will be closed if you'll close it.
On windows explorer navigate to your scrapy project folder and edit the scrapy.cfg file found there: write your deploy target name: [deploy:scrapyd] for instance, and uncomment the url line.
Open a second command line and cd to your scrapy project folder. Type: scrapy deploy -l
Deploy: type: scrapy deploy scrapyd -p project_name (scrapyd is your target). You should get a server response code 200 and an ok status. You can check if deploy was successful also by typing: scrapy deploy -L scrapyd
Scheduling: you need to install curl for windows. Here's how: questions/9507353/steps-to-setup-curl-in-windows
type: curl http://localhost:6800/schedule.json -d project=project_name -d spider=spider_name
Again, you should get an ok status, and in your browser at http://localhost:6800 under jobs you can check whether the job was indeed scheduled.
I hope this helps.
If you install scrapyed-client by using pip install scrapyd-client I suggest try this command:
pip install git+https://github.com/scrapy/scrapyd-client
it worked for me.
thanks to scrapyd-client command not found
For me proposed solution above didn't work.
Below what worked for me: (for scrapy in version 0.24).
Go to C:\Python27\Scripts and create two files:
scrapy.bat
scrapyd-deploy.bat
Edit both files with notepad and paste:
In the scrapy.bat file:
#echo off
c:\Python27\python c:\Python27\Scripts\scrapy %*
In the scrapyd-deploy.bat file:
#echo off
c:\Python27\python c:\Python27\Scripts\scrapyd-deploy %*
Then save these files and restart cmd.
If you have C:\Python27\Python and C:\Python27\Scripts in your PATH then both commands scrapy and scrapyd-deploy should work.
Marcin Rapacz's answer work for me. However, I use anaconda to manage my
python library. So, the files should be in "C:\Program Files (x86)\Anaconda3\Scripts", and content in files should be changed like:
#echo off
"C:\Program Files (x86)\Anaconda3\python.exe" "C:\Program Files
(x86)\Anaconda3\Scripts\scrapyd-deploy" %1 %2 %3 %4 %5 %6 %7 %8 %9
Related
This is my fabric script that runs on the jenkins server.
sudo('/home/myjenkins/killit.sh',pty=False)
sudo('/home/myjenkins/makedir.sh',pty=False)
sudo('/home/myjenkins/runit.sh',pty=False)
This kills the old server, creates a virtualenv, installs the requirements and restarts the server.
The problem is the with the script that starts the server - runit.sh :-
nohup /home/myjenkins/devserver/dev/bin/python /home/myjenkins/devserver
/workspace/manage.py runserver --noreload 0:80 >> jenkins.out &
When the jenkins server that starts the server and I navigate to the homepage, it gives me a 404 Page Not Found. It says /static/index.html not found. But the file exists. When I run 'sudo bash runit.sh' and I access the homepage, It works fine.
mkdir -p /home/myjenkins/devserver
cp -rf /home/myjenkins/workspace /home/jenkins/devserver/
cp -f /home/myjenkins/dev_settings.py /home/myjenkins/devserver/workspace/mywebsite/settings.py
cd /home/myjenkins/devserver
virtualenv -p python3 dev
cd /home/myjenkins/devserver/workspace
../dev/bin/pip install -r requirements.txt
Please ask me for more details if you need it.
EDITED 9/2/18
When I start the script from the folder containing manage.py, the server is able to serve the files. But Jenkins was starting the script from the home folder and if I also start the script from the home folder - the server is not able to find the files. look at my comment for more details. It would be great if someone could explain why this happens even though I've specified the full path in the script.
nohup /home/myjenkins/devserver/dev/bin/python /home/myjenkins/devserver
/workspace/manage.py runserver --noreload 0:80 >> jenkins.out &
Okay I figured out the whole deal. My django server was taking the output of the npm build from the wrong folder.
In the settings.py file, the variable STATICFILES_DIRS was set as:-
STATICFILES_DIRS = ('frontend/dist',)
instead of:-
STATICFILES_DIRS = (os.path.join(BASE_DIR,'frontend/dist'),)
Thus, when Jenkins was running the script, it was doing so from the home folder. This made Django's staticfiles finders to look at /home/myjenkins/frontend/dist instead of the relative '../frontend/dist'
I have installed Mongodb 3.0 using this tutorial -
https://docs.mongodb.com/v3.0/tutorial/install-mongodb-on-amazon/
It has installed fine. I have also given permissions to 'ec2-user' to all the data and log folders ie var/lib/mongo and var/log/mongodb but and have set conf file as well.
Now thing is that mongodb server always fails to start with command
sudo service mongod start
it just say failed, nothing else.
While if I run command -
mongod --dbpath var/lib/mongo
it starts the mongodb server correctly (though I have mentioned same dbpath in .conf file as well)
What is it I am doing wrong here?
When you run sudo mongod it does not load a config file at all, it literally starts with the compiled in defaults - port 27017, database path of /data/db etc. - that is why you got the error about not being able to find that folder. The "Ubuntu default" is only used when you point it at the config file (if you start using the service command, this is done for you behind the scenes).
Next you ran it like this:
sudo mongod -f /etc/mongodb.conf
If there weren't problems before, then there will be now - you have run the process, with your normal config (pointing at your usual dbpath and log) as the root user. That means that there are going to now be a number of files in that normal MongoDB folder with the user:group of root:root.
This will cause errors when you try to start it as a normal service again, because the mongodb user (which the service will attempt to run as) will not have permission to access those root:root files, and most notably, it will probably not be able to write to the log file to give you any information.
Therefore, to run it as a normal service, we need to fix those permissions. First, make sure MongoDB is not currently running as root, then:
cd /var/log/mongodb
sudo chown -R mongodb:mongodb .
cd /var/lib/mongodb
sudo chown -R mongodb:mongodb .
That should fix it up (assuming the user:group is mongodb:mongodb), though it's probably best to verify with an ls -al or similar to be sure. Once this is done you should be able to get the service to start successfully again.
If you’re starting mongod as a service using:
sudo service mongod start
Make sure the directories defined for logpath, dbpath, and pidfilepath in your mongod.conf exist and are owned by mongod:mongod.
I installed supervisor and gunicorn in my virtual environment (venv).
I am using this tutorial: https://realpython.com/blog/python/kickstarting-flask-on-ubuntu-setup-and-deployment/
I'm confused as to where I should be creating the config file for supervisor as the default etc/supervisor won't apply to me.
The supervisorctl file is in the directory:
/home/giri/venv/py2.7/lib/python2.7/site-packages/supervisor
I noticed this line in the supervisorctl file:
Options:
-c/--configuration -- configuration file path (default /etc/supervisord.conf)
Do I need to manually set this flag each time I run the supervisorctl script or is there another way?
Thanks
As found in the docs (http://supervisord.org/configuration.html):
The Supervisor configuration file is conventionally named
supervisord.conf. It is used by both supervisord and supervisorctl. If
either application is started without the -c option (the option which
is used to tell the application the configuration filename
explicitly), the application will look for a file named
supervisord.conf within the following locations, in the specified
order. It will use the first file it finds.
$CWD/supervisord.conf
$CWD/etc/supervisord.conf
/etc/supervisord.conf
So put the supervisor.conf in your current working directory and you're fine.
I download source code of my app from google appengine using this command in cmd:
appcfg.py download_app -A <your_app_id> -V <your_app_version> <output-dir>
But, instead of running the command, it opens the file "appcfg.py". So I don't know what to do now.
Sounds like python files are associated with an editor instead of with the python interpreter.
If so you'll have to change the associations for .py files (found in folder options), or call the python interpreter:
C:\path\to\python appcfg.py download_app -A -V
I'm currently playing around with home networking and getting into servers and I found a python command that will make the directories of you python folder avaliable over your LAN. To execute it in windows, I've just been entering the following into Command Prompt:
cd..
cd..
cd Python33
python -m http.server 8000
This creates the server that can be accessed via x.x.x.x:8000. Instead of manually entering these commands every time, however, I tried to make a batch script that would do this automatically. Below is the script for the batch file. The problem I'm having, is When I run the batch file, command prompt is saying 'python' is not recognized as an internal or external command. I was wondering if someone could tell my why this is happening and offer a solution or reason as to why this couldn't work. Thanks.
#ECHO OFF
cd..
cd..
cd Python33
python -m http.server 8000
pause
EDIT:
Below is the code that works, this will also set the directory to the C drive.
#ECHO OFF
cd C:\
C:\Python33\python -m http.server 8000
Maybe it's because the batch file is located in a directory that is not 2- level deep (C:\path\to\batch.bat)
Specifying the path of the python executable will solve your problem.
#ECHO OFF
C:\python33\python -m http.server 8000
pause
Try this:
#ECHO OFF
cd..
cd..
cd Python33
python -m SimpleHTTPServer 8000
pause