I'm trying to setup Fabric so that I can automatically deploy my Django app to my web server.
What I want to do is to pull the data from my Development machine (os X) to the server.
How do I correctly specify my path in the git url?
This is the error I'm getting:
$ git pull
fatal: '/Users/Bryan/work/tempReview_app/.': unable to chdir or not a git archive
fatal: The remote end hung up unexpectedly
This is .git/config:
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
ignorecase = true
[remote "origin"]
fetch = +refs/heads/*:refs/remotes/origin/*
url = /Users/Bryan/work/my_app/.
[branch "master"]
remote = origin
merge = refs/heads/master
On your server, create a folder called myapp. Chdir to this folder, and then run
server ~/myapp$ git init
Then, let git know about your server. After this, push to the server's repository from your local machine.
local ~/myapp$ git remote add origin user#server:~/myapp.git
local ~/myapp$ git push origin master
Anytime you want to push changes to your server, just run git push. If you make a mistake, just log in to your server and git co last-known-good-commit or something to that effect.
Git hooks are also very useful in situations such as the one you're facing. I would give you pointers on that but I don't know what your workflow is like, so it probably wouldn't be very helpful.
Related
I'm trying to deploy code from my gitlab CI runner to Azure. I'm using the az webapp deployment to do the job:
az webapp deployment source config --branch master --manual-integration --name [myWebApp] --repo-url [git url] --app-working-dir [folder] --resource-group [myResourceGroup]
But there is a problem, the default working directory is the git 's root folder. My application is inside a child folder from root. I check the command configuration and I saw --app-working-dir which does what I want, again, but only works if --cd-project-url is set (I don't need and I'm not using VSTS). Also, there is no example available of using this config. I'm looking to have a command set similar to what I used when deploying via azure cloud shell: cd <app-folder>; az webapp up --sku B1 --name <app-name>. There is a way to do the deploy in a simpler form?
Create a .deployment file in your root directory of your git repository. And add the project sub folder in the configuration. It will work then.
https://github.com/projectkudu/kudu/wiki/Customizing-deployments
I am trying to setup a MLFlow tracking server on a remote machine as a systemd service.
I have a sftp server running and created a SSH key pair.
Everything seems to work fine except the artifact logging. MLFlow seems to not have permissions to list the artifacts saved in the mlruns directory.
I create an experiment and log artifacts in this way:
uri = 'http://192.XXX:8000'
mlflow.set_tracking_uri(uri)
mlflow.create_experiment('test', artifact_location='sftp://192.XXX:_path_to_mlruns_folder_')
experiment=mlflow.get_experiment_by_name('test')
with mlflow.start_run(experiment_id=experiment.experiment_id, run_name=run_name) as run:
mlflow.log_param(_parameter_name_, _parameter_value_)
mlflow.log_artifact(_an_artifact_, _artifact_folder_name_)
I can see the metrics in the UI and the artifacts in the correct destination folder on the remote machine. However, in the UI I receive this message when trying to see the artifacts:
Unable to list artifacts stored
under sftp://192.XXX:path_to_mlruns_folder/run_id/artifacts
for the current run. Please contact your tracking server administrator
to notify them of this error, which can happen when the tracking
server lacks permission to list artifacts under the current run's root
artifact directory.
I cannot figure out why as the mlruns folder has drwxrwxrwx permissions and all the subfolders have drwxrwxr-x. What am I missing?
UPDATE
Looking at it with fresh eyes, it seems weird that it tries to list files through sftp://192.XXX:, it should just look in the folder _path_to_mlruns_folder_/_run_id_/artifacts. However, I still do not know how to circumvent that.
The problem seems to be that by default the systemd service is run by root.
Specifying a user and creating a ssh key pair for that user to access the same remote machine worked.
[Unit]
Description=MLflow server
After=network.target
[Service]
Restart=on-failure
RestartSec=20
User=_user_
Group=_group_
ExecStart=/bin/bash -c 'PATH=_yourpath_/anaconda3/envs/mlflow_server/bin/:$PATH exec mlflow server --backend-store-uri postgresql://mlflow:mlflow#localhost/mlflow --default-artifact-root sftp://_user_#192.168.1.245:_yourotherpath_/MLFLOW_SERVER/mlruns -h 0.0.0.0 -p 8000'
[Install]
WantedBy=multi-user.target
_user_ and _group_ should be the same listed by ls -la in the mlruns directory.
Right now I have an app that write data for users to a database file called users.sql. However, I pushed this to heroku through github and the data doesn't stick. If a certain time has passed, heroku will sleep and all the files will have to be redeployed when another login is detected. I heard that I could add heroku postgres as an addon to heroku but I am not sure what to do to integrate this with my current code. What do I do to set it up?
I stumbled across your question while trying to connect my app to Postgres and getting very frustrated at the lack of clear instructions. So here's the step by step process with all the errors I struggled through (because that's where the tutorials leave you gasping for air while they thunder on obliviously...)
TL;DR
Install Postgres
Install the Heroku CLI
Install psycopg2 or psycopg2-binary and gunicorn
Create an app in Heroku with a Postgres database and save the database url as the SQLALCHEMY_DATABASE_URI
Create Procfile, requirements.txt and runtime.txt
Push to Github and connect repo to Heroku
Deploy branch
from app import db db.create_all()
I work with WSL so all command line syntax in my answer is Bash, and I'm working on Windows 10.
I'm working partly off a youtube tutorial by Traversy Media. Here's the Github repo he works with (I definitely recommend initially working with a basic app instead of struggling with the errors that crop up in a complex app with extra packages).
Before you start
Create an account in Heroku and install the Heroku CLI
Download and install PostgreSQL (this will also give you pgAdmin 4)
Fork and clone the Github repo, and check it runs ok (if you're not working with your own app)
Preliminaries
Create a virtual env and install all packages. Ensure you have psycopg2and gunicorn
ERROR: Failed building wheel for psycopg2-binary
For whatever reason, psycopg refuses to install on my system. You can get round this by using pip3 install psycopg2-binary instead - there's no difference as far as our requirements go.
Connect to Postgres
Open pgAdmin 4 from the start menu. This may prompt you for a password if it's the first time you're logging in or if you were logged out.
could not connect to server: Connection refused (0x0000274D/10061) Is the
server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432?
Use sudo service postgresql status to check if the PostgreSQL server
is running. If it is, try stopping and starting it. If it isn't, start
it.
sudo service postgresql start
sudo service postgresql stop
In the left-hand panel, right-click on Databases and click Create > Database.... Enter your database name (the tutorial uses "lexus") and save.
In app.py (or wherever you do your database config) find the line
if ENV == 'dev':
...
app.config['SQLALCHEMY_DATABASE_URI'] = ''
and put the database connection string inside the quotes. The syntax is
postgresql://[username]:[password]#[host name]/[db name]
# [username] - Right-click on your server in pgAdmin (probably a variation of `PostgreSQL 13`). Go to the Connection tab and check the Username field.
# [password] - The root password you set the first time you logged in to pgAdmin.
# [host name] - In the same place as your username, in the Host name/address field.
# [db name] - The name you entered when creating your database.
In my case, the connection string is:
app.config['SQLALCHEMY_DATABASE_URI'] = 'postgresql://postgres:123456#localhost/lexus'
Important Disclaimer
This solution has you put the database password in code: app.config['SQLALCHEMY_DATABASE_URI'] = 'postgresql://postgres:123456#localhost/lexus'
This saves the password in your version control history, where it can/will eventually allow anyone on the Internet to control your database. Yikes! Get DB user & password from env vars instead.
Thanks #Paul Cantrell!
Enter the python console and create the database:
>>> from app import db
>>> db.create_all()
If you didn't get an error on that, consider yourself the luckiest thing since sliced bread. In pgAdmin go to Databases > [your db name] > Schemas > Tables to check they've been created.
ImportError: cannot import name 'db' from 'app' (unknown location)
db is not located in app in your application. Find where it's
defined and import it from there.
(psycopg2.OperationalError) FATAL: password authentication failed for user "postgres"
You're using the wrong password. I found this confusing
because it's unclear whether you need to use the master password (the
one you set when logging in to pgAdmin for the first time) or the
password for the database you're connecting to. It's the master password that's needed here.
Enter psql -h 127.0.0.1 postgres (psql -h [ip address] [username]) in the command line - the password that authenticates is the one you need for the connection string.
(psycopg2.OperationalError) FATAL: database "lexus" does not exist
You might be connecting to the wrong database, using the wrong
username, or your database hasn't been created. Check your connection string and pgAdmin to confirm that details are correct.
db.create_all() seemed to work, but I have no tables in pgAdmin!
If your database models are stored in a separate file, import them
into app.py after db is initialised. It should look like this:
app = Flask(__name__)
...
db = SQLAlchemy(app)
...
from app.models import *
Send data to Postgres
Run the app and submit some data through the form (or create some data in your table). In pgAdmin, right-click the table you just created and refresh, then right-click and view/edit data.
Could not send data to server: Socket is not connected (0x00002749/10057) could not send SSL negotiation packet: Socket is not connected (0x00002749/10057)
Disconnect and reconnect server (right-click on server and click Disconnect Server, then Connect Server) and refresh table again.
If that didn't work, right-click on server and click Properties, go to the Connection tab and change the Host name/address from
localhost to 127.0.0.1. Then repeat step 1.
Create Heroku app
Since I prefer using Heroku directly with Github instead of the CLI, I'm not using git init here, but note that if your code is not already a Github repo you would need to use it.
In the command line:
heroku login # If you have the Heroku CLI installed correctly, this will open a tab in the browser and log you in to Heroku.
heroku create unique-app-name # (replace `unique-app-name` with your own name). This creates an app on the Heroku server, and it's the place your code is cloned to so that it can be run. Check that it has appeared in your [Heroku dashboard](https://dashboard.heroku.com/).
heroku addons:create heroku-postgresql:hobby-dev -a unique-app-name # This creates an empty postgres database for your app. Heroku uses its own database, not your pgAdmin database.
heroku config -a unique-app-name # The url you set earlier as the `SQLALCHEMY_DATABASE_URI` points to the database you created in pgAdmin, but since your app is now going to be using Heroku's database you need a new url. Copy the DATABASE_URL that is returned.
Go back to the code where you set app.config['SQLALCHEMY_DATABASE_URI'] earlier, and insert the new url inside the else block. It should look more or less like this:
if ENV == 'dev':
app.config['SQLALCHEMY_DATABASE_URI'] = 'postgresql://postgres:123456#localhost/lexus'
else:
app.config['SQLALCHEMY_DATABASE_URI'] = 'postgres://urlyoujustgotfromherokuozftkndvkayeyc:691196bfb1b1ca8318b733935b10c97a19fd41#ec2-52-71-161-140.compute-1.amazonaws.com:5432/d3pofh2b55a2ct'
If you're working with the tutorial's github repo, set ENV = 'prod', otherwise set the config to use the heroku url however you want.
Add the following files to your root directory if they're not yet included in your project files:
# Procfile
web: gunicorn app:app # The first `app` refers to the app.py and the second `app` refers to the app object, ie `app = Flask(__name__)`. Heroku needs this to know how to run your code.
# runtime.txt
python-3.8.5 # Find your python version by entering `python --version` or `python3 --version` in the command line.
# requirements.txt
# Create this by running `pip3 freeze > requirements.txt`. According to the [Heroku docs](https://devcenter.heroku.com/articles/python-runtimes)
# this must be done after changing runtime versions, so it's best to do it last.
Push everything to Github (again, this is if you're already in a repo otherwise you'd be using the Heroku CLI process which I'm not following).
Deploy on Heroku
Click on the app you created previously in the Heroku dashboard and go to the Deploy tab. In the Deployment method options, click Github. Connect your Github account and repository.
Scroll down to the Manual deploy section and select the branch you want to deploy (or just leave it as master if that's the only one). Click Deploy Branch.
Requested runtime (python-3.7.2) is not available for this stack (heroku-20).
! Aborting. More info: https://devcenter.heroku.com/articles/python-support
! Push rejected, failed to compile Python app.
! Push failed
It looks like the python version is wrong. Check that runtime.txt is showing the right python version. If it
isn't, update it, delete and recreate requirements.txt, push to
Github and click Deploy Branch again.
Create Heroku database
Back in the command line:
heroku run python3
>>> from app import db
>>> db.create_all()
To check out your database, enter
heroku pg:psql --app unique-app-name
If you run select * from tablename; nothing will be returned because there isn't any data yet.
Click the View button in Heroku to open your app. Submit the form (or enter data however your app works), then run select * from tablename; again and you'll see the row of data.
In your init.py file you can set up your app.config['SQLALCHEMY_DATABASE_URI'] variable such that it will look for the one Heroku will give your application if it's running on heroku, OR use the configuration for your local database if it's running locally. You can achieve this as such:
app.config['SQLALCHEMY_DATABASE_URI'] = environ.get('DATABASE_URL') or "postgresql://postgres:password#localhost:5432/postgres"
replace what is after the OR w/ the connection details for your local connection
I would suggest considering setting up a local postgres database instead of sql if you're going to use heroku but you can technically use both. You just may run into minor issues, I have ran into issues w/ dates being handled slightly different, for example.
Firstly make heroku login from terminal:
$ heroku login
Now create the database for your app using following command:
$ heroku addons:create heroku-postgresql:hobby-dev --app app_name
Now after your Database has been created, get the URL of your Database using the following command:
$ heroku config --app app_name
Now that you have got your Database URL, replace the value of app.config[‘SQLALCHEMY_DATABASE_URI’] line in the “app.py” file with this Database URL:
app.config['SQLALCHEMY_DATABASE_URI'] = 'heroku_database_url'
And finally push code to heroku.
I searched a lot on this, it may have a simple solution that I am missing.
I have setup scrapy + scrapyd on both my local machine and my server. They work both ok when I try as "scrapyd".
I can deploy to local without a problem, and I can access to localhost:6800 as well from the browser and I can run spiders on local.
After running scrapyd on remote, I try to deploy to http://remoteip:6800/ with the same as I did deploy locally,
I get,
Packing version 1500333306
Deploying to project "projectX" in http://remoteip:6800/addversion.json
Deploy failed: <urlopen error [Errno 111] Connection refused>
I also can't access http://remoteip:6800/ from my local PC, but I can access from ssh on remote PC (with curl)
I opened inbound and outbound connections on the remote server, what else I am missing?
Thanks
First check if its running or not, run curl localhost:6800 on the server where ScrapyD is running
Check if firewall is enabled
sudo ufw status
Ideally, just allow tcp connections to 6800instead of disabling firewall, to do so
sudo ufw allow 6800/tcp
sudo ufw reload
Check your scrapyd.conf please set
bind_address=0.0.0.0
instead of
bind_address=127.x.x.x
0.0.0.0 will make scrapyD accessible for incoming connections outside the server/instance, not only localhost.
Then stop scrapyD, I do killall scrapyd to stop scrapyd
Then restart scrapyD using command scrapyd
Note:
If you want to keep scrapyd running even after you disconnect from server, do this
nohup scrapyd >& /dev/null &
Also see my answer to set ScrapyD as a System Service
I know this answer may be late, but I hope it can help others like me.
From the official documentation, it will search the config file in these places:
/etc/scrapyd/scrapyd.conf (Unix)
c:\scrapyd\scrapyd.conf (Windows)
/etc/scrapyd/conf.d/* (in alphabetical order, Unix) scrapyd.conf
~/.scrapyd.conf (users home directory)
So you need to create a scrapyd.conf file, and put some configurations in it.
Here is an example configuration file with all the defaults from the the documentation:
[scrapyd]
eggs_dir = eggs
logs_dir = logs
items_dir =
jobs_to_keep = 5
dbs_dir = dbs
max_proc = 0
max_proc_per_cpu = 4
finished_to_keep = 100
poll_interval = 5.0
bind_address = 127.0.0.1
http_port = 6800
debug = off
runner = scrapyd.runner
application = scrapyd.app.application
launcher = scrapyd.launcher.Launcher
webroot = scrapyd.website.Root
[services]
schedule.json = scrapyd.webservice.Schedule
cancel.json = scrapyd.webservice.Cancel
addversion.json = scrapyd.webservice.AddVersion
listprojects.json = scrapyd.webservice.ListProjects
listversions.json = scrapyd.webservice.ListVersions
listspiders.json = scrapyd.webservice.ListSpiders
delproject.json = scrapyd.webservice.DeleteProject
delversion.json = scrapyd.webservice.DeleteVersion
listjobs.json = scrapyd.webservice.ListJobs
daemonstatus.json = scrapyd.webservice.DaemonStatus
And what you need to do is: change the bind_address to 0.0.0.0
I'm using elastic beanstalk and django. One of my dependencies in my requirements.txt file has some setup the it performs when it's initially imported. Part of the setup is to check whether a dir exists else it create it. I'm getting permissions errors because the user ( I assume it's wsgi) does not have the permissions to create the dir.
OSError: [Errno 13] Permission denied: '/home/wsgi/.newspaper_scraper/memoized'
How can I setup permissions to allow for these dirs to be created in a way that will be persistent across instances I create in the future?
This is happening because the uWSGI worker is running under a user with limited permissions. You need to create the .newspaper_scraper/memoized directory first, and set the correct permissions on it (allow others to r/w). You can do this on deployment by making a script in .ebextensions that EB executes upon deployment.
Create a file in .ebextensions/setup_newspaper.config and add the following to it:
.ebextensions/setup_newspaper.config
packages:
yum:
libxslt-devel: []
libxml2-devel: []
libjpeg-devel: []
zlib1g-devel: []
libpng12-devel: []
container_commands:
01_setup_newspaper:
command: mkdir -p /home/wsgi/.newspaper_scraper/memoized && chmod 644 /home/wsgi/.newspaper_scraper/memoized
PS: It looks like newspaper requires some extra packages to be installed, so I added them too.
Read more info on .ebextensions here: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create-deploy-python-container.html#create-deploy-python-custom-container