I want to deploy my code in localhost and my live version for this automation i used fabric. My basic fabric file look like:
def localhost():
"Use the local virtual server"
env.hosts = ['127.0.0.1']
env.user = 'user'
env.path = '/var/www/html/{}'.format(env['project_name'])
env.virtualhost_path = env.path
def webserver():
"Use the actual webserver"
env.hosts = ['www.example.com']
env.user = 'username'
env.path = '/var/www/html/{}'.format(env['project_name'])
env.virtualhost_path = env.path
def setup():
require('hosts', provided_by=[localhost])
require('path')
sudo("apt-get update -y")
sudo("apt-get install git -y")
sudo("apt-get install postgresql libpq-dev python-dev python-pip -y")
sudo("apt-get install redis-server -y")
sudo("apt-get install nginx -y")
sudo('aptitude install -y python-setuptools')
sudo('apt-get install python-pip')
sudo('pip install virtualenv virtualenvwrapper')
For now i only want to deploy to my local machine. When I do this it gives me erro saying
The command 'setup' failed because the following required environment variable was not defined:
hosts
Try running the following command prior to this one, to fix the problem:
localhost
What does provided_by=([localhost]) do in here. I guess it should provide the information like hosts and user in localhost.
Why I am getting this error ??
Need help
I'm not sure why that doesn't work other than it's not mentioned in the docs as how host lists get constructed. Your options for setting the host value are:
Specify env.hosts = ['127.0.0.1'] globally in your fabfile
Pass the host to fab: fab -H 127.0.0.1 setup
Call the localhost task: fab localhost setup
Use the #hosts decorator on your setup function
See http://docs.fabfile.org/en/1.10/usage/execution.html#how-host-lists-are-constructed
fabric.operations.require(*keys, **kwargs):
Check for given keys in the shared environment dict and abort if not
found... The optional keyword argument provided_by may be a list of functions or function names or a single function or function name which the user should be able to execute in order to set the key or keys; it will be included in the error output if requirements are not met.
http://docs.fabfile.org/en/1.10/api/core/operations.html?highlight=require#fabric.operations.require
That why you get the error message saying to run localhost first, then setup:
fab localhost setup
Related
I am trying to run a Python script to insert some data into an Oracle table, from a docker image.
I was given following connection string :
jdbc:oracle:thin:#(DESCRIPTION=(CONNECT_TIMEOUT=3)(RETRY_COUNT=2)(FAILOVER=ON)(LOAD_BALANCE=NO) (ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=host_1)(PORT=1521))) (ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=host_2)(PORT=1521))) (CONNECT_DATA=(SERVICE_NAME=service_name)))
I'm trying to connect using cx_Oracle package :
try:
# establish a new connection
with cx_Oracle.connect(self.oracle_user,
self.oracle_pwd,
self.oracle_dsn) as connection:
logger.info('ElasticsearchFinder.oracle_write : connexion established with DB')
with oracle_dsn being the connexion string (minus the jdbc:oracle:thin:# part)
I also tried things like
cx_Oracle.connect(self.oracle_user+'/'+self.oracle_pwd+'#'+self.oracle_dsn)
as seen in some examples, but I always get the following timeout error :
cx_Oracle.DatabaseError: ORA-12170: TNS:Connect timeout occurred
telnet host 1521 works fine, I also tried changing the CONNECT_TIMEOUT value.
I also tried
dsn_tns = cx_Oracle.makedsn(self.oracle_host, self.oracle_port, service_name = self.oracle_service_name)
cx_Oracle.connect(self.oracle_user,self.oracle_pwd,dsn_tns)
as suggested here but then I get
cx_Oracle.DatabaseError: ORA-12514: TNS:listener does not currently know of service requested in connect descriptor
I am running my script from a Docker image build with Dockerfile :
FROM oraclelinux:7.8
RUN yum -y install oracle-release-el7 && \
yum-config-manager --enable ol7_oracle_instantclient && \
yum -y install oracle-instantclient18.3-basic && \
rm -rf /var/cache/yum
COPY ./fetch_session_iptv.py /opt/
COPY ./conf/fetch_session_iptv.conf /opt/conf/
#COPY ./conf/certs/* /opt/conf/certs/
COPY ./logs /opt/logs
RUN yum install -y \
#https://yum.oracle.com/repo/OracleLinux/OL7/developer/x86_64/getPackage/oracle-instantclient18.3-basic-18.3.0.0.0-2.x86_64.rpm \
https://yum.oracle.com/repo/OracleLinux/OL7/developer/x86_64/getPackage/python-cx_Oracle-7.3-1.el7.x86_64.rpm \
https://yum.oracle.com/repo/OracleLinux/OL7/developer/x86_64/getPackage/python36-pytz-2016.10-2.0.1.el7.noarch.rpm
COPY ./python_requirements/elasticsearch-7.8.0-py2.py3-none-any.whl .
COPY ./python_requirements/certifi-2020.4.5.2-py2.py3-none-any.whl .
COPY ./python_requirements/urllib3-1.25.9-py2.py3-none-any.whl .
RUN pip3 install --user \
certifi-2020.4.5.2-py2.py3-none-any.whl \
urllib3-1.25.9-py2.py3-none-any.whl \
elasticsearch-7.8.0-py2.py3-none-any.whl \
cx_Oracle
RUN sh -c "echo /usr/lib/oracle/18.3/client64/lib > /etc/ld.so.conf.d/oracle-instantclient.conf"
RUN ldconfig
RUN export ORACLE_HOME=/usr/lib/oracle/18.3/client64/
RUN yum -y install telnet
#CMD ["/bin/bash"]
CMD [ "python3", "/opt/fetch_session_iptv.py" ]
I can't understand what's going wrong?
[UPDATE] I've set
ORACLE_HOME=/usr/lib/oracle/18.3/client64/bin
TNS_ADMIN=$ORACLE_HOME/admin
LD_LIBRARY_PATH=/usr/lib/oracle/18.3/client64/lib
PATH=$PATH:$ORACLE_HOME
I've added a tnsnames.ora file in TNS_ADMIN directory, with
CNX=(DESCRIPTION=(CONNECT_TIMEOUT=3)(RETRY_COUNT=2)(FAILOVER=ON)(LOAD_BALANCE=NO) (ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=host_1)(PORT=1521))) (ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=host_2)(PORT=1521))) (CONNECT_DATA=(SERVICE_NAME=service_name)))
Now, when I try sqlplus MY_USER#CNX I get the same :
ORA-12170: TNS:Connect timeout occured
but at least it seems to accept the connexion string and user. Prior to setting all the environment variables correctly, I only got different TNS error messages about TNS: listener does not currently know of service requested in connect descriptor or the net service name being incorrect.
[UPDATE 2]
I've checked with the person who gave me all connexion info: user, password and connection string are correct. They're running Oracle Database 12c (12.1.0.2.0), which according to this page is compatible with the 18c client I'm using.
I don't know what else could be a possible reason for this timeout error?
Start by using the equivalent connection string in cx_Oracle:
self.oracle_dsn = "(DESCRIPTION=(CONNECT_TIMEOUT=3)(RETRY_COUNT=2)(FAILOVER=ON)(LOAD_BALANCE=NO) (ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=host_1)(PORT=1521))) (ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=host_2)(PORT=1521))) (CONNECT_DATA=(SERVICE_NAME=service_name)))"
try:
# establish a new connection
with cx_Oracle.connect(self.oracle_user,
self.oracle_pwd,
self.oracle_dsn) as connection:
logger.info('ElasticsearchFinder.oracle_write : connexion established with DB')
The cx_Oracle manual on connecting and connect strings is here.
Personally I'd use 19c Instant Client, which will connect to the same DB versions as 18c - and doesn't need to have ldconfig run. See Docker for Oracle Database Applications in Node.js and Python.
Update your question with information and I can update this answer likewise.
I have upgraded fabric from fabric3==1.14.post1 to fabric=2.5.0. Code works fine with fabric3==1.14.post1.
And now i have a problem while executing tasks inner another tasks.
The problem is that, when i run task, task must update and install soft in remote server which connects without password(with public key, don't ask password!).
Old version(1.14.post1):
from fabric.api import task, sudo, env, run
#task
def install_docker():
sudo('apt-get update && apt-get install -qy docker.io')
#task
def task_queue():
execute(install_docker)
execute(install_some_soft)
How I run it: fab task_queue
New version(2.5.0)
#task
def install_docker(ctx):
ctx.sudo('apt-get update && apt-get install -qy docker.io')
#task(pre=[install_docker])
def task_queue(ctx):
pass
How I run it: fab task-queue
And i'm hiving error like this:
invoke.exceptions.AuthFailure: The password submitted to prompt '[sudo] password: ' was rejected.
Use --prompt-for-sudo-password when you run fab.
I installed odoo 9, and each time i try to run it, this error appears:
raise ValueError('External ID not found in the system: %s' % (xmlid))
QWebTemplateNotFound: External ID not found in the system: web.login
All the modules in requirements.txt were installed including passlib,i also tried to clear history and kill servers. I've created the db role by following this steps:
sudo adduser odoo
sudo apt-get install postgresql
sudo su postgresql
createuser --createdb --username postgres --no-createrole --no-superuser -pwprompt odoo
so what did i miss?
updated
here's openerp-server.config file:
[options]
; This is the password that allows database operations:
; admin_passwd = admin
db_host = False
db_port = False
db_user = odoo
db_password = False
addons_path = /usr/lib/python2.7/dist-packages/openerp/addons
i also tried this command and it's not working:
python odoo.py --addons addons
and
./odoo.py --addons-path=.../Desktop/odoo/addons
I think your addons are not loading when you are starting the server.
What you can do is manually load addons when starting the server:
./odoo-bin --addons=addons/
To access odoo instance from a browser web module is required . So you have to load all the modules and load web module externally.
There are two ways for that:-
1.Load everything as an argument itself
./odoo-bin -d db_name -r db_user -w db_pass --load=web --addons=addons1,addons2
2.Provide a configuration file for above task like this
./odoo-bin -c config-file
This config-file will look like this:-
db_host = False
db_port = False
db_user = deepak
db_password = d33p#k
db_name = proj_db
addons_path = addons1,addons2
server_wide_modules = web
Here odoo-bin is the name of odoo executable for Odoo 10. Change it as per your Odoo version.
I fixed this by installing manualy web and base
./odoo-bin -i web,base
I have a nat and it has various server
So from my local server I want to go to nat and then from nat i have to ssh to other machines
Local-->NAT(abcuser#publicIP with key 1)-->server1(xyzuser#localIP with key 2)
nat has different ssh key
and each of the server has different ssh key
how can i accomplish this type of multihop ssh using fabric
I tried using env.roledefs feature but it doesnt seems to be working
also I am not sure how to define two ssh keys.I know we can define a list of keys with env.key_filename but issue is will it check each key with each server?How can I be more specific and match a key with one server only
I have tried using command from my local machine
fab deploy -g 'ec2-user#54.251.151.39' -i '/home/aman/Downloads/aws_oms.pem'
and my script is
from __future__ import with_statement
from fabric.api import local, run, cd, env, execute
env.hosts=['ubuntu#10.0.0.77']
env.key_filename=['/home/ec2-user/varnish_cache.pem']
def deploy():
run("uname -a")
It's possible. Double hop to 10.0.0.2 (and list files) via gateway hop 10.0.0.1. Basically, you simply nest the connections with the gateway parameter.
# coding: utf-8
from fabric import Connection
path = '/'
conn1 = Connection(host='user1#10.0.0.1', connect_kwargs={'password': '***'})
conn2 = Connection(host='user2#10.0.0.2', connect_kwargs={'password': '***'}, gateway=conn1)
result = conn2.run(f'''cd {path} && ls -al''', hide=True)
conn2.close()
conn1.close()
msg = "Ran {0.command!r} on {0.connection.host}, got stdout:\n{0.stdout}"
print(msg.format(result))
Please remember to run the SSH connection manually once to introduce the servers to each other!
Install via
pip3 install --upgrade fabric
pip3 install cryptography==2.4.2 # optional to hide some annoying warnings
http://docs.fabfile.org/en/latest/concepts/networking.html
Python 3.6+.
In order to connect to remote hosts via an intermediate server, you can use the --gateway command-line option :
http://docs.fabfile.org/en/latest/usage/fab.html#cmdoption-g
Or, alternatively, set the env.gateway variable inside your fabfile :
http://docs.fabfile.org/en/latest/usage/env.html#gateway
For more detail information, see:
http://docs.fabfile.org/en/stable/concepts/networking.html#ssh-gateways
You know that on EC2, there is no password associated with "ubuntu" user. With the following lines, if I try to run :
fab development install_dir
I get :
[ec2-46-51-132-252.eu-west-1.compute.amazonaws.com] sudo: chown -R webadmin:webadmin /var/www
[ec2-46-51-132-252.eu-west-1.compute.amazonaws.com] Login password:
I tried to add shell=False to sudo method (according to Can I prevent fabric from prompting me for a sudo password?), but it doesn't change anything
Any idea ? Thanks a lot !
def development():
env.envname = 'development'
env.user = 'ubuntu'
env.group = 'ubuntu'
env.chuser = 'webadmin'
env.chgroup = 'webadmin'
env.hosts = ['ec2-***.eu-west-1.compute.amazonaws.com']
env.envname_abriev = 'dev'
env.key_filename = '/home/xx/.ssh/xx.pem'
env.postgresql_version = '9.0'
def install_dir():
if not exists('/var/www'):
sudo('mkdir /var/www')
sudo('chown -R %s:%s /var/www' % (env.chuser, env.chgroup))
Download (or create) a keypair file from aws as shown below
Create a file called fabfile.py
and set its contents as follows:
from fabric.context_managers import cd
from fabric.operations import sudo
from fabric.api import run, env
import os
HOME = os.getenv('HOME')
env.user = 'ubuntu'
env.hosts = ['PUBLICDNS.ap-southeast-1.compute.amazonaws.com','ANOTHERSERVER.compute.amazonaws.com'] #can add multiple instances
env.key_filename = [
'%s/<your-keypair-file>.pem'%HOME
] #assuming keypair file is located in HOME
#example code we want to run on remote machine
def update():
with cd('/var/www'):
sudo('svn update')
with cd ('/var/www/cache'):
run('rm -rf *')
sudo('service lighttpd restart')
To run the file, type fab update in the terminal.
You need to specify keypair file name associated with your EC2 instance while running fab command.
Usage: fab [options] <command>[:arg1,arg2=val2,host=foo,hosts='h1;h2',...] ...
Options:
-R ROLES, --roles=ROLES
comma-separated list of roles to operate on
-i KEY_FILENAME path to SSH private key file. May be repeated.