We have python services running in pods in a kubernetes cluster. The services are setup to receive their log-level from an environment variable. Those env vars are set during the deployment of the service in a gitlab pipeline. For debugging purposes I want to be able to just change the env var on a single pod and restart it, without having to redeploy the service from gitlab.
Before we moved to kubernetes, we were running our containers in rancher, where the described change was very easy to do in the GUI. Change the env var -> hit update -> container restarts automatically.
I found this article that suggest to change the replica set using a command like
kubectl set env rs [REPLICASET_NAME] [ENV_VAR]=[VALUE]
And then terminating the pod, after which it will be recreated with the env var set accordingly.
But it also states to
Never do it on a production system.
Never even do this on a dev environment without taking care in how it may impact your deployment workflow.
Is that the only / best way to achieve my goal of quickly changing an env var in a running pod for debug purposes?
Is that the only / best way to achieve my goal of quickly changing an
env var in a running pod for debug purposes?
Short answer: Yes.
Long answer: I've never used or read up on Rancher, but I suspect that it was also changing the ReplicaSet or Deployment template env var, which triggered a Pod update. It's really the only way to change an env var in a Pod. You can't change the env vars on a running container or a running Pod. You can't do that in Docker containers, and you can't do it in Kubernetes, so I assume that you can't do it in Rancher. You can only restart a Pod with a different spec.
Why?
Because containers are just processes running on the host machine. Once the process is started, it's not possible to change a process's environment without resorting to nasty hacks.
If you're just concerned about the warnings that state to not do this in dev or prod, I would say that the same warnings apply to the Rancher workflow you described, so if you were willing to take the risks there, it won't be any different here.
Something I do frequently to define my environment variables in the deployment spec. Then while the deployment is running I am able to just do
kubectl edit deployment <name>
and change the environment variables that I want this will restart the pod though but for my development purposes it's typically okay.
If the environment variable is baked into the image though then you will need to either rebuild the image and restart the pod (which will pull the image) or use some of the suggestions others have stated here.
Related
So currently I am building a docker image for my python flask server, and the way that I am accessing the secret variable inside the code is
app.config["SECRET_KEY"] = os.environ.get("todo_secret_key") or "secret"
Now if I want to run the code without the container, I'd use export command(linux) to temporarily have the secret key in the environment, but I need it to be in the environment of the container.
Now, a few methods that I am aware of are
Pass it with -e in the docker run command docker run {imageName} -e {name}={value} (insecure as I dont want it to be in terminal logs)
Pass it in the dockerfile by specifying the environment variable (definitely insecure as this file would be public)
Apart from these methods, is there a more secure way to pass the variable
P.s It is my first time buidling an image so apologies if it is a silly question
You're trying to tie your runtime configuration to your build time configuration much too tightly if you're worrying about this as you're building your docker image!
Remember that these are discrete, if interconnected, stages.
Develop -> Build -> Test -> Deploy -> Test (Non-functional) -> repeat...
No stage should dictate what future stages do. docker build is the "Build" stage, and so should have no idea what configuration is correct in the environment for when the image is eventually deployed. How could it? That configuration could change over time!
Instead, look to configuration management to handle these sorts of decisions. Your docker scheduler (Kubernetes is common, but other flyweight executors like Docker Swarm or ECS should do this too) should have a way to read a configuration file and inject it into the environment. A full discussion of configuration management could (and has) filled textbooks.
I know that as of now (june 21) PyCharm does not support a fully remote development the way VSCode does; however, it supports a remote interpreter and in the case of ssh connection, it supports an automatic upload of local files to the server.
My question is: can I use an interpreter running in docker container and synchronize my files with the files within the container?
Apparently, I can define a remote docker interpreter and set up a "Mapping":
However, I don't see any way how to automatically upload my local files / changes to the container. When I try to configure a new deployment, I don't see any docker option:
Am I missing something, or is it like PyCharm does not support such "Docker deployment"?
(This might be related, but not a clear conclusion there...)
(UPDATE: after studying more, I'm not even sure if I can use interpreter in a docker container. It seems that pycharm only enables a docker image, and anything I would do inside docker... is gone in the next session. Or am I wrong?)
I have a server and a front end, I would like to get python code from the user in the front end and execute it securely on the backend.
I read this article explaining the problematic aspect of each approach - eg pypl sandbox, docker etc.
I can define the input the code needs, and the output, and can put all of them in a directory, what's important to me is that this code should not be able to hurt my filesystem, and create a disk, memory, cpu overflow (I need to be able to set timeouts, and prevent it from accesing files that are not devoted to it)
What is the best practice here? Docker? Kubernetes? Is there a python module for that?
Thanks.
You can have a python docker container with the volume mount. Volume mount will be the directory in local system for code which will be available in your docker container also. By doing this you have isolated user supplied code only to the container when it runs.
Scan your python container with CIS benchmark for better security
Similar question was asked here, however the solution does not give the shell access to the same environment as the deployment. If I inspect os.environ from within the shell, none of the environment variables appear.
Is there a way to run the manage.py shell with the environment?
PS: As a little side question, I know the mantra for EBS is to stop using eb ssh, but then how would you run one-off management scripts (that you don't want to run on every deploy)?
One of the cases you have to run something once is db schema migrations. Usually you store information about that in the db... So you can use db to sync / ensure that something was triggered only once.
Personally I have nothing against using eb ssh, I see problems with it however. If you want to have CI/CD, that manual operation is against the rules.
Looks like you are referring to WWW/API part of Beanstalk. If you need something that is quite frequent... maybe worker is more suitable? Problem here is that if API goes deployed first you would have wrong schema.
In general you are using EC2, so it's user data stores information that spins up you service. So there you can put your "stuff". Still you need to sync / ensure. Here are docs for beanstalk - for more information how to do that.
Edit
Beanstalk is kind of instrumentation on top of EC2. So there must be a way to work with it, since you have access to user data of that EC2s. No worries you don't need to dig that deep. There is good way of instrumenting your server. It is called ebextensions. It can be used to put files on the server, trigger commands, instrument cron. What ever you want.
You can create ebextension, with container_commands this time Python Configuration Namespaces section. That commands are executed on each deployment. Still, problem is that you need to sync since more then one deployment can go at the same time. Good part is that you can set env in the way you want.
I have no problem to access to the environment variables. How did you get the problem? Try do prepare page with the map.
How should I implement continuous integration on my new application? Currently, this is how we're pushing to production - please bear with me, I know this is far from sane:
From local, git push origin production (the production codebase is kept on the production branch, modifications are either written directly there and committed, or files are checked out individually from another branch. Origin is the remote production server)
On the remote box, sudo stop gunicorn (application is running as a process)
cp ~/flaskgit/application.py ~/flask/applicaion.py (the git push origin from local pushes to an init -bare repo with a post-update hook that populates the files in ~/flaskgit. ~/flask is where the gunicorn service runs the application under a virtualenv)
sudo start gunicorn
we do our testing with the ~/flaskgit code running on a different port. once it looks good we do the CP
I would love to have something more fluid. I have used jenkins in the past, and loved the experience - but didn't set it up.
What resources / utilities should I look up in order to do this well?
Thank you!
buildbot, jenkins/hudson but these give you continuous integration in the sense you can run a "make" equivalent with every code base change through a commit hook. You could also look at vagrant if there is something there for you for creating repeatable vm's wrt to config/setup. Could tie it with a commit hook.