Develop Reference

Test Python shell jobs scripts: dev endpoint?

I would like to set up a Python shell job as an ETL job. This job would do some basic data cleaning.
For now, I have a script that runs locally. I would like to test this script (or parts of it). I tried to setup a dev endpoint like it's explained here: https://docs.aws.amazon.com/glue/latest/dg/dev-endpoint-tutorial-repl.html
Everything went fine, I can ssh into the machine. But with this tutorial, I get a "gluepyspark" shell. And this shell isn't compatible with the AWS data wrangler library: https://github.com/awslabs/aws-data-wrangler.
I would like to know if it's possible to setup a dev endpoint to test python shell jobs. Alternatively, I could also accept a workflow to test python shell jobs.

How to run Python code periodically that connects to the internet

I'm working on a Python script that connects to the Twitter API to pull in some tweets into an array, then pushes this to a mysql database. It's a pretty basic script, but I'd like to set it up to run weekly.
I'd like to know the best way to deploy it so that it can automatically run weekly, so that I don't have to manually run it every week.

This depends on platform where you intend to run your python code. As martin says, this is not a python question and more of scheduling related question.

You can create a batch file that can activate python and run your script and then, use task scheduler to schedule youe batch file execution weekly

How to execute a specific code in a defined period -Django-

I would like to know how to automatically execute a code in Django within a defined period of time.
I'm building a web crawler that collects information and stores it in a JSON file and a function that reads the file and stores the file information in a SQLite database. This information is rendered and visible on my website. At the moment I have to run the crawler and the function that saves the data in a database with a click on a button, but that is very ineffective. It would be better if the database information were updated automatically, about every 6 hours (of course, only if the server is running).

If you want to keep the code on your webserver and you have the permissions then a CRON job (Linux) or Scheduled Task (Windows) will do what you want. Set your cron job to run every six hours and to call your Django script.
You can run the script as a Django command line, eg manage.py mycommand or run as a stand-alone PY file where you include the libs and run django.setup()
Or if you want the crawler to be run from within the context of your webserver then your cron job can initiate a GET request using HTTPIE. That gives you a nice API approach.
When I run this I have a flag in a database (a model called Job) that records what is running just in case my scheduled task is already running when the cron gets called, I have some very long-running asynchronous tasks and I tend to use cron rather than Celery.
If you are running your site on AWS or GCP then both have a cron equivalent. Eg you could create a GCP cloud scheduler link that fires off a GET to your webserver that triggers the code to run the crawler.

Django: Run a script right after runserver

Context:
I have a table on the database that uses values from an external database. This external database updates its values periodically.
Problem:
In order to update my database everytime i start the server, I want to run a script right after the runserver.
Potential Solution:
I have seen that it is possible to run a script from a certain app, which is something I'm interested in. This is achievable by using the django-extensions:
https://django-extensions.readthedocs.io/en/latest/runscript.html
However, this script only runs with the following command:
python manage.py runscript your_script
Is there any other way to run a script from an app and execute it right after the runserver command? I am open to suggestions!
Thanks in advance
Update
Thanks to #Raydel Miranda for the remarks, I feel i left some information behind.
My goal is, once I start the server I'm planning to open a socket to maintain my database updated.

You can execute the code in the top-level urls.py. That module is imported and executed once.
urls.py
from django.confs.urls.defaults import *
from your_script import one_time_startup_function
urlpatterns = ...
one_time_startup_function()

I would recommend to use something like this, lets say you have the script like this:
# abc.py
from your_app.models import do_something
do_something()
Now you can run this script right after runserver(or any other way you are running the django application) like this:
python manage.py runserver & python manage.py shell < abc.py
FYI, it will only work if you have bash in your terminal (like in ie Linux, MacOs).
Update
After reading you problem carefully, I think running a script after runserver might not be the best solution. As you said:
This external database updates its values periodically.
So, I think you need some sort of perodic task to do this update. You can use cronjob or you can use Celery for this.

Running the script after runserver don't seem a very good idea, the main reason is that you will have a window since the server is running (and available for users) till you finish synchronizing your data. Also if you synchronize using a script after runserver you won't get updates from the external db after that.
The best solution for this is to configure multiple databases, you can use the external database with only read access. This way your views will provide really updated data.
On the other hand ...
If want use something like a script is better to write a Django custom command (this way you don't have to deal with initializing django settings and other issues) and execute it using cron or celery as #ruddra states in his/her answer.
Said this, you should see this: https://docs.djangoproject.com/en/2.1/topics/db/multi-db/

This may help.
you can edit yourapp/apps.py
class MyAppConfig(AppConfig):
name = 'myapp'
def ready(self):
# update my database here
pass

Is it possible to manage model data without running the server?

Suppose I have a python module written to do some clean job and daily maintenance. It has no view or template but simply a command line tool. Is it possible to interact with the models and db regardless of whether the server is on?

Yes you can.
Look into the management commands shell and dbshell
You would just do
python manage.py shell #You can call any method, modify Model objects, ...
and
python manage.py dbshell #Gives direct access to the database via command line
And this does not need the server to be running.