Develop Reference

Elasticsearch Data Insertion with Python

I'm brand new to using the Elastic Stack so excuse my lack of knowledge on the subject. I'm running the Elastic Stack on a Windows 10, corporate work computer. I have Git Bash installed for a bash cli, and I can successfully launch the entire Elastic Stack. My task is to take log data that is stored in one of our databases and display it on a Kibana dashboard.
From what my team and I have reasoned, I don't need to use Logstash because the database that the logs are sent to is effectively our 'log stash', so to use the Logstash service would be redundant. I found this nifty diagram
on freecodecamp, and from what I gather, Logstash is just the intermediary for log retrieval different services. So instead of using Logstash, since the log data is already in a database, I could just do something like this
USER ---> KIBANA <---> ELASTICSEARCH <--- My Python Script <--- [DATABASE]
My python script successfully calls our database and retrieves the data, and a function that molds the data into a dict object (as I understand, Elasticsearch takes data in a JSON format).
Now I want to insert all of that data into Elasticsearch - I've been reading the Elastic docs, and there's a lot of talk about indexing that isn't really indexing, and I haven't found any API calls I can use to plug the data right into Elasticsearch. All of the documentation I've found so far concerns the use of Logstash, but since I'm not using Logstash, I'm kind of at a loss here.
If there's anyone who can help me out and point me in the right direction I'd appreciate it. Thanks
-Dan

You ingest data on elasticsearch using the Index API, it is basically a request using the PUT method.
To do that with Python you can use elasticsearch-py, the official python client for elasticsearch.
But sometimes what you need is easier to be done using Logstash, since it can extract the data from your database, format it using many filters and send to elasticsearch.

How to expose Python api for elasticsearch

I have inserted large amount of data(1 million) in EllasticSearch. Now i want to create a REST API to fetch the data from EllasticSearch.
I want to use CURL commands
(eg: curl -i http://localhost:5000/todo/api/v1.0/tasks/2)
for being able to get the json fields having _id=2
I found the following blog https://blog.miguelgrinberg.com/post/designing-a-restful-api-with-python-and-flask
that helped me on how to create REST API, but i am not able to understand how do i extend this for ElasticSearch.

The elasticsearch python API is very convenient to create any kind of operation (inserting or fetching). You can find the doc's here:
https://elasticsearch-py.readthedocs.io/en/master/
Just one hint, in my experience the python api tended to be slower then creating direct curl requests from the command line. Anyhow, it is very convenient to work with. A query is as easy as the following snippet.
from elasticsearch import Elasticsearch
es = Elasticsearch()
res = es.index(index="index-logstash")

How to I use the elasticsearch python api to get an overview of all snapshots?

I'm using the elasticsearch python api to communicate with my elasticsearch database. How can I make a specific GET request to get an overview of all the snapshots that have been created?
The Kibana command for this would be: GET /_snapshot/my_backup/_all.
It seems the Elasticsearch.get() function is only suited to retrieve documents.
I would rather not use the Requests module.
The snapshot helper functions I found only have the option to get an overview of snapshots that are currently running.
from elasticsearch import Elasticsearch
es = Elasticsearch()
es.snapshot.get_repository('my_backup') # configuration information
es.snapshot.status('my_backup') # currently running snapshots

I finally realized you can use the _all keyword when needing all snapshots, in the following way:
all_snapshots = es.snapshot.get(repository = 'my_backup', snapshot = '_all')

Just adding one of my own as this got me in the right path.
If you need to get the general snapshot status i.e if a snapshot is being run:
es_session.snapshot.status('_all')

python elasticsearch use transport client

I am newbie to elasticsearch, I know there is two official client elasticsearch supplies, but when I use the python elasticsearch, i can't find how to use the transport client..
I read the whole doc which is as following:
https://elasticsearch-py.readthedocs.io/en/master/index.html
I also search some docs, i can't find the way to use elasticsearch with python.also, in one doc, it says:
Using the native protocol from anything other than Java is not
recommended, as it would entail implementing a lot of custom
serialization.
does this mean python elasticsearch can't use transport client?

The transport client is written in Java, so the only way to use it from Python is by switching to Jython.

I think the previous answer is out of date now, if this is the transport client you mean.
I've made use of this API to do things like use the _rank_eval API, which is still considered "experimental" so hasn't made it into the official client yet.
def rank_eval(self, query, ratings, metric_name):
res = self.es.transport.perform_request(
"GET",
"/%s/_rank_eval" % INDEX,
body=self.rank_request(query, ratings, metric_name),
)
return res

How to obtain all the hits in elasticsearch for python?

I am currently trying to run a query using a python package for elasticsearch. However, whenever I call es.search(), I only get 10 results, when there should be more than 1M. Can anyone tell me how I can obtain all the hits?

Using the elasticsearch and elasticsearch-dsl libraries:
from elasticsearch import Elasticsearch
from elasticsearch_dsl import Search
client = Elasticsearch(host="localhost")
s = Search(using=client, index="my_index")
for hit in s.scan():
print(hit.title)
See the documentation about pagination.