ievv_elasticsearch — Thin wrapper around pyelasticsearch

The ievv_elasticsearch module is designed to just make it a small bit easier to work with elasticsearch in Django than to just use pyelasticsearch.

Features

• Makes pyelasticsearch easy to use in Django.
• Very thin wrapper around pyelasticsearch. This means that you use the elasticsearch REST API almost directly with just some pythonic glue.
• Automatic indexing of data store data is decoupled from the search/get API.
• Automatic indexing of data store data works with any data store. Our examples use Django ORM, but you can just as easily use a NOSQL database like MongoDB or an XML database. The only difference is the code you provide to convert your data store data into ElasticSearch JSON compatible data structures.
• Small Django/python helpers that enhances the pyelasticsearch API.
• Shortcuts and solutions that makes unit testing easy.

Why not use Haystack?

Haystack is an awesome library for full text document search in a search engine independent manner. But if you want to make full use of ElasticSearch as more than just a document search index, and use it for analytics, document caching and a general purpose nosql data store, haystack just adds an extra layer of complexity that you have to work around. This is, of course, use-case dependent, and many use cases will probably be better served by a combination of ievv_elasticsearch and Haystack.

Note

If considering combining Haystack with ievv_elasticsearch, you should know that ievv_elasticsearch has loose coupling between index definition and querying. Indexe definitions in ievv_elasticsearch are only used to make it easy to sync the backend data store into an elasticseach index. If you define the search indexes in haystack, you can still use the ievv_opensource.ievv_elasticsearch.search API, you just ignore ievv_opensource.ievv_elasticsearch.autoindex.

Getting started

You only need the following to get started:

• ElasticSearch server.
• Configure IEVV_ELASTICSEARCH_URL with the URL of the server.

Then you can start using the ievv_opensource.ievv_elasticsearch.search.Connection API.

Setup for unit-testing and development

First, copy not_for_deploy/elasticsearch.develop.yml and not_for_deploy/elasticsearch.unittest.yml into your own project.

In your test settings, add:

IEVV_ELASTICSEARCH_TESTURL = 'http://localhost:9251'
IEVV_ELASTICSEARCH_TESTMODE = True
IEVV_ELASTICSEARCH_AUTOREFRESH_AFTER_INDEXING = True

In your develop settings, add:

IEVV_ELASTICSEARCH_URL = 'http://localhost:9252'

When this is configured, you can run elasticsearch with ievv devrun — All your development servers in one command if you add the following to IEVVTASKS_DEVRUN_RUNNABLES:

IEVVTASKS_DEVRUN_RUNNABLES = {
    'default': ievvdevrun.config.RunnableThreadList(
        # ...
        ievvdevrun.runnables.elasticsearch.RunnableThread(configpath='not_for_deploy/elasticsearch.unittest.yml'),
        ievvdevrun.runnables.elasticsearch.RunnableThread(configpath='not_for_deploy/elasticsearch.develop.yml'),
    ),
]

(the paths assumes you put the configfiles in the not_for_deploy/ directory in your project).

Automatically update the search indexes

Unless you use ElasticSearch as the primary data source, you will most likely want an easy method of: 1. Update the search index when data in the data store changes. 2. Rebuild the search index from the data in the data store.

This is solved by:

1. Define a ievv_opensource.ievv_elasticsearch.autoindex.AbstractIndex. The ievv_elasticsearch convention is to put search indexes in yourapp.elasticsearch_autoindexes.
2. Register the index class in ievv_opensource.ievv_elasticsearch.autoindex.Registry.
3. Optionally override ievv_opensource.ievv_elasticsearch.autoindex.AbstractIndex.register_index_update_triggers(). and register triggers that react to data store changes and trigger re-indexing.

search API

ievv_opensource.ievv_elasticsearch.search.Connection	Singleton wrapper around pyelasticsearch.ElasticSearch.
ievv_opensource.ievv_elasticsearch.search.SearchResultWrapper	An efficient wrapper around the data returned by pyelasticsearch.ElasticSearch.search().
ievv_opensource.ievv_elasticsearch.search.SearchResultItem	Wrapper around a single dictionary in the hits.hits list of the result returned by pyelasticsearch.ElasticSearch.search().
ievv_opensource.ievv_elasticsearch.search.Paginator	Paginator for ievv_opensource.ievv_elasticsearch.search.SearchResultWrapper.

class ievv_opensource.ievv_elasticsearch.search.IevvElasticSearch(urls='http://localhost', timeout=60, max_retries=0, port=9200, username=None, password=None, ca_certs='/home/docs/checkouts/readthedocs.org/user_builds/ievv-opensource/envs/stable/lib/python3.5/site-packages/certifi/cacert.pem', client_cert=None)

Bases: pyelasticsearch.client.ElasticSearch

Parameters:

• urls – A URL or iterable of URLs of ES nodes. These can be full URLs with port numbers, like http://elasticsearch.example.com:9200, or you can pass the port separately using the port kwarg. To do HTTP basic authentication, you can use RFC-2617-style URLs like http://someuser:somepassword@example.com:9200 or the separate username and password kwargs below.
• timeout – Number of seconds to wait for each request before raising Timeout
• max_retries – How many other servers to try, in series, after a request times out or a connection fails
• username – Authentication username to send via HTTP basic auth
• password – Password to use in HTTP basic auth. If a username and password are embedded in a URL, those are favored.
• port – The default port to connect on, for URLs that don’t include an explicit port
• ca_certs – A path to a bundle of CA certificates to trust. The default is to use Mozilla’s bundle, the same one used by Firefox.
• client_cert – A certificate to authenticate the client to the server

send_request(method, path_components, body='', query_params=None)

Does exactly the same as the method from the superclass, but also prettyprints the request and response if the IEVV_ELASTICSEARCH_PRETTYPRINT_ALL_REQUESTS setting is True.

class ievv_opensource.ievv_elasticsearch.search.Connection

Bases: ievv_opensource.utils.singleton.Singleton

Singleton wrapper around pyelasticsearch.ElasticSearch.

We do not try to wrap everything, instead we use the pyelasticsearch API as it is, and add extra features that makes it easier to use with IEVV and Django. We provide shortcuts for the most commonly used methods of pyelasticsearch.ElasticSearch, some custom methods and we add some code to make unit testing easier.

Usage:

from ievv_opensource.ievv_elasticsearch import search

searchapi = search.Connection.get_instance()
searchapi.bulk_index(
    index='contacts',
    doc_type='person',
    docs=[{'name': 'Joe Tester'},
          {'name': 'Peter The Super Tester'}])
searchresult1 = searchapi.wrapped_search(query='name:joe OR name:freddy', index='contacts')
searchresult2 = searchapi.wrapped_search(query={
    'query': {
        'match': {
            'name': {
                'query': 'Joe'
            }
        }
    }
})
print(searchresult1.total)
for item in searchresult1:
    print(item.doc['name'])

elasticsearch

The pyelasticsearch.ElasticSearch object.

clear_all_data()

Clear all data from ElasticSearch. Perfect for unit tests.

Only allowed when the IEVV_ELASTICSEARCH_TESTMODE-setting is True.

Usage:

class MyTest(TestCase):
    def setUp(self):
        self.searchapi = search.Connection.get_instance()
        self.searchapi.clear_all_data()

index(*args, **kwargs)

Wrapper around pyelasticsearch.ElasticSearch.index().

Works exactly like the wrapped function, except that we provide some extra features that makes testing easier. When the IEVV_ELASTICSEARCH_TESTMODE-setting is True, we automatically run pyelasticsearch.ElasticSearch.refresh() before returning.

bulk_index(*args, **kwargs)

Wrapper around pyelasticsearch.ElasticSearch.bulk_index().

Works exactly like the wrapped function, except that we provide some extra features that makes testing easier. When the IEVV_ELASTICSEARCH_TESTMODE-setting is True, we automatically run pyelasticsearch.ElasticSearch.refresh() before returning.

bulk(*args, **kwargs)

Wrapper around pyelasticsearch.ElasticSearch.bulk().

Works exactly like the wrapped function, except that we provide some extra features that makes testing easier. When the IEVV_ELASTICSEARCH_TESTMODE-setting is True, we automatically run pyelasticsearch.ElasticSearch.refresh() before returning.

search(query, prettyprint_query=False, **kwargs)

Wrapper around pyelasticsearch.ElasticSearch.search().

Works just like the wrapped function, except that query can also be an elasticsearch_dsl.Search object, and you can only specify arguments as kwargs (no positional arguments).

If query is an elasticsearch_dsl.Search object, we convert it to a dict with query.to_dict before forwaring it to the underling pyelasticsearch API.

Parameters:

• query – A string, dict or elasticsearch_dsl.Search object.
• prettyprint_query – If this is True, we prettyprint the query before executing it. Good for debugging.

refresh(*args, **kwargs)

Wrapper around pyelasticsearch.ElasticSearch.refresh().

Works exactly like the wrapped function.

delete_index(*args, **kwargs)

Wrapper around pyelasticsearch.ElasticSearch.delete_index().

Works exactly like the wrapped function.

delete(*args, **kwargs)

Wrapper around pyelasticsearch.ElasticSearch.delete().

Works exactly like the wrapped function.

get(*args, **kwargs)

Wrapper around pyelasticsearch.ElasticSearch.get().

Works exactly like the wrapped function.

wrapped_get(*args, **kwargs)

Just like get(), but we return a SearchResultItem instead of the raw search response.

get_or_none(*args, **kwargs)

Works like get(), but instead of raising an exception, we return None if the requested object does not exist.

wrapped_get_or_none(*args, **kwargs)

Works like wrapped_get(), but instead of raising an exception, we return None if the requested object does not exist.

wrapped_search(*args, **kwargs)

Just like search(), but we return a SearchResultWrapper instead of the raw search response.

paginated_search(query, page_number=0, page_size=100, resultitemwrapper=None, **kwargs)

Performs a search much like wrapped_search(), but limit the size of the result to page_size, and start the results at page_number * page_size.

Parameters:

• page_number – The page number to retrieve.
• page_size – The size of each page.
• query – A query dict for pyelasticsearch.ElasticSearch.search(). We add the size and from keys to this dict (calculated from page_number and page_size.
• resultitemwrapper – Forwarded to Paginator.
• kwargs – Forwarded to wrapped_search() alon with query.

Returns:

The search results wrapped in a Paginator.

Return type:

Paginator

search_all(**kwargs)

Get all documents in the index. Nice for testing and debugging of small datasets. Useless in production.

**kwargs are forwarded to search(), but the query argument is added automatically.

wrapped_search_all(**kwargs)

Just like search_all(), but wraps the results in a SearchResultWrapper.

class ievv_opensource.ievv_elasticsearch.search.SearchResultItem(search_hit)

Bases: object

Wrapper around a single dictionary in the hits.hits list of the result returned by pyelasticsearch.ElasticSearch.search().

id

Returns the value of the _id key of the search hit.

index

Returns the value of the _index key of the search hit.

score

Returns the value of the _score key of the search hit.

source

Returns the value of the _source key of the search hit.

doc_type

Returns the value of the _type key of the search hit.

class ievv_opensource.ievv_elasticsearch.search.SearchResultWrapper(searchresult)

Bases: object

An efficient wrapper around the data returned by pyelasticsearch.ElasticSearch.search().

Parameters:searchresult – Data returned by pyelasticsearch.ElasticSearch.search().

total

Returns the total number of hits.

retrieved_hits_count

Returns the number of retrieved hits.

first()

Shortcut for getting the first search result as a SearchResultItem.

class ievv_opensource.ievv_elasticsearch.search.Paginator(searchresultwrapper, page_number, page_size=100, resultitemwrapper=None)

Bases: object

Paginator for ievv_opensource.ievv_elasticsearch.search.SearchResultWrapper.

The paginator counts the first page as 0, the second as 1, and so on.

Parameters:

• searchresultwrapper – A ievv_opensource.ievv_elasticsearch.search.SearchResultWrapper.
• page_number – The current page number.
• page_size – Number of items per page.
• resultitemwrapper – A class/callable that takes a single item in the searchresultwrapper and does something with it before returning it when iterating the search result. Defaults to just returning the item as returned from searchresultwrapper.__iter__.

total_items

Returns the number of items in total in all pages.

number_of_items_in_current_page

Returns the number of items in the current page.

get_page_startindex(pagenumber)

Get the start index of the given pagenumber.

get_current_page_startindex()

Get the start index of the current page.

page_has_content(pagenumber)

Check if the given pagenumber is within the total number of items in the given searchresultwrapper.

Returns:A boolean.

current_page_has_content()

Check if current page is within the total number of items in the given searchresultwrapper.

Returns:A boolean.

has_next()

Check if we have a next page. Checks if the start index of the next page is lower than searchresultwrapper.total.

Returns:A boolean.

has_previous()

Check if we have a previous page. Checks if the start index of the previous page is larger than 0.

Returns:A boolean.

autoindex API

ievv_opensource.ievv_elasticsearch.autoindex.AbstractIndex	Base class for describing a search index.
ievv_opensource.ievv_elasticsearch.autoindex.AbstractDocument	Base class for indexable documents for AbstractIndex.
ievv_opensource.ievv_elasticsearch.autoindex.AbstractDictDocument	Extends AbstractDocument to make it easy to put dicts in the database.
ievv_opensource.ievv_elasticsearch.autoindex.Registry	Registry of AbstractIndex objects.
ievv_opensource.ievv_elasticsearch.autoindex.MockableRegistry	A non-singleton version of Registry.

This module defines a Registry of objects that takes care of automatically updating the search index when we detect changes to the data in a data store. The data store can be anything you like (Django ORM, MongoDB, …) - our examples use Django ORM.

This is completely decoupled from the ievv_opensource.ievv_elasticsearch.search API.

class ievv_opensource.ievv_elasticsearch.autoindex.AbstractDocumentMeta

Bases: type

Metaclass for AbstractDocument.

class ievv_opensource.ievv_elasticsearch.autoindex.AbstractDocument

Bases: object

Base class for indexable documents for AbstractIndex.

doc_type = None

The document type to store this as in the index.

index_name = None

The name of the index this document belongs to. This is set by AbstractIndex __init__ using set_index_name_for_all_document_classes().

get_document()

Get document for the doc argument of pyelasticsearch.ElasticSearch.index_op().

get_id()

Get the ID to use for the indexed document. Defaults to None, which means that a new document will be added to the index.

get_parent_id()

Get the parent-child mapping parent ID document to use for the indexed document. This should only be overridden if you have a parent specified

Defaults to None, which means that no parent will be sent during indexing operations.

get_index_op_kwargs()

Get kwargs for pyelasticsearch.ElasticSearch.index_op().

You should not need to override this. Override get_document(), get_meta() and doc_type.

classmethod get_mapping_properties()

Get the mapping properties for custom mappings for this document type. You only need to specify those mappings you do not want elasticsearch to create automatically.

If you do not have any mappings, return None (or do not override).

Examples

Simple example:

class MyDocument(autoindex.AbstractDocument):
    @classmethod
    def get_mapping_properties(cls):
        return {
            'slug': {
                'type': 'string',
                'index': 'not_analyzed'
            },
            'author': {
                'username': {
                    'type': 'string',
                    'index': 'not_analyzed'
                }
            }
        }

classmethod get_mapping_parent_type()

Get the type of the parent document for parent-child mapping.

Lets say you have a Movie document, and want to create a parent-child relationship from the Category document with doc_type category to the Movie. In the Movie document class, you would have to:

• Override this method and return "category".
• get_parent_id() and return the ID of the category.

class ievv_opensource.ievv_elasticsearch.autoindex.AbstractDictDocument(document, id)

Bases: ievv_opensource.ievv_elasticsearch.autoindex.AbstractDocument

Extends AbstractDocument to make it easy to put dicts in the database.

Parameters:

• document – A dict that pyelasticsearch can convert to JSON.
• id – The ElasticSearch id of the document. Set to None to autocreate one.

get_document()

Get document for the doc argument of pyelasticsearch.ElasticSearch.index_op().

get_id()

Get the ID to use for the indexed document. Defaults to None, which means that a new document will be added to the index.

class ievv_opensource.ievv_elasticsearch.autoindex.AbstractIndex

Bases: object

Base class for describing a search index.

To register an index:

1. Create a subclass of AbstractIndex and implement iterate_all_documents() and override document_classes.
2. Register the index with Registry.

Examples

Minimal implementation for indexing a Django Product model:

from ievv_opensource.ievv_elasticsearch import searchindex

class ProductDocument(searchindex.AbstractDictDocument):
    doc_type = 'product'

class ProductIndex(searchindex.AbstractIndex):
    name = 'products'
    document_classes = [
        ProductDocument
    ]

    def iterate_all_documents(self):
        for product in Product.objects.iterator():
            yield ProductDocument({
                'name': product.name,
                'price': product.price
            }, id=product.pk)

If you want a more general search index of sellable items, you could do something like this:

from ievv_opensource.ievv_elasticsearch import searchindex

class ProductDocument(searchindex.AbstractDictDocument):
    doc_type = 'product'

class ServiceDocument(searchindex.AbstractDictDocument):
    doc_type = 'service'

class SellableItemIndex(searchindex.AbstractIndex):
    name = 'sellableitems'

    def iterate_all_documents(self):
        for product in Product.objects.iterator():
            yield ProductDocument({
                'name': product.name,
                'price': product.price,
                'quantity': product.quantity
            }, id=product.pk)
        for service in Service.objects.iterator():
            yield ServiceDocument({
                'name': service.name,
                'price': service.price,
            }, id=service.pk)

You could also move the document creation into the index document classes like this:

class ProductDocument(searchindex.AbstractDictDocument):
    doc_type = 'product'

    def __init__(self, product):
        self.product = product

    def get_id(self):
        return self.product.id

    def get_document(self):
       return {
            'name': self.product.name,
            'price': self.product.price,
            'quantity': self.product.quantity
       }

class SellableItemIndex(searchindex.AbstractIndex):
    # ... same as above

    def iterate_all_documents(self):
        for product in Product.objects.iterator():
            yield ProductDocument(product)
        # ...

name = None

The name of the index. Must be set in subclasses.

bulk_index_docs_per_chunk = 500

The number of docs to index per chunk when bulk updating the index.

bulk_index_bytes_per_chunk = 10000

The number of bytes to index per chunk when bulk updating the index.

document_classes = []

The AbstractDocument classes used in this index. Can also be overridden via get_document_classes().

set_index_name_for_all_document_classes()

Called by __init__ to set the AbstractDocument.index_name of all documents in document_classes.

create()

Create the index and put any custom mappings.

You should not need to override this, instead you should override get_document_classes() (and AbstractDocument.get_mapping_properties()), and get_settings().

get_settings()

Override this to provide settings for pyelasticsearch.ElasticSearch.create_index() (which is called by create().

get_document_classes()

Returns an iterable of the AbstractDocument classes used in this index. Defaults to document_classes.

get_document_classes_for_mapping()

Get the document classes for mapping. You normally do not have to override this - it only return get_document_classes() reversed. It is reversed because parent-child mappings have to be created in the child before the parent mapping can be created, but you normally want to index parents before children.

create_mappings()

Create mappings.

You should not need to override this, but instead you should override get_document_classes() (and AbstractDocument.get_mapping_properties()).

iterate_all_documents()

Iterate over all documents returning documents that are ready to be added to the index.

Returns:An iterable of AbstractDocument.

iterate_important_documents()

Just like iterate_all_documents(), but just yield the most important documents in case of a complete search index wipeout/rebuild.

This is typically the newest and most important documents in the database.

Defaults to returning an empty list.

index_items(index_documents)

Index the given index_documents.

Iterates over the given index_documents, and send documents to ievv_opensource.ievv_elasticsearch.search.Connection.bulk() in batches of IEVV_ELASTICSEARCH_INDEX_BATCH_SIZE index_documents.

Parameters:index_documents – An iterable of AbstractDocument.

register_index_update_triggers()

Override this to register behaviors that trigger updates to the index. This is typically something like this:

• Register one or more post_save signals that updates the index in realtime (be very careful with this since it can easily become a bottleneck).
• Register one or more post_save signals that updates the index via a Celery job or some other background queue.

Does nothing by default, so it is up to you to override it if you want to register any triggers.

delete_index()

Delete this index.

classmethod get_instance()

Get an instance of this class.

Use this instead of instanciating the class directly.

rebuild_index()

Rebuild this index completely.

Very useful when writing tests, but probably a bit less than optimal in production code/batch tasks unless you have a really small index. In production you should most likely want to create a management command to rebuild the index with the most recent/most important documents beeing indexed first.

class ievv_opensource.ievv_elasticsearch.autoindex.Registry

Bases: ievv_opensource.utils.singleton.Singleton

Registry of AbstractIndex objects.

Examples

First, define an index (see AbstractIndex).

Register the searchindex with the searchindex registry via an AppConfig for your Django app:

from django.apps import AppConfig
from ievv_opensource.ievv_elasticsearch import searchindex

from myapp import elasticsearch_indexes


class MyAppConfig(AppConfig):
    name = 'myapp'

    def ready(self):
        searchindex.Registry.get_instance().add(elasticsearch_indexes.SellableItemIndex)

add(searchindex_class)

Add the given searchindex_class to the registry.

get(indexname)

Get the index named indexname.

Returns: An AbstractIndex or None if no index matching

the given indexname is found.

get_indexnames()

Get a view with the names of all indexes.

class ievv_opensource.ievv_elasticsearch.autoindex.MockableRegistry

Bases: ievv_opensource.ievv_elasticsearch.autoindex.Registry

A non-singleton version of Registry. For tests.

Typical usage in a test:

class MockSearchIndex(searchindex.AbstractIndex):
    name = 'myindex'
    # ...

mockregistry = searchindex.MockableRegistry()
mockregistry.add(searchindex.MockSearchIndex())

with mock.patch('ievv_opensource.ievv_elasticsearch.searchindex.Registry.get_instance',
                lambda: mockregistry):
    pass  # ... your code here ...

jsondecode API

Utilities for decoding the JSON returned by ElasticSearch.

ievv_opensource.ievv_elasticsearch.jsondecode.datetime(datetimestring)

Convert a datetime object from ElasticSearch (iso format) into a timezone-aware datetime.datetime object.

viewhelpers API

ievv_opensource.ievv_elasticsearch.viewhelpers.searchview.ViewMixin	Makes it a bit easier to search with paging.
ievv_opensource.ievv_elasticsearch.viewhelpers.searchview.View	A Django TemplateView with ViewMixin.
ievv_opensource.ievv_elasticsearch.viewhelpers.searchview.SortMixin	Mixin class for sort-keyword in the querystring based sort (E.g.: ?o=name).
ievv_opensource.ievv_elasticsearch.viewhelpers.searchview.SearchMixin	Mixin class that makes it slightly easier to add search via a querystring attribute (defaults to ?s=<search_string>).

class ievv_opensource.ievv_elasticsearch.viewhelpers.searchview.ViewMixin

Bases: object

Makes it a bit easier to search with paging.

Examples

Minimal example:

class MyView(TemplateView, searchview.ViewMixin):
    template_name = 'myapp/mytemplate.html'

    def get_search_query(self):
        return {
            'match_all': {}
        }

    def get_context_data(self, **kwargs):
        context = super().get_context_data(**kwargs)
        context['searchpaginator'] = self.get_paginator()

A more full featured example:

class MyView(TemplateView, searchview.ViewMixin):
    template_name = 'myapp/mytemplate.html'
    page_size = 30
    paging_querystring_attribute = 'page'

    def get_search_query(self):
        return {
            'match_all': {}
        }

    def get_search_sort(self):
        return {'name': {'order': 'asc'}}

    def get_context_data(self, **kwargs):
        context = super().get_context_data(**kwargs)
        context['searchpaginator'] = self.get_paginator()

You should normally just need to override:

• get_search_query().
• get_search_sort() (optional).
• get_page_size() or page_size (optional).
• get_resultitemwrapper() (optional).
• get_paging_querystring_attribute() or paging_querystring_attribute (optional).

And call get_paginator() to retrieve results that you can iterate and ask for pagination information.

page_size = 100

The number of items per page. Defaults to 100. See get_page_size().

search_index = None

See get_search_index(). Defaults to None.

search_doc_type = None

See get_search_doc_type(). Defaults to None.

paging_querystring_attribute = 'p'

The querystring attribute to use for paging. Defaults to p. Used by get_paging_querystring_attribute().

get_page_size()

Get the page size.

Defaults to returning page_size.

get_paging_querystring_attribute()

The querystring attribute to use for paging. Defaults to paging_querystring_attribute.

get_current_page_number()

Get the current page number from request.GET[self.get_paging_querystring_attribute()].

You can override this if you want to get the page number some other way.

get_search_index()

Get the search index name.

Defaults to search_index.

get_search_doc_type()

Get the document types to search. Can be a single document type or an iterable of document types.

Defaults to search_doc_type.

get_search_query()

Get the query attribute of the elasticsearch query.

You MUST override this.

While get_search_full_query() returns something like:

{
    'query': {
        'match_all': {}
    },
    'size': 20,
    'from': 40
}

This method should only return the value of the query key:

{
    'match_all': {}
}

get_search_sort()

Get the sort dict for the search query.

While get_search_full_query() returns something like:

{
    'query': {
        'match_all': {}
    },
    'sort': {'name': {'order': 'asc'}},
    'size': 20,
    'from': 40
}

This method should only return the value of the sort key:

{'name': {'order': 'asc'}}

Defaults to None, which means no sorting is performed.

get_search_full_query()

Builds the full ElasticSearch query dict including paging.

You should normally not override this directly. Override get_search_query() and get_search_sort() instead.

get_paginated_search_kwargs()

Get the kwargs for ievv_opensource.ievv_elasticsearch.search.Connection#paginated_search().

get_resultitemwrapper()

See the resultitemwrapper argument for ievv_opensource.ievv_elasticsearch.search.Paginator.

get_paginator()

Performs the search and wraps it in a ievv_opensource.ievv_elasticsearch.search.Paginator.

Raises:

• django.http.response.Http404 if the search does not match
• any items. You will typically catch this exception and
• show a message in a normal search setting.

class ievv_opensource.ievv_elasticsearch.viewhelpers.searchview.View(**kwargs)

Bases: django.views.generic.base.TemplateView, ievv_opensource.ievv_elasticsearch.viewhelpers.searchview.ViewMixin

A Django TemplateView with ViewMixin.

For usage example, see ViewMixin (just inherit from this class instead of TemplateView and ViewMixin)

Constructor. Called in the URLconf; can contain helpful extra keyword arguments, and other things.

class ievv_opensource.ievv_elasticsearch.viewhelpers.searchview.SortMixin

Bases: object

Mixin class for sort-keyword in the querystring based sort (E.g.: ?o=name).

This MUST be mixed in before View (or ViewMixin), since it overrides ViewMixin.get_search_sort().

Examples

Simple example of a map where ?o=name sorts by name ascending and ?o=created sorts by created datetime descending:

class MySortView(searchview.SortMixin, searchview.View):
    default_sort_keyword = 'name'

    sort_map = {
        'name': {'name': {'order': 'asc'}},
        'created': {'created_datetime': {'order': 'desc'}},
    }

    def get_search_query(self):
        return {
            'match_all': {}
        }

    def get_context_data(self, **kwargs):
        context = super().get_context_data(**kwargs)
        context['searchpaginator'] = self.get_paginator()

The default if ?o is not specified will be to sort by name ascending.

You should normally just need to override:

• get_sort_map() or sort_map
• get_default_sort_keyword() or default_sort_keyword.

If you do not get the sort keyword from the querystring, you also need to override get_sort_keyword().

If you do not want to use o as the querystring attribute for sort keywords, you need to override get_sort_querystring_attribute() or sort_querystring_attribute.

sort_querystring_attribute = 'o'

The querystring attribute to use for sort. Used by get_sort_querystring_attribute(). Defaults to o (for ordering). We use o instead of s to avoid collision with SearchMixin.

default_sort_keyword = 'default'

The default sort keyword to use when ordering is not specified in the querystring. Defaults to "default".

sort_map = {}

See get_sort_map().

get_sort_querystring_attribute()

The querystring attribute to use for sort. Defaults to sort_querystring_attribute.

get_default_sort_keyword()

Get the default sort keyword to use when ordering is not specified in the querystring. Defaults to default_sort_keyword.

get_sort_keyword()

Get the sort keyword. Defaults to getting it from the querystring argument defined by get_sort_querystring_attribute(), and falling back to get_default_sort_keyword().

get_sort_map()

Get a mapping object that maps keywords to sort dicts compatible with elasticsearch. Defaults to sort_map.

get_search_sort_by_keyword(keyword)

Get the elasticsearch sort dict by looking up the given keyword in get_sort_map().

If the given keyword is not in get_sort_map(), we fall back on get_default_sort_keyword().

get_search_sort()

Overrides ViewMixin.get_search_sort() and gets the value of the sort dict by via get_search_sort_by_keyword() with get_sort_keyword() as the keyword argument.

This means that you should not override this method, but instead override:

• get_sort_map() (or sort_map)
• get_default_sort_keyword() (or default_sort_keyword)

class ievv_opensource.ievv_elasticsearch.viewhelpers.searchview.SearchMixin

Bases: object

Mixin class that makes it slightly easier to add search via a querystring attribute (defaults to ?s=<search_string>).

This is perfect for views that use ElasticSearch for traditional search where a user types in a string, and we wish to search some fields for that string.

This MUST be mixed in before View (or ViewMixin) , since it overrides ViewMixin.get_search_query().

Can safely be used with SortMixin. The order of SortMixin and SearchMixin does not matter, but they must both be mixed in before View (or ViewMixin).

Examples

Minimal example:

class MySearchView(searchview.SearchMixin, searchview.View):
    search_query_fields = ['name', 'email']

    def get_context_data(self, **kwargs):
        context = super().get_context_data(**kwargs)
        context['searchpaginator'] = self.get_paginator()

search_querystring_attribute = 's'

The querystring attribute to use for sort. Used by get_sort_querystring_attribute(). Defaults to s.

search_query_fields = []

List of fields for get_search_query_fields(). Defaults to empty list, so you need to set this, or override get_search_query_fields().

get_search_querystring_attribute()

The querystring attribute to use for search. Defaults to search_querystring_attribute.

clean_search_string(search_string)

Can be overridden to clean/modify the search_string.

Does nothing by default.

Used by get_search_string().

get_search_string()

Get the search string.

We get the search string from the querystring, and clean it with clean_search_string() before returning it.

get_search_query_fields()

Get the fields for the multi_match query performed by get_search_query_with_search_string().

Defaults to search_query_fields

get_search_query_with_search_string(search_string)

Called by get_search_query() when get_search_string() returns something.

Defaults to a multi_matach query with the fields returned by get_search_query_fields().

You will not need to override this for simple cases, but for more complex queries with boosting, filtering, etc. you will most likely have to override this.

get_search_query_without_search_string()

Called by get_search_query() when get_search_string() returns empty string or None.

Defaults to a match_all query.

get_search_query()

Overrides ViewMixin.get_search_query() and splits the logic into two separate states:

1. We have a search string, call get_search_query_with_search_string().
2. We do not have a search string, call get_search_query_without_search_string().

You should not need to override this method, but instead override:

• get_search_fields() or perhaps get_search_query_with_search_string() (for advanced cases).
• Perhaps get_search_query_without_search_string().