2023 Comparative Search Review Raw Results

Introduction

Those are the results of the comparative search experiment described in this post.

Search Queries

3 groups of questions:

• 1 to 4: exact matches
• 5 to 8: reformulation of existing questions
• 9 to 12: non-existing/random questions
• 13 to 16: keywords search

1. How to Use Bluetooth in a Suzuki Swift?
2. What is the best oil type for my Ford Ranger, and is it possible to change the oil myself?
3. Where are Range Rovers made?
4. Who owns Rolls Royce?
5. What is the normal oil for a Honda CR-V and is it an easy DIY job to change?
   • original: What’s the correct oil type for my Honda CR-V, and is it tricky to change it yourself?
6. Is it legal in Victoria to sell a car without a RWC, if so, what are the correct steps?
   • original: In Victoria, can I sell my car without a roadworthy and, if so, what is the correct legal process?
7. Why electric cars are not builtin with PV?
   • original: Why don’t electric cars have solar panels?
8. What does the acronym LDV mean?
   • original: What does LDV stand for?
9. How to change engine oil on my Toyota Prius?
10. What is the difference between Diesel, Regular Petrol and Unleaded 91, Unleaded E10?
11. Why my car won’t start?
12. How to use Bluetooth in a Jeep Willys? ;-)
13. skoda meaning
14. bmw x5 diesel problems
15. best landcruiser engine
16. ford ranger oil

Results

Summary

Spreadsheet: 2023-comparative-search-experiment-results.ods

This spreadsheet has 1 row for each results from each queries from each algo. It contains the columns:

• algorithm: BM25, Google, ELSER or MiniLM
• query type: exact match, keywords, non-existing or reformulation
• search query: one of the 16 listed above
• rank response: 1 to 5 ; rank of the result
• Algorithm score: score given by the algo itself and used to rank the results. No value for Google because it doesn’t give score. MiniLM gives a score between 0 and 1, BM25 and ELSER seems to give a score above 1. Those scores are only interesting to do relative comparison for a same algo.
• Relevance score: 1 or 0: my relevance score of the result for this query.

Detailed results

For the three cloud-based experiments (BM25, ELSER and MiniLM) I have collected as well the raw results from elasticsearch in json format. In the case of ELSER that results contains as well the expanded term of each result:

Located in /assets/docs/2023-comparative-search-review-results-json/: 3 types: bm25, elser, minilm, for number from 01 to 16. Examples:

• BM25: bm25.01.json to bm25.16.json
• ELSER: elser.01.json to elser.16.json
• MiniLM: minilm.01.json to minilm.16.json