ultralytics 8.0.238 Explorer Ask AI feature and fixes (#7408)
Co-authored-by: Kayzwer <68285002+Kayzwer@users.noreply.github.com> Co-authored-by: uwer <uwe.rosebrock@gmail.com> Co-authored-by: Uwe Rosebrock <ro260@csiro.au> Co-authored-by: Ayush Chaurasia <ayush.chaurarsia@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Laughing-q <1182102784@qq.com> Co-authored-by: Muhammad Rizwan Munawar <chr043416@gmail.com> Co-authored-by: AdamP <adamp87hun@gmail.com>
This commit is contained in:
Glenn Jocher
GitHub
parent
e76754eab0
commit
783033fa6b
19 changed files with 387 additions and 76 deletions
|
|
@ -119,7 +119,31 @@ You can also plot the similar images using the `plot_similar` method. This metho
|
|||
plt.show()
|
||||
```
|
||||
|
||||
## 2. SQL Querying
|
||||
## 2. Ask AI (Natural Language Querying)
|
||||
|
||||
This allows you to write how you want to filter your dataset using natural language. You don't have to be proficient in writing SQL queries. Our AI powered query generator will automatically do that under the hood. For example - you can say - "show me 100 images with exactly one person and 2 dogs. There can be other objects too" and it'll internally generate the query and show you those results.
|
||||
Note: This works using LLMs under the hood so the results are probabilistic and might get things wrong sometimes
|
||||
|
||||
!!! Example "Ask AI"
|
||||
|
||||
```python
|
||||
from ultralytics import Explorer
|
||||
from ultralytics.data.explorer import plot_query_result
|
||||
|
||||
|
||||
# create an Explorer object
|
||||
exp = Explorer(data='coco128.yaml', model='yolov8n.pt')
|
||||
exp.create_embeddings_table()
|
||||
|
||||
df = exp.ask_ai("show me 100 images with exactly one person and 2 dogs. There can be other objects too")
|
||||
print(df.head())
|
||||
|
||||
# plot the results
|
||||
plt = plot_query_result(df)
|
||||
plt.show()
|
||||
```
|
||||
|
||||
## 3. SQL Querying
|
||||
|
||||
You can run SQL queries on your dataset using the `sql_query` method. This method takes a SQL query as input and returns a pandas dataframe with the results.
|
||||
|
||||
|
|
@ -153,7 +177,7 @@ You can also plot the results of a SQL query using the `plot_sql_query` method.
|
|||
print(df.head())
|
||||
```
|
||||
|
||||
## 3. Working with embeddings Table (Advanced)
|
||||
## 4. Working with embeddings Table (Advanced)
|
||||
|
||||
You can also work with the embeddings table directly. Once the embeddings table is created, you can access it using the `Explorer.table`
|
||||
|
||||
|
|
@ -210,7 +234,7 @@ When using large datasets, you can also create a dedicated vector index for fast
|
|||
|
||||
Find more details on the type vector indices available and parameters [here](https://lancedb.github.io/lancedb/ann_indexes/#types-of-index) In the future, we will add support for creating vector indices directly from Explorer API.
|
||||
|
||||
## 4. Embeddings Applications
|
||||
## 5. Embeddings Applications
|
||||
|
||||
You can use the embeddings table to perform a variety of exploratory analysis. Here are some examples:
|
||||
|
||||
|
|
|
|||
58
docs/en/datasets/explorer/dashboard.md
Normal file
58
docs/en/datasets/explorer/dashboard.md
Normal file
|
|
@ -0,0 +1,58 @@
|
|||
---
|
||||
comments: 5rue
|
||||
description: Learn about Ultralytics Explorer GUI for semantic search, SQL queries, and AI-powered natural language search in CV datasets.
|
||||
keywords: Ultralytics, Explorer GUI, semantic search, vector similarity search, AI queries, SQL queries, computer vision, dataset exploration, image search, OpenAI integration
|
||||
---
|
||||
|
||||
# Explorer GUI
|
||||
|
||||
Explorer GUI is like a playground build using (Ultralytics Explorer API)[api.md]. It allows you to run semantic/vector similarity search, SQL queries and even search using natural language using our ask AI feature powered by LLMs.
|
||||
|
||||
### Installation
|
||||
|
||||
```bash
|
||||
pip install ultralytics[explorer]
|
||||
```
|
||||
|
||||
!!! note "Note"
|
||||
|
||||
Ask AI feature works using OpenAI, so you'll be prompted to set the api key for OpenAI when you first run the GUI.
|
||||
You can set it like this - `yolo settings openai_api_key="..."`
|
||||
|
||||
## Semantic Search / Vector Similarity Search
|
||||
|
||||
Semantic search is a technique for finding similar images to a given image. It is based on the idea that similar images will have similar embeddings. In the UI, you can select one of more images and search for the images similar to them. This can be useful when you want to find images similar to a given image or a set of images that don't perform as expected.
|
||||
|
||||
For example:
|
||||
In this VOC Exploration dashboard, user selects a couple aeroplane images like this:
|
||||
<p>
|
||||
<img width="1710" alt="Screenshot 2024-01-08 at 8 46 33 PM" src="https://github.com/AyushExel/assets/assets/15766192/da5f1b0a-9eb5-4712-919c-7d5512240dd8">
|
||||
</p>
|
||||
|
||||
On performing similarity search, you should see a similar result:
|
||||
<p>
|
||||
<img width="1710" alt="Screenshot 2024-01-08 at 8 46 46 PM" src="https://github.com/AyushExel/assets/assets/15766192/5e4c6445-8e4e-48bb-a15a-9fb6c6994af8">
|
||||
</p>
|
||||
|
||||
## Ask AI
|
||||
|
||||
This allows you to write how you want to filter your dataset using natural language. You don't have to be proficient in writing SQL queries. Our AI powered query generator will automatically do that under the hood. For example - you can say - "show me 100 images with exactly one person and 2 dogs. There can be other objects too" and it'll internally generate the query and show you those results. Here's an example output when asked to "Show 10 images with exactly 5 persons" and you'll see a result like this:
|
||||
<p>
|
||||
<img width="1709" alt="Screenshot 2024-01-08 at 7 19 48 PM (1)" src="https://github.com/AyushExel/assets/assets/15766192/e536b0eb-6bce-43fe-b800-3e79510d2e5b">
|
||||
</p>
|
||||
|
||||
Note: This works using LLMs under the hood so the results are probabilistic and might get things wrong sometimes
|
||||
|
||||
## Run SQL queries on your CV datasets
|
||||
|
||||
You can run SQL queries on your dataset to filter it. It also works if you only provide the WHERE clause. Example SQL query would show only the images that have at least one 1 person and 1 dog in them:
|
||||
|
||||
```sql
|
||||
WHERE labels LIKE '%person%' AND labels LIKE '%dog%'
|
||||
```
|
||||
|
||||
<p>
|
||||
<img width="1707" alt="Screenshot 2024-01-08 at 8 57 49 PM" src="https://github.com/AyushExel/assets/assets/15766192/71619e16-4db9-4fdb-b951-0d1fdbf59a6a">
|
||||
</p>
|
||||
|
||||
This is a Demo build using the Explorer API. You can use the API to build your own exploratory notebooks or scripts to get insights into your datasets. Learn more about the Explorer API [here](api.md).
|
||||
|
|
@ -109,7 +109,10 @@
|
|||
"metadata": {},
|
||||
"source": [
|
||||
"You can use the also plot the similar samples directly using the `plot_similar` util\n",
|
||||
"<img width=\"689\" alt=\"Screenshot 2024-01-06 at 9 46 48 PM\" src=\"https://github.com/AyushExel/assets/assets/15766192/70e1a4c4-6c67-4664-b77a-ad27b1fba8f8\">\n"
|
||||
"<p>\n",
|
||||
"\n",
|
||||
" <img src=\"https://github.com/AyushExel/assets/assets/15766192/a3c9247b-9271-47df-aaa5-36d96c5034b1\" />\n",
|
||||
"</p>\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -139,17 +142,74 @@
|
|||
"metadata": {},
|
||||
"source": [
|
||||
"<p>\n",
|
||||
"<img width=\"766\" alt=\"Screenshot 2024-01-06 at 10 05 10 PM\" src=\"https://github.com/AyushExel/assets/assets/15766192/faa9c544-d96b-4528-a2ea-95c5d8856744\">\n",
|
||||
"<img src=\"https://github.com/AyushExel/assets/assets/15766192/8e011195-b0da-43ef-b3cd-5fb6f383037e\">\n",
|
||||
"\n",
|
||||
"</p>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0cea63f1-71f1-46da-af2b-b1b7d8f73553",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 2. Ask AI: Search or filter with Natural Language\n",
|
||||
"You can prompt the Explorer object with the kind of data points you want to see and it'll try to return a dataframe with those. Because it is powered by LLMs, it doesn't always get it right. In that case, it'll return None.\n",
|
||||
"<p>\n",
|
||||
"<img width=\"1131\" alt=\"Screenshot 2024-01-07 at 2 34 53 PM\" src=\"https://github.com/AyushExel/assets/assets/15766192/c4a69fd9-e54f-4d6a-aba5-dc9cfae1bc04\">\n",
|
||||
"\n",
|
||||
"</p>\n",
|
||||
"\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "92fb92ac-7f76-465a-a9ba-ea7492498d9c",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"df = exp.ask_ai(\"show me images containing more than 10 objects with at least 2 persons\")\n",
|
||||
"df.head(5)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f2a7d26e-0ce5-4578-ad1a-b1253805280f",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"for plotting these results you can use `plot_query_result` util\n",
|
||||
"Example:\n",
|
||||
"```\n",
|
||||
"plt = plot_query_result(exp.ask_ai(\"show me 10 images containing exactly 2 persons\"))\n",
|
||||
"Image.fromarray(plt)\n",
|
||||
"```\n",
|
||||
"<p>\n",
|
||||
" <img src=\"https://github.com/AyushExel/assets/assets/15766192/2cb780de-d05b-4412-a526-7f7f0f10e669\">\n",
|
||||
"\n",
|
||||
"</p>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "b1cfab84-9835-4da0-8e9a-42b30cf84511",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# plot\n",
|
||||
"from ultralytics.data.explorer import plot_query_result\n",
|
||||
"from PIL import Image\n",
|
||||
"\n",
|
||||
"plt = plot_query_result(exp.ask_ai(\"show me 10 images containing exactly 2 persons\"))\n",
|
||||
"Image.fromarray(plt)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "35315ae6-d827-40e4-8813-279f97a83b34",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 2. Run SQL queries on your Dataset!\n",
|
||||
"## 3. Run SQL queries on your Dataset!\n",
|
||||
"Sometimes you might want to investigate a certain type of entries in your dataset. For this Explorer allows you to execute SQL queries.\n",
|
||||
"It accepts either of the formats:\n",
|
||||
"- Queries beginning with \"WHERE\" will automatically select all columns. This can be thought of as a short-hand query\n",
|
||||
|
|
@ -179,7 +239,7 @@
|
|||
"metadata": {},
|
||||
"source": [
|
||||
"Just like similarity search, you also get a util to directly plot the sql queries using `exp.plot_sql_query`\n",
|
||||
"<img width=\"771\" alt=\"Screenshot 2024-01-06 at 9 48 08 PM\" src=\"https://github.com/AyushExel/assets/assets/15766192/332f5acd-3a4e-462d-a281-5d5effd1886e\">\n"
|
||||
"<img src=\"https://github.com/AyushExel/assets/assets/15766192/f8b66629-8dd0-419e-8f44-9837969ba678\">\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -419,7 +479,7 @@
|
|||
"metadata": {},
|
||||
"source": [
|
||||
"You should see something like this\n",
|
||||
"<img width=\"897\" alt=\"Screenshot 2024-01-06 at 9 50 48 PM\" src=\"https://github.com/AyushExel/assets/assets/15766192/5d3f0e35-2ad4-4a67-8df7-3a4c17867b72\">\n"
|
||||
"<img src=\"https://github.com/AyushExel/assets/assets/15766192/649bc366-ca2d-46ea-bfd9-3097cf575584\">\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
|
|||
|
|
@ -16,6 +16,12 @@ Explorer depends on external libraries for some of its functionality. These are
|
|||
pip install ultralytics[explorer]
|
||||
```
|
||||
|
||||
### Explorer API
|
||||
|
||||
This is a Python API for Exploring your datasets. It also powers the GUI Explorer. You can use this to create your own exploratory notebooks or scripts to get insights into your datasets.
|
||||
|
||||
Learn more about the Explorer API [here](api.md).
|
||||
|
||||
## GUI Explorer Usage
|
||||
|
||||
The GUI demo runs in your browser allowing you to create embeddings for your dataset and search for similar images, run SQL queries and perform semantic search. It can be run using the following command:
|
||||
|
|
@ -24,8 +30,11 @@ The GUI demo runs in your browser allowing you to create embeddings for your dat
|
|||
yolo explorer
|
||||
```
|
||||
|
||||
### Explorer API
|
||||
!!! note "Note"
|
||||
Ask AI feature works using OpenAI, so you'll be prompted to set the api key for OpenAI when you first run the GUI.
|
||||
You can set it like this - `yolo settings openai_api_key="..."`
|
||||
|
||||
This is a Python API for Exploring your datasets. It also powers the GUI Explorer. You can use this to create your own exploratory notebooks or scripts to get insights into your datasets.
|
||||
|
||||
Learn more about the Explorer API [here](api.md).
|
||||
Example
|
||||
<p>
|
||||
<img width="1709" alt="Screenshot 2024-01-08 at 7 19 48 PM (1)" src="https://github.com/AyushExel/assets/assets/15766192/e536b0eb-6bce-43fe-b800-3e79510d2e5b">
|
||||
</p>
|
||||
|
|
|
|||
|
|
@ -99,7 +99,7 @@ A heatmap generated with [Ultralytics YOLOv8](https://github.com/ultralytics/ult
|
|||
fps,
|
||||
(w, h))
|
||||
|
||||
line_points = [(256, 409), (694, 532)] # line for object counting
|
||||
line_points = [(20, 400), (1080, 404)] # line for object counting
|
||||
|
||||
# Init heatmap
|
||||
heatmap_obj = heatmap.Heatmap()
|
||||
|
|
|
|||
|
|
@ -31,7 +31,7 @@ There are two types of instance segmentation tracking available in the Ultralyti
|
|||
from ultralytics import YOLO
|
||||
from ultralytics.utils.plotting import Annotator, colors
|
||||
|
||||
model = YOLO("yolov8n-seg.pt")
|
||||
model = YOLO("yolov8n-seg.pt") # segmentation model
|
||||
names = model.model.names
|
||||
cap = cv2.VideoCapture("path/to/video/file.mp4")
|
||||
w, h, fps = (int(cap.get(x)) for x in (cv2.CAP_PROP_FRAME_WIDTH, cv2.CAP_PROP_FRAME_HEIGHT, cv2.CAP_PROP_FPS))
|
||||
|
|
@ -45,15 +45,15 @@ There are two types of instance segmentation tracking available in the Ultralyti
|
|||
break
|
||||
|
||||
results = model.predict(im0)
|
||||
clss = results[0].boxes.cls.cpu().tolist()
|
||||
masks = results[0].masks.xy
|
||||
|
||||
annotator = Annotator(im0, line_width=2)
|
||||
|
||||
for mask, cls in zip(masks, clss):
|
||||
annotator.seg_bbox(mask=mask,
|
||||
mask_color=colors(int(cls), True),
|
||||
det_label=names[int(cls)])
|
||||
if results[0].masks is not None:
|
||||
clss = results[0].boxes.cls.cpu().tolist()
|
||||
masks = results[0].masks.xy
|
||||
for mask, cls in zip(masks, clss):
|
||||
annotator.seg_bbox(mask=mask,
|
||||
mask_color=colors(int(cls), True),
|
||||
det_label=names[int(cls)])
|
||||
|
||||
out.write(im0)
|
||||
cv2.imshow("instance-segmentation", im0)
|
||||
|
|
@ -77,7 +77,7 @@ There are two types of instance segmentation tracking available in the Ultralyti
|
|||
|
||||
track_history = defaultdict(lambda: [])
|
||||
|
||||
model = YOLO("yolov8n-seg.pt")
|
||||
model = YOLO("yolov8n-seg.pt") # segmentation model
|
||||
cap = cv2.VideoCapture("path/to/video/file.mp4")
|
||||
w, h, fps = (int(cap.get(x)) for x in (cv2.CAP_PROP_FRAME_WIDTH, cv2.CAP_PROP_FRAME_HEIGHT, cv2.CAP_PROP_FPS))
|
||||
|
||||
|
|
@ -93,7 +93,7 @@ There are two types of instance segmentation tracking available in the Ultralyti
|
|||
|
||||
results = model.track(im0, persist=True)
|
||||
|
||||
if results[0].boxes.id is not None:
|
||||
if results[0].boxes.id is not None and results[0].masks is not None:
|
||||
masks = results[0].masks.xy
|
||||
track_ids = results[0].boxes.id.int().cpu().tolist()
|
||||
|
||||
|
|
|
|||
|
|
@ -222,7 +222,7 @@ nav:
|
|||
- Explorer:
|
||||
- datasets/explorer/index.md
|
||||
- Explorer API: datasets/explorer/api.md
|
||||
- GUI Dashboard Demo: datasets/explorer/dash.md
|
||||
- Explorer Dashboard: datasets/explorer/dashboard.md
|
||||
- VOC Exploration Example: datasets/explorer/explorer.ipynb
|
||||
- Detection:
|
||||
- datasets/detect/index.md
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue