ultralytics 8.2.69 FastSAM prompt inference refactor (#14724)
Co-authored-by: UltralyticsAssistant <web@ultralytics.com> Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com>
This commit is contained in:
Laughing
GitHub
parent
82c4bdad10
commit
9532ad7cae
11 changed files with 187 additions and 427 deletions
|
|
@ -8,11 +8,11 @@ keywords: IBM Watsonx, IBM Watsonx AI, What is Watson?, IBM Watson Integration,
|
|||
|
||||
Nowadays, scalable [computer vision solutions](../guides/steps-of-a-cv-project.md) are becoming more common and transforming the way we handle visual data. A great example is IBM Watsonx, an advanced AI and data platform that simplifies the development, deployment, and management of AI models. It offers a complete suite for the entire AI lifecycle and seamless integration with IBM Cloud services.
|
||||
|
||||
You can train [Ultralytics YOLOv8 models](https://github.com/ultralytics/ultralytics) using IBM Watsonx. It's a good option for enterprises interested in efficient [model training](../modes/train.md), fine-tuning for specific tasks, and improving [model performance](../guides/model-evaluation-insights.md) with robust tools and a user-friendly setup. In this guide, we'll walk you through the process of training YOLOv8 with IBM Watsonx, covering everything from setting up your environment to evaluating your trained models. Let’s get started!
|
||||
You can train [Ultralytics YOLOv8 models](https://github.com/ultralytics/ultralytics) using IBM Watsonx. It's a good option for enterprises interested in efficient [model training](../modes/train.md), fine-tuning for specific tasks, and improving [model performance](../guides/model-evaluation-insights.md) with robust tools and a user-friendly setup. In this guide, we'll walk you through the process of training YOLOv8 with IBM Watsonx, covering everything from setting up your environment to evaluating your trained models. Let's get started!
|
||||
|
||||
## What is IBM Watsonx?
|
||||
|
||||
[Watsonx](https://www.ibm.com/watsonx) is IBM's cloud-based platform designed for commercial generative AI and scientific data. IBM Watsonx’s three components - watsonx.ai, watsonx.data, and watsonx.governance - come together to create an end-to-end, trustworthy AI platform that can accelerate AI projects aimed at solving business problems. It provides powerful tools for building, training, and [deploying machine learning models](../guides/model-deployment-options.md) and makes it easy to connect with various data sources.
|
||||
[Watsonx](https://www.ibm.com/watsonx) is IBM's cloud-based platform designed for commercial generative AI and scientific data. IBM Watsonx's three components - watsonx.ai, watsonx.data, and watsonx.governance - come together to create an end-to-end, trustworthy AI platform that can accelerate AI projects aimed at solving business problems. It provides powerful tools for building, training, and [deploying machine learning models](../guides/model-deployment-options.md) and makes it easy to connect with various data sources.
|
||||
|
||||
<p align="center">
|
||||
<img width="800" src="https://cdn.stackoverflow.co/images/jo7n4k8s/production/48b67e6aec41f89031a3426cbd1f78322e6776cb-8800x4950.jpg?auto=format" alt="Overview of IBM Watsonx">
|
||||
|
|
@ -22,7 +22,7 @@ Its user-friendly interface and collaborative capabilities streamline the develo
|
|||
|
||||
## Key Features of IBM Watsonx
|
||||
|
||||
IBM Watsonx is made of three main components: watsonx.ai, watsonx.data, and watsonx.governance. Each component offers features that cater to different aspects of AI and data management. Let’s take a closer look at them.
|
||||
IBM Watsonx is made of three main components: watsonx.ai, watsonx.data, and watsonx.governance. Each component offers features that cater to different aspects of AI and data management. Let's take a closer look at them.
|
||||
|
||||
### [Watsonx.ai](https://www.ibm.com/products/watsonx-ai)
|
||||
|
||||
|
|
@ -42,11 +42,11 @@ You can use IBM Watsonx to accelerate your YOLOv8 model training workflow.
|
|||
|
||||
### Prerequisites
|
||||
|
||||
You need an [IBM Cloud account](https://cloud.ibm.com/registration) to create a [watsonx.ai](https://www.ibm.com/products/watsonx-ai) project, and you’ll also need a [Kaggle](./kaggle.md) account to load the data set.
|
||||
You need an [IBM Cloud account](https://cloud.ibm.com/registration) to create a [watsonx.ai](https://www.ibm.com/products/watsonx-ai) project, and you'll also need a [Kaggle](./kaggle.md) account to load the data set.
|
||||
|
||||
### Step 1: Set Up Your Environment
|
||||
|
||||
First, you’ll need to set up an IBM account to use a Jupyter Notebook. Log in to [watsonx.ai](https://eu-de.dataplatform.cloud.ibm.com/registration/stepone?preselect_region=true) using your IBM Cloud account.
|
||||
First, you'll need to set up an IBM account to use a Jupyter Notebook. Log in to [watsonx.ai](https://eu-de.dataplatform.cloud.ibm.com/registration/stepone?preselect_region=true) using your IBM Cloud account.
|
||||
|
||||
Then, create a [watsonx.ai project](https://www.ibm.com/docs/en/watsonx/saas?topic=projects-creating-project), and a [Jupyter Notebook](https://www.ibm.com/docs/en/watsonx/saas?topic=editor-creating-managing-notebooks).
|
||||
|
||||
|
|
@ -88,7 +88,7 @@ Then, you can import the needed packages.
|
|||
|
||||
For this tutorial, we will use a [marine litter dataset](https://www.kaggle.com/datasets/atiqishrak/trash-dataset-icra19) available on Kaggle. With this dataset, we will custom-train a YOLOv8 model to detect and classify litter and biological objects in underwater images.
|
||||
|
||||
We can load the dataset directly into the notebook using the Kaggle API. First, create a free Kaggle account. Once you have created an account, you’ll need to generate an API key. Directions for generating your key can be found in the [Kaggle API documentation](https://github.com/Kaggle/kaggle-api/blob/main/docs/README.md) under the section "API credentials".
|
||||
We can load the dataset directly into the notebook using the Kaggle API. First, create a free Kaggle account. Once you have created an account, you'll need to generate an API key. Directions for generating your key can be found in the [Kaggle API documentation](https://github.com/Kaggle/kaggle-api/blob/main/docs/README.md) under the section "API credentials".
|
||||
|
||||
Copy and paste your Kaggle username and API key into the following code. Then run the code to install the API and load the dataset into Watsonx.
|
||||
|
||||
|
|
@ -248,7 +248,7 @@ Run the following command-line code to fine tune a pretrained default YOLOv8 mod
|
|||
!yolo task=detect mode=train data={work_dir}/trash_ICRA19/config.yaml model=yolov8s.pt epochs=2 batch=32 lr0=.04 plots=True
|
||||
```
|
||||
|
||||
Here’s a closer look at the parameters in the model training command:
|
||||
Here's a closer look at the parameters in the model training command:
|
||||
|
||||
- **task**: It specifies the computer vision task for which you are using the specified YOLO model and data set.
|
||||
- **mode**: Denotes the purpose for which you are loading the specified model and data. Since we are training a model, it is set to "train." Later, when we test our model's performance, we will set it to "predict."
|
||||
|
|
@ -257,7 +257,7 @@ Here’s a closer look at the parameters in the model training command:
|
|||
- **lr0**: Specifies the model's initial learning rate.
|
||||
- **plots**: Directs YOLO to generate and save plots of our model's training and evaluation metrics.
|
||||
|
||||
For a detailed understanding of the model training process and best practices, refer to the [YOLOv8 Model Training guide](../modes/train.md). This guide will help you get the most out of your experiments and ensure you’re using YOLOv8 effectively.
|
||||
For a detailed understanding of the model training process and best practices, refer to the [YOLOv8 Model Training guide](../modes/train.md). This guide will help you get the most out of your experiments and ensure you're using YOLOv8 effectively.
|
||||
|
||||
### Step 6: Test the Model
|
||||
|
||||
|
|
@ -312,7 +312,7 @@ Unlike precision, recall moves in the opposite direction, showing greater recall
|
|||
|
||||
### Step 8: Calculating Intersection Over Union
|
||||
|
||||
You can measure the prediction accuracy by calculating the IoU between a predicted bounding box and a ground truth bounding box for the same object. Check out [IBM’s tutorial on training YOLOv8](https://developer.ibm.com/tutorials/awb-train-yolo-object-detection-model-in-python/) for more details.
|
||||
You can measure the prediction accuracy by calculating the IoU between a predicted bounding box and a ground truth bounding box for the same object. Check out [IBM's tutorial on training YOLOv8](https://developer.ibm.com/tutorials/awb-train-yolo-object-detection-model-in-python/) for more details.
|
||||
|
||||
## Summary
|
||||
|
||||
|
|
|
|||
|
|
@ -66,7 +66,6 @@ To perform object detection on an image, use the `predict` method as shown below
|
|||
|
||||
```python
|
||||
from ultralytics import FastSAM
|
||||
from ultralytics.models.fastsam import FastSAMPrompt
|
||||
|
||||
# Define an inference source
|
||||
source = "path/to/bus.jpg"
|
||||
|
|
@ -77,23 +76,17 @@ To perform object detection on an image, use the `predict` method as shown below
|
|||
# Run inference on an image
|
||||
everything_results = model(source, device="cpu", retina_masks=True, imgsz=1024, conf=0.4, iou=0.9)
|
||||
|
||||
# Prepare a Prompt Process object
|
||||
prompt_process = FastSAMPrompt(source, everything_results, device="cpu")
|
||||
# Run inference with bboxes prompt
|
||||
results = model(source, bboxes=[439, 437, 524, 709])
|
||||
|
||||
# Everything prompt
|
||||
results = prompt_process.everything_prompt()
|
||||
# Run inference with points prompt
|
||||
results = model(source, points=[[200, 200]], labels=[1])
|
||||
|
||||
# Bbox default shape [0,0,0,0] -> [x1,y1,x2,y2]
|
||||
results = prompt_process.box_prompt(bbox=[200, 200, 300, 300])
|
||||
# Run inference with texts prompt
|
||||
results = model(source, texts="a photo of a dog")
|
||||
|
||||
# Text prompt
|
||||
results = prompt_process.text_prompt(text="a photo of a dog")
|
||||
|
||||
# Point prompt
|
||||
# points default [[0,0]] [[x1,y1],[x2,y2]]
|
||||
# point_label default [0] [1,0] 0:background, 1:foreground
|
||||
results = prompt_process.point_prompt(points=[[200, 200]], pointlabel=[1])
|
||||
prompt_process.plot(annotations=results, output="./")
|
||||
# Run inference with bboxes and points and texts prompt at the same time
|
||||
results = model(source, bboxes=[439, 437, 524, 709], points=[[200, 200]], labels=[1], texts="a photo of a dog")
|
||||
```
|
||||
|
||||
=== "CLI"
|
||||
|
|
@ -105,6 +98,28 @@ To perform object detection on an image, use the `predict` method as shown below
|
|||
|
||||
This snippet demonstrates the simplicity of loading a pre-trained model and running a prediction on an image.
|
||||
|
||||
!!! Example "FastSAMPredictor example"
|
||||
|
||||
This way you can run inference on image and get all the segment `results` once and run prompts inference multiple times without running inference multiple times.
|
||||
|
||||
=== "Prompt inference"
|
||||
|
||||
```python
|
||||
from ultralytics.models.fastsam import FastSAMPredictor
|
||||
|
||||
# Create FastSAMPredictor
|
||||
overrides = dict(conf=0.25, task="segment", mode="predict", model="FastSAM-s.pt", save=False, imgsz=1024)
|
||||
predictor = FastSAMPredictor(overrides=overrides)
|
||||
|
||||
# Segment everything
|
||||
everything_results = predictor("ultralytics/assets/bus.jpg")
|
||||
|
||||
# Prompt inference
|
||||
bbox_results = predictor.prompt(everything_results, bboxes=[[200, 200, 300, 300]])
|
||||
point_results = predictor.prompt(everything_results, points=[200, 200])
|
||||
text_results = predictor.prompt(everything_results, texts="a photo of a dog")
|
||||
```
|
||||
|
||||
!!! Note
|
||||
|
||||
All the returned `results` in above examples are [Results](../modes/predict.md#working-with-results) object which allows access predicted masks and source image easily.
|
||||
|
|
@ -270,7 +285,6 @@ To use FastSAM for inference in Python, you can follow the example below:
|
|||
|
||||
```python
|
||||
from ultralytics import FastSAM
|
||||
from ultralytics.models.fastsam import FastSAMPrompt
|
||||
|
||||
# Define an inference source
|
||||
source = "path/to/bus.jpg"
|
||||
|
|
@ -281,21 +295,17 @@ model = FastSAM("FastSAM-s.pt") # or FastSAM-x.pt
|
|||
# Run inference on an image
|
||||
everything_results = model(source, device="cpu", retina_masks=True, imgsz=1024, conf=0.4, iou=0.9)
|
||||
|
||||
# Prepare a Prompt Process object
|
||||
prompt_process = FastSAMPrompt(source, everything_results, device="cpu")
|
||||
# Run inference with bboxes prompt
|
||||
results = model(source, bboxes=[439, 437, 524, 709])
|
||||
|
||||
# Everything prompt
|
||||
ann = prompt_process.everything_prompt()
|
||||
# Run inference with points prompt
|
||||
results = model(source, points=[[200, 200]], labels=[1])
|
||||
|
||||
# Bounding box prompt
|
||||
ann = prompt_process.box_prompt(bbox=[200, 200, 300, 300])
|
||||
# Run inference with texts prompt
|
||||
results = model(source, texts="a photo of a dog")
|
||||
|
||||
# Text prompt
|
||||
ann = prompt_process.text_prompt(text="a photo of a dog")
|
||||
|
||||
# Point prompt
|
||||
ann = prompt_process.point_prompt(points=[[200, 200]], pointlabel=[1])
|
||||
prompt_process.plot(annotations=ann, output="./")
|
||||
# Run inference with bboxes and points and texts prompt at the same time
|
||||
results = model(source, bboxes=[439, 437, 524, 709], points=[[200, 200]], labels=[1], texts="a photo of a dog")
|
||||
```
|
||||
|
||||
For more details on inference methods, check the [Predict Usage](#predict-usage) section of the documentation.
|
||||
|
|
|
|||
|
|
@ -1,16 +0,0 @@
|
|||
---
|
||||
description: Explore the FastSAM prompt module for image annotation and visualization in Ultralytics, detailed with class methods and attributes.
|
||||
keywords: Ultralytics, FastSAM, image annotation, image visualization, FastSAMPrompt, YOLO, python script
|
||||
---
|
||||
|
||||
# Reference for `ultralytics/models/fastsam/prompt.py`
|
||||
|
||||
!!! Note
|
||||
|
||||
This file is available at [https://github.com/ultralytics/ultralytics/blob/main/ultralytics/models/fastsam/prompt.py](https://github.com/ultralytics/ultralytics/blob/main/ultralytics/models/fastsam/prompt.py). If you spot a problem please help fix it by [contributing](https://docs.ultralytics.com/help/contributing/) a [Pull Request](https://github.com/ultralytics/ultralytics/edit/main/ultralytics/models/fastsam/prompt.py) 🛠️. Thank you 🙏!
|
||||
|
||||
<br>
|
||||
|
||||
## ::: ultralytics.models.fastsam.prompt.FastSAMPrompt
|
||||
|
||||
<br><br>
|
||||
Loading…
Add table
Add a link
Reference in a new issue