From 31de5d044f7aedfc2a760dd02fad693d48709a16 Mon Sep 17 00:00:00 2001 From: Francesco Mattioli Date: Thu, 20 Jun 2024 18:41:19 +0200 Subject: [PATCH] ROS quickstart Docs page cleanup (#13835) Co-authored-by: UltralyticsAssistant Co-authored-by: Glenn Jocher --- README.md | 16 +- README.zh-CN.md | 104 ++++----- docs/en/guides/ros-quickstart.md | 302 ++++++++++++++++++------- docs/en/integrations/tensorrt.md | 4 +- docs/en/integrations/weights-biases.md | 14 +- docs/en/models/yolov9.md | 2 +- docs/en/usage/cli.md | 1 - 7 files changed, 282 insertions(+), 161 deletions(-) diff --git a/README.md b/README.md index d781b999..42f814eb 100644 --- a/README.md +++ b/README.md @@ -104,13 +104,13 @@ See YOLOv8 [Python Docs](https://docs.ultralytics.com/usage/python) for more exa Ultralytics provides interactive notebooks for YOLOv8, covering training, validation, tracking, and more. Each notebook is paired with a [YouTube](https://youtube.com/ultralytics?sub_confirmation=1) tutorial, making it easy to learn and implement advanced YOLOv8 features. -| Docs | Notebook | YouTube | -| --------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | -| YOLOv8 Train, Val, Predict and Export Modes | Open In Colab |
Ultralytics Youtube Video
 | -| Ultralytics HUB QuickStart | Open In Colab |
Ultralytics Youtube Video
 | -| YOLOv8 Multi-Object Tracking in Videos | Open In Colab |
Ultralytics Youtube Video
 | -| YOLOv8 Object Counting in Videos | Open In Colab |
Ultralytics Youtube Video
 | -| YOLOv8 Heatmaps in Videos | Open In Colab |
Ultralytics Youtube Video
 | +| Docs | Notebook | YouTube | +| ---------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | +| YOLOv8 Train, Val, Predict and Export Modes | Open In Colab |
Ultralytics Youtube Video
 | +| Ultralytics HUB QuickStart | Open In Colab |
Ultralytics Youtube Video
 | +| YOLOv8 Multi-Object Tracking in Videos | Open In Colab |
Ultralytics Youtube Video
 | +| YOLOv8 Object Counting in Videos | Open In Colab |
Ultralytics Youtube Video
 | +| YOLOv8 Heatmaps in Videos | Open In Colab |
Ultralytics Youtube Video
 | | Ultralytics Datasets Explorer with SQL and OpenAI Integration 🚀 New | Open In Colab |
Ultralytics Youtube Video
 | ##
Models
@@ -248,7 +248,7 @@ Our key integrations with leading AI platforms extend the functionality of Ultra NeuralMagic logo -| Roboflow | ClearML ⭐ NEW | Comet ⭐ NEW | Neural Magic ⭐ NEW | +| Roboflow | ClearML ⭐ NEW | Comet ⭐ NEW | Neural Magic ⭐ NEW | | :--------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------: | | Label and export your custom datasets directly to YOLOv8 for training with [Roboflow](https://roboflow.com/?ref=ultralytics) | Automatically track, visualize and even remotely train YOLOv8 using [ClearML](https://clear.ml/) (open-source!) | Free forever, [Comet](https://bit.ly/yolov8-readme-comet) lets you save YOLOv8 models, resume training, and interactively visualize and debug predictions | Run YOLOv8 inference up to 6x faster with [Neural Magic DeepSparse](https://bit.ly/yolov5-neuralmagic) | diff --git a/README.zh-CN.md b/README.zh-CN.md index 74e78d46..769c9e1e 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -106,13 +106,13 @@ success = model.export(format="onnx") # 将模型导出为 ONNX 格式 Ultralytics 提供了 YOLOv8 的交互式笔记本，涵盖训练、验证、跟踪等内容。每个笔记本都配有 [YouTube](https://youtube.com/ultralytics?sub_confirmation=1) 教程，使学习和实现高级 YOLOv8 功能变得简单。 -| 文档 | 笔记本 | YouTube | -| ------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | -| YOLOv8 训练、验证、预测和导出模式 | 在 Colab 中打开 |
Ultralytics Youtube 视频
 | -| Ultralytics HUB 快速开始 | 在 Colab 中打开 |
Ultralytics Youtube 视频
 | -| YOLOv8 视频中的多对象跟踪 | 在 Colab 中打开 |
Ultralytics Youtube 视频
 | -| YOLOv8 视频中的对象计数 | 在 Colab 中打开 |
Ultralytics Youtube 视频
 | -| YOLOv8 视频中的热图 | 在 Colab 中打开 |
Ultralytics Youtube 视频
 | +| 文档 | 笔记本 | YouTube | +| ----------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | +| YOLOv8 训练、验证、预测和导出模式 | 在 Colab 中打开 |
Ultralytics Youtube 视频
 | +| Ultralytics HUB 快速开始 | 在 Colab 中打开 |
Ultralytics Youtube 视频
 | +| YOLOv8 视频中的多对象跟踪 | 在 Colab 中打开 |
Ultralytics Youtube 视频
 | +| YOLOv8 视频中的对象计数 | 在 Colab 中打开 |
Ultralytics Youtube 视频
 | +| YOLOv8 视频中的热图 | 在 Colab 中打开 |
Ultralytics Youtube 视频
 | | Ultralytics 数据集浏览器，集成 SQL 和 OpenAI 🚀 New | 在 Colab 中打开 |
Ultralytics Youtube Video
 | ##
模型
@@ -127,13 +127,13 @@ Ultralytics 提供了 YOLOv8 的交互式笔记本，涵盖训练、验证、跟 查看[检测文档](https://docs.ultralytics.com/tasks/detect/)以获取这些在[COCO](https://docs.ultralytics.com/datasets/detect/coco/)上训练的模型的使用示例，其中包括80个预训练类别。 -| 模型 | 尺寸
(像素) | mAPval
50-95 | 速度
CPU ONNX
(ms) | 速度
A100 TensorRT
(ms) | 参数
(M) | FLOPs
(B) | -| ------------------------------------------------------------------------------------ | --------------- | -------------------- | --------------------------- | -------------------------------- | -------------- | ----------------- | -| [YOLOv8n](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8n.pt) | 640 | 37.3 | 80.4 | 0.99 | 3.2 | 8.7 | -| [YOLOv8s](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8s.pt) | 640 | 44.9 | 128.4 | 1.20 | 11.2 | 28.6 | -| [YOLOv8m](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8m.pt) | 640 | 50.2 | 234.7 | 1.83 | 25.9 | 78.9 | -| [YOLOv8l](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8l.pt) | 640 | 52.9 | 375.2 | 2.39 | 43.7 | 165.2 | -| [YOLOv8x](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8x.pt) | 640 | 53.9 | 479.1 | 3.53 | 68.2 | 257.8 | +| 模型 | 尺寸
(像素) | mAPval
50-95 | 速度
CPU ONNX
(ms) | 速度
A100 TensorRT
(ms) | 参数
(M) | FLOPs
(B) | +| ------------------------------------------------------------------------------------ | ------------------- | -------------------- | ----------------------------- | ---------------------------------- | ---------------- | ----------------- | +| [YOLOv8n](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8n.pt) | 640 | 37.3 | 80.4 | 0.99 | 3.2 | 8.7 | +| [YOLOv8s](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8s.pt) | 640 | 44.9 | 128.4 | 1.20 | 11.2 | 28.6 | +| [YOLOv8m](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8m.pt) | 640 | 50.2 | 234.7 | 1.83 | 25.9 | 78.9 | +| [YOLOv8l](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8l.pt) | 640 | 52.9 | 375.2 | 2.39 | 43.7 | 165.2 | +| [YOLOv8x](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8x.pt) | 640 | 53.9 | 479.1 | 3.53 | 68.2 | 257.8 | - **mAPval** 值是基于单模型单尺度在 [COCO val2017](https://cocodataset.org) 数据集上的结果。
通过 `yolo val detect data=coco.yaml device=0` 复现 - **速度** 是使用 [Amazon EC2 P4d](https://aws.amazon.com/ec2/instance-types/p4/) 实例对 COCO val 图像进行平均计算的。
通过 `yolo val detect data=coco.yaml batch=1 device=0|cpu` 复现 @@ -144,13 +144,13 @@ Ultralytics 提供了 YOLOv8 的交互式笔记本，涵盖训练、验证、跟 查看[检测文档](https://docs.ultralytics.com/tasks/detect/)以获取这些在[Open Image V7](https://docs.ultralytics.com/datasets/detect/open-images-v7/)上训练的模型的使用示例，其中包括600个预训练类别。 -| 模型 | 尺寸
(像素) | mAP验证
50-95 | 速度
CPU ONNX
(毫秒) | 速度
A100 TensorRT
(毫秒) | 参数
(M) | 浮点运算
(B) | -| ----------------------------------------------------------------------------------------- | --------------- | ------------------- | --------------------------- | -------------------------------- | -------------- | ---------------- | -| [YOLOv8n](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8n-oiv7.pt) | 640 | 18.4 | 142.4 | 1.21 | 3.5 | 10.5 | -| [YOLOv8s](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8s-oiv7.pt) | 640 | 27.7 | 183.1 | 1.40 | 11.4 | 29.7 | -| [YOLOv8m](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8m-oiv7.pt) | 640 | 33.6 | 408.5 | 2.26 | 26.2 | 80.6 | -| [YOLOv8l](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8l-oiv7.pt) | 640 | 34.9 | 596.9 | 2.43 | 44.1 | 167.4 | -| [YOLOv8x](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8x-oiv7.pt) | 640 | 36.3 | 860.6 | 3.56 | 68.7 | 260.6 | +| 模型 | 尺寸
(像素) | mAP验证
50-95 | 速度
CPU ONNX
(毫秒) | 速度
A100 TensorRT
(毫秒) | 参数
(M) | 浮点运算
(B) | +| ----------------------------------------------------------------------------------------- | ------------------- | --------------------- | ------------------------------- | ------------------------------------ | ---------------- | -------------------- | +| [YOLOv8n](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8n-oiv7.pt) | 640 | 18.4 | 142.4 | 1.21 | 3.5 | 10.5 | +| [YOLOv8s](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8s-oiv7.pt) | 640 | 27.7 | 183.1 | 1.40 | 11.4 | 29.7 | +| [YOLOv8m](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8m-oiv7.pt) | 640 | 33.6 | 408.5 | 2.26 | 26.2 | 80.6 | +| [YOLOv8l](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8l-oiv7.pt) | 640 | 34.9 | 596.9 | 2.43 | 44.1 | 167.4 | +| [YOLOv8x](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8x-oiv7.pt) | 640 | 36.3 | 860.6 | 3.56 | 68.7 | 260.6 | - **mAP验证** 值适用于在[Open Image V7](https://docs.ultralytics.com/datasets/detect/open-images-v7/)数据集上的单模型单尺度。
通过 `yolo val detect data=open-images-v7.yaml device=0` 以复现。 - **速度** 在使用[Amazon EC2 P4d](https://aws.amazon.com/ec2/instance-types/p4/)实例对Open Image V7验证图像进行平均测算。
通过 `yolo val detect data=open-images-v7.yaml batch=1 device=0|cpu` 以复现。 @@ -161,13 +161,13 @@ Ultralytics 提供了 YOLOv8 的交互式笔记本，涵盖训练、验证、跟 查看[分割文档](https://docs.ultralytics.com/tasks/segment/)以获取这些在[COCO-Seg](https://docs.ultralytics.com/datasets/segment/coco/)上训练的模型的使用示例，其中包括80个预训练类别。 -| 模型 | 尺寸
(像素) | mAPbox
50-95 | mAPmask
50-95 | 速度
CPU ONNX
(ms) | 速度
A100 TensorRT
(ms) | 参数
(M) | FLOPs
(B) | -| -------------------------------------------------------------------------------------------- | --------------- | -------------------- | --------------------- | --------------------------- | -------------------------------- | -------------- | ----------------- | -| [YOLOv8n-seg](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8n-seg.pt) | 640 | 36.7 | 30.5 | 96.1 | 1.21 | 3.4 | 12.6 | -| [YOLOv8s-seg](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8s-seg.pt) | 640 | 44.6 | 36.8 | 155.7 | 1.47 | 11.8 | 42.6 | -| [YOLOv8m-seg](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8m-seg.pt) | 640 | 49.9 | 40.8 | 317.0 | 2.18 | 27.3 | 110.2 | -| [YOLOv8l-seg](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8l-seg.pt) | 640 | 52.3 | 42.6 | 572.4 | 2.79 | 46.0 | 220.5 | -| [YOLOv8x-seg](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8x-seg.pt) | 640 | 53.4 | 43.4 | 712.1 | 4.02 | 71.8 | 344.1 | +| 模型 | 尺寸
(像素) | mAPbox
50-95 | mAPmask
50-95 | 速度
CPU ONNX
(ms) | 速度
A100 TensorRT
(ms) | 参数
(M) | FLOPs
(B) | +| -------------------------------------------------------------------------------------------- | ------------------- | -------------------- | --------------------- | ----------------------------- | ---------------------------------- | ---------------- | ----------------- | +| [YOLOv8n-seg](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8n-seg.pt) | 640 | 36.7 | 30.5 | 96.1 | 1.21 | 3.4 | 12.6 | +| [YOLOv8s-seg](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8s-seg.pt) | 640 | 44.6 | 36.8 | 155.7 | 1.47 | 11.8 | 42.6 | +| [YOLOv8m-seg](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8m-seg.pt) | 640 | 49.9 | 40.8 | 317.0 | 2.18 | 27.3 | 110.2 | +| [YOLOv8l-seg](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8l-seg.pt) | 640 | 52.3 | 42.6 | 572.4 | 2.79 | 46.0 | 220.5 | +| [YOLOv8x-seg](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8x-seg.pt) | 640 | 53.4 | 43.4 | 712.1 | 4.02 | 71.8 | 344.1 | - **mAPval** 值是基于单模型单尺度在 [COCO val2017](https://cocodataset.org) 数据集上的结果。
通过 `yolo val segment data=coco-seg.yaml device=0` 复现 - **速度** 是使用 [Amazon EC2 P4d](https://aws.amazon.com/ec2/instance-types/p4/) 实例对 COCO val 图像进行平均计算的。
通过 `yolo val segment data=coco-seg.yaml batch=1 device=0|cpu` 复现 @@ -178,14 +178,14 @@ Ultralytics 提供了 YOLOv8 的交互式笔记本，涵盖训练、验证、跟 查看[姿态文档](https://docs.ultralytics.com/tasks/pose/)以获取这些在[COCO-Pose](https://docs.ultralytics.com/datasets/pose/coco/)上训练的模型的使用示例，其中包括1个预训练类别，即人。 -| 模型 | 尺寸
(像素) | mAPpose
50-95 | mAPpose
50 | 速度
CPU ONNX
(ms) | 速度
A100 TensorRT
(ms) | 参数
(M) | FLOPs
(B) | -| ---------------------------------------------------------------------------------------------------- | --------------- | --------------------- | ------------------ | --------------------------- | -------------------------------- | -------------- | ----------------- | -| [YOLOv8n-pose](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8n-pose.pt) | 640 | 50.4 | 80.1 | 131.8 | 1.18 | 3.3 | 9.2 | -| [YOLOv8s-pose](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8s-pose.pt) | 640 | 60.0 | 86.2 | 233.2 | 1.42 | 11.6 | 30.2 | -| [YOLOv8m-pose](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8m-pose.pt) | 640 | 65.0 | 88.8 | 456.3 | 2.00 | 26.4 | 81.0 | -| [YOLOv8l-pose](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8l-pose.pt) | 640 | 67.6 | 90.0 | 784.5 | 2.59 | 44.4 | 168.6 | -| [YOLOv8x-pose](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8x-pose.pt) | 640 | 69.2 | 90.2 | 1607.1 | 3.73 | 69.4 | 263.2 | -| [YOLOv8x-pose-p6](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8x-pose-p6.pt) | 1280 | 71.6 | 91.2 | 4088.7 | 10.04 | 99.1 | 1066.4 | +| 模型 | 尺寸
(像素) | mAPpose
50-95 | mAPpose
50 | 速度
CPU ONNX
(ms) | 速度
A100 TensorRT
(ms) | 参数
(M) | FLOPs
(B) | +| ---------------------------------------------------------------------------------------------------- | ------------------- | --------------------- | ------------------ | ----------------------------- | ---------------------------------- | ---------------- | ----------------- | +| [YOLOv8n-pose](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8n-pose.pt) | 640 | 50.4 | 80.1 | 131.8 | 1.18 | 3.3 | 9.2 | +| [YOLOv8s-pose](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8s-pose.pt) | 640 | 60.0 | 86.2 | 233.2 | 1.42 | 11.6 | 30.2 | +| [YOLOv8m-pose](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8m-pose.pt) | 640 | 65.0 | 88.8 | 456.3 | 2.00 | 26.4 | 81.0 | +| [YOLOv8l-pose](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8l-pose.pt) | 640 | 67.6 | 90.0 | 784.5 | 2.59 | 44.4 | 168.6 | +| [YOLOv8x-pose](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8x-pose.pt) | 640 | 69.2 | 90.2 | 1607.1 | 3.73 | 69.4 | 263.2 | +| [YOLOv8x-pose-p6](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8x-pose-p6.pt) | 1280 | 71.6 | 91.2 | 4088.7 | 10.04 | 99.1 | 1066.4 | - **mAPval** 值是基于单模型单尺度在 [COCO Keypoints val2017](https://cocodataset.org) 数据集上的结果。
通过 `yolo val pose data=coco-pose.yaml device=0` 复现 - **速度** 是使用 [Amazon EC2 P4d](https://aws.amazon.com/ec2/instance-types/p4/) 实例对 COCO val 图像进行平均计算的。
通过 `yolo val pose data=coco-pose.yaml batch=1 device=0|cpu` 复现 @@ -196,13 +196,13 @@ Ultralytics 提供了 YOLOv8 的交互式笔记本，涵盖训练、验证、跟 查看[旋转检测文档](https://docs.ultralytics.com/tasks/obb/)以获取这些在[DOTAv1](https://docs.ultralytics.com/datasets/obb/dota-v2/#dota-v10/)上训练的模型的使用示例，其中包括15个预训练类别。 -| 模型 | 尺寸
(像素) | mAPtest
50 | 速度
CPU ONNX
(ms) | 速度
A100 TensorRT
(ms) | 参数
(M) | FLOPs
(B) | -| -------------------------------------------------------------------------------------------- | --------------- | ------------------ | --------------------------- | -------------------------------- | -------------- | ----------------- | -| [YOLOv8n-obb](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8n-obb.pt) | 1024 | 78.0 | 204.77 | 3.57 | 3.1 | 23.3 | -| [YOLOv8s-obb](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8s-obb.pt) | 1024 | 79.5 | 424.88 | 4.07 | 11.4 | 76.3 | -| [YOLOv8m-obb](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8m-obb.pt) | 1024 | 80.5 | 763.48 | 7.61 | 26.4 | 208.6 | -| [YOLOv8l-obb](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8l-obb.pt) | 1024 | 80.7 | 1278.42 | 11.83 | 44.5 | 433.8 | -| [YOLOv8x-obb](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8x-obb.pt) | 1024 | 81.36 | 1759.10 | 13.23 | 69.5 | 676.7 | +| 模型 | 尺寸
(像素) | mAPtest
50 | 速度
CPU ONNX
(ms) | 速度
A100 TensorRT
(ms) | 参数
(M) | FLOPs
(B) | +| -------------------------------------------------------------------------------------------- | ------------------- | ------------------ | ----------------------------- | ---------------------------------- | ---------------- | ----------------- | +| [YOLOv8n-obb](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8n-obb.pt) | 1024 | 78.0 | 204.77 | 3.57 | 3.1 | 23.3 | +| [YOLOv8s-obb](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8s-obb.pt) | 1024 | 79.5 | 424.88 | 4.07 | 11.4 | 76.3 | +| [YOLOv8m-obb](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8m-obb.pt) | 1024 | 80.5 | 763.48 | 7.61 | 26.4 | 208.6 | +| [YOLOv8l-obb](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8l-obb.pt) | 1024 | 80.7 | 1278.42 | 11.83 | 44.5 | 433.8 | +| [YOLOv8x-obb](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8x-obb.pt) | 1024 | 81.36 | 1759.10 | 13.23 | 69.5 | 676.7 | - **mAPval** 值是基于单模型多尺度在 [DOTAv1](https://captain-whu.github.io/DOTA/index.html) 数据集上的结果。
通过 `yolo val obb data=DOTAv1.yaml device=0 split=test` 复现 - **速度** 是使用 [Amazon EC2 P4d](https://aws.amazon.com/ec2/instance-types/p4/) 实例对 COCO val 图像进行平均计算的。
通过 `yolo val obb data=DOTAv1.yaml batch=1 device=0|cpu` 复现 @@ -213,13 +213,13 @@ Ultralytics 提供了 YOLOv8 的交互式笔记本，涵盖训练、验证、跟 查看[分类文档](https://docs.ultralytics.com/tasks/classify/)以获取这些在[ImageNet](https://docs.ultralytics.com/datasets/classify/imagenet/)上训练的模型的使用示例，其中包括1000个预训练类别。 -| 模型 | 尺寸
(像素) | acc
top1 | acc
top5 | 速度
CPU ONNX
(ms) | 速度
A100 TensorRT
(ms) | 参数
(M) | FLOPs
(B) at 640 | -| -------------------------------------------------------------------------------------------- | --------------- | ---------------- | ---------------- | --------------------------- | -------------------------------- | -------------- | ------------------------ | -| [YOLOv8n-cls](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8n-cls.pt) | 224 | 69.0 | 88.3 | 12.9 | 0.31 | 2.7 | 4.3 | -| [YOLOv8s-cls](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8s-cls.pt) | 224 | 73.8 | 91.7 | 23.4 | 0.35 | 6.4 | 13.5 | -| [YOLOv8m-cls](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8m-cls.pt) | 224 | 76.8 | 93.5 | 85.4 | 0.62 | 17.0 | 42.7 | -| [YOLOv8l-cls](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8l-cls.pt) | 224 | 78.3 | 94.2 | 163.0 | 0.87 | 37.5 | 99.7 | -| [YOLOv8x-cls](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8x-cls.pt) | 224 | 79.0 | 94.6 | 232.0 | 1.01 | 57.4 | 154.8 | +| 模型 | 尺寸
(像素) | acc
top1 | acc
top5 | 速度
CPU ONNX
(ms) | 速度
A100 TensorRT
(ms) | 参数
(M) | FLOPs
(B) at 640 | +| -------------------------------------------------------------------------------------------- | ------------------- | ---------------- | ---------------- | ----------------------------- | ---------------------------------- | ---------------- | ------------------------ | +| [YOLOv8n-cls](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8n-cls.pt) | 224 | 69.0 | 88.3 | 12.9 | 0.31 | 2.7 | 4.3 | +| [YOLOv8s-cls](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8s-cls.pt) | 224 | 73.8 | 91.7 | 23.4 | 0.35 | 6.4 | 13.5 | +| [YOLOv8m-cls](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8m-cls.pt) | 224 | 76.8 | 93.5 | 85.4 | 0.62 | 17.0 | 42.7 | +| [YOLOv8l-cls](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8l-cls.pt) | 224 | 78.3 | 94.2 | 163.0 | 0.87 | 37.5 | 99.7 | +| [YOLOv8x-cls](https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8x-cls.pt) | 224 | 79.0 | 94.6 | 232.0 | 1.01 | 57.4 | 154.8 | - **acc** 值是模型在 [ImageNet](https://www.image-net.org/) 数据集验证集上的准确率。
通过 `yolo val classify data=path/to/ImageNet device=0` 复现 - **速度** 是使用 [Amazon EC2 P4d](https://aws.amazon.com/ec2/instance-types/p4/) 实例对 ImageNet val 图像进行平均计算的。
通过 `yolo val classify data=path/to/ImageNet batch=1 device=0|cpu` 复现 @@ -250,8 +250,8 @@ Ultralytics 提供了 YOLOv8 的交互式笔记本，涵盖训练、验证、跟 NeuralMagic logo -| Roboflow | ClearML ⭐ NEW | Comet ⭐ NEW | Neural Magic ⭐ NEW | -| :--------------------------------------------------------------------------------: | :--------------------------------------------------------: | :----------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------: | +| Roboflow | ClearML ⭐ NEW | Comet ⭐ NEW | Neural Magic ⭐ NEW | +| :-------------------------------------------------------------------------------------------------------: | :------------------------------------------------------------------------------: | :------------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------------: | | 使用 [Roboflow](https://roboflow.com/?ref=ultralytics) 将您的自定义数据集直接标记并导出至 YOLOv8 进行训练 | 使用 [ClearML](https://clear.ml/)（开源！）自动跟踪、可视化，甚至远程训练 YOLOv8 | 免费且永久，[Comet](https://bit.ly/yolov8-readme-comet) 让您保存 YOLOv8 模型、恢复训练，并以交互式方式查看和调试预测 | 使用 [Neural Magic DeepSparse](https://bit.ly/yolov5-neuralmagic) 使 YOLOv8 推理速度提高多达 6 倍 | ##
Ultralytics HUB
diff --git a/docs/en/guides/ros-quickstart.md b/docs/en/guides/ros-quickstart.md index dd0abd17..21ded5ae 100644 --- a/docs/en/guides/ros-quickstart.md +++ b/docs/en/guides/ros-quickstart.md @@ -9,7 +9,7 @@ keywords: Ultralytics, YOLO, object detection, deep learning, machine learning,

ROS Introduction (captioned) from Open Robotics on Vimeo.

-### What is ROS? +## What is ROS? The [Robot Operating System (ROS)](https://www.ros.org/) is an open-source framework widely used in robotics research and industry. ROS provides a collection of [libraries and tools](https://www.ros.org/blog/ecosystem/) to help developers create robot applications. ROS is designed to work with various [robotic platforms](https://robots.ros.org/), making it a flexible and powerful tool for roboticists. @@ -56,12 +56,13 @@ This guide has been tested using [this ROS environment](https://github.com/ambit Apart from the ROS environment, you will need to install the following dependencies: - **[ROS Numpy package](https://github.com/eric-wieser/ros_numpy)**: This is required for fast conversion between ROS Image messages and numpy arrays. + ```bash pip install ros_numpy ``` - **Ultralytics package**: - + ```bash pip install ultralytics ``` @@ -74,57 +75,56 @@ The `sensor_msgs/Image` [message type](https://docs.ros.org/en/api/sensor_msgs/h Detection and Segmentation in ROS Gazebo

-!!! Example "Usage" +### Step-by-Step Usage - The following code snippet demonstrates how to use the Ultralytics YOLO package with ROS. In this example, we subscribe to a camera topic, process the incoming image using YOLO, and publish the detected objects to new topics for [detection](../tasks/detect.md) and [segmentation](../tasks/segment.md). +The following code snippet demonstrates how to use the Ultralytics YOLO package with ROS. In this example, we subscribe to a camera topic, process the incoming image using YOLO, and publish the detected objects to new topics for [detection](../tasks/detect.md) and [segmentation](../tasks/segment.md). - ### Step-by-Step Explanation - First, we import the necessary libraries and instantiate two models: one for [segmentation](../tasks/segment.md) and one for [detection](../tasks/detect.md). Next, we initialize a ROS node (with the name `ultralytics`) to enable communication with the ROS master. To ensure a stable connection, we include a brief pause, giving the node sufficient time to establish the connection before proceeding. +First, import the necessary libraries and instantiate two models: one for [segmentation](../tasks/segment.md) and one for [detection](../tasks/detect.md). Initialize a ROS node (with the name `ultralytics`) to enable communication with the ROS master. To ensure a stable connection, we include a brief pause, giving the node sufficient time to establish the connection before proceeding. - ```python - import time +```python +import time - import rospy +import rospy - from ultralytics import YOLO +from ultralytics import YOLO - detection_model = YOLO("yolov8m.pt") - segmentation_model = YOLO("yolov8m-seg.pt") - rospy.init_node("ultralytics") - time.sleep(1) - ``` +detection_model = YOLO("yolov8m.pt") +segmentation_model = YOLO("yolov8m-seg.pt") +rospy.init_node("ultralytics") +time.sleep(1) +``` - We initialize two ROS topics: one for [detection](../tasks/detect.md) and one for [segmentation](../tasks/segment.md). These topics will be used to publish the annotated images, making them accessible for further processing. The communication between nodes is facilitated using `sensor_msgs/Image` messages. +Initialize two ROS topics: one for [detection](../tasks/detect.md) and one for [segmentation](../tasks/segment.md). These topics will be used to publish the annotated images, making them accessible for further processing. The communication between nodes is facilitated using `sensor_msgs/Image` messages. - ```python - det_image_pub = rospy.Publisher("/ultralytics/detection/image", Image, queue_size=5) - seg_image_pub = rospy.Publisher("/ultralytics/segmentation/image", Image, queue_size=5) - ``` +```python +det_image_pub = rospy.Publisher("/ultralytics/detection/image", Image, queue_size=5) +seg_image_pub = rospy.Publisher("/ultralytics/segmentation/image", Image, queue_size=5) +``` - Finally, we create a subscriber that listens to messages on the `/camera/color/image_raw` topic and calls a callback function for each new message. This callback function receives messages of type `sensor_msgs/Image`, converts them into a numpy array using `ros_numpy`, processes the images with the previously instantiated YOLO models, annotates the images, and then publishes them back to the respective topics: `/ultralytics/detection/image` for detection and `/ultralytics/segmentation/image` for segmentation. - - ```python - def callback(data): - """Callback function to process image and publish annotated images.""" - array = ros_numpy.numpify(data) - if det_image_pub.get_num_connections(): - det_result = detection_model(array) - det_annotated = det_result[0].plot(show=False) - det_image_pub.publish(ros_numpy.msgify(Image, det_annotated, encoding="rgb8")) +Finally, create a subscriber that listens to messages on the `/camera/color/image_raw` topic and calls a callback function for each new message. This callback function receives messages of type `sensor_msgs/Image`, converts them into a numpy array using `ros_numpy`, processes the images with the previously instantiated YOLO models, annotates the images, and then publishes them back to the respective topics: `/ultralytics/detection/image` for detection and `/ultralytics/segmentation/image` for segmentation. - if seg_image_pub.get_num_connections(): - seg_result = segmentation_model(array) - seg_annotated = seg_result[0].plot(show=False) - seg_image_pub.publish(ros_numpy.msgify(Image, seg_annotated, encoding="rgb8")) +```python +def callback(data): + """Callback function to process image and publish annotated images.""" + array = ros_numpy.numpify(data) + if det_image_pub.get_num_connections(): + det_result = detection_model(array) + det_annotated = det_result[0].plot(show=False) + det_image_pub.publish(ros_numpy.msgify(Image, det_annotated, encoding="rgb8")) + + if seg_image_pub.get_num_connections(): + seg_result = segmentation_model(array) + seg_annotated = seg_result[0].plot(show=False) + seg_image_pub.publish(ros_numpy.msgify(Image, seg_annotated, encoding="rgb8")) - rospy.Subscriber("/camera/color/image_raw", Image, callback) +rospy.Subscriber("/camera/color/image_raw", Image, callback) - while True: - rospy.spin() - ``` +while True: + rospy.spin() +``` - ### Complete code +??? Example "Complete code" ```python import time @@ -174,45 +174,47 @@ The `sensor_msgs/Image` [message type](https://docs.ros.org/en/api/sensor_msgs/h 4. For more complex visualizations, such as 3D representations, you can use [RViz](https://wiki.ros.org/rviz). RViz (ROS Visualization) is a powerful 3D visualization tool for ROS. It allows you to visualize the state of your robot and its environment in real-time. With RViz, you can view sensor data (e.g. `sensors_msgs/Image`), robot model states, and various other types of information, making it easier to debug and understand the behavior of your robotic system. ### Publish Detected Classes with `std_msgs/String` + Standard ROS messages also include `std_msgs/String` messages. In many applications, it is not necessary to republish the entire annotated image; instead, only the classes present in the robot's view are needed. The following example demonstrates how to use `std_msgs/String` [messages](https://docs.ros.org/en/noetic/api/std_msgs/html/msg/String.html) to republish the detected classes on the `/ultralytics/detection/classes` topic. These messages are more lightweight and provide essential information, making them valuable for various applications. #### Example Use Case + Consider a warehouse robot equipped with a camera and object [detection model](../tasks/detect.md). Instead of sending large annotated images over the network, the robot can publish a list of detected classes as `std_msgs/String` messages. For instance, when the robot detects objects like "box", "pallet" and "forklift" it publishes these classes to the `/ultralytics/detection/classes` topic. This information can then be used by a central monitoring system to track the inventory in real-time, optimize the robot's path planning to avoid obstacles, or trigger specific actions such as picking up a detected box. This approach reduces the bandwidth required for communication and focuses on transmitting critical data. -!!! Example "Usage" +### Step-by-Step Usage - ```python - import time +This example demonstrates how to use the Ultralytics YOLO package with ROS. In this example, we subscribe to a camera topic, process the incoming image using YOLO, and publish the detected objects to new topic `/ultralytics/detection/classes` using `std_msgs/String` messages. The `ros_numpy` package is used to convert the ROS Image message to a numpy array for processing with YOLO. - import ros_numpy - import rospy - from sensor_msgs.msg import Image - from std_msgs.msg import String +```python +import time - from ultralytics import YOLO +import ros_numpy +import rospy +from sensor_msgs.msg import Image +from std_msgs.msg import String - detection_model = YOLO("yolov8m.pt") - rospy.init_node("ultralytics") - time.sleep(1) - classes_pub = rospy.Publisher("/ultralytics/detection/classes", String, queue_size=5) +from ultralytics import YOLO + +detection_model = YOLO("yolov8m.pt") +rospy.init_node("ultralytics") +time.sleep(1) +classes_pub = rospy.Publisher("/ultralytics/detection/classes", String, queue_size=5) - def callback(data): - """Callback function to process image and publish detected classes.""" - array = ros_numpy.numpify(data) - if classes_pub.get_num_connections(): - det_result = detection_model(array) - classes = det_result[0].boxes.cls.cpu().numpy().astype(int) - names = [det_result[0].names[i] for i in classes] - classes_pub.publish(String(data=str(names))) +def callback(data): + """Callback function to process image and publish detected classes.""" + array = ros_numpy.numpify(data) + if classes_pub.get_num_connections(): + det_result = detection_model(array) + classes = det_result[0].boxes.cls.cpu().numpy().astype(int) + names = [det_result[0].names[i] for i in classes] + classes_pub.publish(String(data=str(names))) - rospy.Subscriber("/camera/color/image_raw", Image, callback) - while True: - rospy.spin() - ``` - - This example demonstrates how to use the Ultralytics YOLO package with ROS. In this example, we subscribe to a camera topic, process the incoming image using YOLO, and publish the detected objects to new topic `/ultralytics/detection/classes` using `std_msgs/String` messages. The `ros_numpy` package is used to convert the ROS Image message to a numpy array for processing with YOLO. +rospy.Subscriber("/camera/color/image_raw", Image, callback) +while True: + rospy.spin() +``` ## Use Ultralytics with ROS Depth Images @@ -220,7 +222,7 @@ In addition to RGB images, ROS supports [depth images](https://en.wikipedia.org/ A depth image is an image where each pixel represents the distance from the camera to an object. Unlike RGB images that capture color, depth images capture spatial information, enabling robots to perceive the 3D structure of their environment. -!!! tip "Obtaining Depth Images" +!!! tip "Obtaining Depth Images" Depth images can be obtained using various sensors: @@ -228,22 +230,77 @@ A depth image is an image where each pixel represents the distance from the came 2. [Time-of-Flight (ToF) Cameras](https://en.wikipedia.org/wiki/Time-of-flight_camera): Measure the time light takes to return from an object. 3. [Structured Light Sensors](https://en.wikipedia.org/wiki/Structured-light_3D_scanner): Project a pattern and measure its deformation on surfaces. -#### Using YOLO with Depth Images +### Using YOLO with Depth Images + In ROS, depth images are represented by the `sensor_msgs/Image` message type, which includes fields for encoding, height, width, and pixel data. The encoding field for depth images often uses a format like "16UC1", indicating a 16-bit unsigned integer per pixel, where each value represents the distance to the object. Depth images are commonly used in conjunction with RGB images to provide a more comprehensive view of the environment. Using YOLO, it is possible to extract and combine information from both RGB and depth images. For instance, YOLO can detect objects within an RGB image, and this detection can be used to pinpoint corresponding regions in the depth image. This allows for the extraction of precise depth information for detected objects, enhancing the robot's ability to understand its environment in three dimensions. !!! warning "RGB-D Cameras" - When working with depth images, it is essential to ensure that the RGB and depth images are correctly aligned. RGB-D cameras, such as the [Intel RealSense](https://www.intelrealsense.com/) series, provide synchronized RGB and depth images, making it easier to combine information from both sources. If using separate RGB and depth cameras, it is crucial to calibrate them to ensure accurate alignment. -!!! Example "Usage" + When working with depth images, it is essential to ensure that the RGB and depth images are correctly aligned. RGB-D cameras, such as the [Intel RealSense](https://www.intelrealsense.com/) series, provide synchronized RGB and depth images, making it easier to combine information from both sources. If using separate RGB and depth cameras, it is crucial to calibrate them to ensure accurate alignment. - In this example, we use YOLO to segment an image and apply the extracted mask to segment the object in the depth image. This allows us to determine the distance of each pixel of the object of interest from the camera's focal center. By obtaining this distance information, we can calculate the distance between the camera and the specific object in the scene. We begin by importing the necessary libraries, creating a ROS node, and instantiating a segmentation model and a ROS topic. +#### Step-by-Step Usage - ```python - import time +In this example, we use YOLO to segment an image and apply the extracted mask to segment the object in the depth image. This allows us to determine the distance of each pixel of the object of interest from the camera's focal center. By obtaining this distance information, we can calculate the distance between the camera and the specific object in the scene. Begin by importing the necessary libraries, creating a ROS node, and instantiating a segmentation model and a ROS topic. +```python +import time + +import rospy +from std_msgs.msg import String + +from ultralytics import YOLO + +rospy.init_node("ultralytics") +time.sleep(1) + +segmentation_model = YOLO("yolov8m-seg.pt") + +classes_pub = rospy.Publisher("/ultralytics/detection/distance", String, queue_size=5) +``` + +Next, define a callback function that processes the incoming depth image message. The function waits for the depth image and RGB image messages, converts them into numpy arrays, and applies the segmentation model to the RGB image. It then extracts the segmentation mask for each detected object and calculates the average distance of the object from the camera using the depth image. Most sensors have a maximum distance, known as the clip distance, beyond which values are represented as inf (`np.inf`). Before processing, it is important to filter out these null values and assign them a value of `0`. Finally, it publishes the detected objects along with their average distances to the `/ultralytics/detection/distance` topic. + +```python +import numpy as np +import ros_numpy +from sensor_msgs.msg import Image + + +def callback(data): + """Callback function to process depth image and RGB image.""" + image = rospy.wait_for_message("/camera/color/image_raw", Image) + image = ros_numpy.numpify(image) + depth = ros_numpy.numpify(data) + result = segmentation_model(image) + + for index, cls in enumerate(result[0].boxes.cls): + class_index = int(cls.cpu().numpy()) + name = result[0].names[class_index] + mask = result[0].masks.data.cpu().numpy()[index, :, :].astype(int) + obj = depth[mask == 1] + obj = obj[~np.isnan(obj)] + avg_distance = np.mean(obj) if len(obj) else np.inf + + classes_pub.publish(String(data=str(all_objects))) + + +rospy.Subscriber("/camera/depth/image_raw", Image, callback) + +while True: + rospy.spin() +``` + +??? Example "Complete code" + +````python +import time + + import numpy as np + import ros_numpy import rospy + from sensor_msgs.msg import Image from std_msgs.msg import String from ultralytics import YOLO @@ -254,11 +311,8 @@ Using YOLO, it is possible to extract and combine information from both RGB and segmentation_model = YOLO("yolov8m-seg.pt") classes_pub = rospy.Publisher("/ultralytics/detection/distance", String, queue_size=5) - ``` - Next, we define a callback function that processes the incoming depth image message. The function waits for the depth image and RGB image messages, converts them into numpy arrays, and applies the segmentation model to the RGB image. It then extracts the segmentation mask for each detected object and calculates the average distance of the object from the camera using the depth image. Most sensors have a maximum distance, known as the clip distance, beyond which values are represented as inf (`np.inf`). Before processing, it is important to filter out these null values and assign them a value of `0`. Finally, it publishes the detected objects along with their average distances to the `/ultralytics/detection/distance` topic. - ```python def callback(data): """Callback function to process depth image and RGB image.""" image = rospy.wait_for_message("/camera/color/image_raw", Image) @@ -294,9 +348,10 @@ The `sensor_msgs/PointCloud2` [message type](https://docs.ros.org/en/api/sensor_ A point cloud is a collection of data points defined within a three-dimensional coordinate system. These data points represent the external surface of an object or a scene, captured via 3D scanning technologies. Each point in the cloud has `X`, `Y`, and `Z` coordinates, which correspond to its position in space, and may also include additional information such as color and intensity. !!! warning "reference frame" + When working with `sensor_msgs/PointCloud2`, it's essential to consider the reference frame of the sensor from which the point cloud data was acquired. The point cloud is initially captured in the sensor's reference frame. You can determine this reference frame by listening to the `/tf_static` topic. However, depending on your specific application requirements, you might need to convert the point cloud into another reference frame. This transformation can be achieved using the `tf2_ros` package, which provides tools for managing coordinate frames and transforming data between them. -!!! tip "Obtaining Point clouds" +!!! tip "Obtaining Point clouds" Point Clouds can be obtained using various sensors: @@ -305,19 +360,97 @@ A point cloud is a collection of data points defined within a three-dimensional 3. **Stereo Cameras**: Utilize two or more cameras to obtain depth information through triangulation. 4. **Structured Light Scanners**: Project a known pattern onto a surface and measure the deformation to calculate depth. -#### Using YOLO with Point Clouds +### Using YOLO with Point Clouds To integrate YOLO with `sensor_msgs/PointCloud2` type messages, we can employ a method similar to the one used for depth maps. By leveraging the color information embedded in the point cloud, we can extract a 2D image, perform segmentation on this image using YOLO, and then apply the resulting mask to the three-dimensional points to isolate the 3D object of interest. For handling point clouds, we recommend using Open3D (`pip install open3d`), a user-friendly Python library. Open3D provides robust tools for managing point cloud data structures, visualizing them, and executing complex operations seamlessly. This library can significantly simplify the process and enhance our ability to manipulate and analyze point clouds in conjunction with YOLO-based segmentation. -!!! Example "Usage" +#### Step-by-Step Usage - Import the necessary libraries and instantiate the YOLO model for segmentation. +Import the necessary libraries and instantiate the YOLO model for segmentation. +```python +import time - ```python - import time +import rospy +from ultralytics import YOLO + +rospy.init_node("ultralytics") +time.sleep(1) +segmentation_model = YOLO("yolov8m-seg.pt") +```` + +Create a function `pointcloud2_to_array`, which transforms a `sensor_msgs/PointCloud2` message into two numpy arrays. The `sensor_msgs/PointCloud2` messages contain `n` points based on the `width` and `height` of the acquired image. For instance, a `480 x 640` image will have `307,200` points. Each point includes three spatial coordinates (`xyz`) and the corresponding color in `RGB` format. These can be considered as two separate channels of information. + +The function returns the `xyz` coordinates and `RGB` values in the format of the original camera resolution (`width x height`). Most sensors have a maximum distance, known as the clip distance, beyond which values are represented as inf (`np.inf`). Before processing, it is important to filter out these null values and assign them a value of `0`. + +```python +import numpy as np +import ros_numpy + + +def pointcloud2_to_array(pointcloud2: PointCloud2) -> tuple: + """ + Convert a ROS PointCloud2 message to a numpy array. + + Args: + pointcloud2 (PointCloud2): the PointCloud2 message + + Returns: + (tuple): tuple containing (xyz, rgb) + """ + pc_array = ros_numpy.point_cloud2.pointcloud2_to_array(pointcloud2) + split = ros_numpy.point_cloud2.split_rgb_field(pc_array) + rgb = np.stack([split["b"], split["g"], split["r"]], axis=2) + xyz = ros_numpy.point_cloud2.get_xyz_points(pc_array, remove_nans=False) + xyz = np.array(xyz).reshape((pointcloud2.height, pointcloud2.width, 3)) + nan_rows = np.isnan(xyz).all(axis=2) + xyz[nan_rows] = [0, 0, 0] + rgb[nan_rows] = [0, 0, 0] + return xyz, rgb +``` + +Next, subscribe to the `/camera/depth/points` topic to receive the point cloud message and convert the `sensor_msgs/PointCloud2` message into numpy arrays containing the XYZ coordinates and RGB values (using the `pointcloud2_to_array` function). Process the RGB image using the YOLO model to extract segmented objects. For each detected object, extract the segmentation mask and apply it to both the RGB image and the XYZ coordinates to isolate the object in 3D space. + +Processing the mask is straightforward since it consists of binary values, with `1` indicating the presence of the object and `0` indicating the absence. To apply the mask, simply multiply the original channels by the mask. This operation effectively isolates the object of interest within the image. Finally, create an Open3D point cloud object and visualize the segmented object in 3D space with associated colors. + +```python +import sys + +import open3d as o3d + +ros_cloud = rospy.wait_for_message("/camera/depth/points", PointCloud2) +xyz, rgb = pointcloud2_to_array(ros_cloud) +result = segmentation_model(rgb) + +if not len(result[0].boxes.cls): + print("No objects detected") + sys.exit() + +classes = result[0].boxes.cls.cpu().numpy().astype(int) +for index, class_id in enumerate(classes): + mask = result[0].masks.data.cpu().numpy()[index, :, :].astype(int) + mask_expanded = np.stack([mask, mask, mask], axis=2) + + obj_rgb = rgb * mask_expanded + obj_xyz = xyz * mask_expanded + + pcd = o3d.geometry.PointCloud() + pcd.points = o3d.utility.Vector3dVector(obj_xyz.reshape((ros_cloud.height * ros_cloud.width, 3))) + pcd.colors = o3d.utility.Vector3dVector(obj_rgb.reshape((ros_cloud.height * ros_cloud.width, 3)) / 255) + o3d.visualization.draw_geometries([pcd]) +``` + +??? Example "Complete code" + +````python +import sys +import time + + import numpy as np + import open3d as o3d + import ros_numpy import rospy from ultralytics import YOLO @@ -325,11 +458,8 @@ For handling point clouds, we recommend using Open3D (`pip install open3d`), a u rospy.init_node("ultralytics") time.sleep(1) segmentation_model = YOLO("yolov8m-seg.pt") - ``` - Let's create a function `pointcloud2_to_array`, which transforms a `sensor_msgs/PointCloud2` message into two numpy arrays. The `sensor_msgs/PointCloud2` messages contain `n` points based on the `width` and `height` of the acquired image. For instance, a `480 x 640` image will have `307,200` points. Each point includes three spatial coordinates (`xyz`) and the corresponding color in `RGB` format. These can be considered as two separate channels of information. The function returns the `xyz` coordinates and `RGB` values in the format of the original camera resolution (`width x height`). Most sensors have a maximum distance, known as the clip distance, beyond which values are represented as inf (`np.inf`). Before processing, it is important to filter out these null values and assign them a value of `0`. - ```python def pointcloud2_to_array(pointcloud2: PointCloud2) -> tuple: """ Convert a ROS PointCloud2 message to a numpy array. @@ -349,11 +479,8 @@ For handling point clouds, we recommend using Open3D (`pip install open3d`), a u xyz[nan_rows] = [0, 0, 0] rgb[nan_rows] = [0, 0, 0] return xyz, rgb - ``` - Next, we subscribe to the `/camera/depth/points` topic to receive the point cloud message. We then convert the `sensor_msgs/PointCloud2` message into numpy arrays containing the XYZ coordinates and RGB values (using the `pointcloud2_to_array` function). We process the RGB image using the YOLO model to extract segmented objects. For each detected object, we extract the segmentation mask and apply it to both the RGB image and the XYZ coordinates to isolate the object in 3D space. Processing the mask is straightforward since it consists of binary values, with `1` indicating the presence of the object and `0` indicating the absence. To apply the mask, simply multiply the original channels by the mask. This operation effectively isolates the object of interest within the image. Finally, we create an Open3D point cloud object and visualize the segmented object in 3D space with associated colors. - ```python ros_cloud = rospy.wait_for_message("/camera/depth/points", PointCloud2) xyz, rgb = pointcloud2_to_array(ros_cloud) result = segmentation_model(rgb) @@ -379,3 +506,4 @@ For handling point clouds, we recommend using Open3D (`pip install open3d`), a u

Point Cloud Segmentation with Ultralytics

+```` diff --git a/docs/en/integrations/tensorrt.md b/docs/en/integrations/tensorrt.md index 583bef10..b37d6615 100644 --- a/docs/en/integrations/tensorrt.md +++ b/docs/en/integrations/tensorrt.md @@ -137,9 +137,9 @@ The arguments provided when using [export](../modes/export.md) for an Ultralytic - `batch` : The maximum batch-size that will be used for inference. During inference smaller batches can be used, but inference will not accept batches any larger than what is specified. - !!! note +!!! note - During calibration, twice the `batch` size provided will be used. Using small batches can lead to inaccurate scaling during calibration. This is because the process adjusts based on the data it sees. Small batches might not capture the full range of values, leading to issues with the final calibration, so the `batch` size is doubled automatically. If no batch size is specified `batch=1`, calibration will be run at `batch=1 * 2` to reduce calibration scaling errors. + During calibration, twice the `batch` size provided will be used. Using small batches can lead to inaccurate scaling during calibration. This is because the process adjusts based on the data it sees. Small batches might not capture the full range of values, leading to issues with the final calibration, so the `batch` size is doubled automatically. If no batch size is specified `batch=1`, calibration will be run at `batch=1 * 2` to reduce calibration scaling errors. Experimentation by NVIDIA led them to recommend using at least 500 calibration images that are representative of the data for your model, with INT8 quantization calibration. This is a guideline and not a _hard_ requirement, and **you will need to experiment with what is required to perform well for your dataset**. Since the calibration data is required for INT8 calibration with TensorRT, make certain to use the `data` argument when `int8=True` for TensorRT and use `data="my_dataset.yaml"`, which will use the images from [validation](../modes/val.md) to calibrate with. When no value is passed for `data` with export to TensorRT with INT8 quantization, the default will be to use one of the ["small" example datasets based on the model task](../datasets/index.md) instead of throwing an error. diff --git a/docs/en/integrations/weights-biases.md b/docs/en/integrations/weights-biases.md index 834b2685..0458c5f6 100644 --- a/docs/en/integrations/weights-biases.md +++ b/docs/en/integrations/weights-biases.md @@ -13,7 +13,7 @@ This guide showcases Ultralytics YOLOv8 integration with Weights & Biases' for e ## Weights & Biases

- Weights & Biases Overview + Weights & Biases Overview

[Weights & Biases](https://wandb.ai/site) is a cutting-edge MLOps platform designed for tracking, visualizing, and managing machine learning experiments. It features automatic logging of training metrics for full experiment reproducibility, an interactive UI for streamlined data analysis, and efficient model management tools for deploying across various environments. @@ -126,25 +126,19 @@ After running the usage code snippet, you can access the Weights & Biases (W&B) ## Key Features of the Weights & Biases Dashboard -- **Real-Time Metrics Tracking**: Observe metrics like loss, accuracy, and validation scores as they evolve during the training, offering immediate insights for model tuning. - - +- **Real-Time Metrics Tracking**: Observe metrics like loss, accuracy, and validation scores as they evolve during the training, offering immediate insights for model tuning. [See how experiments are tracked using Weights & Biases](https://imgur.com/D6NVnmN). - **Hyperparameter Optimization**: Weights & Biases aids in fine-tuning critical parameters such as learning rate, batch size, and more, enhancing the performance of YOLOv8. - **Comparative Analysis**: The platform allows side-by-side comparisons of different training runs, essential for assessing the impact of various model configurations. -- **Visualization of Training Progress**: Graphical representations of key metrics provide an intuitive understanding of the model's performance across epochs. - - +- **Visualization of Training Progress**: Graphical representations of key metrics provide an intuitive understanding of the model's performance across epochs. [See how Weights & Biases helps you visualize validation results](https://imgur.com/a/kU5h7W4). - **Resource Monitoring**: Keep track of CPU, GPU, and memory usage to optimize the efficiency of the training process. - **Model Artifacts Management**: Access and share model checkpoints, facilitating easy deployment and collaboration. -- **Viewing Inference Results with Image Overlay**: Visualize the prediction results on images using interactive overlays in Weights & Biases, providing a clear and detailed view of model performance on real-world data. For more detailed information on Weights & Biases' image overlay capabilities, check out this [link](https://docs.wandb.ai/guides/track/log/media#image-overlays). - - +- **Viewing Inference Results with Image Overlay**: Visualize the prediction results on images using interactive overlays in Weights & Biases, providing a clear and detailed view of model performance on real-world data. For more detailed information on Weights & Biases' image overlay capabilities, check out this [link](https://docs.wandb.ai/guides/track/log/media#image-overlays). [See how Weights & Biases' image overlays helps visualize model inferences](https://imgur.com/a/UTSiufs). By using these features, you can effectively track, analyze, and optimize your YOLOv8 model's training, ensuring the best possible performance and efficiency. diff --git a/docs/en/models/yolov9.md b/docs/en/models/yolov9.md index 3c5c6f3a..249b2634 100644 --- a/docs/en/models/yolov9.md +++ b/docs/en/models/yolov9.md @@ -149,7 +149,7 @@ This example provides simple YOLOv9 training and inference examples. For full do The YOLOv9 series offers a range of models, each optimized for high-performance [Object Detection](../tasks/detect.md). These models cater to varying computational needs and accuracy requirements, making them versatile for a wide array of applications. | Model | Filenames | Tasks | Inference | Validation | Training | Export | -|------------|---------------------------------------------------------|----------------------------------------------|-----------|------------|----------|--------| +| ---------- | ------------------------------------------------------- | -------------------------------------------- | --------- | ---------- | -------- | ------ | | YOLOv9 | `yolov9t` `yolov9s` `yolov9m` `yolov9c.pt` `yolov9e.pt` | [Object Detection](../tasks/detect.md) | ✅ | ✅ | ✅ | ✅ | | YOLOv9-seg | `yolov9c-seg.pt` `yolov9e-seg.pt` | [Instance Segmentation](../tasks/segment.md) | ✅ | ✅ | ✅ | ✅ | diff --git a/docs/en/usage/cli.md b/docs/en/usage/cli.md index 2bf11b68..2cc25e90 100644 --- a/docs/en/usage/cli.md +++ b/docs/en/usage/cli.md @@ -78,7 +78,6 @@ Where: - `TASK` (optional) is one of `[detect, segment, classify]`. If it is not passed explicitly YOLOv8 will try to guess the `TASK` from the model type. - `MODE` (required) is one of `[train, val, predict, export, track]` - `ARGS` (optional) are any number of custom `arg=value` pairs like `imgsz=320` that override defaults. For a full list of available `ARGS` see the [Configuration](cfg.md) page and `defaults.yaml` - GitHub [source](https://github.com/ultralytics/ultralytics/blob/main/ultralytics/cfg/default.yaml). !!! Warning "Warning"