AXIS Scene Metadata integration

AXIS Scene Metadata enhances scene understanding, providing critical details such as object classes (human or vehicle), clothing and vehicle colors, license plates, and speed data. This in turn enables rapid decision-making, automated actions, and simplified search. Seamlessly integrated with third-party solutions through standardized methods, and delivered directly from Axis cameras, AXIS Scene Metadata helps reduce system and operational costs while ensuring efficiency and precision.

AXIS Scene Metadata includes information about the object types as well as other specific attributes such as clothing and vehicle color, license plate information, speed, location, and timestamp. AI-based analytics, featuring object detection and classification, lets you filter the attained metadata, trigger events, and examine specific elements of the scene.

A complete list of supported object classes and attributes is available below, but please note that the detection and classification capabilities are camera dependent. For detailed information about which object classes and attributes a specific camera model supports, please refer to the camera's data sheet.

• Human
• Vehicle
  • Car
  • Bus
  • Truck
  • Bike (Motorcycle/Bicycle)
  • Other
• Human Face
• License Plate

• Clothing Color (Upper/Lower)
• Vehicle Color

In addition to the listed object attributes, the analytics metadata stream may also provide information about other properties of the detected objects such as likelihood, duration, shape, images, speed*, geocoordinates* and position. 

*Requires radar or radar-video fusion camera integration.

It is also worth noting that AXIS Scene Metadata generates information such as shape and position about non-classified moving objects in the video scene as well. 

Analytics metadata not only provides details about objects in a scene. It also provides context to events and allows large amounts of footage to be quickly sorted and searched. This enables functions that can be broadly categorized into the areas of post-event forensic search, real-time use, and identification of trends, patterns, and insights.

Axis devices generate analytics metadata that is conformant with ONVIF Profile M streaming over RTSP to support use-cases related to post-event forensic search. However, this metadata is also accessible through alternative communication protocols and file formats, enabling a straight forward integration with a wide variety of systems covering a large array of use-cases. This guide provides important information such as suggested architecture, development examples, and recommended considerations when designing a solution that consumes this analytics metadata.

This guide also assumes that you are familiar with the concept of video metadata and more specifically with the capabilities that Axis cameras posses for detection of classified objects in a scene. In order to better understand the added value of video metadata, as well as an introduction to the capabilities of Axis cameras in this context, consider visiting the AXIS Scene Metadata product page and reading our white paper if you haven’t done so already.

Axis devices with AXIS OS 11.11 or later support two different methods of generating analytics metadata, each being designed and optimized to support specific use-cases, such as real-time alarm notifications or post-event forensic search and identification of trends, patterns, and insights. To retrieve information regarding the metadata analytics producers on the Axis camera and the kind of metadata they can produce, please refer to the Analytics Metadata Producer Configuration API available in the VAPIX Library.

Consider whether your application requires the device to deliver the information in real-time when selecting which method is most suitable to fulfill your specific use-case. In the following section, we will take a deeper dive into each of the available methods.

The analytics metadata streams can also be configured to include cropped images of detected classified objects using the Best Snapshot feature.

The image is represented as a base-64 encoded string within the metadata output. For examples, please refer to sample data frames which include images in the Ways of accessing the metadata section. The Best Snapshot feature must be enabled manually by issuing the following request to the camera:

In this section, we will provide examples of how different consumer types are capable of accessing and consuming AXIS Scene Metadata.

AXIS Camera Application Platform (ACAP) is an open application platform from Axis. It provides a development platform for software-based solutions and systems built around Axis devices. ACAP is available for various types of Axis products such as cameras, speakers and intercoms.

The ACAP Native SDK is targeted towards users that want to develop plug-in style, event generating applications that fit well into a Video Management System centric environment. From AXIS OS 11.9, ACAP applications can consume AXIS Scene Metadata leveraging the Message Broker to further apply logical filters and rules to the information about the object in the scene in order to, for example, trigger actions based on defined thresholds or specific behaviors.

Example

This example showcases how an ACAP application can consume frame-by-frame or consolidated analytics metadata using the Message Broker API.

The available topics to subscribe to are:

Please refer to the following resource for more detailed information on available topics and terminology.

Axis devices send both the analytics metadata and video streams to the Video management system (vms) to enable forensic search integrations. Two examples of these integrations are AXIS Optimizer forensic search for Milestone plugin and AXIS Forensic Search for Genetec. 

The Analytics Scene Description metadata stream can be retrieved from an Axis device by opening an RTSP stream that uses the TCP transport protocol according to the following example:

1) Information about the type of metadata stream generated by the device.

2) Frame timestamp crucial for syncing metadata with video (or audio) when you play or query and source field.

3) Information for bounding boxes and polygons, represented in ONVIF coordinate system which is -1 to 1 in the X and Y axes.

4) Bounding boxes and polygons are currently the same if you use Analytics Scene Description as a source.

5) Represents the color (of the car) and the probability value of the object classification. The color is presented before object class due to the ONVIF Profile M format. Object class can be found in section 6.

6) Object class and probability value of the object classification, such as vehicle in this example.

7) In addition to the main category, it presents a sub-category, such as a car in this example.

Additional parameters such as "camera=2" can be added to the above request to e.g. receive metadata events from a different video channel. This is useful when the Axis device supports more than one video source.

Please visit the AXIS OS knowledge base for additional information.

Some applications require a combination of edge-based and server-based processing to perform more advanced analyses. Pre-processing can be performed on the camera and further processing on a server. Such a hybrid system can facilitate cost-efficient scaling of analytics by streaming only relevant video or images along with metadata to the server.

MQTT is a standard messaging protocol that facilitates efficient and reliable exchange of data between IoT devices and cloud applications. It allows devices (through their MQTT clients) to publish messages to a common MQTT broker (server) that mediates communication with other devices. The broker keeps track of who is publishing what and who wants to see the data, forwarding messages to only the clients that subscribe to the right topic. In a typical VMS ecosystem, Axis event notifications from devices are traditionally streamed to a single destination via VAPIX/ONVIF API interface using the RTSP streaming protocol. But the same notifications can be distributed using the MQTT protocol via the device’s built-in MQTT client (applicable for devices running AXIS OS 9.80 or a later version). This is possible both within VMS ecosystems and outside of them, and is particularly useful over the internet. This guide does not provide an overview of the MQTT protocol or specific configuration of the MQTT client on the camera. To find out more about these topics, please visit Device integration with MQTT and VAPIX MQTT Client API.

The capability of obtaining analytics metadata using the MQTT client available on Axis cameras is available by enabling a feature flag through the Feature Flag Service API. Feature flags can be used to, among other things, toggle experimental features on/off and to enable prototyping a solution early on in the development process to get feedback from developers.

Developers interested in exploring this capability can obtain detailed instructions on how to enable this feature by creating a support helpdesk ticket specifying AXIS Scene Metadata in the product category field. Please include information about your project and use-case in the description field.

You now posses the necessary information to develop solutions based on the analytics metadata content generated by Axis cameras. The components making up a solution can vary drastically depending on the specific needs of your project.

Analytics metadata can be used in real time to help operators respond quickly to situational changes. It can also provide valuable input to support decision making or enable automated action.

Real time edge analytics that work with high-quality metadata can help you secure people, sites, and buildings and protect them from intentional or accidental harm. You can rapidly detect, verify, and evaluate threats so they can be efficiently handled.

The analytics metadata is typical stored in a database type component which can be queried regularly to extract the data that is of interest and visually present it in dashboard.

In the case of leveraging cloud services to perform advanced analysis on a detected object, however, the database component might not be required if there is a need to present the data in as close to real-time as possible. 

An example implementation is represented below.

1) An Axis camera with MLPU or DLPU, generating analytics metadata

2) The analytics metadata is transmitted to consumers through the available communication protocols

3) The analytics metadata is further processed and stored, then consumed by different applications

4a) Collected data is visualized in a graphical dashboard to analyze trends and gain insights

4b) Access to a sensitive area is restricted based on license plate information included in the analytics metadata