How Do Generative Image AIs Work? Searching Query Using Scene Graph Part 1.

With the rise of image-generating AI, more people are curious about the capabilities of graph databases. These databases can decode the intricate behaviors of images, and scene graphs play a pivotal role in searching for images and identifying those with remarkable similarities.

This article delves into the innovative methods that revolutionize image analysis and the retrieval of visual data in the digital landscape.

What is a Scene Graph?

A scene graph is a graph-based analysis method that represents the attribute information inherent in images or image data as a graph. It captures not only objects but also the relationships between them. For example, it uses the Subject-Predicate-Object (SPO) relationship to describe scenes, as shown in Figure 1.

Figure 1. Example of Scene Graph (Source: Scene Graph Generation by Interactive Message Passing)

Searching for Subject-Object-Predicate Relationships

To efficiently search for images and objects using a scene graph, the graph modeling consists of nodes and edges. The nouns of the image and the object are represented in nodes. The edge that connects the image to the noun is represented as an 'eg_connect' edge, while an 'eg_predicate' edge connects the noun to the object as a predicate, shown in Figure 2.

Figure 2. SPO representation

The 'eg_predicate' edge can include a list of images as "img_id_arr" to retrieve images from the predicate without unnecessary navigation. This allows for a focused search on relationships related to the images and predicate relationships between objects.

Logic Behind the Search Queries

Below are the logics for retrieving the SPO corresponding to an image or searching for images that contain SPOs

Figure 3. SPO from Image (Image->SPO) Query Logic

Search Query: "Show me the SPO results for image No.1"

• Search for the noun connected to the 'Image 1' node

• Find all the 'Noun nodes' connected to the 'Image 1' node

• Filter the SPOs that contain 'Image 1' in the property of the 'Predicate edge' connected between the 'Noun nodes'

• Extract the SPOs corresponding to the 'image 1 node'

This method can scan for unique predicate edges, resulting in faster computation compared to a relational database that must search for all instances.

Figure 4. Image Query with SPO (SPO-> Image) Query Logic

Search Query: "Show me photos that contain the information 'man (S) driving (P) Car (O)"

• Perform the SPO search for each noun node and predicate edge

• List the images stored in the predicate edge property

This method also scans for unique predicate edges, ensuring faster computation.

Performance of Complex Search Queries

Graph databases shine when tasked with searching for intricate SPO relationships involving multiple hops. For example, a query seeking “pictures of a man wearing Nike shoes while playing tennis" will significantly degrade query performance in a relational database due to numerous join relationships. In contrast, graph databases handle these queries efficiently.

Figure 5. Example of Complex Query

In this article, we explored the logic of search queries to retrieve and extract 'Subject-Predicate-Object' relationships between images or objects in an image using scene graphs.

Part 2 will explain how to extract similar images with a graph database's scene graph and how efficient it is compared to traditional relational databases.

Stay tuned for the next blog to learn more about the usage of scene graphs!

If you have any questions or want to query your data in graph databases, contact us today to get started!