Adding nullable column to postgres images table with type Vector to s… #357

ahmednasserswe · 2023-11-08T19:55:07Z

Description

Adding nullable column to postgres images table with type Vector to store sscd embedding vectors.
This included changes to both Postgress dockerfile and Alegre dockerfile.

Reference: TICKET-ID CV2-3917

How has this been tested?

tested locally

Have you considered secure coding practices when writing this code?

No potential security threads

…tore sscd embedding vectors.

skyemeedan · 2023-11-09T09:46:58Z

This looks really promising, probably lots of cool things we can do with direct access to the vector. But it looks like we need to first get Alegre migrated to python 3.8? Build is failing with

ERROR: Ignored the following versions that require a different python version: 0.2.0 Requires-Python >=3.8; 0.2.1 Requires-Python >=3.8; 0.2.2 Requires-Python >=3.8; 0.2.3 Requires-Python >=3.8

ERROR: Could not find a version that satisfies the requirement pgvector==0.2.3 (from versions: 0.1.0, 0.1.1, 0.1.2, 0.1.3, 0.1.4, 0.1.5, 0.1.6, 0.1.7, 0.1.8)

ERROR: No matching distribution found for pgvector==0.2.3

DGaffney · 2023-11-09T14:27:29Z

Upgrading to 3.8 is a good idea - I'm usually against over-eager version bumps, but this code is now very long in the tooth and 3.8 is very old.

@skyemeedan I wonder if we should hold on the version bump until we release all the dependencies off alegre? That would very likely make a bump much easier...

skyemeedan · 2023-11-09T14:55:54Z

@ahmednasserswe are there earlier versions of pgvector that provide functionality but don't require python 3.8?

…ions, making sscd vector size 512, and removing version number from pgvector in requirements.tx

codeclimate · 2023-11-12T19:40:41Z

app/main/model/image.py

@@ -43,6 +44,7 @@ def from_url(url, doc_id, context={}, created_at=None):
    raw = remote_response.read()
    im = Image.open(io.BytesIO(raw)).convert('RGB')
    phash = compute_phash_int(im)
+    sscd = None


Unused variable 'sscd'

codeclimate · 2023-11-12T19:40:41Z

manage.py

@@ -19,6 +19,7 @@
 from app.main.lib.language_analyzers import init_indices
 from app.main.lib.image_hash import compute_phash_int
 from PIL import Image
+from sqlalchemy import text


Imports from package sqlalchemy are not grouped

… `callback_add_item` to avoid error

codeclimate · 2023-11-20T13:04:01Z

app/main/model/image.py


+from app.main.lib.presto import Presto
+import json


standard import import json should be placed before from flask import current_app as app

codeclimate · 2023-11-20T13:04:01Z

app/main/model/image.py


+from app.main.lib.presto import Presto


Imports from package app are not grouped

…['phash']' to 'post' '/image/similarity/' in test_image_similarity.py'

codeclimate · 2023-11-27T22:39:33Z

app/main/lib/image_similarity.py

+  #   models = ['phash']
+  try:
+    models = [m.lower() for m in models]
+  except:


No exception type(s) specified

codeclimate · 2023-11-27T22:39:33Z

app/main/lib/image_similarity.py

  if not context:
    context = {}
  if not threshold:
    threshold = 0.9
  if url:
-    image = ImageModel.from_url(url, None)
+    result = []
+    image = ImageModel.from_url(url, None,models = models)


No space allowed around keyword argument assignment

codeclimate · 2023-11-27T22:39:33Z

app/main/lib/image_similarity.py

  if not context:
    context = {}
  if not threshold:
    threshold = 0.9
  if url:
-    image = ImageModel.from_url(url, None)
+    result = []
+    image = ImageModel.from_url(url, None,models = models)


Exactly one space required after comma

codeclimate · 2023-11-27T22:39:33Z

app/main/model/image.py

@@ -31,23 +34,51 @@ class ImageModel(db.Model):
  )

  @staticmethod
-  def from_url(url, doc_id, context={}, created_at=None):
+  def from_url(url, doc_id, context={}, created_at=None, models=None):


Too many local variables (18/15)

codeclimate · 2023-11-27T22:39:35Z

Code Climate has analyzed commit 8b59feb and detected 9 issues on this pull request.

Here's the issue category breakdown:

Category	Count
Complexity	1
Style	5
Bug Risk	3

The test coverage on the diff in this pull request is 41.3% (50% is the threshold).

This pull request will bring the total coverage in the repository to 86.4%.

View more on Code Climate.

Adding nullable column to postgres images table with type Vector to s…

1b189c9

…tore sscd embedding vectors.

ahmednasserswe requested review from DGaffney, skyemeedan and computermacgyver as code owners November 8, 2023 19:55

moving CREATE EXTENSION IF NOT EXISTS vector; under init_perl_funct…

ccb4cb9

…ions, making sscd vector size 512, and removing version number from pgvector in requirements.tx

codeclimate bot reviewed Nov 12, 2023

View reviewed changes

ahmednasserswe added 4 commits November 12, 2023 20:55

setting pgvector version to pgvector==0.1.8

ee54b8c

Call presto from image.py to calculate SSCD

22a5360

adding elif similarity_type == "image_sscd__Model": return None to…

b74be8c

… `callback_add_item` to avoid error

adding e = sys.exc_info()[0] to sscd expect

04dd0c8

codeclimate bot reviewed Nov 20, 2023

View reviewed changes

ahmednasserswe and others added 6 commits November 20, 2023 16:58

Update image_similarity.py to search postgres by sscd cosine sim

61e5e97

Add models parameter to image GET and POST

538f3b5

Passing 'models' params in 'add_image'

b217ffc

adding create_sscd_column migration script

4e6798f

passing 'models = models' as paramter 'ImageModel.from_url'

6278ab5

adding 'models = ['phash']' to ' ImageModel.from_url' and ''models': …

8b59feb

…['phash']' to 'post' '/image/similarity/' in test_image_similarity.py'

codeclimate bot reviewed Nov 27, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding nullable column to postgres images table with type Vector to s… #357

Adding nullable column to postgres images table with type Vector to s… #357

ahmednasserswe commented Nov 8, 2023

skyemeedan commented Nov 9, 2023

DGaffney commented Nov 9, 2023

skyemeedan commented Nov 9, 2023

codeclimate bot Nov 12, 2023

codeclimate bot Nov 12, 2023

codeclimate bot Nov 20, 2023

codeclimate bot Nov 20, 2023

codeclimate bot Nov 27, 2023

codeclimate bot Nov 27, 2023

codeclimate bot Nov 27, 2023

codeclimate bot Nov 27, 2023

codeclimate bot commented Nov 27, 2023

Adding nullable column to postgres images table with type Vector to s… #357

Are you sure you want to change the base?

Adding nullable column to postgres images table with type Vector to s… #357

Conversation

ahmednasserswe commented Nov 8, 2023

Description

How has this been tested?

Have you considered secure coding practices when writing this code?

skyemeedan commented Nov 9, 2023

DGaffney commented Nov 9, 2023

skyemeedan commented Nov 9, 2023

codeclimate bot Nov 12, 2023

Choose a reason for hiding this comment

codeclimate bot Nov 12, 2023

Choose a reason for hiding this comment

codeclimate bot Nov 20, 2023

Choose a reason for hiding this comment

codeclimate bot Nov 20, 2023

Choose a reason for hiding this comment

codeclimate bot Nov 27, 2023

Choose a reason for hiding this comment

codeclimate bot Nov 27, 2023

Choose a reason for hiding this comment

codeclimate bot Nov 27, 2023

Choose a reason for hiding this comment

codeclimate bot Nov 27, 2023

Choose a reason for hiding this comment

codeclimate bot commented Nov 27, 2023