Simply run python ./gradio_demo.py

[av]

Unfortunately installing the dependencies and running the demo wasn't enough:

• Python 3.12.2 (installed via pyenv) needs bziplib2 (libbz2-dev on Deb), if it's not installed, after installation Python itself needs to be rebuilt/reinstalled
• The repo requires model weights to be downloaded from the official HF repo and put into weights folder
• Downloaded weights are in .safetensors - convert with python ./weights/convert_safetensor_to_pt.py (after HF repo download)
• gradio_demo.py has a bug in Checkbox signature, there's no keyed arg named "default", should be manually changed from default=True to value=True
  • Fixed in the latest main

[yadong-lu]

Hi , for 2,3,4 please checkout the guide in readme and pull the latest version of the code. For 1, I wonder has anyone encountered similar issue?

[MeDott29]

here you go https://www.kaggle.com/code/athviii/omniparser-demo

[nmstoker]

@av do you have the errors that mafe you think bziplib2 was required? It's quite an odd sounding issue.

What's your setup?
And are you running it with your system python or one you installed from somewhere?
If elsewhere can you give details.

[abrichr]

@av @yadong-lu @MeDott29 @nmstoker for demo see also #52:

% python deploy.py start
...
% python client.py http://44.198.58.162:7861 ~/Desktop/screenshot.png                                                                                                             
Loaded as API: http://44.198.58.162:7861/ ✔                                                                                                                                                                                                               
2024-10-29 18:55:51.336 | INFO     | __main__:predict:61 - Prediction completed successfully                                                                                                                                                              
2024-10-29 18:55:51.337 | INFO     | __main__:predict:70 - label_coordinates={'0': [0.12276943773031235, 0.002296212362125516, 0.045681655406951904, 0.028702640905976295], '1': [0.48429691791534424, 0.008610793389379978, 0.030335474759340286, 0.01607
3478385806084], '2': [0.0060670948587358, 0.043628014624118805, 0.02890792302787304, 0.016073478385806084], '3': [0.049250528216362, 0.043628014624118805, 0.02855103462934494, 0.016073478385806084], '4': [0.0935046374797821, 0.043628014624118805, 0.0
242683794349432, 0.016073478385806084], '5': [0.13276232779026031, 0.04247990995645523, 0.057102058082818985, 0.02238805964589119], '6': [0.20628122985363007, 0.043628014624118805, 0.04282655194401741, 0.016073478385806084], '7': [0.26552459597587585
, 0.043628014624118805, 0.021770164370536804, 0.016073478385806084], '8': [0.3029978573322296, 0.04247990995645523, 0.03283369168639183, 0.017221584916114807], '9': [0.3508208394050598, 0.043628014624118805, 0.02319771610200405, 0.016073478385806084]
, '10': [0.39971449971199036, 0.043628014624118805, 0.03354746475815773, 0.016073478385806084], '11': [0.9625267386436462, 0.04649827629327774, 0.026409707963466644, 0.016073478385806084], '12': [0.08422555029392242, 0.08553387224674225, 0.0578158460
5574608, 0.01836968958377838], '13': [0.4057815968990326, 0.08610792458057404, 0.03854389861226082, 0.017221584916114807], '14': [0.4921484589576721, 0.08553387224674225, 0.03354746475815773, 0.016073478385806084], '15': [0.012491078115999699, 0.1194
0297484397888, 0.021770164370536804, 0.014925390481948853], '16': [0.08529621362686157, 0.11997702717781067, 0.06495360285043716, 0.02238805964589119], '17': [0.40613847970962524, 0.1228473111987114, 0.0239114910364151, 0.016073478385806084], '18': [
0.4382583796977997, 0.1228473111987114, 0.026409707963466644, 0.014925372786819935], '19': [0.6145610213279724, 0.11710677295923233, 0.04318344220519066, 0.017221584916114807], '20': [0.6623840928077698, 0.11940297484397888, 0.027837172150611877, 0.0
14925390481948853], '21': [0.7041398882865906, 0.11997703462839127, 0.015703069046139717, 0.01205510925501585], '22': [0.7494646906852722, 0.11997703462839127, 0.022483939304947853, 0.01205510925501585], '23': [0.7847965955734253, 0.11997703462839127
, 0.0239114910364151, 0.01205510925501585], '24': [0.8183440566062927, 0.1188289225101471, 0.02855103462934494, 0.015499443747103214], '25': [0.8954318165779114, 0.11997703462839127, 0.017130620777606964, 0.01205510925501585], '26': [0.93076366186141
97, 0.11997703462839127, 0.016416845843195915, 0.01205510925501585], '27': [0.6145610213279724, 0.13318024575710297, 0.04139900207519531, 0.016073478385806084], '28': [0.6602426767349243, 0.13375429809093475, 0.031406138092279434, 0.01205510925501585
], '29': [0.699857234954834, 0.13318024575710297, 0.0242683794349432, 0.014925372786819935], '30': [0.8979300260543823, 0.13375429809093475, 0.01927194930613041, 0.01205510925501585], '31': [0.9304068684577942, 0.13318024575710297, 0.0249821562319993
97, 0.014925372786819935], '32': [0.02105638198554516, 0.16590125858783722, 0.059957172721624374, 0.01836968958377838], '33': [0.08422557264566422, 0.16532720625400543, 0.3875803053379059, 0.01894374191761017], '34': [0.9107779264450073, 0.1659012585
8783722, 0.07708779722452164, 0.017221584916114807], '35': [0.050321198999881744, 0.19517795741558075, 0.07566024363040924, 0.024684270843863487], '36': [0.18022841215133667, 0.30022960901260376, 0.10706637054681778, 0.051090698689222336], '37': [0.3
1334760785102844, 0.3053961396217346, 0.12312633544206619, 0.04477611929178238], '38': [0.4464668035507202, 0.3071182668209076, 0.055317629128694534, 0.03731343150138855], '39': [0.581013560295105, 0.305970162153244, 0.06745181977748871, 0.0373134315
0138855], '40': [0.014989293180406094, 0.3639494776725769, 0.013918629847466946, 0.029276693239808083], '41': [0.09671662747859955, 0.3559127151966095, 0.08065667748451233, 0.0482204370200634], '42': [0.29407569766044617, 0.3639494776725769, 0.013918
586075305939, 0.029276693239808083], '43': [0.4275517165660858, 0.3639494776725769, 0.013204853981733322, 0.029276693239808083], '44': [0.014989293180406094, 0.4161882996559143, 0.013918629847466946, 0.02985074557363987], '45': [0.09743039309978485, 
0.4087255895137787, 0.07994289696216583, 0.0482204370200634], '46': [0.29407569766044617, 0.4161882996559143, 0.013918586075305939, 0.02985074557363987], '47': [0.4275517463684082, 0.4161882996559143, 0.013918629847466946, 0.02985074557363987], '48':
 [0.0970735102891922, 0.4609643816947937, 0.08065667748451233, 0.04879448935389519], '49': [0.29407569766044617, 0.46785303950309753, 0.013918586075305939, 0.030998852103948593], '50': [0.4275517463684082, 0.4690011441707611, 0.013918629847466946, 0.
02985074557363987], '51': [0.5613847374916077, 0.4690011441707611, 0.013561741448938847, 0.02985074557363987], '52': [0.6948608160018921, 0.46785303950309753, 0.013918629847466946, 0.030998852103948593], '53': [0.09671661257743835, 0.5143513679504395
, 0.08137047290802002, 0.0482204370200634], '54': [0.29407569766044617, 0.5206659436225891, 0.013918586075305939, 0.030998852103948593], '55': [0.4275517463684082, 0.5206659436225891, 0.013918629847466946, 0.030998852103948593], '56': [0.561384737491
6077, 0.5206659436225891, 0.013561741448938847, 0.030998852103948593], '57': [0.6948608160018921, 0.5206659436225891, 0.013918629847466946, 0.030998852103948593], '58': [0.008922198787331581, 0.7824339866638184, 0.02783725969493389, 0.036739379167556
76], '59': [0.008208423852920532, 0.8363949656486511, 0.027123482897877693, 0.034443169832229614], '60': [0.008922199718654156, 0.8892077803611755, 0.027123482897877693, 0.034443169832229614], '61': [0.055317629128694534, 0.9500574469566345, 0.029621
697962284088, 0.014925372786819935], '62': [0.015346181578934193, 0.9787600636482239, 0.027480371296405792, 0.017221584916114807], '63': [0.06780871003866196, 0.9787600636482239, 0.10135617107152939, 0.017221584916114807], '64': [0.9586009979248047, 
0.9787600636482239, 0.024625267833471298, 0.014925372786819935], '65': [0.6651304364204407, 0.08138538151979446, 0.022024760022759438, 0.03468729183077812], '66': [0.7468552589416504, 0.07959043234586716, 0.023525850847363472, 0.028781019151210785], 
'67': [0.6991221308708191, 0.08180632442235947, 0.018437061458826065, 0.03362924978137016], '68': [0.7826605439186096, 0.0806342288851738, 0.02190661057829857, 0.027261780574917793], '69': [0.6214930415153503, 0.08046738058328629, 0.01830035261809826
, 0.031031927093863487], '70': [0.8183373212814331, 0.07944951951503754, 0.021337995305657387, 0.03104429505765438], '71': [0.09984152764081955, 0.9439650177955627, 0.02320893481373787, 0.028721420094370842], '72': [0.38827988505363464, 0.11905978620
052338, 0.01550249382853508, 0.02412971667945385], '73': [0.866210401058197, 0.08966667205095291, 0.015609142370522022, 0.03238759562373161], '74': [0.3112151622772217, 0.07979363203048706, 0.01715545356273651, 0.030124405398964882], '75': [0.0003453
716926742345, 0.9426923394203186, 0.044026125222444534, 0.029541436582803726], '76': [0.6377907395362854, 0.2446022927761078, 0.015758832916617393, 0.03145069256424904], '77': [0.3110503554344177, 0.12188976258039474, 0.017567103728652, 0.02876365743
5774803], '78': [0.8960394263267517, 0.08187582343816757, 0.020451003685593605, 0.02726871706545353], '79': [0.9309751987457275, 0.0796903744339943, 0.02361646667122841, 0.029221685603260994]}                                                          
2024-10-29 18:55:51.339 | INFO     | __main__:predict:71 - parsed_content_list=['Text Box ID 0: 5 v G', 'Text Box ID 1: Book1', 'Text Box ID 2: Home', 'Text Box ID 3: Insert', 'Text Box ID 4: Draw', 'Text Box ID 5: Page Layout', 'Text Box ID 6: Formu
las', 'Text Box ID 7: Data', 'Text Box ID 8: Review', 'Text Box ID 9: View', 'Text Box ID 10: Tell me', 'Text Box ID 11: Share', 'Text Box ID 12: Calibri (Body)', 'Text Box ID 13: Wrap Text', 'Text Box ID 14: General', 'Text Box ID 15: Paste', 'Text 
Box ID 16: B   I W ~ [', 'Text Box ID 17: Merge', 'Text Box ID 18: Center', 'Text Box ID 19: Conditional', 'Text Box ID 20: Format', 'Text Box ID 21: Cell', 'Text Box ID 22: Insert', 'Text Box ID 23: Delete', 'Text Box ID 24: Format', 'Text Box ID 25
: Sort', 'Text Box ID 26: Find', 'Text Box ID 27: Formatting', 'Text Box ID 28: as Table', 'Text Box ID 29: Styles', 'Text Box ID 30: Filter', 'Text Box ID 31: Select', 'Text Box ID 32: Office Update', 'Text Box ID 33: To keep up-to-date with securit
y updates, fixes, and improvements, choose Check for Updates:', 'Text Box ID 34: Check for Updates', 'Text Box ID 35: 4* v fx', 'Text Box ID 36: Marketing', 'Text Box ID 37: Engineering', 'Text Box ID 38: Sales', 'Text Box ID 39: Admin', 'Text Box ID
 40: 2', 'Text Box ID 41: 13-May', 'Text Box ID 42: 2', 'Text Box ID 43: 3', 'Text Box ID 44: 3', 'Text Box ID 45: 14-May', 'Text Box ID 46: 3', 'Text Box ID 47: 2', 'Text Box ID 48: 15-May', 'Text Box ID 49: 3', 'Text Box ID 50: 3', 'Text Box ID 51:
 3', 'Text Box ID 52: 3', 'Text Box ID 53: 16-May', 'Text Box ID 54: 2', 'Text Box ID 55: 2', 'Text Box ID 56: 2', 'Text Box ID 57: 1', 'Text Box ID 58: 10', 'Text Box ID 59: 11', 'Text Box ID 60: 12', 'Text Box ID 61: Sheet1', 'Text Box ID 62: Ready
', 'Text Box ID 63: Accessibility: Good to go', 'Text Box ID 64: 287%', 'Icon Box ID 65: a drawing or painting tool.', 'Icon Box ID 66: a layout or table layout.', 'Icon Box ID 67: a drawing or painting tool.', 'Icon Box ID 68: expanding or opening a
 window or view.', 'Icon Box ID 69: a table or spreadsheet.', 'Icon Box ID 70: a table or spreadsheet.', 'Icon Box ID 71: adding or creating a new item.', 'Icon Box ID 72: Decrease', 'Icon Box ID 73: Download', 'Icon Box ID 74: a menu or list.', 'Ico
n Box ID 75: navigating to the previous item or screen.', 'Icon Box ID 76: the "E" function.', 'Icon Box ID 77: Justified', 'Icon Box ID 78: Text formatting options.', 'Icon Box ID 79: a search function.']
2024-10-29 18:55:51.341 | INFO     | __main__:predict:77 - Parsed content saved to: result_data_20241029_185545.json         
2024-10-29 18:55:51.343 | INFO     | __main__:predict:83 - Output image saved to: output_image_20241029_185545.png  
% open output_image.png 
% head result_data_20241029_185545.json                                              
{                                                                                                                            
    "label_coordinates": {                                                                                                   
        "0": [                                                                                                               
            0.12276943773031235,                                                                                             
            0.002296212362125516,                                                                                            
            0.045681655406951904,                                                                                            
            0.028702640905976295                                                                                             
        ],                                                                                                                   
        "1": [                                                                                                               
            0.48429691791534424,                                                                                             
(venv) abrichr@MacBook-Pro-5 OmniParser % tail result_data_20241029_185545.json                                              
        "Icon Box ID 72: Decrease",                                                                                          
        "Icon Box ID 73: Download",                                                                                          
        "Icon Box ID 74: a menu or list.",                                                                                   
        "Icon Box ID 75: navigating to the previous item or screen.",                                                        
        "Icon Box ID 76: the \"E\" function.",                                                                               
        "Icon Box ID 77: Justified",                                                                                         
        "Icon Box ID 78: Text formatting options.",                                                                          
        "Icon Box ID 79: a search function."                                                                                 
    ]                                                                                                                        
}%     

>>> from client import predict
>>> from PIL import Image
>>> 
>>> result = predict("http://98.80.191.41:7861", "~/oa/OpenAdapt/tests/assets/excel.png")
Loaded as API: http://98.80.191.41:7861/ ✔
>>> result_data = result["result_data"]
>>> label_coordinates = result_data["label_coordinates"]
>>> parsed_content_list = result_data["parsed_content_list"]
>>> len(label_coordinates), len(parsed_content_list)
(80, 80)
>>> output_image_path = result['output_image']
>>> Image.open(output_image_path).show()

[av]

Hey, sorry if my issue seemed mean, I meant to document the steps I had to take to make it up and running. Sorry for overlooking the download instructions, totally on me due to "quick-reading" through README.

#52 looks great, I'm looking forward towards a dockerized version

[nmstoker]

Not at all, I'm delighted your issues seem like they'll be resolved 🙂