Custom Projects List Feature #14

mrthankyou · 2021-02-17T20:12:51Z

Issues Resolved

What Does This PR Do

This PR adds an optional choice for users to add projects they follow to custom lists. The way this works is as follows:

# This command finds repositories based on a search term. The projects are then followed. Please note that the custom list name will be the name of the custom list you want these projects to eventually go to.
python3 follow_repos_by_search_term.py javascript <keyword_search> <custom_list_name>

# This command actually moves the projects to your custom list and then unfollows the projects.
python3 move_repos_to_lgtm_lists.py

One neat feature is that if a project is still being built and we can't move it to the custom list, the move_repos_to_lgtm_lists.py script will stop and report that the project is still being built. This informs the user that they must wait at a later time to re-run the script

Besides this major new feature, there are numerous under-the-hood/misc changes:

The SimpleProject class was updated to handle processing LGTM projects for our new feature. It could use some refactoring down the road but for now it works.
Code in lgtm.py was refactored to allow this new feature to take advantage of some of the baked-in logic in dealing with proto vs. real projects.
A new utils/cacher.py was added to handle caching projects in a txt file. This txt file can then later be used in the move_repos_to_lgtm_lists.py script.
A unfollow_all_followed_projects.py script was added. I mostly used this for testing purposes but since I wrote it felt like I should keep it in the PR.
Added Python documentation to several methods.
All cached files are now placed in the cache folder.

Anything Else We Should Know

This took way longer than expected. As I wrap up this PR I want to note that so far I've experienced no errors in testing this new functionality. If possible, I encourage others to test this as well. Although I think all the kinks are out I've grown weary of LGTM and their internal workings.

I also have to admit that this code can be better. But I've spent more than two weeks on this and am a bit exhausted and would like to go back to writing CodeQL queries. If you decide to shelve this for some time as you want to let this feature "bake" so to speak, that's fine with me. I'll be using this script on a daily basis so if there are more bugs I'll find them.

mrthankyou · 2021-02-17T20:14:04Z

@JLLeitschuh,

If you want to, you can start taking a look at the work I've done. It would be appreciated. That way, I can catch any errors early on before I invest more time in polishing/testing code.

mrthankyou · 2021-02-17T20:15:43Z

lgtm.py

@@ -104,14 +108,15 @@ def force_rebuild_project(self, simple_project: 'SimpleProject'):
        except LGTMRequestException:
            print('Failed rebuilding project. This may be because it is already being built. `%s`' % simple_project)

-    def follow_repository(self, repository_url: str):
+    def follow_repository(self, repository_url: str) -> dict:


I checked and can't find any case where returning data in the function breaks the code.

mrthankyou · 2021-02-17T20:16:09Z

test.py

@@ -0,0 +1,36 @@
+from typing import List


Will remove this file once the code is more polished.

JLLeitschuh · 2021-02-17T20:16:12Z

README.md

+To resolve this, we decided to create a feature users can opt-in. This feature called "Custom Projects Lists" does the
+following:
+
+- Follows all repos (aka project) in your LGTM account.
+- Stores every project you follow in a txt file.
+- At a later date (we suggest 24 hours), the user may run a follow-up command that will take the repos followed, add them to a LGTM custom list, and finally unfollow the projects in the user's LGTM account.


Big fan of this!

lgtm-com · 2021-02-17T20:18:46Z

This pull request introduces 4 alerts when merging e69003a into 00273ac - view on LGTM.com

new alerts:

2 for Unused import
1 for Unused local variable
1 for Wrong number of arguments in a call

JLLeitschuh · 2021-02-17T20:19:03Z

lgtm.py

@@ -35,6 +35,7 @@ def _make_lgtm_get(self, url: str) -> dict:
        )
        return r.json()

+    # Retrieves a user's projects


Can you use the python standard for API doc comments?

Can you point me in that direction? I can google it but want to make sure we are on the same page. I'm not native to python :/

@JLLeitschuh

Can you send that link to me? I tried looking online but can't find any defacto API doc comment rules.

I think this covers it.

https://www.programiz.com/python-programming/docstrings

test.py

JLLeitschuh · 2021-02-17T20:20:10Z

I'm a fan of this! Nice work!

lgtm-com · 2021-02-17T22:00:40Z

This pull request introduces 2 alerts when merging 69543fb into 00273ac - view on LGTM.com

new alerts:

2 for Unused import

I refactored the SimpleProject and LGTMDataFilters section in order to take advantage of some code that determines whether a project is a real project or a proto project. I also added two attributes to SimpleProject to get the refactor complete.

lgtm-com · 2021-03-03T21:40:09Z

This pull request introduces 2 alerts when merging f44b959 into 4cbdb21 - view on LGTM.com

new alerts:

2 for Unused import

mrthankyou · 2021-03-04T15:24:28Z

@JLLeitschuh This is now ready for review. I've updated the PR description with details on this PR request. Thank you.

JLLeitschuh · 2021-03-10T14:01:25Z

lgtm.py

+    def is_protoproject(self):
+        # The values for project_type should be hardcoded in one central location
+        return self.project_type == "protoproject"


Are there other project types other than protoprojects and non-protoprojects?

As far as I know, no. But I will take a deeper look and get back to you on this.

JLLeitschuh · 2021-03-10T14:04:11Z

utils/cacher.py

    file = open(cached_file, "r")

-    project_data = file.read().split("\n")
+    cached_projects = file.read().split("\n")

-    while("" in project_data):
-        project_data.remove("")
+    while("" in cached_projects):
+        cached_projects.remove("")
+
+    for i, project in enumerate(cached_projects):
+        cached_projects[i] = ProjectBuild(
+            display_name=project.split(",")[0],
+            key=project.split(",")[1],
+            project_type=project.split(",")[2],
+            is_valid_project=True,
+            org="",
+            state=""
+        )

-    for i, project in enumerate(project_data):
-        project_data[i] = ProjectBuild({
-            "name": project.split(",")[0],
-            "id": project.split(",")[1],
-            "type": project.split(",")[2],
-        })
+            # display_name: str
+            # key: str
+            # project_type: str
+            # is_valid_project: bool
+            # org: str
+            # state: str

    file.close()


Consider refactoring to with open(cached_file, "r") as file: so that close is automatically called even if an exception is thrown.

JLLeitschuh · 2021-03-10T14:05:14Z

utils/cacher.py

+    file = open("cache/" + file_name + ".txt", "a")
+
+    for project_key in project_keys:
+        file.write(project_key + "\n")
+
+    file.close()


This should use the with open style so that you don't need to call close

mrthankyou · 2021-03-16T21:16:51Z

I'm holding off on fixing these changes. I've been exploring other code options and would like to field-test this code a bit more before I commit a thumps-up from me for this PR. I hope that's ok with you.

mrthankyou · 2021-04-21T16:39:54Z

Short update: I'm actively using these scripts and have squashed most potential bugs that come from it. I think I want to spend a little more time working out these scripts. Interestingly enough I'm using a script that utilizes ghtopdep to collect repos based on a libraries dependencies and have found great results from that. Almost every query I'm running produces hits. Obviously this only works if you have a dependency library you're query identifies vulnerabilities in.

mrthankyou added 4 commits February 11, 2021 13:17

Add fork:false to Github queries

d26d6b1

Initial work setting up custom LGTM project list curation

2e58640

Clean up code and get basic cache parsing file setup

d4098d3

Continued work on custom project lists feature

e69003a

mrthankyou commented Feb 17, 2021

View reviewed changes

test.py Outdated

@@ -0,0 +1,36 @@

from typing import List

Copy link

Contributor Author

mrthankyou Feb 17, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will remove this file once the code is more polished.

JLLeitschuh reviewed Feb 17, 2021

View reviewed changes

test.py Outdated Show resolved Hide resolved

mrthankyou added 8 commits February 17, 2021 16:03

Fix misc issues

02f16a3

Add comment and ignore cache files

d3898cb

Refactor code

8e839a8

Reword text

5c996fc

Revert stars to accurate count

7bd1bf0

Remove comment

3f8d336

Update README.md

ca2bbc6

Add custom project list feature to search term script

69543fb

mrthankyou added 9 commits February 17, 2021 18:00

Save only real projects to LGTM project lists

a687d35

Remove unnecessary modules

a4133cc

Create cache folder if it already doesn't exist

b5ecc8a

Add draft for build in progress guard clause

c770a2f

Accept both proto and real projects

b83bacf

Add ProjectBuild and ProjectBuilds classes

1b2982a

Remove logs and add new request for proto projects

0bdd4cc

Save more project data to cache files

88e3793

Refactor how we move repos to LGTM lists

f515563

mrthankyou added 18 commits March 3, 2021 13:39

Clarify API call to LGTM

cbe5fa5

Refactor how we build SimpleProjects

1e40f13

I refactored the SimpleProject and LGTMDataFilters section in order to take advantage of some code that determines whether a project is a real project or a proto project. I also added two attributes to SimpleProject to get the refactor complete.

Rename method

aa14305

Remove useless code

bedc587

Rename ProjectBuild#name and refactor code

e362ef8

Add SimpleProject#project_type method

fda2a9f

Continue refactoring how we determine LGTM project types

b51fced

Rename ProjectBuild#id to #key

0b182b9

Update comment on refactoring

c6db487

Refactor SimpleProject to store the project type

9eb9c4a

Simplify logic in determining project state

0ba1576

Add comments

476faeb

Refactor logic with guard clauses

2c0d44b

Add unfollow_all_followed_projects.py script

574d0f6

Convert ProjectBuild to a subclass of SimpleProject

803ebd7

Refactor simple project build to not raise error

69b6614

Add checks confirming LGTM project is valid

89a25d7

Fix misc errors

f44b959

mrthankyou added 3 commits March 4, 2021 10:04

Reword comment

9ba28cf

Remove unnecessary code

2e1d595

Remove comments

5ea2b9a

mrthankyou marked this pull request as ready for review March 4, 2021 15:23

mrthankyou mentioned this pull request Mar 5, 2021

Collect a repos network dependencies #9

Open

JLLeitschuh reviewed Mar 10, 2021

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Custom Projects List Feature #14

Custom Projects List Feature #14

mrthankyou commented Feb 17, 2021 •

edited

Loading

mrthankyou commented Feb 17, 2021

mrthankyou Feb 17, 2021

mrthankyou Feb 17, 2021

JLLeitschuh Feb 17, 2021

lgtm-com bot commented Feb 17, 2021

JLLeitschuh Feb 17, 2021

mrthankyou Feb 17, 2021 •

edited

Loading

mrthankyou Feb 19, 2021

JLLeitschuh Feb 20, 2021

JLLeitschuh commented Feb 17, 2021

lgtm-com bot commented Feb 17, 2021

lgtm-com bot commented Mar 3, 2021

mrthankyou commented Mar 4, 2021

JLLeitschuh Mar 10, 2021

mrthankyou Mar 10, 2021 •

edited

Loading

JLLeitschuh Mar 10, 2021

JLLeitschuh Mar 10, 2021

mrthankyou commented Mar 16, 2021

mrthankyou commented Apr 21, 2021

Custom Projects List Feature #14

Are you sure you want to change the base?

Custom Projects List Feature #14

Conversation

mrthankyou commented Feb 17, 2021 • edited Loading

Issues Resolved

What Does This PR Do

Anything Else We Should Know

mrthankyou commented Feb 17, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lgtm-com bot commented Feb 17, 2021

Choose a reason for hiding this comment

mrthankyou Feb 17, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JLLeitschuh commented Feb 17, 2021

lgtm-com bot commented Feb 17, 2021

lgtm-com bot commented Mar 3, 2021

mrthankyou commented Mar 4, 2021

Choose a reason for hiding this comment

mrthankyou Mar 10, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrthankyou commented Mar 16, 2021

mrthankyou commented Apr 21, 2021

mrthankyou commented Feb 17, 2021 •

edited

Loading

mrthankyou Feb 17, 2021 •

edited

Loading

mrthankyou Mar 10, 2021 •

edited

Loading