Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

s3 touch not properly evaluated for url encoded filters #5

Open
trentgerman opened this issue Dec 3, 2019 · 2 comments
Open

s3 touch not properly evaluated for url encoded filters #5

trentgerman opened this issue Dec 3, 2019 · 2 comments

Comments

@trentgerman
Copy link

trentgerman commented Dec 3, 2019

S3Touch.evaluate_filter improperly returns false for filters that require url encoding.

For example ->

afilter = {'Key': {'FilterRules': [{'Name': 'Prefix', 'Value': 'shopping_cart_svc/payload_version%3D1/event%3Ditem_note_removed'}]}}We have used these filters in production for some time and they work for the s3 key below. 

akey = {'Key': 'shopping_cart_svc/payload_version=1/event=item_note_added/2019/03/13/19/analytics_events-1-2019-03-13-19-13-30-14b0720a-dbda-4469-b441-eec5cab90fce.gz', 'LastModified': '2019-12-02T22:30:45.000Z', 'ETag': '"af8b97e1eb90540e0a15113a3432a317"', 'Size': 2402, 'StorageClass': 'STANDARD'}

import boto3
from awscli.plugins.s3touch import S3Touch
cmd = S3Touch(boto3.Session())

cmd.evaluate_filter(afilter, akey)

This returns False

Suggested fix -> use urllib.parse.quote when doing string comparison.

@trentgerman
Copy link
Author

trentgerman commented Dec 3, 2019

Here is a quick fix I used locally that fixed the issue ->

def evaluate_filter(self, filter, file):
    import urllib.parse
    for attr in filter:
        rules = filter[attr]['FilterRules']
        for rule in rules:
            if rule['Name'] == 'Prefix' and not urllib.parse.quote(file[attr]).startswith(rule['Value']):
                return False
            if rule['Name'] == 'Suffix' and not urllib.parse.quote(file[attr]).endswith(rule['Value']):
                return False
     return True```

@john-heinnickel

This comment was marked as outdated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants