Skip to content

Commit

Permalink
feat: commit with entirely functional code for bug repro
Browse files Browse the repository at this point in the history
  • Loading branch information
pedrohbtp committed Nov 7, 2019
1 parent 4757a64 commit e984e2e
Show file tree
Hide file tree
Showing 10 changed files with 172 additions and 0 deletions.
12 changes: 12 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
*.pyc
/.vscode/*
.vscode/*
/.vscode
aws_env.env
*.tar.gz
# package directories
node_modules
jspm_packages
# Serverless directories
.serverless
.requirements.*
20 changes: 20 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Python support can be specified down to the minor or micro version
# (e.g. 3.6 or 3.6.3).
# OS Support also exists for jessie & stretch (slim and full).
# See https://hub.docker.com/r/library/python/ for all supported Python
# tags from Docker Hub.
FROM python:3.6
WORKDIR /usr/app
ADD . /usr/app

# install necessary node things
RUN apt-get update
RUN apt-get -y install curl gnupg
RUN curl -sL https://deb.nodesource.com/setup_11.x | bash -
RUN apt-get -y install nodejs
RUN npm install -g serverless
RUN npm config set bin-links false
RUN npm install
# install necessary python things
RUN pip install -r requirements.txt

44 changes: 44 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,2 +1,46 @@
# bug-aws-lambda-infinite-timeout
Repository to reproduce the aws lambda bug of non-stop timeout after a first timeout

# Bug Trigger

When the lambda package is too large, after the lambda times out once, it times out forever. It does not matter if you increase the timeout or memory in AWS.

In this repo, the dependencies are large when zipped to upload to AWS in order to reproduce the bug.

# How to run it

copy the aws credentials file like below

``` bash
cp aws_env.env.rename aws_env.env
```

fill the credentials file that will be used inside your container

Starting the container

``` bash
docker-compose run --rm bug-lambda-service bash
```

Deploying the application

``` bash
serverless deploy
```

Run the script to continuously invoke the lambda function

``` bash
python main.py
```


Then go to AWS console and reduce the timeout to **1 seconds**. The function will start to timeout.
After it starts to timeout, increate the timeout in AWS' console to something greater than 3 seconds. Even putting it to 30 seconds. It will continue to timeout.

The image below shows the execution of the lamda.
Even though it is only a sleep of **2 seconds**, after it starts timing out, it never recovers. Even after **10 seconds** of waiting time.


![image](execution-output.JPG)
3 changes: 3 additions & 0 deletions aws_env.env.rename
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_DEFAULT_REGION=
15 changes: 15 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
version: '2.1'

services:
bug-lambda-service:
image: bug-lambda-service
build: .
volumes:
- .:/usr/app/
env_file:
- aws_env.env
environment:
# needs this for deploying using windows
- VIRTUALENV_ALWAYS_COPY=1
command: python main.py

Binary file added execution-output.JPG
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
22 changes: 22 additions & 0 deletions main.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
try:
import unzip_requirements
except ImportError:
pass
import time
import boto3
import json

def handler(event, context):
time.sleep(2)
return 'finished'


if __name__ == '__main__':
lambda_client = boto3.client('lambda')
while True:
lambda_request = lambda_client.invoke(
FunctionName='bug-lambda-timeout-dev-bug-lambda-event-function'
)
print('published: ', time.time())
print(lambda_request['Payload'].read())
time.sleep(0.5)
20 changes: 20 additions & 0 deletions package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{
"name": "bug-lambda-timeout",
"version": "1.0.0",
"description": "",
"main": "handler.js",
"dependencies": {},
"devDependencies": {
"serverless-python-requirements": "^5.0.1"
},
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"repository": {
"type": "git",
"url": ""
},
"author": "Pedro Torres",
"license": "",
"homepage": ""
}
10 changes: 10 additions & 0 deletions requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
boto3==1.10.12
botocore==1.13.12
docutils==0.15.2
jmespath==0.9.4
lambda-packages==0.20.0
numpy==1.17.3
python-dateutil==2.8.0
s3transfer==0.2.1
six==1.13.0
urllib3==1.25.6
26 changes: 26 additions & 0 deletions serverless.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
service: bug-lambda-timeout

package:
exclude:
- aws_env.env

custom:
pythonRequirements:
zip: true
useDownloadCache: false
useStaticCache: false

plugins:
- serverless-python-requirements

provider:
name: aws
runtime: python3.6
memorySize: 2048 # optional, in MB, default is 1024


functions:
bug-lambda-event-function:
timeout: 3
handler: main.handler
description: lambda to test aws timeout bug

0 comments on commit e984e2e

Please sign in to comment.