torchserve_perf.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: pip in /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages (21.1.3)\r\n"
     ]
    }
   ],
   "source": [
    "!pip install --upgrade pip\n",
    "!pip -q install sagemaker awscli boto3 pandas --upgrade "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example: TorchServe Performance Tuning on Amazon SageMaker"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example, we’ll show you how you can tune TorchServe performance, build a TorchServe container and host it using Amazon SageMaker. With Amazon SageMaker hosting you get a fully-managed hosting experience. Just specify the type of instance, and the maximum and minimum number desired, and SageMaker takes care of the rest.\n",
    "\n",
    "There are two options to tune TorchServe performance on SageMaker:\n",
    "\n",
    "1. Tune the following TorchServe's parameters in config.properties.\n",
    "[TorchServe configuration](https://github.com/pytorch/serve/blob/master/docs/configuration.md#other-properties)\n",
    "\n",
    "* number_of_netty_threads\n",
    "* netty_client_threads\n",
    "* async_logging\n",
    "* minWorkers\n",
    "* maxWorkers\n",
    "* batchSize\n",
    "\n",
    "2. [SageMaker batch transform](https://docs.aws.amazon.com/sagemaker/latest/dg/batch-transform.html)\n",
    "\n",
    "SageMaker batch transform job provides two stragties to send Http request to TorchServe, and one parameter to adjust the concurrency.\n",
    "\n",
    "* 2 Strategies:\n",
    "1) SingleRecord: a single HTTP request contains one record.\n",
    "2) MultiRecord: a single HTTP request contains multiple records. This is a client side batching and requires a model handler to split the requests in a batch.\n",
    "\n",
    "* Concurrency parameter:\n",
    "[MaxConcurrentTransforms](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateTransformJob.html#sagemaker-CreateTransformJob-request-MaxConcurrentTransforms): The maximum number of parallel requests that can be sent to each instance in a transform job.\n",
    "\n",
    "In summary, we recommend setting TorchServe on SageMaker as the following:\n",
    "1. TorchSeve dynamic batching\n",
    "2. SageMaker batch transform SingleRecord strategy\n",
    "3. Set MaxConcurrentTransforms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TorchServe dynamic batching"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### config.properties"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "inference_address=http://0.0.0.0:8080\r\n",
      "management_address=http://0.0.0.0:8081\r\n",
      "number_of_netty_threads=32\r\n",
      "job_queue_size=1000\r\n",
      "model_store=/opt/ml/model\r\n",
      "load_models=all\r\n",
      "install_py_dep_per_model=true\r\n",
      "default_response_timeout=300\r\n",
      "unregister_model_timeout=300\r\n",
      "-XX:-UseContainerSupport -XX:+UnlockDiagnosticVMOptions -XX:+PrintActiveCpus\r\n",
      "models={\\\r\n",
      "  \"TransformerEn2Fr\": {\\\r\n",
      "    \"1.0\": {\\\r\n",
      "        \"defaultVersion\": true,\\\r\n",
      "        \"marName\": \"TransformerEn2Fr.mar\",\\\r\n",
      "        \"minWorkers\": 1,\\\r\n",
      "        \"maxWorkers\": 4,\\\r\n",
      "        \"batchSize\": 4,\\\r\n",
      "        \"maxBatchDelay\": 500,\\\r\n",
      "        \"responseTimeout\": 120\\\r\n",
      "    }\\\r\n",
      "  }\\\r\n",
      "}\r\n",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!cat config.properties"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clone the TorchServe repository"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!git clone https://github.com/pytorch/serve.git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cd /home/ec2-user/SageMaker/torchserve_perf/serve"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download a PyTorch model "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TransformerEn2Fr.mar\r\n"
     ]
    }
   ],
   "source": [
    "model_name = \"TransformerEn2Fr\"\n",
    "mar_file = f'{model_name}.mar'\n",
    "mar_url = f'https://torchserve.pytorch.org/mar_files/{mar_file}'\n",
    "!wget -q {mar_url}\n",
    "!ls *.mar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Upload the TransformerEn2Fr.mar archive file to Amazon S3\n",
    "Create a compressed tar.gz file from the TransformerEn2Fr.mar file since Amazon SageMaker expects that models are in a tar.gz file. \n",
    "Uploads the model to your default Amazon SageMaker S3 bucket under the models directory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a boto3 session and get specify a role with SageMaker access"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3, time, json\n",
    "sess    = boto3.Session()\n",
    "sm      = sess.client('sagemaker')\n",
    "region  = sess.region_name\n",
    "account = boto3.client('sts').get_caller_identity().get('Account')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "role = sagemaker.get_execution_role()\n",
    "sagemaker_session = sagemaker.Session(boto_session=sess)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "bucket_name = sagemaker_session.default_bucket()\n",
    "prefix = 'torchserve'\n",
    "\n",
    "!tar cvfz {model_name}.tar.gz {mar_file}\n",
    "!aws s3 cp {model_name}.tar.gz s3://{bucket_name}/{prefix}/models/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create an Amazon ECR registry\n",
    "Create a new docker container registry for your torchserve container images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r\n",
      "An error occurred (RepositoryAlreadyExistsException) when calling the CreateRepository operation: The repository with name 'torchserve-perf' already exists in the registry with id '057122759684'\r\n"
     ]
    }
   ],
   "source": [
    "registry_name = 'torchserve-perf'\n",
    "!aws ecr create-repository --repository-name {registry_name}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build a TorchServe Docker container and push it to Amazon ECR"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "image_label = 'v1'\n",
    "image = f'{account}.dkr.ecr.{region}.amazonaws.com/{registry_name}:{image_label}'\n",
    "\n",
    "!docker build -t {registry_name}:{image_label} .\n",
    "!$(aws ecr get-login --no-include-email --region {region})\n",
    "!docker tag {registry_name}:{image_label} {image}\n",
    "!docker push {image}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deploy endpoint and make prediction using Amazon SageMaker SDK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.model import Model\n",
    "from sagemaker.predictor import Predictor\n",
    "\n",
    "model_data = f's3://{bucket_name}/{prefix}/models/{model_name}.tar.gz'\n",
    "sm_model_name = f'torchserve-{model_name}'\n",
    "\n",
    "torchserve_model = Model(model_data = model_data, \n",
    "                         image_uri = image,\n",
    "                         role  = role,\n",
    "                         predictor_cls=Predictor,\n",
    "                         name  = sm_model_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "endpoint_name = 'torchserve-endpoint-' + time.strftime(\"%Y-%m-%d-%H-%M-%S\", time.gmtime())\n",
    "\n",
    "predictor = torchserve_model.deploy(instance_type='ml.g4dn.xlarge',\n",
    "                                    initial_instance_count=1,\n",
    "                                    endpoint_name = endpoint_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test the TorchServe hosted model\n",
    "\n",
    "TorchServe dynamic batching is transparent to client side. It aggregates a model's incoming prediction requests together, processes in batch and distributes response to clients."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "payload = \"Hi James, when are you coming back home? I am waiting for you. Please come as soon as possible.\"    \n",
    "response = predictor.predict(data=payload)\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SageMaker Batch Transform Jobs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'s3://sagemaker-us-east-2-057122759684/TransformerEn2Fr/batch_transform_torchserve_sagemaker_input'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch_input = f's3://{bucket_name}/{model_name}/batch_transform_torchserve_sagemaker_input/'\n",
    "batch_output = f's3://{bucket_name}/{model_name}/batch_transform_torchserve_sagemaker_output/'\n",
    "batch_job_name = f'{model_name}-batch-' + time.strftime(\"%Y-%m-%d-%H-%M-%S\", time.gmtime())\n",
    "transform_input = sagemaker_session.upload_data('serve/examples/nmt_transformer/model_input/', \n",
    "                                                bucket=bucket_name, \n",
    "                                                key_prefix=f'{model_name}/batch_transform_torchserve_sagemaker_input')\n",
    "transform_input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "transformer = sagemaker.transformer.Transformer(model_name=sm_model_name, \n",
    "                                                instance_count=1, \n",
    "                                                instance_type='ml.m4.xlarge',\n",
    "                                                strategy=\"SingleRecord\",\n",
    "                                                max_concurrent_transforms=2,\n",
    "                                                assemble_with=None, \n",
    "                                                output_path=batch_output, \n",
    "                                                sagemaker_session=sagemaker_session)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "...........................................\u001b[34mCUDNN_VERSION=7.6.5.32\u001b[0m\n",
      "\u001b[34mPYTHONUNBUFFERED=TRUE\u001b[0m\n",
      "\u001b[34mLD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64\u001b[0m\n",
      "\u001b[34mSAGEMAKER_MAX_CONCURRENT_TRANSFORMS=2\u001b[0m\n",
      "\u001b[34mSAGEMAKER_BATCH_STRATEGY=SINGLE_RECORD\u001b[0m\n",
      "\u001b[34mLANG=C.UTF-8\u001b[0m\n",
      "\u001b[34mSAGEMAKER_SAFE_PORT_RANGE=10000-10999\u001b[0m\n",
      "\u001b[34mHOSTNAME=c166cabe72bf\u001b[0m\n",
      "\u001b[34mPYTHONIOENCODING=UTF-8\u001b[0m\n",
      "\u001b[34mNVIDIA_VISIBLE_DEVICES=all\u001b[0m\n",
      "\u001b[34mNCCL_VERSION=2.10.3\u001b[0m\n",
      "\u001b[34mPWD=/home/model-server\u001b[0m\n",
      "\u001b[34mHOME=/root\u001b[0m\n",
      "\u001b[34mSAGEMAKER_BATCH=true\u001b[0m\n",
      "\u001b[34mAWS_REGION=us-east-2\u001b[0m\n",
      "\u001b[34mSAGEMAKER_BIND_TO_PORT=8080\u001b[0m\n",
      "\u001b[34mCUDA_PKG_VERSION=10-2=10.2.89-1\u001b[0m\n",
      "\u001b[34mCUDA_VERSION=10.2.89\u001b[0m\n",
      "\u001b[34mNVIDIA_DRIVER_CAPABILITIES=compute,utility\u001b[0m\n",
      "\u001b[34mSHLVL=1\u001b[0m\n",
      "\u001b[34mNVIDIA_REQUIRE_CUDA=cuda>=10.2 brand=tesla,driver>=396,driver<397 brand=tesla,driver>=410,driver<411 brand=tesla,driver>=418,driver<419 brand=tesla,driver>=440,driver<441\u001b[0m\n",
      "\u001b[34mAWS_CONTAINER_CREDENTIALS_RELATIVE_URI=/v2/credentials/54OQ8-83Gr-VPt9TmQCLjSHUs8Vb2IlZq3gZDJOtcl8\u001b[0m\n",
      "\u001b[34mTEMP=/home/model-server/tmp\u001b[0m\n",
      "\u001b[34mPATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\u001b[0m\n",
      "\u001b[34m_=/usr/bin/printenv\u001b[0m\n",
      "\u001b[34mml\u001b[0m\n",
      "\u001b[34m2021-07-14 21:49:26,175 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager - Initializing plugins manager...\u001b[0m\n",
      "\u001b[34m2021-07-14 21:49:26,383 [INFO ] main org.pytorch.serve.ModelServer - \u001b[0m\n",
      "\u001b[34mTorchserve version: 0.4.0\u001b[0m\n",
      "\u001b[34mTS Home: /usr/local/lib/python3.6/dist-packages\u001b[0m\n",
      "\u001b[34mCurrent directory: /home/model-server\u001b[0m\n",
      "\u001b[34mTemp directory: /home/model-server/tmp\u001b[0m\n",
      "\u001b[34mNumber of GPUs: 0\u001b[0m\n",
      "\u001b[34mNumber of CPUs: 4\u001b[0m\n",
      "\u001b[34mMax heap size: 3090 M\u001b[0m\n",
      "\u001b[34mPython executable: /usr/bin/python3\u001b[0m\n",
      "\u001b[34mConfig file: /home/model-server/config.properties\u001b[0m\n",
      "\u001b[34mInference address: http://0.0.0.0:8080\u001b[0m\n",
      "\u001b[34mManagement address: http://0.0.0.0:8081\u001b[0m\n",
      "\u001b[34mMetrics address: http://127.0.0.1:8082\u001b[0m\n",
      "\u001b[34mModel Store: /opt/ml/model\u001b[0m\n",
      "\u001b[34mInitial Models: all\u001b[0m\n",
      "\u001b[34mLog dir: /home/model-server/logs\u001b[0m\n",
      "\u001b[34mMetrics dir: /home/model-server/logs\u001b[0m\n",
      "\u001b[34mNetty threads: 32\u001b[0m\n",
      "\u001b[34mNetty client threads: 0\u001b[0m\n",
      "\u001b[34mDefault workers per model: 4\u001b[0m\n",
      "\u001b[34mBlacklist Regex: N/A\u001b[0m\n",
      "\u001b[34mMaximum Response Size: 6553500\u001b[0m\n",
      "\u001b[34mMaximum Request Size: 6553500\u001b[0m\n",
      "\u001b[34mPrefer direct buffer: false\u001b[0m\n",
      "\u001b[34mAllowed Urls: [file://.*|http(s)?://.*]\u001b[0m\n",
      "\u001b[34mCustom python dependency for model allowed: true\u001b[0m\n",
      "\u001b[34mMetrics report format: prometheus\u001b[0m\n",
      "\u001b[34mEnable metrics API: true\u001b[0m\n",
      "\u001b[34mWorkflow Store: /opt/ml/model\u001b[0m\n",
      "\u001b[34mModel config: {\"TransformerEn2Fr\": {\"1.0\": {\"defaultVersion\": true,\"marName\": \"TransformerEn2Fr.mar\",\"minWorkers\": 1,\"maxWorkers\": 4,\"batchSize\": 4,\"maxBatchDelay\": 500,\"responseTimeout\": 120}}}\u001b[0m\n",
      "\u001b[34m2021-07-14 21:49:26,396 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager -  Loading snapshot serializer plugin...\u001b[0m\n",
      "\u001b[34m2021-07-14 21:49:26,400 [DEBUG] main org.pytorch.serve.ModelServer - Loading models from model store: TransformerEn2Fr.mar\u001b[0m\n",
      "\u001b[34m2021-07-14 21:50:35,076 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Adding new version 1.0 for model TransformerEn2Fr\u001b[0m\n",
      "\u001b[34m2021-07-14 21:50:35,077 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 1.0 for model TransformerEn2Fr\u001b[0m\n",
      "\u001b[34m2021-07-14 21:50:35,077 [INFO ] main org.pytorch.serve.wlm.ModelManager - Model TransformerEn2Fr loaded.\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:50,312 [DEBUG] main org.pytorch.serve.wlm.ModelManager - updateModel: TransformerEn2Fr, count: 1\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:50,332 [INFO ] main org.pytorch.serve.ModelServer - Initialize Inference server with: EpollServerSocketChannel.\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:50,439 [INFO ] main org.pytorch.serve.ModelServer - Inference API bind to: http://0.0.0.0:8080\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:50,439 [INFO ] main org.pytorch.serve.ModelServer - Initialize Management server with: EpollServerSocketChannel.\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:50,445 [INFO ] main org.pytorch.serve.ModelServer - Management API bind to: http://0.0.0.0:8081\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:50,446 [INFO ] main org.pytorch.serve.ModelServer - Initialize Metrics server with: EpollServerSocketChannel.\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:50,447 [INFO ] main org.pytorch.serve.ModelServer - Metrics API bind to: http://127.0.0.1:8082\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:50,641 [INFO ] pool-1-thread-2 ACCESS_LOG - /169.254.255.130:41532 \"GET /ping HTTP/1.1\" 200 10\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:50,650 [INFO ] pool-1-thread-2 TS_METRICS - Requests2XX.Count:1|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:50,682 [INFO ] epollEventLoopGroup-3-2 ACCESS_LOG - /169.254.255.130:41536 \"GET /execution-parameters HTTP/1.1\" 404 2\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:50,683 [INFO ] epollEventLoopGroup-3-2 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:50,312 [DEBUG] main org.pytorch.serve.wlm.ModelManager - updateModel: TransformerEn2Fr, count: 1\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:50,332 [INFO ] main org.pytorch.serve.ModelServer - Initialize Inference server with: EpollServerSocketChannel.\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:50,439 [INFO ] main org.pytorch.serve.ModelServer - Inference API bind to: http://0.0.0.0:8080\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:50,439 [INFO ] main org.pytorch.serve.ModelServer - Initialize Management server with: EpollServerSocketChannel.\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:50,445 [INFO ] main org.pytorch.serve.ModelServer - Management API bind to: http://0.0.0.0:8081\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:50,446 [INFO ] main org.pytorch.serve.ModelServer - Initialize Metrics server with: EpollServerSocketChannel.\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:50,447 [INFO ] main org.pytorch.serve.ModelServer - Metrics API bind to: http://127.0.0.1:8082\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:50,641 [INFO ] pool-1-thread-2 ACCESS_LOG - /169.254.255.130:41532 \"GET /ping HTTP/1.1\" 200 10\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:50,650 [INFO ] pool-1-thread-2 TS_METRICS - Requests2XX.Count:1|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:50,682 [INFO ] epollEventLoopGroup-3-2 ACCESS_LOG - /169.254.255.130:41536 \"GET /execution-parameters HTTP/1.1\" 404 2\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:50,683 [INFO ] epollEventLoopGroup-3-2 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[34mModel server started.\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:50,807 [WARN ] pool-2-thread-1 org.pytorch.serve.metrics.MetricCollector - worker pid is not available yet.\u001b[0m\n",
      "\u001b[35mModel server started.\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:50,807 [WARN ] pool-2-thread-1 org.pytorch.serve.metrics.MetricCollector - worker pid is not available yet.\u001b[0m\n",
      "\u001b[32m2021-07-14T21:52:50.698:[sagemaker logs]: MaxConcurrentTransforms=2, MaxPayloadInMB=6, BatchStrategy=SINGLE_RECORD\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:50,929 [INFO ] pool-2-thread-1 TS_METRICS - CPUUtilization.Percent:0.0|#Level:Host|#hostname:c166cabe72bf,timestamp:1626299570\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:50,929 [INFO ] pool-2-thread-1 TS_METRICS - DiskAvailable.Gigabytes:40.294002532958984|#Level:Host|#hostname:c166cabe72bf,timestamp:1626299570\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:50,929 [INFO ] pool-2-thread-1 TS_METRICS - DiskUsage.Gigabytes:15.621910095214844|#Level:Host|#hostname:c166cabe72bf,timestamp:1626299570\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:50,930 [INFO ] pool-2-thread-1 TS_METRICS - DiskUtilization.Percent:27.9|#Level:Host|#hostname:c166cabe72bf,timestamp:1626299570\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:50,930 [INFO ] pool-2-thread-1 TS_METRICS - MemoryAvailable.Megabytes:14635.0390625|#Level:Host|#hostname:c166cabe72bf,timestamp:1626299570\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:50,930 [INFO ] pool-2-thread-1 TS_METRICS - MemoryUsed.Megabytes:1095.14453125|#Level:Host|#hostname:c166cabe72bf,timestamp:1626299570\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:50,931 [INFO ] pool-2-thread-1 TS_METRICS - MemoryUtilization.Percent:8.8|#Level:Host|#hostname:c166cabe72bf,timestamp:1626299570\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:51,218 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - Listening on port: /home/model-server/tmp/.ts.sock.9000\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:51,219 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - [PID]138\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:51,219 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - Torch worker started.\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:51,219 [DEBUG] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-TransformerEn2Fr_1.0 State change null -> WORKER_STARTED\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:51,219 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - Python runtime: 3.6.9\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:51,223 [INFO ] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /home/model-server/tmp/.ts.sock.9000\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:51,233 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - Connection accepted: /home/model-server/tmp/.ts.sock.9000.\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:51,256 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - batchSize: 4\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:51,746 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - loading archive file /home/model-server/tmp/models/0c2b4435393041bb91a6a960eaa61f42\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:51,746 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - loading archive file /home/model-server/tmp/models/0c2b4435393041bb91a6a960eaa61f42\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:50,929 [INFO ] pool-2-thread-1 TS_METRICS - CPUUtilization.Percent:0.0|#Level:Host|#hostname:c166cabe72bf,timestamp:1626299570\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:50,929 [INFO ] pool-2-thread-1 TS_METRICS - DiskAvailable.Gigabytes:40.294002532958984|#Level:Host|#hostname:c166cabe72bf,timestamp:1626299570\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:50,929 [INFO ] pool-2-thread-1 TS_METRICS - DiskUsage.Gigabytes:15.621910095214844|#Level:Host|#hostname:c166cabe72bf,timestamp:1626299570\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:50,930 [INFO ] pool-2-thread-1 TS_METRICS - DiskUtilization.Percent:27.9|#Level:Host|#hostname:c166cabe72bf,timestamp:1626299570\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:50,930 [INFO ] pool-2-thread-1 TS_METRICS - MemoryAvailable.Megabytes:14635.0390625|#Level:Host|#hostname:c166cabe72bf,timestamp:1626299570\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:50,930 [INFO ] pool-2-thread-1 TS_METRICS - MemoryUsed.Megabytes:1095.14453125|#Level:Host|#hostname:c166cabe72bf,timestamp:1626299570\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:50,931 [INFO ] pool-2-thread-1 TS_METRICS - MemoryUtilization.Percent:8.8|#Level:Host|#hostname:c166cabe72bf,timestamp:1626299570\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:51,218 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - Listening on port: /home/model-server/tmp/.ts.sock.9000\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:51,219 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - [PID]138\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:51,219 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - Torch worker started.\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:51,219 [DEBUG] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-TransformerEn2Fr_1.0 State change null -> WORKER_STARTED\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:51,219 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - Python runtime: 3.6.9\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:51,223 [INFO ] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /home/model-server/tmp/.ts.sock.9000\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:51,233 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - Connection accepted: /home/model-server/tmp/.ts.sock.9000.\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:51,256 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - batchSize: 4\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:51,746 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - loading archive file /home/model-server/tmp/models/0c2b4435393041bb91a6a960eaa61f42\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:51,746 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - loading archive file /home/model-server/tmp/models/0c2b4435393041bb91a6a960eaa61f42\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:07,761 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - Requests2XX.Count:1|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:07,761 [DEBUG] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.job.Job - Waiting time ns: 14300586193, Backend time ns: 2689227249\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:07,761 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - QueueTime.ms:14300|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:07,761 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - Requests2XX.Count:1|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:07,761 [DEBUG] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.job.Job - Waiting time ns: 14300586193, Backend time ns: 2689227249\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:07,761 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - QueueTime.ms:14300|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:07,762 [INFO ] W-9000-TransformerEn2Fr_1.0 ACCESS_LOG - /169.254.255.130:41544 \"POST /invocations HTTP/1.1\" 200 16959\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:07,762 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - Requests2XX.Count:1|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:07,763 [DEBUG] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.job.Job - Waiting time ns: 14268455993, Backend time ns: 2690601390\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:07,763 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - QueueTime.ms:14268|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:07,763 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - WorkerThreadTime.ms:4|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:07,762 [INFO ] W-9000-TransformerEn2Fr_1.0 ACCESS_LOG - /169.254.255.130:41544 \"POST /invocations HTTP/1.1\" 200 16959\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:07,762 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - Requests2XX.Count:1|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:07,763 [DEBUG] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.job.Job - Waiting time ns: 14268455993, Backend time ns: 2690601390\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:07,763 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - QueueTime.ms:14268|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:07,763 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - WorkerThreadTime.ms:4|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:08,369 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - preprocess data: [{'body': bytearray(b'Hi James, when are you coming back home? I am waiting for you.\\nPlease come as soon as possible.\\n')}, {'body': bytearray(b'I\\xe2\\x80\\x99m sorry, I don\\xe2\\x80\\x99t remember your name. You are you?\\n')}]\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:08,370 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - preprocessed data: ['Hi James, when are you coming back home? I am waiting for you.\\nPlease come as soon as possible.\\n', 'I’m sorry, I don’t remember your name. You are you?\\n']\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:08,369 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - preprocess data: [{'body': bytearray(b'Hi James, when are you coming back home? I am waiting for you.\\nPlease come as soon as possible.\\n')}, {'body': bytearray(b'I\\xe2\\x80\\x99m sorry, I don\\xe2\\x80\\x99t remember your name. You are you?\\n')}]\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:08,370 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - preprocessed data: ['Hi James, when are you coming back home? I am waiting for you.\\nPlease come as soon as possible.\\n', 'I’m sorry, I don’t remember your name. You are you?\\n']\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:10,995 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - Model translated: ['Bonjour James, quand rentrerez-vous chez vous, je vous attends et je vous prie de venir le plus tôt possible.', \"Je vous prie de m'excuser, je ne me souviens pas de votre nom.\"]\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:10,995 [INFO ] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 2627\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:10,996 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_METRICS - HandlerTime.Milliseconds:2625.98|#ModelName:TransformerEn2Fr,Level:Model|#hostname:c166cabe72bf,requestID:70314e99-eff3-4e9e-b414-d5c455553fa7,1a6c1d6c-b123-446f-b2cb-f24c9cbb9f35,timestamp:1626299590\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:10,996 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_METRICS - PredictionTime.Milliseconds:2626.03|#ModelName:TransformerEn2Fr,Level:Model|#hostname:c166cabe72bf,requestID:70314e99-eff3-4e9e-b414-d5c455553fa7,1a6c1d6c-b123-446f-b2cb-f24c9cbb9f35,timestamp:1626299590\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:10,996 [INFO ] W-9000-TransformerEn2Fr_1.0 ACCESS_LOG - /169.254.255.130:41544 \"POST /invocations HTTP/1.1\" 200 3129\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:10,996 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - Requests2XX.Count:1|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:10,997 [DEBUG] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.job.Job - Waiting time ns: 500693981, Backend time ns: 2629150981\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:10,997 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - QueueTime.ms:500|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:10,997 [INFO ] W-9000-TransformerEn2Fr_1.0 ACCESS_LOG - /169.254.255.130:41540 \"POST /invocations HTTP/1.1\" 200 3130\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:10,997 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - Requests2XX.Count:1|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:10,997 [DEBUG] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.job.Job - Waiting time ns: 500358482, Backend time ns: 2630042648\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:10,995 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - Model translated: ['Bonjour James, quand rentrerez-vous chez vous, je vous attends et je vous prie de venir le plus tôt possible.', \"Je vous prie de m'excuser, je ne me souviens pas de votre nom.\"]\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:10,995 [INFO ] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 2627\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:10,996 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_METRICS - HandlerTime.Milliseconds:2625.98|#ModelName:TransformerEn2Fr,Level:Model|#hostname:c166cabe72bf,requestID:70314e99-eff3-4e9e-b414-d5c455553fa7,1a6c1d6c-b123-446f-b2cb-f24c9cbb9f35,timestamp:1626299590\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:10,996 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_METRICS - PredictionTime.Milliseconds:2626.03|#ModelName:TransformerEn2Fr,Level:Model|#hostname:c166cabe72bf,requestID:70314e99-eff3-4e9e-b414-d5c455553fa7,1a6c1d6c-b123-446f-b2cb-f24c9cbb9f35,timestamp:1626299590\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:10,996 [INFO ] W-9000-TransformerEn2Fr_1.0 ACCESS_LOG - /169.254.255.130:41544 \"POST /invocations HTTP/1.1\" 200 3129\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:10,996 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - Requests2XX.Count:1|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:10,997 [DEBUG] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.job.Job - Waiting time ns: 500693981, Backend time ns: 2629150981\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:10,997 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - QueueTime.ms:500|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:10,997 [INFO ] W-9000-TransformerEn2Fr_1.0 ACCESS_LOG - /169.254.255.130:41540 \"POST /invocations HTTP/1.1\" 200 3130\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:10,997 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - Requests2XX.Count:1|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:10,997 [DEBUG] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.job.Job - Waiting time ns: 500358482, Backend time ns: 2630042648\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:10,998 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - QueueTime.ms:500|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:10,998 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - WorkerThreadTime.ms:4|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:11,505 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - preprocess data: [{'body': bytearray(b'Hello world\\nGood morning\\nGood evening\\nThis is a dog\\nThis is a cat\\nGo swimming\\nGo to Paris\\nThank you\\n')}, {'body': bytearray(b'I\\xe2\\x80\\x99m well. How are you?\\nIt\\xe2\\x80\\x99s going well, thank you. How are you doing?\\nFine, thanks. And yourself?\\n')}]\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:11,505 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - preprocessed data: ['Hello world\\nGood morning\\nGood evening\\nThis is a dog\\nThis is a cat\\nGo swimming\\nGo to Paris\\nThank you\\n', 'I’m well. How are you?\\nIt’s going well, thank you. How are you doing?\\nFine, thanks. And yourself?\\n']\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:10,998 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - QueueTime.ms:500|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:10,998 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - WorkerThreadTime.ms:4|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:11,505 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - preprocess data: [{'body': bytearray(b'Hello world\\nGood morning\\nGood evening\\nThis is a dog\\nThis is a cat\\nGo swimming\\nGo to Paris\\nThank you\\n')}, {'body': bytearray(b'I\\xe2\\x80\\x99m well. How are you?\\nIt\\xe2\\x80\\x99s going well, thank you. How are you doing?\\nFine, thanks. And yourself?\\n')}]\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:11,505 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - preprocessed data: ['Hello world\\nGood morning\\nGood evening\\nThis is a dog\\nThis is a cat\\nGo swimming\\nGo to Paris\\nThank you\\n', 'I’m well. How are you?\\nIt’s going well, thank you. How are you doing?\\nFine, thanks. And yourself?\\n']\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:14,255 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - Model translated: [\"Bonjour Monde Bonjour Bonsoir C'est un chien C'est un chat Allez nager Allez à Paris Merci\", 'Je me sens bien. Comment allez-vous ? Ça va bien, merci. Comment allez-vous ?']\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:14,255 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_METRICS - HandlerTime.Milliseconds:2750.51|#ModelName:TransformerEn2Fr,Level:Model|#hostname:c166cabe72bf,requestID:f7a442f1-21c7-473e-8fb7-272c6742540f,810fe8e0-e625-4d97-afed-034bfcd07c74,timestamp:1626299594\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:14,256 [INFO ] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 2752\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:14,256 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_METRICS - PredictionTime.Milliseconds:2750.57|#ModelName:TransformerEn2Fr,Level:Model|#hostname:c166cabe72bf,requestID:f7a442f1-21c7-473e-8fb7-272c6742540f,810fe8e0-e625-4d97-afed-034bfcd07c74,timestamp:1626299594\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:14,256 [INFO ] W-9000-TransformerEn2Fr_1.0 ACCESS_LOG - /169.254.255.130:41540 \"POST /invocations HTTP/1.1\" 200 3253\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:14,256 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - Requests2XX.Count:1|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:14,257 [DEBUG] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.job.Job - Waiting time ns: 500307809, Backend time ns: 2753299510\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:14,257 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - QueueTime.ms:500|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:14,257 [INFO ] W-9000-TransformerEn2Fr_1.0 ACCESS_LOG - /169.254.255.130:41544 \"POST /invocations HTTP/1.1\" 200 3252\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:14,257 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - Requests2XX.Count:1|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:14,257 [DEBUG] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.job.Job - Waiting time ns: 498379862, Backend time ns: 2754086106\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:14,258 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - QueueTime.ms:498|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:14,258 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - WorkerThreadTime.ms:3|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:14,255 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - Model translated: [\"Bonjour Monde Bonjour Bonsoir C'est un chien C'est un chat Allez nager Allez à Paris Merci\", 'Je me sens bien. Comment allez-vous ? Ça va bien, merci. Comment allez-vous ?']\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:14,255 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_METRICS - HandlerTime.Milliseconds:2750.51|#ModelName:TransformerEn2Fr,Level:Model|#hostname:c166cabe72bf,requestID:f7a442f1-21c7-473e-8fb7-272c6742540f,810fe8e0-e625-4d97-afed-034bfcd07c74,timestamp:1626299594\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:14,256 [INFO ] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 2752\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:14,256 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_METRICS - PredictionTime.Milliseconds:2750.57|#ModelName:TransformerEn2Fr,Level:Model|#hostname:c166cabe72bf,requestID:f7a442f1-21c7-473e-8fb7-272c6742540f,810fe8e0-e625-4d97-afed-034bfcd07c74,timestamp:1626299594\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:14,256 [INFO ] W-9000-TransformerEn2Fr_1.0 ACCESS_LOG - /169.254.255.130:41540 \"POST /invocations HTTP/1.1\" 200 3253\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:14,256 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - Requests2XX.Count:1|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:14,257 [DEBUG] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.job.Job - Waiting time ns: 500307809, Backend time ns: 2753299510\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:14,257 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - QueueTime.ms:500|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:14,257 [INFO ] W-9000-TransformerEn2Fr_1.0 ACCESS_LOG - /169.254.255.130:41544 \"POST /invocations HTTP/1.1\" 200 3252\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:14,257 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - Requests2XX.Count:1|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:14,257 [DEBUG] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.job.Job - Waiting time ns: 498379862, Backend time ns: 2754086106\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:14,258 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - QueueTime.ms:498|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:14,258 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - WorkerThreadTime.ms:3|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\n",
      "\u001b[34mCUDNN_VERSION=7.6.5.32\u001b[0m\n",
      "\u001b[34mPYTHONUNBUFFERED=TRUE\u001b[0m\n",
      "\u001b[34mLD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64\u001b[0m\n",
      "\u001b[34mSAGEMAKER_MAX_CONCURRENT_TRANSFORMS=2\u001b[0m\n",
      "\u001b[34mSAGEMAKER_BATCH_STRATEGY=SINGLE_RECORD\u001b[0m\n",
      "\u001b[34mLANG=C.UTF-8\u001b[0m\n",
      "\u001b[35mCUDNN_VERSION=7.6.5.32\u001b[0m\n",
      "\u001b[35mPYTHONUNBUFFERED=TRUE\u001b[0m\n",
      "\u001b[35mLD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64\u001b[0m\n",
      "\u001b[35mSAGEMAKER_MAX_CONCURRENT_TRANSFORMS=2\u001b[0m\n",
      "\u001b[35mSAGEMAKER_BATCH_STRATEGY=SINGLE_RECORD\u001b[0m\n",
      "\u001b[35mLANG=C.UTF-8\u001b[0m\n",
      "\u001b[34mSAGEMAKER_SAFE_PORT_RANGE=10000-10999\u001b[0m\n",
      "\u001b[34mHOSTNAME=c166cabe72bf\u001b[0m\n",
      "\u001b[34mPYTHONIOENCODING=UTF-8\u001b[0m\n",
      "\u001b[34mNVIDIA_VISIBLE_DEVICES=all\u001b[0m\n",
      "\u001b[34mNCCL_VERSION=2.10.3\u001b[0m\n",
      "\u001b[34mPWD=/home/model-server\u001b[0m\n",
      "\u001b[34mHOME=/root\u001b[0m\n",
      "\u001b[34mSAGEMAKER_BATCH=true\u001b[0m\n",
      "\u001b[34mAWS_REGION=us-east-2\u001b[0m\n",
      "\u001b[35mSAGEMAKER_SAFE_PORT_RANGE=10000-10999\u001b[0m\n",
      "\u001b[35mHOSTNAME=c166cabe72bf\u001b[0m\n",
      "\u001b[35mPYTHONIOENCODING=UTF-8\u001b[0m\n",
      "\u001b[35mNVIDIA_VISIBLE_DEVICES=all\u001b[0m\n",
      "\u001b[35mNCCL_VERSION=2.10.3\u001b[0m\n",
      "\u001b[35mPWD=/home/model-server\u001b[0m\n",
      "\u001b[35mHOME=/root\u001b[0m\n",
      "\u001b[35mSAGEMAKER_BATCH=true\u001b[0m\n",
      "\u001b[35mAWS_REGION=us-east-2\u001b[0m\n",
      "\u001b[34mSAGEMAKER_BIND_TO_PORT=8080\u001b[0m\n",
      "\u001b[34mCUDA_PKG_VERSION=10-2=10.2.89-1\u001b[0m\n",
      "\u001b[34mCUDA_VERSION=10.2.89\u001b[0m\n",
      "\u001b[34mNVIDIA_DRIVER_CAPABILITIES=compute,utility\u001b[0m\n",
      "\u001b[34mSHLVL=1\u001b[0m\n",
      "\u001b[34mNVIDIA_REQUIRE_CUDA=cuda>=10.2 brand=tesla,driver>=396,driver<397 brand=tesla,driver>=410,driver<411 brand=tesla,driver>=418,driver<419 brand=tesla,driver>=440,driver<441\u001b[0m\n",
      "\u001b[34mAWS_CONTAINER_CREDENTIALS_RELATIVE_URI=/v2/credentials/54OQ8-83Gr-VPt9TmQCLjSHUs8Vb2IlZq3gZDJOtcl8\u001b[0m\n",
      "\u001b[34mTEMP=/home/model-server/tmp\u001b[0m\n",
      "\u001b[34mPATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\u001b[0m\n",
      "\u001b[34m_=/usr/bin/printenv\u001b[0m\n",
      "\u001b[34mml\u001b[0m\n",
      "\u001b[35mSAGEMAKER_BIND_TO_PORT=8080\u001b[0m\n",
      "\u001b[35mCUDA_PKG_VERSION=10-2=10.2.89-1\u001b[0m\n",
      "\u001b[35mCUDA_VERSION=10.2.89\u001b[0m\n",
      "\u001b[35mNVIDIA_DRIVER_CAPABILITIES=compute,utility\u001b[0m\n",
      "\u001b[35mSHLVL=1\u001b[0m\n",
      "\u001b[35mNVIDIA_REQUIRE_CUDA=cuda>=10.2 brand=tesla,driver>=396,driver<397 brand=tesla,driver>=410,driver<411 brand=tesla,driver>=418,driver<419 brand=tesla,driver>=440,driver<441\u001b[0m\n",
      "\u001b[35mAWS_CONTAINER_CREDENTIALS_RELATIVE_URI=/v2/credentials/54OQ8-83Gr-VPt9TmQCLjSHUs8Vb2IlZq3gZDJOtcl8\u001b[0m\n",
      "\u001b[35mTEMP=/home/model-server/tmp\u001b[0m\n",
      "\u001b[35mPATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\u001b[0m\n",
      "\u001b[35m_=/usr/bin/printenv\u001b[0m\n",
      "\u001b[35mml\u001b[0m\n",
      "\u001b[34m2021-07-14 21:49:26,175 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager - Initializing plugins manager...\u001b[0m\n",
      "\u001b[34m2021-07-14 21:49:26,383 [INFO ] main org.pytorch.serve.ModelServer - \u001b[0m\n",
      "\u001b[34mTorchserve version: 0.4.0\u001b[0m\n",
      "\u001b[34mTS Home: /usr/local/lib/python3.6/dist-packages\u001b[0m\n",
      "\u001b[34mCurrent directory: /home/model-server\u001b[0m\n",
      "\u001b[34mTemp directory: /home/model-server/tmp\u001b[0m\n",
      "\u001b[34mNumber of GPUs: 0\u001b[0m\n",
      "\u001b[34mNumber of CPUs: 4\u001b[0m\n",
      "\u001b[34mMax heap size: 3090 M\u001b[0m\n",
      "\u001b[34mPython executable: /usr/bin/python3\u001b[0m\n",
      "\u001b[34mConfig file: /home/model-server/config.properties\u001b[0m\n",
      "\u001b[34mInference address: http://0.0.0.0:8080\u001b[0m\n",
      "\u001b[34mManagement address: http://0.0.0.0:8081\u001b[0m\n",
      "\u001b[34mMetrics address: http://127.0.0.1:8082\u001b[0m\n",
      "\u001b[34mModel Store: /opt/ml/model\u001b[0m\n",
      "\u001b[34mInitial Models: all\u001b[0m\n",
      "\u001b[34mLog dir: /home/model-server/logs\u001b[0m\n",
      "\u001b[35m2021-07-14 21:49:26,175 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager - Initializing plugins manager...\u001b[0m\n",
      "\u001b[35m2021-07-14 21:49:26,383 [INFO ] main org.pytorch.serve.ModelServer - \u001b[0m\n",
      "\u001b[35mTorchserve version: 0.4.0\u001b[0m\n",
      "\u001b[35mTS Home: /usr/local/lib/python3.6/dist-packages\u001b[0m\n",
      "\u001b[35mCurrent directory: /home/model-server\u001b[0m\n",
      "\u001b[35mTemp directory: /home/model-server/tmp\u001b[0m\n",
      "\u001b[35mNumber of GPUs: 0\u001b[0m\n",
      "\u001b[35mNumber of CPUs: 4\u001b[0m\n",
      "\u001b[35mMax heap size: 3090 M\u001b[0m\n",
      "\u001b[35mPython executable: /usr/bin/python3\u001b[0m\n",
      "\u001b[35mConfig file: /home/model-server/config.properties\u001b[0m\n",
      "\u001b[35mInference address: http://0.0.0.0:8080\u001b[0m\n",
      "\u001b[35mManagement address: http://0.0.0.0:8081\u001b[0m\n",
      "\u001b[35mMetrics address: http://127.0.0.1:8082\u001b[0m\n",
      "\u001b[35mModel Store: /opt/ml/model\u001b[0m\n",
      "\u001b[35mInitial Models: all\u001b[0m\n",
      "\u001b[35mLog dir: /home/model-server/logs\u001b[0m\n",
      "\u001b[34mMetrics dir: /home/model-server/logs\u001b[0m\n",
      "\u001b[34mNetty threads: 32\u001b[0m\n",
      "\u001b[34mNetty client threads: 0\u001b[0m\n",
      "\u001b[34mDefault workers per model: 4\u001b[0m\n",
      "\u001b[34mBlacklist Regex: N/A\u001b[0m\n",
      "\u001b[34mMaximum Response Size: 6553500\u001b[0m\n",
      "\u001b[34mMaximum Request Size: 6553500\u001b[0m\n",
      "\u001b[34mPrefer direct buffer: false\u001b[0m\n",
      "\u001b[34mAllowed Urls: [file://.*|http(s)?://.*]\u001b[0m\n",
      "\u001b[34mCustom python dependency for model allowed: true\u001b[0m\n",
      "\u001b[34mMetrics report format: prometheus\u001b[0m\n",
      "\u001b[34mEnable metrics API: true\u001b[0m\n",
      "\u001b[34mWorkflow Store: /opt/ml/model\u001b[0m\n",
      "\u001b[34mModel config: {\"TransformerEn2Fr\": {\"1.0\": {\"defaultVersion\": true,\"marName\": \"TransformerEn2Fr.mar\",\"minWorkers\": 1,\"maxWorkers\": 4,\"batchSize\": 4,\"maxBatchDelay\": 500,\"responseTimeout\": 120}}}\u001b[0m\n",
      "\u001b[34m2021-07-14 21:49:26,396 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager -  Loading snapshot serializer plugin...\u001b[0m\n",
      "\u001b[35mMetrics dir: /home/model-server/logs\u001b[0m\n",
      "\u001b[35mNetty threads: 32\u001b[0m\n",
      "\u001b[35mNetty client threads: 0\u001b[0m\n",
      "\u001b[35mDefault workers per model: 4\u001b[0m\n",
      "\u001b[35mBlacklist Regex: N/A\u001b[0m\n",
      "\u001b[35mMaximum Response Size: 6553500\u001b[0m\n",
      "\u001b[35mMaximum Request Size: 6553500\u001b[0m\n",
      "\u001b[35mPrefer direct buffer: false\u001b[0m\n",
      "\u001b[35mAllowed Urls: [file://.*|http(s)?://.*]\u001b[0m\n",
      "\u001b[35mCustom python dependency for model allowed: true\u001b[0m\n",
      "\u001b[35mMetrics report format: prometheus\u001b[0m\n",
      "\u001b[35mEnable metrics API: true\u001b[0m\n",
      "\u001b[35mWorkflow Store: /opt/ml/model\u001b[0m\n",
      "\u001b[35mModel config: {\"TransformerEn2Fr\": {\"1.0\": {\"defaultVersion\": true,\"marName\": \"TransformerEn2Fr.mar\",\"minWorkers\": 1,\"maxWorkers\": 4,\"batchSize\": 4,\"maxBatchDelay\": 500,\"responseTimeout\": 120}}}\u001b[0m\n",
      "\u001b[35m2021-07-14 21:49:26,396 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager -  Loading snapshot serializer plugin...\u001b[0m\n",
      "\u001b[34m2021-07-14 21:49:26,400 [DEBUG] main org.pytorch.serve.ModelServer - Loading models from model store: TransformerEn2Fr.mar\u001b[0m\n",
      "\u001b[35m2021-07-14 21:49:26,400 [DEBUG] main org.pytorch.serve.ModelServer - Loading models from model store: TransformerEn2Fr.mar\u001b[0m\n",
      "\u001b[34m2021-07-14 21:50:35,076 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Adding new version 1.0 for model TransformerEn2Fr\u001b[0m\n",
      "\u001b[35m2021-07-14 21:50:35,076 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Adding new version 1.0 for model TransformerEn2Fr\u001b[0m\n",
      "\u001b[34m2021-07-14 21:50:35,077 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 1.0 for model TransformerEn2Fr\u001b[0m\n",
      "\u001b[34m2021-07-14 21:50:35,077 [INFO ] main org.pytorch.serve.wlm.ModelManager - Model TransformerEn2Fr loaded.\u001b[0m\n",
      "\u001b[35m2021-07-14 21:50:35,077 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 1.0 for model TransformerEn2Fr\u001b[0m\n",
      "\u001b[35m2021-07-14 21:50:35,077 [INFO ] main org.pytorch.serve.wlm.ModelManager - Model TransformerEn2Fr loaded.\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:50,312 [DEBUG] main org.pytorch.serve.wlm.ModelManager - updateModel: TransformerEn2Fr, count: 1\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:50,332 [INFO ] main org.pytorch.serve.ModelServer - Initialize Inference server with: EpollServerSocketChannel.\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:50,439 [INFO ] main org.pytorch.serve.ModelServer - Inference API bind to: http://0.0.0.0:8080\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:50,439 [INFO ] main org.pytorch.serve.ModelServer - Initialize Management server with: EpollServerSocketChannel.\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:50,445 [INFO ] main org.pytorch.serve.ModelServer - Management API bind to: http://0.0.0.0:8081\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:50,446 [INFO ] main org.pytorch.serve.ModelServer - Initialize Metrics server with: EpollServerSocketChannel.\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:50,447 [INFO ] main org.pytorch.serve.ModelServer - Metrics API bind to: http://127.0.0.1:8082\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:50,641 [INFO ] pool-1-thread-2 ACCESS_LOG - /169.254.255.130:41532 \"GET /ping HTTP/1.1\" 200 10\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:50,650 [INFO ] pool-1-thread-2 TS_METRICS - Requests2XX.Count:1|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:50,682 [INFO ] epollEventLoopGroup-3-2 ACCESS_LOG - /169.254.255.130:41536 \"GET /execution-parameters HTTP/1.1\" 404 2\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:50,683 [INFO ] epollEventLoopGroup-3-2 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:50,312 [DEBUG] main org.pytorch.serve.wlm.ModelManager - updateModel: TransformerEn2Fr, count: 1\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:50,332 [INFO ] main org.pytorch.serve.ModelServer - Initialize Inference server with: EpollServerSocketChannel.\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:50,439 [INFO ] main org.pytorch.serve.ModelServer - Inference API bind to: http://0.0.0.0:8080\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:50,439 [INFO ] main org.pytorch.serve.ModelServer - Initialize Management server with: EpollServerSocketChannel.\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:50,445 [INFO ] main org.pytorch.serve.ModelServer - Management API bind to: http://0.0.0.0:8081\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:50,446 [INFO ] main org.pytorch.serve.ModelServer - Initialize Metrics server with: EpollServerSocketChannel.\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:50,447 [INFO ] main org.pytorch.serve.ModelServer - Metrics API bind to: http://127.0.0.1:8082\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:50,641 [INFO ] pool-1-thread-2 ACCESS_LOG - /169.254.255.130:41532 \"GET /ping HTTP/1.1\" 200 10\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:50,650 [INFO ] pool-1-thread-2 TS_METRICS - Requests2XX.Count:1|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:50,682 [INFO ] epollEventLoopGroup-3-2 ACCESS_LOG - /169.254.255.130:41536 \"GET /execution-parameters HTTP/1.1\" 404 2\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:50,683 [INFO ] epollEventLoopGroup-3-2 TS_METRICS - Requests4XX.Count:1|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[34mModel server started.\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:50,807 [WARN ] pool-2-thread-1 org.pytorch.serve.metrics.MetricCollector - worker pid is not available yet.\u001b[0m\n",
      "\u001b[35mModel server started.\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:50,807 [WARN ] pool-2-thread-1 org.pytorch.serve.metrics.MetricCollector - worker pid is not available yet.\u001b[0m\n",
      "\u001b[32m2021-07-14T21:52:50.698:[sagemaker logs]: MaxConcurrentTransforms=2, MaxPayloadInMB=6, BatchStrategy=SINGLE_RECORD\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:50,929 [INFO ] pool-2-thread-1 TS_METRICS - CPUUtilization.Percent:0.0|#Level:Host|#hostname:c166cabe72bf,timestamp:1626299570\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:50,929 [INFO ] pool-2-thread-1 TS_METRICS - DiskAvailable.Gigabytes:40.294002532958984|#Level:Host|#hostname:c166cabe72bf,timestamp:1626299570\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:50,929 [INFO ] pool-2-thread-1 TS_METRICS - DiskUsage.Gigabytes:15.621910095214844|#Level:Host|#hostname:c166cabe72bf,timestamp:1626299570\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:50,930 [INFO ] pool-2-thread-1 TS_METRICS - DiskUtilization.Percent:27.9|#Level:Host|#hostname:c166cabe72bf,timestamp:1626299570\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:50,930 [INFO ] pool-2-thread-1 TS_METRICS - MemoryAvailable.Megabytes:14635.0390625|#Level:Host|#hostname:c166cabe72bf,timestamp:1626299570\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:50,930 [INFO ] pool-2-thread-1 TS_METRICS - MemoryUsed.Megabytes:1095.14453125|#Level:Host|#hostname:c166cabe72bf,timestamp:1626299570\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:50,931 [INFO ] pool-2-thread-1 TS_METRICS - MemoryUtilization.Percent:8.8|#Level:Host|#hostname:c166cabe72bf,timestamp:1626299570\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:51,218 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - Listening on port: /home/model-server/tmp/.ts.sock.9000\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:51,219 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - [PID]138\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:51,219 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - Torch worker started.\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:51,219 [DEBUG] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-TransformerEn2Fr_1.0 State change null -> WORKER_STARTED\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:51,219 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - Python runtime: 3.6.9\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:51,223 [INFO ] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /home/model-server/tmp/.ts.sock.9000\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:51,233 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - Connection accepted: /home/model-server/tmp/.ts.sock.9000.\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:51,256 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - batchSize: 4\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:51,746 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - loading archive file /home/model-server/tmp/models/0c2b4435393041bb91a6a960eaa61f42\u001b[0m\n",
      "\u001b[34m2021-07-14 21:52:51,746 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - loading archive file /home/model-server/tmp/models/0c2b4435393041bb91a6a960eaa61f42\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:50,929 [INFO ] pool-2-thread-1 TS_METRICS - CPUUtilization.Percent:0.0|#Level:Host|#hostname:c166cabe72bf,timestamp:1626299570\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:50,929 [INFO ] pool-2-thread-1 TS_METRICS - DiskAvailable.Gigabytes:40.294002532958984|#Level:Host|#hostname:c166cabe72bf,timestamp:1626299570\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:50,929 [INFO ] pool-2-thread-1 TS_METRICS - DiskUsage.Gigabytes:15.621910095214844|#Level:Host|#hostname:c166cabe72bf,timestamp:1626299570\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:50,930 [INFO ] pool-2-thread-1 TS_METRICS - DiskUtilization.Percent:27.9|#Level:Host|#hostname:c166cabe72bf,timestamp:1626299570\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:50,930 [INFO ] pool-2-thread-1 TS_METRICS - MemoryAvailable.Megabytes:14635.0390625|#Level:Host|#hostname:c166cabe72bf,timestamp:1626299570\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:50,930 [INFO ] pool-2-thread-1 TS_METRICS - MemoryUsed.Megabytes:1095.14453125|#Level:Host|#hostname:c166cabe72bf,timestamp:1626299570\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:50,931 [INFO ] pool-2-thread-1 TS_METRICS - MemoryUtilization.Percent:8.8|#Level:Host|#hostname:c166cabe72bf,timestamp:1626299570\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:51,218 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - Listening on port: /home/model-server/tmp/.ts.sock.9000\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:51,219 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - [PID]138\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:51,219 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - Torch worker started.\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:51,219 [DEBUG] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-TransformerEn2Fr_1.0 State change null -> WORKER_STARTED\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:51,219 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - Python runtime: 3.6.9\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:51,223 [INFO ] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /home/model-server/tmp/.ts.sock.9000\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:51,233 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - Connection accepted: /home/model-server/tmp/.ts.sock.9000.\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:51,256 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - batchSize: 4\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:51,746 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - loading archive file /home/model-server/tmp/models/0c2b4435393041bb91a6a960eaa61f42\u001b[0m\n",
      "\u001b[35m2021-07-14 21:52:51,746 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - loading archive file /home/model-server/tmp/models/0c2b4435393041bb91a6a960eaa61f42\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:00,794 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - [en] dictionary: 44512 types\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:00,794 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - [fr] dictionary: 44512 types\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:00,794 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - [en] dictionary: 44512 types\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:00,794 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - [fr] dictionary: 44512 types\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:04,387 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - Namespace(activation_dropout=0.0, activation_fn='relu', adam_betas='(0.9, 0.98)', adam_eps=1e-08, adaptive_input=False, adaptive_softmax_cutoff=None, adaptive_softmax_dropout=0, arch='transformer_vaswani_wmt_en_de_big', attention_dropout=0.0, batch_size=None, bpe='subword_nmt', bpe_codes='/home/model-server/tmp/models/0c2b4435393041bb91a6a960eaa61f42/bpecodes', bpe_separator='@@', clip_norm=0.0, criterion='label_smoothed_cross_entropy', cross_self_attention=False, data='/home/model-server/tmp/models/0c2b4435393041bb91a6a960eaa61f42', decoder_attention_heads=16, decoder_embed_dim=1024, decoder_embed_path=None, decoder_ffn_embed_dim=4096, decoder_input_dim=1024, decoder_layerdrop=0, decoder_layers=6, decoder_layers_to_keep=None, decoder_learned_pos=False, decoder_normalize_before=False, decoder_output_dim=1024, device_id=0, distributed_backend='nccl', distributed_init_method='tcp://learnfair0253:58342', distributed_port=58342, distributed_rank=0, distributed_world_size=128, dropout=0.1, encoder_attention_heads=16, encoder_embed_dim=1024, encoder_embed_path=None, encoder_ffn_embed_dim=4096, encoder_layerdrop=0, encoder_layers=6, encoder_layers_to_keep=None, encoder_learned_pos=False, encoder_normalize_before=False, eval_bleu_detok='space', eval_bleu_remove_bpe=None, eval_tokenized_bleu=False, fp16=True, ignore_prefix_size=0, label_smoothing=0.1, layernorm_embedding=False, left_pad_source=True, left_pad_target=False, log_format='json', log_interval=10, lr=[0.0007], lr_scheduler='inverse_sqrt', lr_shrink=0.1, max_epoch=0, max_sentences=None, max_sentences_valid=None, max_source_positions=1024, max_target_positions=1024, max_tokens=5120, max_update=80000, min_lr=1e-09, momentum=0.99, moses_no_dash_splits=False, moses_no_escape=False, no_cross_attention=False, no_epoch_checkpoints=False, no_progress_bar=False, no_save=False, no_scale_embedding=False, no_token_positional_embeddings=False, num_batch_buckets=0, optimizer='adam', quant_noise_pq=0, quant_noise_pq_block_size=8, quant_noise_scalar=0, relu_dropout=0.0, restore_file='checkpoint_last.pt', sample_without_replacement=0, save_dir='/checkpoint02/myleott/2018-05-19/wmt14_en_fr.fp16_allreduce.fp16.maxupd80000.transformer_vaswani_wmt_en_de_big.shareemb.adam.beta0.9,0.98.initlr1e-07.warmup4000.lr0.0007.clip0.0.drop0.1.wd0.0.ls0.1.maxtok5120.seed2.ngpu128', save_interval=1, seed=2, sentence_avg=False, share_all_embeddings=True, share_decoder_input_output_embed=True, skip_invalid_size_inputs_valid_test=False, source_lang='en', target_lang='fr', task='translation', tie_adaptive_weights=False, tokenizer='moses', train_subset='train', truncate_source=False, update_freq=[1.0], upsample_primary=1, use_old_adam=False, valid_subset='valid', validate_interval=1, warmup_init_lr=1e-07, warmup_updates=4000, weight_decay=0.0)\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:04,571 [INFO ] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 13315\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:04,387 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - Namespace(activation_dropout=0.0, activation_fn='relu', adam_betas='(0.9, 0.98)', adam_eps=1e-08, adaptive_input=False, adaptive_softmax_cutoff=None, adaptive_softmax_dropout=0, arch='transformer_vaswani_wmt_en_de_big', attention_dropout=0.0, batch_size=None, bpe='subword_nmt', bpe_codes='/home/model-server/tmp/models/0c2b4435393041bb91a6a960eaa61f42/bpecodes', bpe_separator='@@', clip_norm=0.0, criterion='label_smoothed_cross_entropy', cross_self_attention=False, data='/home/model-server/tmp/models/0c2b4435393041bb91a6a960eaa61f42', decoder_attention_heads=16, decoder_embed_dim=1024, decoder_embed_path=None, decoder_ffn_embed_dim=4096, decoder_input_dim=1024, decoder_layerdrop=0, decoder_layers=6, decoder_layers_to_keep=None, decoder_learned_pos=False, decoder_normalize_before=False, decoder_output_dim=1024, device_id=0, distributed_backend='nccl', distributed_init_method='tcp://learnfair0253:58342', distributed_port=58342, distributed_rank=0, distributed_world_size=128, dropout=0.1, encoder_attention_heads=16, encoder_embed_dim=1024, encoder_embed_path=None, encoder_ffn_embed_dim=4096, encoder_layerdrop=0, encoder_layers=6, encoder_layers_to_keep=None, encoder_learned_pos=False, encoder_normalize_before=False, eval_bleu_detok='space', eval_bleu_remove_bpe=None, eval_tokenized_bleu=False, fp16=True, ignore_prefix_size=0, label_smoothing=0.1, layernorm_embedding=False, left_pad_source=True, left_pad_target=False, log_format='json', log_interval=10, lr=[0.0007], lr_scheduler='inverse_sqrt', lr_shrink=0.1, max_epoch=0, max_sentences=None, max_sentences_valid=None, max_source_positions=1024, max_target_positions=1024, max_tokens=5120, max_update=80000, min_lr=1e-09, momentum=0.99, moses_no_dash_splits=False, moses_no_escape=False, no_cross_attention=False, no_epoch_checkpoints=False, no_progress_bar=False, no_save=False, no_scale_embedding=False, no_token_positional_embeddings=False, num_batch_buckets=0, optimizer='adam', quant_noise_pq=0, quant_noise_pq_block_size=8, quant_noise_scalar=0, relu_dropout=0.0, restore_file='checkpoint_last.pt', sample_without_replacement=0, save_dir='/checkpoint02/myleott/2018-05-19/wmt14_en_fr.fp16_allreduce.fp16.maxupd80000.transformer_vaswani_wmt_en_de_big.shareemb.adam.beta0.9,0.98.initlr1e-07.warmup4000.lr0.0007.clip0.0.drop0.1.wd0.0.ls0.1.maxtok5120.seed2.ngpu128', save_interval=1, seed=2, sentence_avg=False, share_all_embeddings=True, share_decoder_input_output_embed=True, skip_invalid_size_inputs_valid_test=False, source_lang='en', target_lang='fr', task='translation', tie_adaptive_weights=False, tokenizer='moses', train_subset='train', truncate_source=False, update_freq=[1.0], upsample_primary=1, use_old_adam=False, valid_subset='valid', validate_interval=1, warmup_init_lr=1e-07, warmup_updates=4000, weight_decay=0.0)\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:04,571 [INFO ] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 13315\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:04,571 [DEBUG] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-TransformerEn2Fr_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:04,571 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - W-9000-TransformerEn2Fr_1.0.ms:14250|#Level:Host|#hostname:c166cabe72bf,timestamp:1626299584\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:04,572 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - WorkerThreadTime.ms:22|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:04,571 [DEBUG] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-TransformerEn2Fr_1.0 State change WORKER_STARTED -> WORKER_MODEL_LOADED\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:04,571 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - W-9000-TransformerEn2Fr_1.0.ms:14250|#Level:Host|#hostname:c166cabe72bf,timestamp:1626299584\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:04,572 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - WorkerThreadTime.ms:22|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:05,074 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - preprocess data: [{'body': bytearray(b'Hi James, when are you coming back home? I am waiting for you.\\nPlease come as soon as possible.')}, {'body': bytearray(b'Hello World !!!\\n')}]\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:05,075 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - preprocessed data: ['Hi James, when are you coming back home? I am waiting for you.\\nPlease come as soon as possible.', 'Hello World !!!\\n']\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:05,516 [WARN ] W-9000-TransformerEn2Fr_1.0-stderr MODEL_LOG - /home/model-server/tmp/models/0c2b4435393041bb91a6a960eaa61f42/torch/_tensor.py:575: UserWarning: floor_divide is deprecated, and will be removed in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values.\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:05,517 [WARN ] W-9000-TransformerEn2Fr_1.0-stderr MODEL_LOG - To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). (Triggered internally at  /pytorch/aten/src/ATen/native/BinaryOps.cpp:467.)\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:05,518 [WARN ] W-9000-TransformerEn2Fr_1.0-stderr MODEL_LOG -   return torch.floor_divide(self, other)\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:05,074 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - preprocess data: [{'body': bytearray(b'Hi James, when are you coming back home? I am waiting for you.\\nPlease come as soon as possible.')}, {'body': bytearray(b'Hello World !!!\\n')}]\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:05,075 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - preprocessed data: ['Hi James, when are you coming back home? I am waiting for you.\\nPlease come as soon as possible.', 'Hello World !!!\\n']\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:05,516 [WARN ] W-9000-TransformerEn2Fr_1.0-stderr MODEL_LOG - /home/model-server/tmp/models/0c2b4435393041bb91a6a960eaa61f42/torch/_tensor.py:575: UserWarning: floor_divide is deprecated, and will be removed in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values.\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:05,517 [WARN ] W-9000-TransformerEn2Fr_1.0-stderr MODEL_LOG - To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). (Triggered internally at  /pytorch/aten/src/ATen/native/BinaryOps.cpp:467.)\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:05,518 [WARN ] W-9000-TransformerEn2Fr_1.0-stderr MODEL_LOG -   return torch.floor_divide(self, other)\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:07,759 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - Model translated: ['Bonjour James, quand rentrerez-vous chez vous, je vous attends et je vous prie de venir le plus tôt possible.', 'Bonjour le monde ! ! !']\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:07,760 [INFO ] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 2687\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:07,759 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_METRICS - HandlerTime.Milliseconds:2684.97|#ModelName:TransformerEn2Fr,Level:Model|#hostname:c166cabe72bf,requestID:38c5f9f4-4b73-4ca0-af21-058e0ba5acb7,bd6594b3-a5b8-4d63-8afb-b6ee9c6160b5,timestamp:1626299587\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:07,761 [INFO ] W-9000-TransformerEn2Fr_1.0 ACCESS_LOG - /169.254.255.130:41540 \"POST /invocations HTTP/1.1\" 200 17001\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:07,761 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_METRICS - PredictionTime.Milliseconds:2685.05|#ModelName:TransformerEn2Fr,Level:Model|#hostname:c166cabe72bf,requestID:38c5f9f4-4b73-4ca0-af21-058e0ba5acb7,bd6594b3-a5b8-4d63-8afb-b6ee9c6160b5,timestamp:1626299587\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:07,761 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - Requests2XX.Count:1|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:07,761 [DEBUG] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.job.Job - Waiting time ns: 14300586193, Backend time ns: 2689227249\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:07,761 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - QueueTime.ms:14300|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:07,759 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - Model translated: ['Bonjour James, quand rentrerez-vous chez vous, je vous attends et je vous prie de venir le plus tôt possible.', 'Bonjour le monde ! ! !']\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:07,760 [INFO ] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 2687\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:07,759 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_METRICS - HandlerTime.Milliseconds:2684.97|#ModelName:TransformerEn2Fr,Level:Model|#hostname:c166cabe72bf,requestID:38c5f9f4-4b73-4ca0-af21-058e0ba5acb7,bd6594b3-a5b8-4d63-8afb-b6ee9c6160b5,timestamp:1626299587\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:07,761 [INFO ] W-9000-TransformerEn2Fr_1.0 ACCESS_LOG - /169.254.255.130:41540 \"POST /invocations HTTP/1.1\" 200 17001\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:07,761 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_METRICS - PredictionTime.Milliseconds:2685.05|#ModelName:TransformerEn2Fr,Level:Model|#hostname:c166cabe72bf,requestID:38c5f9f4-4b73-4ca0-af21-058e0ba5acb7,bd6594b3-a5b8-4d63-8afb-b6ee9c6160b5,timestamp:1626299587\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:07,761 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - Requests2XX.Count:1|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:07,761 [DEBUG] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.job.Job - Waiting time ns: 14300586193, Backend time ns: 2689227249\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:07,761 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - QueueTime.ms:14300|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:07,762 [INFO ] W-9000-TransformerEn2Fr_1.0 ACCESS_LOG - /169.254.255.130:41544 \"POST /invocations HTTP/1.1\" 200 16959\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:07,762 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - Requests2XX.Count:1|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:07,763 [DEBUG] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.job.Job - Waiting time ns: 14268455993, Backend time ns: 2690601390\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:07,763 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - QueueTime.ms:14268|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:07,763 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - WorkerThreadTime.ms:4|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:07,762 [INFO ] W-9000-TransformerEn2Fr_1.0 ACCESS_LOG - /169.254.255.130:41544 \"POST /invocations HTTP/1.1\" 200 16959\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:07,762 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - Requests2XX.Count:1|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:07,763 [DEBUG] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.job.Job - Waiting time ns: 14268455993, Backend time ns: 2690601390\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:07,763 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - QueueTime.ms:14268|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:07,763 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - WorkerThreadTime.ms:4|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:08,369 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - preprocess data: [{'body': bytearray(b'Hi James, when are you coming back home? I am waiting for you.\\nPlease come as soon as possible.\\n')}, {'body': bytearray(b'I\\xe2\\x80\\x99m sorry, I don\\xe2\\x80\\x99t remember your name. You are you?\\n')}]\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:08,370 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - preprocessed data: ['Hi James, when are you coming back home? I am waiting for you.\\nPlease come as soon as possible.\\n', 'I’m sorry, I don’t remember your name. You are you?\\n']\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:08,369 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - preprocess data: [{'body': bytearray(b'Hi James, when are you coming back home? I am waiting for you.\\nPlease come as soon as possible.\\n')}, {'body': bytearray(b'I\\xe2\\x80\\x99m sorry, I don\\xe2\\x80\\x99t remember your name. You are you?\\n')}]\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:08,370 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - preprocessed data: ['Hi James, when are you coming back home? I am waiting for you.\\nPlease come as soon as possible.\\n', 'I’m sorry, I don’t remember your name. You are you?\\n']\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:10,995 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - Model translated: ['Bonjour James, quand rentrerez-vous chez vous, je vous attends et je vous prie de venir le plus tôt possible.', \"Je vous prie de m'excuser, je ne me souviens pas de votre nom.\"]\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:10,995 [INFO ] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 2627\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:10,996 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_METRICS - HandlerTime.Milliseconds:2625.98|#ModelName:TransformerEn2Fr,Level:Model|#hostname:c166cabe72bf,requestID:70314e99-eff3-4e9e-b414-d5c455553fa7,1a6c1d6c-b123-446f-b2cb-f24c9cbb9f35,timestamp:1626299590\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:10,996 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_METRICS - PredictionTime.Milliseconds:2626.03|#ModelName:TransformerEn2Fr,Level:Model|#hostname:c166cabe72bf,requestID:70314e99-eff3-4e9e-b414-d5c455553fa7,1a6c1d6c-b123-446f-b2cb-f24c9cbb9f35,timestamp:1626299590\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:10,996 [INFO ] W-9000-TransformerEn2Fr_1.0 ACCESS_LOG - /169.254.255.130:41544 \"POST /invocations HTTP/1.1\" 200 3129\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:10,996 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - Requests2XX.Count:1|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:10,997 [DEBUG] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.job.Job - Waiting time ns: 500693981, Backend time ns: 2629150981\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:10,997 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - QueueTime.ms:500|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:10,997 [INFO ] W-9000-TransformerEn2Fr_1.0 ACCESS_LOG - /169.254.255.130:41540 \"POST /invocations HTTP/1.1\" 200 3130\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:10,997 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - Requests2XX.Count:1|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:10,997 [DEBUG] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.job.Job - Waiting time ns: 500358482, Backend time ns: 2630042648\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:10,995 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - Model translated: ['Bonjour James, quand rentrerez-vous chez vous, je vous attends et je vous prie de venir le plus tôt possible.', \"Je vous prie de m'excuser, je ne me souviens pas de votre nom.\"]\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:10,995 [INFO ] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 2627\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:10,996 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_METRICS - HandlerTime.Milliseconds:2625.98|#ModelName:TransformerEn2Fr,Level:Model|#hostname:c166cabe72bf,requestID:70314e99-eff3-4e9e-b414-d5c455553fa7,1a6c1d6c-b123-446f-b2cb-f24c9cbb9f35,timestamp:1626299590\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:10,996 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_METRICS - PredictionTime.Milliseconds:2626.03|#ModelName:TransformerEn2Fr,Level:Model|#hostname:c166cabe72bf,requestID:70314e99-eff3-4e9e-b414-d5c455553fa7,1a6c1d6c-b123-446f-b2cb-f24c9cbb9f35,timestamp:1626299590\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:10,996 [INFO ] W-9000-TransformerEn2Fr_1.0 ACCESS_LOG - /169.254.255.130:41544 \"POST /invocations HTTP/1.1\" 200 3129\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:10,996 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - Requests2XX.Count:1|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:10,997 [DEBUG] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.job.Job - Waiting time ns: 500693981, Backend time ns: 2629150981\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:10,997 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - QueueTime.ms:500|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:10,997 [INFO ] W-9000-TransformerEn2Fr_1.0 ACCESS_LOG - /169.254.255.130:41540 \"POST /invocations HTTP/1.1\" 200 3130\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:10,997 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - Requests2XX.Count:1|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:10,997 [DEBUG] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.job.Job - Waiting time ns: 500358482, Backend time ns: 2630042648\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:10,998 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - QueueTime.ms:500|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:10,998 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - WorkerThreadTime.ms:4|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:11,505 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - preprocess data: [{'body': bytearray(b'Hello world\\nGood morning\\nGood evening\\nThis is a dog\\nThis is a cat\\nGo swimming\\nGo to Paris\\nThank you\\n')}, {'body': bytearray(b'I\\xe2\\x80\\x99m well. How are you?\\nIt\\xe2\\x80\\x99s going well, thank you. How are you doing?\\nFine, thanks. And yourself?\\n')}]\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:11,505 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - preprocessed data: ['Hello world\\nGood morning\\nGood evening\\nThis is a dog\\nThis is a cat\\nGo swimming\\nGo to Paris\\nThank you\\n', 'I’m well. How are you?\\nIt’s going well, thank you. How are you doing?\\nFine, thanks. And yourself?\\n']\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:10,998 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - QueueTime.ms:500|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:10,998 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - WorkerThreadTime.ms:4|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:11,505 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - preprocess data: [{'body': bytearray(b'Hello world\\nGood morning\\nGood evening\\nThis is a dog\\nThis is a cat\\nGo swimming\\nGo to Paris\\nThank you\\n')}, {'body': bytearray(b'I\\xe2\\x80\\x99m well. How are you?\\nIt\\xe2\\x80\\x99s going well, thank you. How are you doing?\\nFine, thanks. And yourself?\\n')}]\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:11,505 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - preprocessed data: ['Hello world\\nGood morning\\nGood evening\\nThis is a dog\\nThis is a cat\\nGo swimming\\nGo to Paris\\nThank you\\n', 'I’m well. How are you?\\nIt’s going well, thank you. How are you doing?\\nFine, thanks. And yourself?\\n']\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:14,255 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - Model translated: [\"Bonjour Monde Bonjour Bonsoir C'est un chien C'est un chat Allez nager Allez à Paris Merci\", 'Je me sens bien. Comment allez-vous ? Ça va bien, merci. Comment allez-vous ?']\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:14,255 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_METRICS - HandlerTime.Milliseconds:2750.51|#ModelName:TransformerEn2Fr,Level:Model|#hostname:c166cabe72bf,requestID:f7a442f1-21c7-473e-8fb7-272c6742540f,810fe8e0-e625-4d97-afed-034bfcd07c74,timestamp:1626299594\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:14,256 [INFO ] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 2752\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:14,256 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_METRICS - PredictionTime.Milliseconds:2750.57|#ModelName:TransformerEn2Fr,Level:Model|#hostname:c166cabe72bf,requestID:f7a442f1-21c7-473e-8fb7-272c6742540f,810fe8e0-e625-4d97-afed-034bfcd07c74,timestamp:1626299594\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:14,256 [INFO ] W-9000-TransformerEn2Fr_1.0 ACCESS_LOG - /169.254.255.130:41540 \"POST /invocations HTTP/1.1\" 200 3253\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:14,256 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - Requests2XX.Count:1|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:14,257 [DEBUG] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.job.Job - Waiting time ns: 500307809, Backend time ns: 2753299510\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:14,257 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - QueueTime.ms:500|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:14,257 [INFO ] W-9000-TransformerEn2Fr_1.0 ACCESS_LOG - /169.254.255.130:41544 \"POST /invocations HTTP/1.1\" 200 3252\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:14,257 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - Requests2XX.Count:1|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:14,257 [DEBUG] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.job.Job - Waiting time ns: 498379862, Backend time ns: 2754086106\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:14,258 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - QueueTime.ms:498|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[34m2021-07-14 21:53:14,258 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - WorkerThreadTime.ms:3|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:14,255 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_LOG - Model translated: [\"Bonjour Monde Bonjour Bonsoir C'est un chien C'est un chat Allez nager Allez à Paris Merci\", 'Je me sens bien. Comment allez-vous ? Ça va bien, merci. Comment allez-vous ?']\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:14,255 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_METRICS - HandlerTime.Milliseconds:2750.51|#ModelName:TransformerEn2Fr,Level:Model|#hostname:c166cabe72bf,requestID:f7a442f1-21c7-473e-8fb7-272c6742540f,810fe8e0-e625-4d97-afed-034bfcd07c74,timestamp:1626299594\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:14,256 [INFO ] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 2752\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:14,256 [INFO ] W-9000-TransformerEn2Fr_1.0-stdout MODEL_METRICS - PredictionTime.Milliseconds:2750.57|#ModelName:TransformerEn2Fr,Level:Model|#hostname:c166cabe72bf,requestID:f7a442f1-21c7-473e-8fb7-272c6742540f,810fe8e0-e625-4d97-afed-034bfcd07c74,timestamp:1626299594\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:14,256 [INFO ] W-9000-TransformerEn2Fr_1.0 ACCESS_LOG - /169.254.255.130:41540 \"POST /invocations HTTP/1.1\" 200 3253\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:14,256 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - Requests2XX.Count:1|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:14,257 [DEBUG] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.job.Job - Waiting time ns: 500307809, Backend time ns: 2753299510\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:14,257 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - QueueTime.ms:500|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:14,257 [INFO ] W-9000-TransformerEn2Fr_1.0 ACCESS_LOG - /169.254.255.130:41544 \"POST /invocations HTTP/1.1\" 200 3252\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:14,257 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - Requests2XX.Count:1|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:14,257 [DEBUG] W-9000-TransformerEn2Fr_1.0 org.pytorch.serve.job.Job - Waiting time ns: 498379862, Backend time ns: 2754086106\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:14,258 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - QueueTime.ms:498|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n",
      "\u001b[35m2021-07-14 21:53:14,258 [INFO ] W-9000-TransformerEn2Fr_1.0 TS_METRICS - WorkerThreadTime.ms:3|#Level:Host|#hostname:c166cabe72bf,timestamp:null\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "transformer.transform(data=transform_input)\n",
    "transformer.wait()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s3://sagemaker-us-east-2-057122759684/TransformerEn2Fr/batch_transform_torchserve_sagemaker_output/\n"
     ]
    }
   ],
   "source": [
    "print(transformer.output_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "download: s3://sagemaker-us-east-2-057122759684/TransformerEn2Fr/batch_transform_torchserve_sagemaker_output/sample2.txt.out to ./sample2.txt.out\n",
      "download: s3://sagemaker-us-east-2-057122759684/TransformerEn2Fr/batch_transform_torchserve_sagemaker_output/sample.txt.out to ./sample.txt.out\n",
      "download: s3://sagemaker-us-east-2-057122759684/TransformerEn2Fr/batch_transform_torchserve_sagemaker_output/sample5.txt.out to ./sample5.txt.out\n",
      "download: s3://sagemaker-us-east-2-057122759684/TransformerEn2Fr/batch_transform_torchserve_sagemaker_output/sample3.txt.out to ./sample3.txt.out\n",
      "download: s3://sagemaker-us-east-2-057122759684/TransformerEn2Fr/batch_transform_torchserve_sagemaker_output/sample1.txt.out to ./sample1.txt.out\n",
      "download: s3://sagemaker-us-east-2-057122759684/TransformerEn2Fr/batch_transform_torchserve_sagemaker_output/sample4.txt.out to ./sample4.txt.out\n"
     ]
    }
   ],
   "source": [
    "!aws s3 cp --recursive $transformer.output_path ./"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==> sample1.txt.out <==\r\n",
      "{\"input\": \"Hello World !!!\\n\", \"french_output\": \"Bonjour le monde ! ! !\"}\r\n",
      "==> sample2.txt.out <==\r\n",
      "{\"input\": \"Hi James, when are you coming back home? I am waiting for you.\\nPlease come as soon as possible.\\n\", \"french_output\": \"Bonjour James, quand rentrerez-vous chez vous, je vous attends et je vous prie de venir le plus t\\u00f4t possible.\"}\r\n",
      "==> sample3.txt.out <==\r\n",
      "{\"input\": \"I\\u2019m sorry, I don\\u2019t remember your name. You are you?\\n\", \"french_output\": \"Je vous prie de m'excuser, je ne me souviens pas de votre nom.\"}\r\n",
      "==> sample4.txt.out <==\r\n",
      "{\"input\": \"I\\u2019m well. How are you?\\nIt\\u2019s going well, thank you. How are you doing?\\nFine, thanks. And yourself?\\n\", \"french_output\": \"Je me sens bien. Comment allez-vous ? \\u00c7a va bien, merci. Comment allez-vous ?\"}\r\n",
      "==> sample5.txt.out <==\r\n",
      "{\"input\": \"Hello world\\nGood morning\\nGood evening\\nThis is a dog\\nThis is a cat\\nGo swimming\\nGo to Paris\\nThank you\\n\", \"french_output\": \"Bonjour Monde Bonjour Bonsoir C'est un chien C'est un chat Allez nager Allez \\u00e0 Paris Merci\"}\r\n",
      "==> sample.txt.out <==\r\n",
      "{\"input\": \"Hi James, when are you coming back home? I am waiting for you.\\nPlease come as soon as possible.\", \"french_output\": \"Bonjour James, quand rentrerez-vous chez vous, je vous attends et je vous prie de venir le plus t\\u00f4t possible.\"}"
     ]
    }
   ],
   "source": [
    "!head sample*.txt.out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}