Hi, for inference I created a deployment on AWS which is working fine when used for summarization It works fine on some texts, but as soon as I use a particular long text I get an error and I am not able to run any summarization taks again, even the ones with short input texts are no longer workin…

Were you able to resolve this ? This seems problamatic for production deployments. Running into a similar issue myself.

hello Aarya, please check the link sent by Philipp above, with two alternative solution!

Sagemaker HuggingFaceModel crashed with CUDA error

philschmid May 17, 2023, 12:35pm 2

Topic		Replies	Views
Hugging Face BART Text Summarization Model-Sage Maker deployed after a month giving me an 400 error Models	1	600	May 2, 2023
CUDA error for inference on GPU instance Amazon SageMaker	2	817	May 16, 2023
Inference failed for FLAN-UL2(20B) on SageMaker Amazon SageMaker	6	2237	April 4, 2023
Calling Sagemaker Endpoint for fine-tuned summarization model Amazon SageMaker	15	5190	March 22, 2024
Inference error for FLAN-UL2 on AWS SageMaker Amazon SageMaker	1	992	April 3, 2023