IncompleteRead When Downloading From 🪣 AWS S3

aws s3
http
error
Published

June 20, 2024

Set keep alive connection to avoid IncompleteRead that is due to connection error

image.png

My challenge today was figuring out why my code could not run. I got IncompleteRead error when trying to load data from AWS S3. The file is only 400 MB. I tried multiple solutions from github issues to stackoverflow answers. The only workaround that worked for me was setting these in terminal:

sudo sysctl net.inet.tcp.keepintvl=200000
sudo sysctl net.inet.tcp.keepidle=200000
sudo sysctl net.inet.tcp.keepinit=200000
sudo sysctl net.inet.tcp.always_keepalive=1

And run again, and again. Now, my code doesn’t exit out anymore and I still don’t know why 😅

Some useful references:
1. https://docs.aws.amazon.com/redshift/latest/mgmt/connecting-firewall-guidance.html
2. https://github.com/boto/boto3/issues/2424