OpenID
Log In | Register
Skip to main content
Topic: The subprocess ended prematurely because it received the “KILL” (9) signal.  (Read 8869 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.
The subprocess ended prematurely because it received the “KILL” (9) signal.
Hello,

Thanks for this wonderful script.   Recently i've been having an issue with a larger account.  The strange thing is the backup completes alright - I use SFTP site and the file gets transferred okay..

But I received an alert that one account was out of disk space..  I noticed a large backup file that was sitting in the home directory...  and the user was out of disk space.  So i increased the disk space but the problem still happens.  I noticed this in the generated backup email.. below:

The backup file is named “backup-1.23.2020_01-00-02_xxxxxxx.tar.gz”
The backup transferred via “FTP” to the server “xxxxxxxx.canadacentral.cloudapp.azure.com” as user “xxxxxxxxxx” in the “/” directory.
The backup process encountered the following error:
The subprocess ended prematurely because it received the “KILL” (9) signal.

When i examine the log file.. the last line instead of saying the file was transferred via ftp successfully.. it says
The subprocess ended prematurely because it received the “KILL” (9) signal with no ftp tranfer successfull message.

But strangely enough - the file appears to be transferred to the ftp site and it does say it was transferred in the email..     But because it doesn't think it has been transferred it wont remove the file from the account and disk space is getting chewed up.. that or the clean up script is failing - if it deletes the file does it permanently delete the file or put into a recycleing bin.. Maybe its not able to delete the file because of a space issue (if it's not permanently deleting the file).  If i delete the file manually it asks if i want to permanently delete the file i always choose yes.

So for now i've been manually remove the backup file - so there is no inturrption in email flow to the account (so it doesn't run out of disk space) .. but any idea what could be the issue?

I've check my ftp server - there is no limit on data transfer that i can see I'm using filezilla server - i see a file sitting there at 12GB's.. and its been growing.. after backups.

Is there anything in the script that would have a limit or ftp transfer?  I guess my next step would be to enable logging on filezilla.. i've yet to get the host involved; as the backup runs fine; i'm not sure it has anything to do with them at this point.

Anyone seen this error before - or have any idea what the problem might be.  At first i thought its the cleanup script? or a memory issue? but now i'm leaning towards an ftp transfer issue; although the file looks intact on the other side.. from what i can tell.

Re: The subprocess ended prematurely because it received the “KILL” (9) signal.
Reply #1 –
I'm just going to post some updates here in the hopes to help someone else with this problem.

After examining the email I receive from the problem account - the subject and the first line say..  "A full backup has completed, but the upload has failed. "

The backup does complete though - the file is there on the ftp server and intact.... 

So i did a little research with my FTP server - it appears that connection timeout setting on the server may come into play here.   My guess is the client (whm/cpanel server in this case)  is not giving a reply because the connection was closed (as it takes awhile for the large file transfer).  Cpanel assumes the transfer failed.   So i checked the connection timeout setting; it was set to 120 seconds on the server by default.   I've increased this to 1 hour (could disable it entirely) although I figure an hour is enough for me the transfer only takes a few mins.    This may not help though because firewalls or routers could force the connection to close earlier.  There are keep alive settings but i dont see this option on the server side.     

I'm hoping this will resolve the problem; i will try and post back if it does.

The biggest problem is that cpanel will store the file locally in a shared server/reseller environment (with no way to configure this); in the clients account if it fails.. so its going to run out of disk space.  If disk space runs out then the client wont recieve emails and this is a big problem    It's expensive to just double up all users disks just because of backups (incase one fails).  It'd be nice if there was a n on or off switch for this; or if it could be scripted.. but seems like this is a cpanel thing not configurable or if it is; it's not available to resellers...

Seems a tad dangerous to run this way.. could use rsync alternatively i guess.. and just do file level backups from another server.. But this starts to defeats the purpose i guess - as you may as well just run vps at this point.. 

i will post any updates - hopefully this resolves the issue.
Re: The subprocess ended prematurely because it received the “KILL” (9) signal.
Reply #2 –
Quote from: rob_van on
So i did a little research with my FTP server - it appears that connection timeout setting on the server may come into play here.   My guess is the client (whm/cpanel server in this case)  is not giving a reply because the connection was closed (as it takes awhile for the large file transfer).

sorry i mean - the client is given a reply from the FTP server

i'll post any updates to see if the connection time out fixes the problem
Re: The subprocess ended prematurely because it received the “KILL” (9) signal.
Reply #3 –
Quote from: rob_van on
Quote from: rob_van on
So i did a little research with my FTP server - it appears that connection timeout setting on the server may come into play here.   My guess is the client (whm/cpanel server in this case)  is not giving a reply because the connection was closed (as it takes awhile for the large file transfer).

sorry i mean - the client is given a reply from the FTP server

i'll post any updates to see if the connection time out fixes the problem

err..  no reply is given to the client rather.  Sorry not sure if i can edit a post.. :)  or delete them.. 

Hopefully this fixes.. will update tomorrow
Re: The subprocess ended prematurely because it received the “KILL” (9) signal.
Reply #4 –
Quote from: rob_van on
Quote from: rob_van on
Quote from: rob_van on
So i did a little research with my FTP server - it appears that connection timeout setting on the server may come into play here.   My guess is the client (whm/cpanel server in this case)  is not giving a reply because the connection was closed (as it takes awhile for the large file transfer).

sorry i mean - the client is given a reply from the FTP server

i'll post any updates to see if the connection time out fixes the problem

err..  no reply is given to the client rather.  Sorry not sure if i can edit a post.. :)  or delete them.. 

Hopefully this fixes.. will update tomorrow

unfortunately this did not resolve the issue..  still getting upload failed..  i will continue to investigate.. if anyone has any ideas please let me know..
Re: The subprocess ended prematurely because it received the “KILL” (9) signal.
Reply #5 –
I enabled logging on my FTP server to see if that would shed some light.

I noticed in the log that because this backup is much larger than the rest of them.. the next backup starts before this one completes.  I had them spaced out 10 mins apart (the one cron job is set to run every 10 mins).   Normally they complete within a min or two including ftp upload.  But this one takes 13 mins to complete from start to finish.   It takes 6 mins to build the backup then another 7mins to transfer and the next one is already running by the time this one starts.

I also noticed that the upload does complete but there is no goodbye message just shows it completed then disconnected.  The other backups show a goodbye message.  So i'm hoping this is because it has something to do with the next backup already starting from the same server.  

Would be nice if you could disable keeping the backup file in the home directory weather it thinks the backup successfully transferred or not.  If anyone knows if this could be scripted; let me know as i have no idea.  But I can see problems for others if an ftp backup transfer gets interrupted or something happens.. most likely it would be more important to discard the backup then to clog up the home directory.

So for now i've adjusted the backups to run every 30 mins instead of 10.  Giving large backup plenty of time to complete.   I will report back if adjusting the backup period resolves the issue.  I'm hoping that it does. 
Re: The subprocess ended prematurely because it received the “KILL” (9) signal.
Reply #6 –
Quote from: rob_van on
I enabled logging on my FTP server to see if that would shed some light.

I noticed in the log that because this backup is much larger than the rest of them.. the next backup starts before this one completes.  I had them spaced out 10 mins apart (the one cron job is set to run every 10 mins).   Normally they complete within a min or two including ftp upload.  But this one takes 13 mins to complete from start to finish.   It takes 6 mins to build the backup then another 7mins to transfer and the next one is already running by the time this one starts.

I also noticed that the upload does complete but there is no goodbye message just shows it completed then disconnected.  The other backups show a goodbye message.  So i'm hoping this is because it has something to do with the next backup already starting from the same server.  

Would be nice if you could disable keeping the backup file in the home directory weather it thinks the backup successfully transferred or not.  If anyone knows if this could be scripted; let me know as i have no idea.  But I can see problems for others if an ftp backup transfer gets interrupted or something happens.. most likely it would be more important to discard the backup then to clog up the home directory.

So for now i've adjusted the backups to run every 30 mins instead of 10.  Giving large backup plenty of time to complete.   I will report back if adjusting the backup period resolves the issue.  I'm hoping that it does.

It's still happening..  so this has nothing to do with it.    I'm sure the whm/cpanel is not getting the successful transfer backup message so after much more searching..   I found that azure closes there connections after 4 mins.  They rely on keep alive.  With no way to support keepalive on a reseller server - I was able to find a way to change this setting - I used the max of 30 mins.

https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-tcp-idle-timeout#set-the-tcp-timeout-on-a-load-balanced-rule-to-15-minutes

I will post back if this resolve the problem
Re: The subprocess ended prematurely because it received the “KILL” (9) signal.
Reply #7 –
Quote from: rob_van on

It's still happening..  so this has nothing to do with it.    I'm sure the whm/cpanel is not getting the successful transfer backup message so after much more searching..   I found that azure closes there connections after 4 mins.  They rely on keep alive.  With no way to support keepalive on a reseller server - I was able to find a way to change this setting - I used the max of 30 mins.

https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-tcp-idle-timeout#set-the-tcp-timeout-on-a-load-balanced-rule-to-15-minutes

I will post back if this resolve the problem

Okay this did infact solve the problem.  Hope this helps someone else..   So the problem was with my router (azure) time out setting on the ftp server side.  But upon closer examination.. I can see the backup file for each account is dumped into the users home directory while its being transferred (or at least that is how it configured on my reseller host).  This may pose a problem if the account is more then 3/4 of the way full.  As the compressed backup may push the account out of space.  I assumed that a temp file was written somewhere but apparently not.  I guess I could just write a script that removes the backups daily.  Actually its' not a bad idea to run it just incase the FTP transfer fails and the backup is not removed (the account could then run out of space). 

I guess the take away or warning to anyone else using this method of backup - Make sure you have enough space in the home dir to cover the backup while it's running or emails or uploads to the account are going to fail. 
Re: The subprocess ended prematurely because it received the “KILL” (9) signal.
Reply #8 –
Hi,

I'm glad you were able to identify and resolve the issue.
Unfortunately the issue with regards to requiring home directory storage space is something that i've looked at several times with no workaround possible. The only thing I could do is build in storage usage indication into the script. Which i'm currently looking into.