Improve error logging for incremental backups
Would like to have separate logging of errors for incremental backups in the new backup system.
The code for this already exists - well partially - in the new backup script(via the parse_log_and_notify function), to send errors and notification emails when a backup run completes. However, this function currently is limited to only the first error and is not a good indicator of the full issue. (see bug 91769) Also, relying on email when backing up several hundred servers is not always a viable option.
I would be great if any failures/errors could just be added to the existing file backup_meta_data for each account. The addition of this new file was great to see and can be used for much more, however, at this time that file is updating even when an account fails to backup.(see bug 91773)
Alternatively, all errors could be parsed out to one log within the backup target directory.
If all of these issues were addressed and some form of logging existed for errors only; one could quickly determining the following using common languages like bash/perl/php:
-If all incremental account backups completed successfully
-if incremental backups ran in the requested time frame
-If an incremental backup exists for all accounts
These logs could also be used to display a warning via WHM, such as is done when upcp fails, but that is a whole other request.
I like this idea 
cPanel, WebHost Manager and WHM are registered trademarks of cPanel, L.L.C. for providing its computer software that facilitates the management and configuration of Internet web servers.
While it is unlikely that we would implement a verbose logging system that has an expectation of sysadmins custom writing their own bash/perl/php scripts to parse them and make sense of them, log clarity in general is a worthy goal.
I'd like to focus on recommendations that you, and others, have in relation to improving error/log clarity with regards to backups in general. Like I mentioned, though, they should be complete solutions and not something that stops short and expects custom scripts to handle them to make sense of them.
Additionally, logs are great for deep investigation -- but I can't imagine most sysadmins routinely logging in and reviewing verbose logs to determine the success of their backups.
Some form of automated notification would be necessary, such as the existing email notifications.
It sounds like you personally would like to have some way to retrieve this type of failure/success state information on-demand, however -- such as through an API call.
Would it be reasonable to say that your feature request involves two major aspects?
[1] Increase verbosity of errors (Citing Case 91769 as cause for when they can be misleading)
[2] In *addition* to 1, provide a means of exposing this information such as through an API.
If so, please elaborate a bit further on examples/use cases of how you see those notifications being handled or how you would plan on using such APIs.
I encourage further input from others on this as well so we can try and see specifically how individuals would want to/plan to use this data.
While it is unlikely that we would implement a verbose logging system that has an expectation of sysadmins custom writing their own bash/perl/php scripts to parse them and make sense of them, log clarity in general is a worthy goal.
I'd like to focus on recommendations that you, and others, have in relation to improving error/log clarity with regards to backups in general. Like I mentioned, though, they should be complete solutions and not something that stops short and expects custom scripts to handle them to make sense of them.
Additionally, logs are great for deep investigation -- but I can't imagine most sysadmins routinely logging in and reviewing verbose logs to determine the success of their backups.
Some form of automated notification would be necessary, such as the existing email notifications.
It sounds like you personally would like to have some way to retrieve this type of failure/success state information on-demand, however -- such as through an API call.
Would it be reasonable to say that your feature request involves two major aspects?
[1] Increase verbosity of errors (Citing Case 91769 as cause for when they can be misleading)
[2] In *addition* to 1, provide a means of exposing this information such as through an API.
If so, please elaborate a bit further on examples/use cases of how you see those notifications being handled or how you would plan on using such APIs.
I encourage further input from others on this as well so we can try and see specifically how individuals would want to/plan to use this data.
While it is unlikely that we would implement a verbose logging system that has an expectation of sysadmins custom writing their own bash/perl/php scripts to parse them and make sense of them, log clarity in general is a worthy goal.
I'd like to focus on recommendations that you, and others, have in relation to improving error/log clarity with regards to backups in general. Like I mentioned, though, they should be complete solutions and not something that stops short and expects custom scripts to handle them to make sense of them.
Additionally, logs are great for deep investigation -- but I can't imagine most sysadmins routinely logging in and reviewing verbose logs to determine the success of their backups.
Some form of automated notification would be necessary, such as the existing email notifications.
It sounds like you personally would like to have some way to retrieve this type of failure/success state information on-demand, however -- such as through an API call.
Would it be reasonable to say that your feature request involves two major aspects?
[1] Increase verbosity of errors (Citing Case 91769 as cause for when they can be misleading)
[2] In *addition* to 1, provide a means of exposing this information such as through an API.
If so, please elaborate a bit further on examples/use cases of how you see those notifications being handled or how you would plan on using such APIs.
I encourage further input from others on this as well so we can try and see specifically how individuals would want to/plan to use this data.
While it is unlikely that we would implement a verbose logging system that has an expectation of sysadmins custom writing their own bash/perl/php scripts to parse them and make sense of them, log clarity in general is a worthy goal.
I'd like to focus on recommendations that you, and others, have in relation to improving error/log clarity with regards to backups in general. Like I mentioned, though, they should be complete solutions and not something that stops short and expects custom scripts to handle them to make sense of them.
Additionally, logs are great for deep investigation -- but I can't imagine most sysadmins routinely logging in and reviewing verbose logs to determine the success of their backups.
Some form of automated notification would be necessary, such as the existing email notifications.
It sounds like you personally would like to have some way to retrieve this type of failure/success state information on-demand, however -- such as through an API call.
Would it be reasonable to say that your feature request involves two major aspects?
[1] Increase verbosity of errors (Citing Case 91769 as cause for when they can be misleading)
[2] In *addition* to 1, provide a means of exposing this information such as through an API.
If so, please elaborate a bit further on examples/use cases of how you see those notifications being handled or how you would plan on using such APIs.
I encourage further input from others on this as well so we can try and see specifically how individuals would want to/plan to use this data.
In all fairness, I would not expect that many admins would want to parse such logs, or have to routinely check them manually, but I am sure many currently are.There really is no other choice at this time due to the lack of a centralized location or method within cPanel to review backup errors. I do agree that some form of notification is required, but email is only worthwhile when you are dealing with a few servers.
>"Would it be reasonable to say that your feature request involves two major aspects?
[1] Increase verbosity of errors (Citing Case 91769 as cause for when they can be misleading)
[2] In *addition* to 1, provide a means of exposing this information such as through an API.
"
Correct, although I don't personally see the need for the API, however, I can see how the API could be beneficial to many. I would be happy with a simple file such as the one used for upcp issues (/var/cpanel/update_blocks.config). I realize that cPanel didn't create files like this to be used for anything other than their intended purpose, but it is a resource none the less. There are many files in cPanel that are as simple as this one that can be used to determine configuration, status, or failure issues - without logging into WHM of reviewing a full log.
As for an example of use, we currently maintain an externally centralized system to monitor the following:
-If all account backups completed successfully
-if backups ran in the requested time frame
-If a backup exists for all accounts
We have been doing this for years using a modified version of cpbackup that writes out the relevant information/return values to logs in our backup directory, which is then feed to our monitoring system. This need to modify cpbackup is what I would like to move away from but the aforementioned bugs, lack of verbosity, and decentralized logging in the new system make this a daunting task.
So with our current setup for backups and while unrelated upcp errors, I know within minutes of issues. All without having to read a few hundred emails or log into a few hundred servers.
While this setup may beyond the standard use for some, the basis of this request would benefit many.
In an effort to summarize all of this rambling, either of the following options would be acceptable.
[1] Create a single file in the target backup directory after a backup run, in this case /incremental/, that is populated with any errors only. This file would be overwritten upon each new run with errors or removed if no errors are found. The existence of this file would then be used to notify of an issue upon login to WHM.
[2] Some sort of API
In all fairness, I would not expect that many admins would want to parse such logs, or have to routinely check them manually, but I am sure many currently are.There really is no other choice at this time due to the lack of a centralized location or method within cPanel to review backup errors. I do agree that some form of notification is required, but email is only worthwhile when you are dealing with a few servers.
>"Would it be reasonable to say that your feature request involves two major aspects?
[1] Increase verbosity of errors (Citing Case 91769 as cause for when they can be misleading)
[2] In *addition* to 1, provide a means of exposing this information such as through an API.
"
Correct, although I don't personally see the need for the API, however, I can see how the API could be beneficial to many. I would be happy with a simple file such as the one used for upcp issues (/var/cpanel/update_blocks.config). I realize that cPanel didn't create files like this to be used for anything other than their intended purpose, but it is a resource none the less. There are many files in cPanel that are as simple as this one that can be used to determine configuration, status, or failure issues - without logging into WHM of reviewing a full log.
As for an example of use, we currently maintain an externally centralized system to monitor the following:
-If all account backups completed successfully
-if backups ran in the requested time frame
-If a backup exists for all accounts
We have been doing this for years using a modified version of cpbackup that writes out the relevant information/return values to logs in our backup directory, which is then feed to our monitoring system. This need to modify cpbackup is what I would like to move away from but the aforementioned bugs, lack of verbosity, and decentralized logging in the new system make this a daunting task.
So with our current setup for backups and while unrelated upcp errors, I know within minutes of issues. All without having to read a few hundred emails or log into a few hundred servers.
While this setup may beyond the standard use for some, the basis of this request would benefit many.
In an effort to summarize all of this rambling, either of the following options would be acceptable.
[1] Create a single file in the target backup directory after a backup run, in this case /incremental/, that is populated with any errors only. This file would be overwritten upon each new run with errors or removed if no errors are found. The existence of this file would then be used to notify of an issue upon login to WHM.
[2] Some sort of API
Replies have been locked on this page!