I noticed that a task is executed one at a time. Compress, Verify, Upload, Post Event.
One thing that could be very helpful, is to pipeline them somewhat, or give the option to, if there are multiple tasks in the queue.
A very basic type of pipelining would be would be task granular level pipelining.
For example: task 1 is being uploaded to a server, while task 2 is verifying it's archive, while task 3 is compressing, while task 4 is gathering its files, etc. All at the same time. This is ideal if there's no contention.
There might be contention if the tasks are all backing up data on the same drive though, so at least pipelining uploads and compression tasks would be useful by itself. Since uploading could be slow, so while it's uploading, the system could be compressing/archiving something else while it's waiting to upload.
Thoughts?
One thing that could be very helpful, is to pipeline them somewhat, or give the option to, if there are multiple tasks in the queue.
A very basic type of pipelining would be would be task granular level pipelining.
For example: task 1 is being uploaded to a server, while task 2 is verifying it's archive, while task 3 is compressing, while task 4 is gathering its files, etc. All at the same time. This is ideal if there's no contention.
There might be contention if the tasks are all backing up data on the same drive though, so at least pipelining uploads and compression tasks would be useful by itself. Since uploading could be slow, so while it's uploading, the system could be compressing/archiving something else while it's waiting to upload.
Thoughts?
Statistics: Posted by SleepySpider — 03 Jan 2023, 00:02