Last week's tip presented you with a hypothetical data collection and processing requirement, and asked you to think about how it could be optimized for a large server environment using co-processes.

As mentioned last week, collecting a significant amount of data from each server in an environment containing many servers in a serial fashion (one after another) may not be an acceptable option, especially if the collecting and subsequent processing needs to be performed within a limited amount of time.  Running multiple co-processes from a parent script on the "master" system, with each co-process performing the data collection for one server in the environment, would be one possible option.

The most outer loop of the parent script could be a "for" or "while" loop, and would be used to cycle through each server (host) in the environment.  Within this loop, you would spawn a co-process for the server and bump your co-process counter.

This outer loop would also contain an inner loop used to read from each outstanding co-process when your co-process counter was equal to the max co-process variable.  After each outstanding co-process had been accounted for, the co-process counter would be re-initialized and this inner "harvesting" loop would not run again until the counter becomes equal to the max co-process variable again.

Once a co-process is started for the last host in the environment, code would need to be executed to account for any outstanding co-processes.  At this point the data could be consolidated/processed on the master system as needed.  Notice that the max co-process variable could be tweaked based on how much of a load you want to put on the master server, and how quickly you want the parent script to execute.

This is obviously a very high-level view of one potential solution, but it will hopefully get you thinking about how co-processes can be creatively used to enhance your shell scripts.