Logs for Auto-Scaling
When mlcp scales-out or scales-in, there will be a log message on INFO level to notify user about the scaling process. If the thread count has reached the maximum value, it will also be logged on INFO level. For every periodic polling, mlcp will log new available server threads on DEBUG level. If mlcp decides to scale-out or scale-in, the assigned or deducted threads for each input split will also be logged on DEBUG level.
The following messages are an example of common log messages a user may get in an auto-scaling process. Timestamps have been removed.
DEBUG contentpump.ThreadManager: Initial thread pool size: 32 DEBUG contentpump.ThreadManager: Thread pool will auto-scale based on available server threads. DEBUG contentpump.ThreadManager: Running with MultithreadedMapper. Initial thread count for split #0: 11 DEBUG contentpump.ThreadManager: Running with MultithreadedMapper. Initial thread count for split #1: 11 DEBUG contentpump.ThreadManager: Running with MultithreadedMapper. Initial thread count for split #2: 10 INFO contentpump.LocalJobRunner: completed 0% DEBUG contentpump.ThreadManager: New available server threads: 32 DEBUG contentpump.ThreadManager: New available server threads: 32 DEBUG contentpump.ThreadManager: New available server threads: 16 INFO contentpump.ThreadManager: Thread pool is scaling-in. New thread pool size: 16 DEBUG contentpump.ThreadManager: Running with MultithreadedMapper. New thread count for split #0: 6 DEBUG contentpump.ThreadManager: Running with MultithreadedMapper. New thread count for split #1: 5 DEBUG contentpump.ThreadManager: Running with MultithreadedMapper. New thread count for split #2: 5 DEBUG contentpump.ThreadManager: New available server threads: 16