uefi-ref/edk2
1
0
Fork 0
mirror of https://github.com/tianocore/edk2 synced 2026-08-27 00:23:19 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions

BaseTools: Cap AutoGen thread count to avoid file descriptor exhaustion

Browse source 
When the number of build threads multiplied by per-thread file
descriptor usage exceeds the system's open file descriptor limit,
some threads may fail to acquire necessary resources (e.g., pipes
or semaphores), leading to deadlocks or hangs during parallel builds.

To prevent this situation, calculate the safety upper limit of
concurrency by dividing the system's maximum file descriptor limit by
3 (An empirical value derived from balancing performance overhead
against the theoretical number of file descriptors consumed per thread).
The actual thread count is then clamped to this safe value.

Other usages of ThreadNum()—such as during actual compilation or log
queue creation—do not significantly contribute to file descriptor
consumption. Therefore, adjusting ThreadNum() globally would be
unwarranted, as it could unnecessarily restrict parallelism in stages
that are not FD-bound.

This ensures stable parallel builds even under constrained resource
limits.

Signed-off-by: Ayden Meng <mengxiangdong@loongson.cn>
This commit is contained in:
Ayden Meng 2025-11-27 20:57:59 +08:00 • committed by mergify[bot]
parent 7aaf742d2d
commit a058a2856d
1 changed files with 10 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

11
BaseTools/Source/Python/build/build.py
View file

@ -838,13 +838,22 @@ class Build():
try:
if SkipAutoGen:
return True,0
if sys.platform == "win32":
SafeThreadNumber = self.ThreadNumber
else:
import resource
soft = resource.getrlimit(resource.RLIMIT_NOFILE)[0]
SafeThreadNumber = min(self.ThreadNumber, soft // 3)
if SafeThreadNumber < self.ThreadNumber:
EdkLogger.verbose("AutoGen workers limited to %d to avoid file descriptor exhaustion" % SafeThreadNumber)
feedback_q = mp.Queue()
error_event = mp.Event()
FfsCmd = DataPipe.Get("FfsCommand")
if FfsCmd is None:
FfsCmd = {}
GlobalData.FfsCmd = FfsCmd
auto_workers = [AutoGenWorkerInProcess(mqueue,DataPipe.dump_file,feedback_q,GlobalData.file_lock,cqueue,self.log_q,error_event) for _ in range(self.ThreadNumber)]
auto_workers = [AutoGenWorkerInProcess(mqueue,DataPipe.dump_file,feedback_q,GlobalData.file_lock,cqueue,self.log_q,error_event) for _ in range(SafeThreadNumber)]
self.AutoGenMgr = AutoGenManager(auto_workers,feedback_q,error_event)
self.AutoGenMgr.start()
for w in auto_workers: