1.Add FFS file type check: Only allow rebase operation for
EFI_FV_FILETYPE_SECURITY_CORE, EFI_FV_FILETYPE_PEI_CORE,
EFI_FV_FILETYPE_DXE_CORE, EFI_FV_FILETYPE_PEIM, EFI_FV_FILETYPE_DRIVER,
EFI_FV_FILETYPE_COMBINED_PEIM_DRIVER,EFI_FV_FILETYPE_FIRMWARE_VOLUME_IMAGE
types, improving safety and compliance.
2.Automatically detect and complete the PE/COFF or TE image
relocation table (reloc section) to ensure integrity and compatibility
of the rebase operation.
3.After rebase, automatically update FFS checksum and FV header
information to ensure correct data structure.
4.Support recursive processing for nested FVs, ensuring all related FFS
files' PE/TE images are properly rebased and reloc tables are completed.
5.Use table-driven architecture for relocation types, making it easier to
extend to more platforms.
6.Improve error handling and logging for better robustness and
maintainability.
Please attention, only IA32 and X64 PE/COFF image are supported now.
For other Arch, will support it after testing.
Signed-off-by: Yuwei Chen <yuwei.chen@intel.com>
This patch adds and improves the rebase functionality
for firmware images (PE/COFF/TE) in the FMMT tool.
Key features include:
1.Automatically rebases PE/COFF/TE images within FFS files
when the firmware volume (FV) layout is adjusted or FFS files
are moved, ensuring correct loading and execution at new addresses.
2.Implements recursive rebase logic for nested sections,
guaranteeing all relevant images are properly relocated.
3.Adds support for rebasing subsequent FFS files within the same FV,
enhancing compatibility and stability during firmware layout changes.
4.Core code changes are mainly in FvHandler.py, BiosTreeNode.py,
and BinaryFactoryProduct.py, covering rebase flag detection, address calculation,
and actual relocation operations.
5.This feature improves the flexibility of firmware space management
and enhances the automation and reliability of the FMMT tool.
Co-Auther: Ashraf Ali S <ashraf.ali.s@intel.com>
Signed-off-by: Yuwei Chen <yuwei.chen@intel.com>
There is another issue in FvHander.py similar to the one fixed by
a60334ad59 ("BaseTools: Fix FMMT FvHandler Padding operation issue").
For a guided section (ParTree.Data.Type == 0x02), the length of
ParTree.Data.OriData is used to obtain the original size of the data even
after ParTree.Data.OriData has changed, which is incorrect.
This caused the following issue I observed. I built OVMF image for Aarch64
and then tried to add a couple FFS modules to it with 'FMMT -a [...]'. The
resulting image turned out to be invalid: the total size of the image was
less than the size of the firmware volume within it.
UEFITool failed to load such firmware image and complained:
"parseRawArea: one of objects inside overlaps the end of data".
This patch fixes the issue.
Signed-off-by: Evgenii Shatokhin <euspectre@gmail.com>
This commit is the first in a series of multiple commits to fix various
typos in the code, originating mostly from copy&paste over the years.
Most of them only affect documentation and not code.
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
The DXE_SAL_DRIVER module type was introduced to support
Itanium (IPF) platforms. Since support for Itanium processors has been
dropped, the instances of DXE_SAL_DRIVER have been removed.
Ref: [3cb0a311cb]
Cc: Sachin Ganesh <sachinganesh@ami.com>
Signed-off-by: Sathya Ravichandran <sathyar@ami.com>
- Str is not being used in FvHeader.py
- Removed import of ast.Str as it was removed in Python 3.14.
- Ensures compatibility with Python 3.14 and later.
- https://docs.python.org/3/whatsnew/3.14.html#id9
This addresses ImportError caused by removal of deprecated AST classes
including ast.Str.
Signed-off-by: Ashraf Ali S <ashraf.ali.s@intel.com>
Currently the FMMT tool supports CRC32 GUID section (Ref. GuidTools.py).
But it is found that there are errors for the FV with CRC32 section.
For example, the errors are checked on the following commands.
--v : Show FV information except for the FV
--a : Show "Target Fv not found!!!" when adding an FFS file to the FV
--d : Show "Target Ffs not found!!!" when deleting an FFS file in the FV
They are caused by the mismatch for CRC32 section data between the FMMT
and GenCrc32 tools. The FMMT tool returns CRC32 section data without CRC
checksum field (4 bytes). The GenCrc32 tool (with -d option, verify CRC32
value for the input file) requires CRC32 section data including CRC
checksum field (4 bytes).
Fix the issue through adjusting the section data to include CRC checksum
field. Currently DataOffset field for CRC32 GUID section is reported as
0x1C differently (GUID Section header length: 0x18 + Checksum field: 0x4).
Instead of DataOffset field that includes CRC checksum field, configure
the section data based on the offset from section header length (0x18)
that was previously calculated. This update enables GUID sections to use
the same offset consistently.
Signed-off-by: Phil Noh <Phil.Noh@amd.com>
Set BUILD_TIME_EPOCH to the current UTC timestamp if not already present
in the environment.
Use the resulting value to print the "Build start time:" message.
Signed-off-by: Leif Lindholm <leif.lindholm@oss.qualcomm.com>
The function CheckEnvVariable in fact checks several environment
variables. And the comment at its invocation enumerates a specific set of
variables, which defeats half the point of abstracting it out into a
helper function.
Rename the function to the plural form and turn the comment into a list
of examples.
Signed-off-by: Leif Lindholm <leif.lindholm@oss.qualcomm.com>
edk2 is dropping support for the ARM32 architecture. This
commit removes ARM32 code from BaseTools.
Signed-off-by: Oliver Smith-Denny <osde@microsoft.com>
Commit f0a2015373 ("UefiPayloadPkg: Add AARCH64 support") changed
`[Components.X64]` to `[Components.X64, Components.AARCH64]` which
resulted in the following code within that section to not work as
expected (the code wasn't there, just providing a real world example
that uncovered the issue):
[Components.X64, Components.AARCH64]
FmpDevicePkg/FmpDxe/FmpDxe.inf {
...
<PcdsFixedAtBuild>
!include .../...PcdFmpDevicePkcs7CertBufferXdr.inc
...
}
At the same time `[Components.X64]` or even `[Components.AARCH64,
Components.X64]` (notice the swapped order) worked fine for X64 target.
The cause of the issue turned out to be skipping includes inside
`_PostProcess()` method of `DscParser` class. This method processes
list of items stored in a database filled on the first pass through a
DSC file in `Start()` method. One of the fields stored in each row
of a table is link to a parent object (owner). A section like
`[Components.X64, Components.AARCH64]` creates two objects and all of
its subelements are duplicated for both X64 and AARCH64. This was not
happening for !include statement in the example above.
Because `_PostProcess()` contracted a sequence of !include objects
disregarding their owner, it did not create instance for each of the
requested targets. Codewise, `self._ContentIndex` was incremented more
than once, while `__ProcessDirective()` method (invoked indirectly as
`Processer[self._ItemType]()`) queried owner of the current directive
as:
if self._InSubsection:
Owner = self._Content[self._ContentIndex - 1][8]
else: # not taken in this case
This is why order of targets made a difference, only the last was fully
initialized in this case.
An alternative fix is completely dropping merging of !include
directives, but hard to say whether it still has some utility (the code
is complex, hard to follow and barely documented). Safer to keep it, in
the worst case it doesn't do anything now.
Signed-off-by: Sergii Dmytruk <sergii.dmytruk@3mdeb.com>
When using the FMMT FvHandler function, new padding size
should be calculated correctly comparing with origin ffs
and new ffs, else it will cause the binary size changes.
This patch is used to fix the bug.
Signed-off-by: Yuwei Chen <yuwei.chen@intel.com>
Build report would not detect a nested FV if the nested
FV was not in a subsection of an FFS statement.
Modify the build report to better handle some of the
variations of nested FVs.
Failing Example:
[Fv.FvName1]
INF <path to some driver>.inf
[Fv.FvName0]
FILE FV_IMAGE = B25ACDEF-39CE-4FA5-B50A-33E24DB1BDDF {
SECTION FV_IMAGE = FvName1
}
Working Example:
[Fv.FvName1]
INF <path to some driver>.inf
[Fv.FvName0]
FILE FV_IMAGE = DA04F6BF-A0FD-47EC-928B-5101A6C95026 {
SECTION GUIDED EE4E5898-3914-4259-9D6E-DC7BD79403CF
PROCESSING_REQUIRED = TRUE {
SECTION FV_IMAGE = FvName1
}
}
Signed-off-by: Aaron Pop <aaronpop@microsoft.com>
Running the vulture tool on the UPT/Xml folder gave the following
report. Remove the unnecessary code.
- UPT/Xml/CommonXml.py:585:
unused attribute 'LangDefsList' (60% confidence)
Signed-off-by: Pierre Gondois <pierre.gondois@arm.com>
Running the vulture tool on the UPT/PomAdapter folder gave the following
report. Remove the unnecessary code.
- UPT/PomAdapter/DecPomAlignment.py:898:
unused method 'ShowPackage' (60% confidence)
Signed-off-by: Pierre Gondois <pierre.gondois@arm.com>
Running the vulture tool on the Capsule folder gave the following
report. Remove the unnecessary code.
- Capsule/WindowsCapsuleSupportHelper.py:26:
unused method 'RegisterHelpers' (60% confidence)
Signed-off-by: Pierre Gondois <pierre.gondois@arm.com>
Running the vulture tool gave the following report.
Remove the unreachable code.
- TargetTool/TargetTool.py:49:
unreachable code after 'raise' (100% confidence)
- UPT/Library/UniClassObject.py:137:
unreachable code after 'return' (100% confidence)
- UPT/Object/Parser/InfDefineObject.py:795:
unreachable code after 'if' (100% confidence)
- Ecc/Check.py:1504:
unreachable code after 'return' (100% confidence)
Signed-off-by: Pierre Gondois <pierre.gondois@arm.com>
Adds support for building the C language BaseTools for Windows using
toolchains based on mingw-w64.
Mingw-w64 is a collection of header files, libraries, and tools that
when combined with a compiler enable development of Windows software.
Mingw-w64 is a fork of the original MinGW (Minimalist GNU for Windows).
Most active development on MinGW has ceased and mingw-w64 is now the
actively maintained successor. Mingw-w64 provides a libc implementation
built on top of Microsoft's UCRT (Universal C Runtime) with all
nessesary compiler bindings needed to support the C++11 feature set.
Modern mingw-w64 development appears to have coalesced around MSYS2,
which produces a distributions of both GCC and LLVM/Clang that use
mingw-w64 to target the Windows OS. This MSYS2 Clang distribution has a
UNIX-like directory layout and includes Windows binaries of GNU Make.
Combined with the open source licensing, MSYS2's Clang distribution is a
highly attractive choice as an alternative Windows SDK for open source
projects such as TianoCore.
If one wishes to use EDK II to build UEFI firmware on the Windows
platform, then the C BaseTools need to be compiled as Windows
applications. This includes the PcdValueInit.exe program, which needs
to be recompiled every time a firmware build is run in order to
regenerate the initial values for structured PCDs. Currently, BaseTools
only supports the Visual C++ toolchain on the Windows platform. The
following new features have been added to enable usage of the toolchains
derived from mingw-w64:
- Fixes to the BaseTools C source code to support the use of a
GCC-style compiler on the Windows OS.
- The GNU Make-style Makefiles for the C BaseTools have been modified
to support Windows. Both GCC + mingw-w64 and Clang + mingw-w64 have
been tested and confirmed to build a working BaseTools.
- BaseTools now supports generating GNU Make-style Makefiles on the
Windows platform for the purpose of building firmware.
- edksetup.bat has been modified to optionally build BaseTools via
mingw-w64. There is no impact to the existing support for Visual C++
and Visual C++ remains the default toolchain.
Usage Instructions:
For the vast majority of users, the only system setup change nessesary
to use a mingw-w64 toolchain is to set the BASETOOLS_MINGW_PATH to the
directory containing the desired mingw-w64 based toolchain.
A new command line argument has been added to edksetup.bat: Mingw-w64
If this command line argument is set, then the script will set the
BASETOOLS_MINGW_BUILD environment variable. The user can also opt to set
this environment variable manually before running edksetup.bat
If BASETOOLS_MINGW_BUILD is defined, then the BASETOOLS_MINGW_PATH
environment variable must point to the directory containing the
mingw-w64 toolchain.
If CLANG_BIN is not defined and %BASETOOLS_MINGW_PATH%\bin\clang.exe
exists, then edksetup.bat will set CLANG_BIN=%BASETOOLS_MINGW_PATH%\bin\
This removes the requirement to configure the CLANG_BIN environment
variable manually in order to run a CLANGPDB or CLANGDWARF build if one
has the MSYS2 Clang distribution installed. If one wishes to use a
different copy of Clang (for example official LLVM binaries) to build
firmware and only use the MSYS2 Clang to build BaseTools, then one can
continue to set the CLANG_BIN environment variable, same as before. I
have tested the MSYS2 Clang distribution against the official LLVM
distribution and can confirm that if the compiler version is the same
the emitted machine code is identical between the two. Interestingly,
the MSYS2 Clang distribution emits the path to the PDB file using "/" as
the path seperator instead of "\". That appears to be the only
difference in output. Therefore, using the MSYS2 Clang distribution to
compile firmware seems a reasonable choice.
If CLANG_HOST_BIN is not defined and BASETOOLS_MINGW_BUILD is defined
and %BASETOOLS_MINGW_PATH%\bin\mingw32-make.exe exists, then
edksetup.bat will add %BASETOOLS_MINGW_PATH%\bin\ to the PATH and set
CLANG_HOST_BIN=mingw32-
This enable usage of the GNU Make included in the mingw-w64 toolchain
to build firmware in addition to BaseTools. if BASETOOLS_MINGW_BUILD is
not defined, edksetup.bat will continue to set CLANG_HOST_BIN=n, which
uses nmake to build firmware. This behavior can be overridden by
manually setting the value of CLANG_HOST_BIN before executing
edksetup.bat if one wishes to use a specific Make utility for the
CLANGPDB/CLANGDWARF toolchains.
References:
- https://www.mingw-w64.org/
- https://www.msys2.org/
Co-authored-by: Sandesh Jain <sandesh.jain@intel.com>
Signed-off-by: Nate DeSimone <nathaniel.l.desimone@intel.com>
Add GUID section for build report.
Also, change the GUID format to string format to be easier to parse
Signed-off-by: Zhiguang Liu <zhiguang.liu@intel.com>
BaseTools currently does not expand macros for component architecture when
nested !include directives are used. This breaks use cases like
[Component.$(DXE_ARCH)].
The fix is to add explicit macro expansion when computing the Arch value
for component lines in DscParser.
Signed-off-by: Nate DeSimone <nathaniel.l.desimone@intel.com>
The following example fails to be parsed correctly due to Size
being used in the outer scope but initialized in the inner
scope
```
gPlatformPkgTokenSpaceGuid.PcdSecureBootDbxBinaryFile|{}
```
Problematic code:
```python
for Item in NewPcdValueList:
Size = 0
# ....
if Size > 0:
PcdValue = '{' + ', '.join(AllPcdValueList) + '}'
````
Signed-off-by: Doug Flick <dougflick@microsoft.com>
When using the replace function, if the replaced file
located in a compress section, then the tool will re-build
the section. As the compress status set in the wrong place,
in some situation, the compress will do twice times which is
a wrong behavior.
This patch is used to fix this issue.
Cc: Rebecca Cran <rebecca@bsdio.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Ashraf Ali S <ashraf.ali.s@intel.com>
Signed-off-by: Yuwei Chen <yuwei.chen@intel.com>
When a module M depends on L1, which depends on L2,
which depends on L3, the build fails when the library instance
of L3 cannot be found according to the library class-instance
mapping configuration specified in the DSC file.
When such failure happens, the build tool only prints that the
instance of L3 required by module M cannot be found. But it
does not tell how L3 is required by M.
The change enhances build tool to print the entire dependency
chain when such failure happens.
With the change, the new error message will be as follows:
<dsc-path>(...): error 4000: Instance of library class [L3] is not
found for module [M], [L3] is:
consumed by <instance of L2>
consumed by <instance of L1>
Signed-off-by: Ray Ni <ray.ni@intel.com>
If an FV_SPARE_SPACE_THRESHOLD is enabled and an FV
is 100% full with 0 bytes free, then this is likely
a special FV that may have alignment requirements
for the FFS file for both the start address and the
length and the FFS file consumes all the available
FV space.
Reduce FV_SPARE_SPACE_THRESHOLD from an error to a
warning if this FV 0 bytes free condition is
detected.
PR #10828 introduced the generation of these error
conditions for an FV with large alignment requirements.
The pad region before the aligned FFS file used to
be counted as free space even though it could never
be used due to the alignment requirements. There was
actually no free space available. PR #10828 fixed the
free space calculation to properly show it as 0 bytes
free, and this change then caused build error when
FV_SPARE_SPACE_THRESHOLD feature was enabled. The
reduction to a warning for this condition allows the
build to complete with errors and also provides a
build log warning message for review.
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
- FMMT tool would use the "PATH" environment variable for locating
the required GUID tool.
- On Windows-like system, batch file not found in the "PATH" environment
variable when "shell=False".
- This issue required commands to include program extensions or
absolute paths.
- This patch sets "shell=True" to extend the support for batch files,
including scripts in BinWrappers under BaseTools.
- Converted input commands from lists to strings to ensure proper
argument interpretation in POSIX-like shell scripts.
Signed-off-by: Jason1 Lin <jason1.lin@intel.com>
If command line options are moved into a response file
of a GCC family build, then the file path separators are
converted from '\' to '/'. However, this can corrupt
command line options that are quoted strings.
Update GenMake to no convert '\' to '/' in quoted strings.
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
This patch addresses an issue in the FMMT operations where the parent
reference is not checked for NoneType. This oversight can lead to an
AttributeError: 'NoneType' object has no attribute 'Name' when
attempting to access the parent reference. The fix involves adding a
check for NoneType before accessing the parent reference to ensure that
the operations handle such cases gracefully.
The affected functions include:
- AddNewFfs
- ReplaceFfs
- ExtractFfs
These functions now include proper checks to prevent the AttributeError.
Signed-off-by: Ashraf Ali S <ashraf.ali.s@intel.com>
Refer to the docs of python, `os.path.normcase(path)` function:
"Normalize the case of a pathname. On Windows, convert all characters in
the pathname to lowercase, and also convert forward slashes to backward
slashes. On other operating systems, return the path unchanged."
`os.path.normpath(path)` also convert forward slashes to backward slashes.
So call `os.path.normcase` after `os.path.normpath` just convert path to
lowercase on Windows(only).
And Windows is case-insensitive but case-preserving.
So the usage of `os.path.normcase(os.path.normpath(path))` can be
simplified to `os.path.normpath(path)`. Then we can use case-preserving
paths rather than lowercase paths in compile_commands.json file
or build log.
But this patch continue to use `os.path.normcase`
when comparing/searching paths.
Signed-off-by: Yang Gang <yanggang@byosoft.com.cn>
In Python 3.12 invalid escape sequences in strings moved from
DeprecationWarning to SyntaxWarning
(ref https://docs.python.org/3/whatsnew/changelog.html#python-3-12-0-final
and search for gh-98401). In a future Python version this will become
SyntaxError.
Multiple instances of these SyntaxWarnings are currently printed when
running the BaseTools tests using Python 3.12 (though without actually
failing the affected tests).
This commit updates all lines which were causing this type of warning.
Typical examples which needed fixing are:
- "BaseTools\Source\Python" representing a path: "\S" and "\P" are invalid
escape sequences, therefore left unchanged, therefore the test works
(with a warning in Python 3.12). r"BaseTools\Source\Python" represents
the same string, but with escapes turned off completely thus no warning.
- Where '\t\s' is used as a regex pattern, then chr(9) + '\\s' is sent
to the regex parser (with a warning in Python 3.12) since '\s' is not a
valid Python escape sequence. This works correctly, though arguably for
the wrong reasons. r'\t\s' sends the same as '\\t\\s', as originally
intended and with no warning.
(Note that ' and " are not fundamentally different in Python.)
Signed-off-by: Mike Beaton <mjsbeaton@gmail.com>
This patch updates the GenC logic to generate a random stack cookie value
for the stack check libraries. These random values improve security
for modules which cannot update the global intrinsics.
If the stack cookie value is randomized in the AutoGen.h file each
build, the build system will determine the module/library must be
rebuilt causing effectively a clean build every time. This also makes
binary reproducibility impossible.
This patch updates the early build scripts to create 32 and 64-bit JSON
files in the build output directory which each contain 100 randomized
stack cookie values for each bitwidth. If the JSON files are already
present, then they are not recreated which allows them to be stored and
moved to other builds for binary reproducibility. Because they are in
the build directory, a clean build will cause the values to be
regenerated.
The logic which creates AutoGen.h will read these JSON files and use a
hash of the module GUID (the hash seed is fixed in Basetools) to index
into the array of stack cookie values for the module bitwidth. This
model is necessary because there isn't thread-consistent data so we
cannot use a locking mechanism to ensure only one thread is writing to
the stack cookie files at a time. With this model, the build threads
only need to read from the files.
Signed-off-by: Oliver Smith-Denny <osde@linux.microsoft.com>
