Headless Analyzer

- Ghidra's Headless Analyzer allows users to run Ghidra from the command line without invoking the user interface.
- Can be run on one file or a directory of files (including subdirectories).
- Can either import programs to a project or process programs in an existing project.
- Any Ghidra script including those that invoke the GUI can be run in headless mode.

How to Run the Headless Analyzer

- ghidra_<version>/support/analyzeHeadless.bat|sh

      analyzeHeadless [<project_location> <project_name>[/<folder_path>]] |
        [ghidra://<server>[:<port>]/<repository_name>[/<folder_path>]]
           [[-import [<directory>|<file>]+] | [-process [<project_file>]]]
           [-preScript <ScriptName> [<arg>]*]
           [-postScript <ScriptName> [<arg>]*]
           [-scriptPath "<path1>[;<path2>...]"]
           [-propertiesPath "<path1>[;<path2>...]"]
           [-log <path to log file>] [-scriptlog <path to script log file>]
           [-overwrite] [-recursive [<depth>]] [-readOnly] [-deleteProject]
           [-noanalysis] [-librarySearchPaths <path1>[;<path2>...]]
           [-processor <languageID>] [-cspec <compilerSpecID>]
           [-analysisTimeoutPerFile <timeout in seconds>]
           [-keystore <KeystorePath>] [-connect [<userID>]]
           [-p] [-commit ["<comment>"]] [-okToDelete]
           [-max-cpu <max cpu cores to use>] [-loader <desired loader name>]

- NOTE: The analyzeHeadlessREADME.html has many more details and examples.

<project_location>

- The directory that either contains a current Ghidra project or will contain a newly-created project.
NOTE: This directory must already exist; it will not be created for you.

<project_name>[/<folder_path>]

- The name of either an existing project or new project you wish to create in the <project_location> directory.
- If the optional folder path is included, import(s) will be rooted under this project folder.

In the normal case, if you don't have an existing project of the name "project_name", then one will be created by the Headless Analyzer.

In the server case, a project must already exist.

ghidra://<server>[:<port>]/
<repository_name>[/<folder_path>]

- Used to specify a Ghidra Server repository URL and folder path instead of a local project_location and project_name.
- If the optional folder path is used, imports will be rooted under this directory (subfolders will be created if they don't already exist).
- The named repository must already exist on the Ghidra Server.

In the normal case, if you don't have an existing project of the name "project_name", then one will be created by the Headless Analyzer.

In the server case, a project must already exist.

-import [<directory>|<file>]+

- Specifies one or more executables (or directories of executables) to import.
- May use wildcard characters to specify a directory or file (valid characters and behavior subject to the host OS interpretation of wildcards).
- This option may be repeated to specify additional imports.
- NOTE: -import and -process can not both be present in the parameters list.

- If importing a directory:

- Creates directory folder in project if it doesn't already exist.
- When using -recursive, imports files in subdirectories as well; directory structure will be recreated in the Ghidra project.

- Wildcard characters can be used to specify multiple files (i.e., "-import /home/files/n*"). Valid wildcard characters are determined by the host OS. For example:

- Linux:

- Allows * and ?
- Allows ranges of characters: [a-z] or [!a-z]
- Wildcards can expand to either directories or files

- Windows:

- Allows * and ?
- Wildcards can only expand to files

- Can specify multiple imports two ways:

- Space-separated paths:

- -import [directory or file] [directory or file] [directory or file]

- Repeating the "-import" parameter

- -import [directory or file] –import [directory or file]

-process [<project_file>]

- Runs scripts and/or analysis on existing project file(s).
- When [project_file] is included, it specifies existing project executable(s).
- When [project_file] is omitted, runs on all the files in the project <folder_path>.
- May use '*' and '?' wildcards (wildcard string must be surrounded by single quotes)
- Unlike -import, -process may only be used once.

- [project_file] specifies a file only (can not contain a path to a file). Use <folder_path> in the "<project_name>[/<folder_path>]" specification to identify the path to the file.
- Wildcard characters are limited to "?" and "*", and strings should be surrounded by single quotes (to prevent host OS from prematurely expanding the wildcard before it is passed to the Headless Analyzer).
- Can use -recursive to run scripts/analysis over subfolders of a directory.
- Using -recursive with wildcard string searches subfolders for files that match the wildcard string.
- Examples:

- Find all .exe files starting with 'a' in the root directory
analyzeHeadless /home/usr/ghidra/projects TestProj -process 'a*.exe'
- Find all .exe files starting with 'a' in the root directory and child dirs
analyzeHeadless /home/usr/ghidra/projects TestProj -process 'a*.exe' -recursive

-preScript <ScriptName.ext> [<arg>]*

- Identifies the name of a script (including file extension, such as MyScript.java) that will execute before analysis. No path is necessary (script locations are specified using -scriptPath).
- Parameters to the script may be passed to the script.
- This option must be repeated to specify additional pre-scripts.

Headless Analyzer will look for script in default script directories and those specified by -scriptPath

-postScript <ScriptName> [<arg>]*

- Identifies the name of a script (including file extension, such as MyScript.java) that will execute after analysis. No path is necessary (script locations are specified using -scriptPath).
- Parameters to the script may be passed to the script.
- This option must be repeated to specify additional post-scripts.

Headless Analyzer will look for script in default script directories and those specified by -scriptPath

-scriptPath "<path1>[;<path2>...]"

- Specifies script search path(s).
- A path may start with $GHIDRA_SCRIPT (the Ghidra installation directory) or $USER_HOME (the user's home directory).

- On Unix systems, these home variables must be escaped with a '\' character.

- Examples:

- Windows:
-scriptPath "$GHIDRA_HOME/Ghidra/Features/Base/ghidra_scripts;/myscripts"
- Unix:
-scriptPath "\$GHIDRA_HOME/Ghidra/Features/Base/ghidra_scripts;/myscripts"

Specifies script paths for all these types of scripts: Pre, post, primary, and secondary

-propertiesPath "<path1>[;<path2>...]"

- Specifies *.properties file search path(s).
- .properties files are used to pass parameters to scripts
- A path may start with $GHIDRA_SCRIPT (the Ghidra installation directory) or $USER_HOME (the user's home directory).
- NOTE: .properties files are a legacy way to pass parameters to scripts. It is now preferred to pass arguments as mentioned on previous slides.

More detail in later slide and analyzeHeadlessREADME.html file.

-log <path to log file>

- Sets the location of the file that stores logging output from importing, processing, and analysis.
- If not used, logging output is written to the "application.log" file in the user directory.

Captures general-purpose logging for Ghidra. Script outputs can be redirected using the "-scriptlog" parameter.

-scriptlog <path to script log file>

- Sets the location of the file that stores logging output from pre- and post-scripts.
- If not used, script logging output is written to the "script.log" file in the user directory.

Captures script logging for Ghidra.

-overwrite

- Overwrites existing project files that conflict with an import file.
- If not used, conflicting import files are skipped.

Allows an existing project file to be overwritten if it shares the same name as the imported binary.

-recursive [<depth>]

- Enables recursive descent into directories and project sub-folders when a directory has been specified in -import or -process mode.
- If not used, only the immediate files contained within the specified directory are imported or processed.
- Specify a value for the optional depth argument to recurse into various levels of nested supported container files (e.g., zip, tar, .a, etc.).

Affects -import or -process modes – used for determining whether to process files in current directory or files in current directory + all its subfolders.

-readOnly

- In -import mode, imported files are not saved to the project after scripts and/or analysis.
- In -process mode, any changes made to files by scripts or analysis are discarded.
- The -overwrite option will be ignored if this option is specified during import operations.

For now, -readOnly must be used with -process when operating on a shared project.

-deleteProject

- Deletes the Ghidra project after scripts and/or analysis have completed.
NOTE: This only applies if the project has been created in the current session with -import; existing projects are never deleted!

-noanalysis

- Turns off analysis for executables.

-processor <languageID>

- Sets the processor information to be used in -import mode (and any subsequent analysis).
- If not used, Ghidra uses header info (if available) to determine the processor.
- languageID can be found in the
  ..ghidra_<version>\Ghidra\Processors\proc_name\data\languages\proc_name.ldefs file

- Example (path for x86 processor):
  ..ghidra_<version>\Ghidra\Processors\x86\data\languages\x86.ldefs

If Ghidra recognizes processor it uses the recognized one, if no processor recognized it uses the one specified by -processor option.

-processor <languageID>

Or, in Help -> About <name of program>

-cspec <compilerSpecID>

- Sets the compiler specification to be used in -import mode (and any subsequent analysis).
- 'compilerSpecID' can be found in the
  ..ghidra_<version>\Ghidra\Processors\proc_name\data\languages\proc_name.ldefs file

If Ghidra recognizes processor it uses the preferred or default cspec for the recognized processor spec, if no processor recognized it uses the one specified by combination –processor and –spec options

-cspec <compilerSpecID>
-analysisTimeoutPerFile
<timeout in seconds>

- Sets a timeout value, in seconds, for analysis
- If analysis on a file exceeds the specified time, analysis is interrupted and processing continues as scheduled
- Can use postScripts to detect if analysis has timed out (see analysisHeadlessREADME.html)

-keystore <KeystorePath>

- When using a Ghidra Server with PKI or SSH authentication, allows specification of a suitable private keystore file. The file should rely on file system protection only to avoid prompting for a password.

Better if you do it in the tool first to get it set up.

-connect [<userID>]

- Allows the process owner's default userID to be overridden with the given <userID> when connecting to a Ghidra Server (provided the server has been configured to allow this).

You can configure the Ghidra server to specify a different user ID. Ex: people who log in locally as root – wanted way to specify different users on Ghidra server

If you use the URL project path this option is implied – you don’t need to use this option unless you want to change the user name

-p

- When connecting to a server, allows interactive prompting for a password via the console.
NOTE: In rare cases, password text will be echoed to the console (warning will show at password prompt).

Used when connecting to a server that needs a password.

This method of authentication is normally discouraged – but if not used, the server connection will likely fail authentication if a password is required.

In some cases, password text will be echoed back (NOT masked) when the user is typing (there will be a warning). Password masking seems to work when using the Ghidra jarFile, but not when using analyzeHeadless.bat/sh or eclipse launcher.

-commit ["<comment>"]

- Commits all imported/processed files in a shared project to the project's underlying repository.
- The <comment> is optional and is saved with all commits in the headless session.
- Commits do not apply when the -readOnly parameter has been enabled.

If you use the URL project path, this option is implied – you don’t need to use this option unless you want to commit a comment when you commit.

Commits are currently not allowed for -process mode on shared projects (which must run in -readOnly mode).

-okToDelete

- When running in headless mode, some scripts exist that mark programs for deletion. This deletes all versions of a program in shared project mode and cannot be undone
- Therefore, so that programs are not accidentally deleted, this option is required to be set if user wants program deleted in headless mode.

-max-cpu <max cpu cores to use>

- Sets the maximum number of CPU cores to use during headless processing

-loader <desired loader name>

- Forces the file to be imported using a specific loader.
- Loaders can take additional arguments that they apply during the import process for things like block name, base address, file offset, length, etc... . See analyzeHeadlessREADME.html for full list.

General Notes

- Make sure the specified project is not already open in the Ghidra GUI.
- When importing in bulk (i.e., specifying a directory or wildcard string), files starting with '.' are ignored.

- Import can be forced by naming the file during import:
-import /Users/user/.hidden.exe

Shared Project Notes

- Avoid using the script API method 'setServerCredentials'.
- Specify -connect with userID to override default userID (subject to server settings).
- If server/keystore requires password, -connect will force prompting via stdin.
- It is not possible to create a shared project via command line. See the GhidraProject class for a simple API that allows shared project creation using a script.

- Why should setServerCredentials be avoided:

- Because the headless analyzer already does it for you; when you use setServerCredentials, the two actions conflict with each other

- Note: be careful with password – it will show up in the clear (except when using ghidra.jar).

Exercise 1

- Create a directory in your home directory called "MyPrograms"
- Copy some of the class exercise programs to the "MyPrograms" directory
- Run Ghidra in headless so that it:

- Analyzes the programs in the "MyPrograms" directory
- Creates a project called "<your_name>_HeadlessProj" in your home directory that is saved after the script finishes.
- Writes script output to a log called "myLog" in your home directory.
- Calls the ReportPercentDisassembled.java script on each program after analysis is finished.

- After the script finishes, verify that it created a project and log in the correct directory. Open the log and verify that the log contains lines that start with "REPORT DISASSEMBLY..." for each program in your "MyPrograms" directory. Open the project to verify that it contains analyzed programs from your "MyPrograms" directory.

Headless Scripting Capabilities

- Can do just about anything you can do in normal scripts including GUI-dependent actions
- Can make scripts behave correctly in both GUI and headless environment
- Any number Ghidra scripts can be run before and/or after analysis.
- For multiple pre/post-scripts, scripts are executed in the order specified on the command line.

When running scripts without -import or -process, only the specified pre/post-scripts will be executed. In this case, all scripts must execute in a program-independent manner, or errors will occur. Use -process for scripts that are program-dependent.

Headless Scripting Capabilities

- Selections made in scripts will carry through to following scripts unless changed/cleared.
- When running in -import mode, any pre- or post-script may invoke the setTemporary() method on currentProgram to prevent that import from being saved.
- Advanced control for deleting programs or canceling future scripts is discussed in the "Control Follow-On Program Processing" section.

Why setTemporary() is useful: if you are using scripts to determine which binaries are interesting, you can save the interesting ones in your project and ditch the others.

Headless Scripting Capabilities

- Some useful features...

- Run other scripts from a script
- Get current and default analysis options
- Set analysis options

- Also reset to default

- Create and use selections
- Detect auto-analysis timeout
- Pass parameter values to a script
- Control follow-on processing of a program after the current script (whether to abort future processing and/or delete program from project).

New in 6.0: build-in methods for getting/setting analysis options.

Making a Script behave in
GUI and Headless environments

Use:

       if ( isRunningHeadless() ) { }

to check whether running headless mode

Calling a Script from a Script

Example:

       runScript("HelloWorldScript.java");

- (See analyzeHeadlessREADME.html for different runScript options)

Get Analyzer Information

      // Print analyzer names, their current values,
      // and whether set to default value

      Map<String, String> analyzers =
          getCurrentAnalysisOptionsAndValues(currentProgram);

      for (String analyzer : analyzers.keySet()) {
        String analyzerValue = analyzers.get(analyzer);
        boolean isDefault =
            isAnalysisOptionDefaultValue(currentProgram, analyzer,
              analyzerValue);
    	println(analyzer + " : " + analyzerValue +
              (isDefault ? " (default)" : ""));
      }

- There are also built-in functions to:

- get option choices (for those analyzers that can only be set to certain values)
- get descriptions for options (as provided by the analyzer)
- reset some/all analysis options to default
- get default values for analysis options

Set Analysis Options

Example (change one option):

      // Turn off Stack Analyzer

      setAnalysisOption(currentProgram, "Stack", "false");

Example (change multiple options):

      // Turn off Stack Analyzer,
      // Turn on Decompiler Param ID

      Map<String,String> optionsToSet =
               new HashMap<String, String>();
      optionsToSet.put("Stack", "false");
      optionsToSet.put("Decompiler Parameter ID", "true");
      setAnalysisOptions(currentProgram, optionsToSet);

Note that option names and values must be strings (code will attempt to convert the value to the correct type).

Reset Analysis Options

Example (reset all analysis options to default):

      resetAllAnalysisOptions(currentProgram);

Example (reset some analysis options to default):

      List<String> opts = Arrays.asList(new String[] {
    	"Stack", "Decompiler Parameter ID"});

      resetAnalysisOption(currentProgram, opts);

Create a Selection

Example:

      MemoryBlock block =
         currentProgram.getMemory().getBlock(".data");

      AddressSet set = createAddressSet();

      set.addRange(block.getStart(),block.getEnd());

      createSelection(set);

This selection will stay set and can be used by subsequently-called scripts until it is changed or cleared.

Control Follow-On Program
Processing from Scripts

Options (behavior after current script completes):

      CONTINUE
      // (default) Continue scheduled analysis/scripts,
      // save program

      CONTINUE_THEN_DELETE
      // Continue scheduled analysis/scripts, do not
      // import/save program

      ABORT
      // Abort scheduled analysis/scripts,
      // save program

      ABORT_AND_DELETE
      // Abort scheduled analysis/scripts,
      // do not import/save program

Control Follow-On Program
Processing from Scripts

Example:

      // Abort processing for the current file
      // because its version number is older than 8.1

      GhidraState currState = getState();

      if ( !getGhidraVersion().startsWith("8.1") ) {
          currState.addEnvironmentVar(
              SCRIPT_SET_CONTINUATION_STATUS,
              HeadlessContinuationOption.ABORT);
      }

More specifics on usage in analyzeHeadlessREADME.html

How to write a Headless Script

- Exactly the same as writing any Ghidra script!
- Best way: Use Eclipse Framework + GhidraDev plugin
- GhidraDev plugin provides Headless Ghidra run configuration (requires customization of program arguments)
- See ghidra_<version>/Extensions/Eclipse/GhidraDev/GhidraDev_README.html for more info.

There are two ways to install the GhidraDev plugin into your Eclipse:
- Link Ghidra with an Eclipse in Front-End tool options and edit a script from the Ghidra script manager. Ghidra will offer to install GhidraDev for you if not present.
- Install directly into Eclipse using the file found at ghidra_<version>/Extensions/Eclipse/GhidraDev/GhidraDev-x.x.x.zip
See the GhidraDev_README.html for pros and cons of each.
(Optional) Exercise 2

- Repeat the same exercise as in Exercise 1 but instead of using the -import to run against a non-shared project, run it against all programs in a shared project.

More Headless Information

- analyzeHeadlessREADME.html in the ghidra_<version>/support/ folder contains many more details, lots of examples, and is usually more up to date than these slides!