Symbolicate Crashlogs

This is useful for symbolicating crashlogs you have collected from a device you don't have access to

ipsw will detect the dyld_shared_cache needed to symbolicate the userspace crashlog

❯ ipsw symbolicate solitaire-2021-02-23-185510.ips

   ⨯ please supply a dyld_shared_cache for iPhone12,1 running 14.5 (18E5154f)

You can download the required dyld_shared_cache like so

❯ ipsw download ipsw --device iPhone12,1 --version 14.5

To extract the dyld_shared_cache from the downloaded IPSW file

❯ ipsw extract --dyld <IPSW.ipsw>

You can download the current beta shared caches like so

❯ ipsw download ota --platform ios --device iPhone12,1 --beta --dyld
? You are about to download 1 OTA files. Continue? Yes
   • Parsing remote OTA        build=18E5154f device=iPhone12,1 iPhone11,8 version=iOS145DevBeta2
   • Extracting remote dyld_shared_cache (can be a bit CPU intensive)

Now you can symbolicate the crashlog

❯ ipsw symbolicate solitaire-2021-02-23-185510.ips dyld_shared_cache_arm64e

Process:             solitaire [12345]
Hardware Model:      iPhone12,1
OS Version:          14.5
BuildID:             18E5154f

Exception Type:      EXC_BAD_ACCESS (SIGSEGV)
Exception Subtype:
KERN_INVALID_ADDRESS at 0x0000020000000010 -> 0x0000000000000010 (possible pointer authentication failure)
VM Region Info: 0x10 is not in any region.  Bytes before following region: 4363091952
      REGION TYPE                 START - END      [ VSIZE] PRT/MAX SHRMOD  REGION DETAIL
      UNUSED SPACE AT START
--->
      __TEXT                   1040f8000-1040fc000 [   16K] r-x/r-x SM=COW  ...app/solitaire

Termination Signal:  Segmentation fault: 11
Termination Reason:  Namespace SIGNAL, Code 0xb
Terminating Process: exc handler [12345]
Triggered by Thread: 45

Thread 45 name: Dispatch queue: CTTelephonyNetworkInfo
Thread 45 Crashed:
  0: libobjc.A.dylib         (slide=0x27010000) 0x1bc39e1e0 _objc_msgSend + 32
  1: CoreFoundation          (slide=0x27010000) 0x1a734e76c -[__NSDictionaryM objectForKeyedSubscript:] + 184
  2: CoreTelephony           (slide=0x27010000) 0x1a79d1230 -[CTTelephonyNetworkInfo updateRat:descriptor:] + 144
  3: CoreTelephony           (slide=0x27010000) 0x1a79d1114 -[CTTelephonyNetworkInfo queryRatForDescriptor:] + 164
  4: CoreTelephony           (slide=0x27010000) 0x1a79cfe9c -[CTTelephonyNetworkInfo connectionStateChanged:connection:dataConnectionStatusInfo:] + 72
  5: CoreFoundation          (slide=0x27010000) 0x1a7476894 ___invoking___ + 148
  6: CoreFoundation          (slide=0x27010000) 0x1a734a054 -[NSInvocation invoke] + 380
  7: CoreFoundation          (slide=0x27010000) 0x1a734a658 -[NSInvocation invokeWithTarget:] + 80
  8: CoreTelephony           (slide=0x27010000) 0x1a79d9fb0 __ZZN8dispatch5asyncIZ50-[CoreTelephonyClientMux sink:handleNotification:]E3$_2EEvP16dispatch_queue_sNSt3__110unique_ptrIT_NS4_14default_deleteIS6_EEEEENUlPvE_8__invokeESA_ + 44
  9: libdispatch.dylib       (slide=0x27010000) 0x1a705e878 __dispatch_client_callout + 20
 10: libdispatch.dylib       (slide=0x27010000) 0x1a7066060 __dispatch_lane_serial_drain + 620
 11: libdispatch.dylib       (slide=0x27010000) 0x1a7066c5c __dispatch_lane_invoke + 404
 12: libdispatch.dylib       (slide=0x27010000) 0x1a7071518 __dispatch_workloop_worker_thread + 764
 13: libsystem_pthread.dylib (slide=0x27010000) 0x1f33ba7a4 __pthread_wqthread + 276
 14: libsystem_pthread.dylib (slide=0x27010000) 0x1f33c174c _start_wqthread + 8

Thread 45 State:
    x0: 0x00000002803e0047   x1: 0x00000001f6df3124   x2: 0x0000000281ed4000   x3: 0x00000001b6efb344
    x4: 0x00000000000062dc   x5: 0x0000000000000001   x6: 0x3130303030303030   x7: 0x0000000000000000
    x8: 0x00000001f6df3000   x9: 0xdb6091bce8b38b12  x10: 0x6ae10002803e0047  x11: 0x0000000281ed4021
   x12: 0x0000000281ed4021  x13: 0x0000020000000000  x14: 0x00000001a7604012  x15: 0x0000020000000000
   x16: 0x0000000000000000  x17: 0xd0357901a735c988  x18: 0x0000000000000000  x19: 0x0000000281ed4000
   x20: 0x00000001f6df3124  x21: 0x0000000000000003  x22: 0x00000002801de580  x23: 0x0000000000000003
   x24: 0x0000000000000001  x25: 0x000000020abc3520  x26: 0x0000000000000003  x27: 0x0000000000000000
   x28: 0x0000000000000104   fp: 0x000000016de0e780   lr: 0x00000001a734e76c
    sp: 0x000000016de0e740   pc: 0x00000001bc39e1e0 cpsr: 0x20001000
   esr: 0x92000004

info

You can use the --unslide flag to unslide the crashlog for easier static analysis

🆕 Symbol Server

Setup Server

Databases supported:

in-mem
sqlite
postgres

info

Want another database driver supported? Make a feature request

Install `postgres`

brew install postgresql@16

Start postgres either as service:

brew services start postgresql@16

Or manually:

LC_ALL="C" /opt/homebrew/opt/postgresql@16/bin/postgres -D /opt/homebrew/var/postgresql@16

Create a `~/.config/ipsw/config.yml` with the following database info

database:
  driver: postgres
  name: postgres
  host: localhost
  port: 5432
  user: blacktop

info

To add kernel symbolication

Get the signatures

git clone https://github.com/blacktop/symbolicator.git

Add them to your ~/.config/ipsw/config.yml

daemon:
  sigs-dir: /path/to/blacktop/symbolicator/kernel

For more information see Kernel Symbolication

Start `ipswd`

ipswd is a daemon that exposes a subset of ipsw's functionality as a RESTful API to allow for easier automation and use in large scale pipelines.

ipswd start

Here are the ipswd API Docs

Scan an IPSW

Using httpie

http POST 'localhost:3993/v1/syms/scan' path==./IPSWs/iPad_Pro_HFR_17.4_21E219_Restore.ipsw

Symbolicate a `panic`

The symbolicate command now supports the NEW panic/crash JSON format

❯ ipsw symbolicate --help

Symbolicate ARM 64-bit crash logs (similar to Apple's symbolicatecrash)

Usage:
  ipsw symbolicate <CRASHLOG> [IPSW|DSC] [flags]

Aliases:
  symbolicate, sym

Flags:
  -a, --all                 Show all threads in crashlog
  -t, --api-token string    Symbol Server API bearer token
  -d, --demangle            Demangle symbol names
      --dsc-slide string    Rebase dyld_shared_cache frames onto this base for static analysis (hex, e.g. 0x180000000)
  -x, --extra string        Path to folder with extra files for symbolication
      --force               Force using the supplied IPSW even if it doesn't match the crashlog (e.g. for vPhone)
  -h, --help                help for symbolicate
      --hex                 Display function offsets in hexadecimal
      --ida                 Generate IDAPython script to mark panic frames in IDA Pro
      --kc-slide string     Apply custom KASLR slide to kernelcache frames for live debugging (hex, e.g. 0x14f74000)
      --peek                Show disassembly instructions around each panicked frame
      --peek-count int      Number of instructions to show with --peek (centered on frame, respects function boundaries) (default 5)
      --pem-db string       AEA pem DB JSON file
  -p, --proc string         Filter crashlog by process name
  -r, --running             Show all running (TH_RUN) threads in crashlog
  -s, --server string       Symbol Server DB URL
      --signatures string   Path to signatures folder
  -u, --unslide             Unslide user-space addresses for static analysis (kernel frames are always unslid)

Global Flags:
      --color           colorize output
      --config string   config file (default is $HOME/.config/ipsw/config.yaml)
      --no-color        disable colorize output
  -V, --verbose         verbose output

The full example list lives on the CLI reference page.

info

Swagger Docs for the symbol server API are here Syms API

Symbolicate a 210 panic using the symbol server

❯ ipsw symbolicate --server 'http://localhost:3993' panic-full-2023-08-04-191003.000.ips

   • Symbolicating 210 Panic with Symbol Server server=http://localhost:3993

syms-panic

NOTE: panic is from here

Authenticated symbol server

ipswd itself does no auth. If your symbol server sits behind a reverse proxy or API gateway that wants a bearer token, give the token to ipsw and it is sent as Authorization: Bearer <token> on every request.

Three ways to set it, highest priority first:

# 1. flag (lands in your shell history, so prefer one of the other two)
❯ ipsw symbolicate --server https://syms.example.com --api-token "$TOKEN" panic-full-2023-08-04-191003.000.ips

# 2. environment variable
❯ env IPSW_SYMBOLICATE_API_TOKEN="$TOKEN" ipsw symbolicate --server https://syms.example.com panic-full-2023-08-04-191003.000.ips

# 3. ~/.config/ipsw/config.yml
symbolicate:
  server: https://syms.example.com
  api-token: <token>

A few rules apply once a token is set:

The server URL has to be https://. Plain http:// only works for localhost and loopback addresses, so a token never crosses the network in the clear.
Redirects are refused, so the token can't be replayed to some other host the server points at.
The token is not path-expanded like other config values, is never logged, and is scrubbed from error messages if a server echoes it back.
Whitespace, control characters and commas are rejected before any request goes out.

🆕 Symbol Server​

Setup Server​

Install postgres​

Create a ~/.config/ipsw/config.yml with the following database info​

To add kernel symbolication​

Start ipswd​

Scan an IPSW​

Symbolicate a panic​

Authenticated symbol server​

🆕 Symbol Server

Setup Server

Install `postgres`

Create a `~/.config/ipsw/config.yml` with the following database info

To add kernel symbolication

Start `ipswd`

Scan an IPSW

Symbolicate a `panic`

Authenticated symbol server