Linux Shipping Server Not Reading Debug Symbol

after we upgraded to UE 4.20, Linux Shipping Package Server not load debug symbol file,

so the crash stack looks like below:

 [2018.10.31-10.00.53:516][584]LogCore: === Critical error: ===
 Unhandled Exception: SIGSEGV: invalid attempt to read memory at address 0x000000000000009a
 
 [2018.10.31-10.00.53:516][584]LogCore: 0x000000000601ba2f MyGameServer!_ZN17FObjectReplicator19ReplicatePropertiesER9FOutBunch17FReplicationFlags(+0x2f)
 0x000000000601a9b7 MyGameServer!_ZN13UActorChannel14ReplicateActorEv(+0x966)
 0x00000000042e4ed8 MyGameServer!_ZN17UReplicationGraph20ReplicateSingleActorEP6AActorR31FConnectionReplicationActorInfoR27FGlobalActorReplicationInfoR26FPerConnectionActorInfoMapP14UNetConnectionj(+0xc7)
 0x00000000042e2b56 MyGameServer!_ZN17UReplicationGraph21ServerReplicateActorsEf(+0x6a5)
 0x00000000062fb094 MyGameServer!_ZN10UNetDriver9TickFlushEf(+0xc3)
 0x0000000006325c16 MyGameServer!_ZNK34TBaseUObjectMethodDelegateInstanceILb0E10UNetDriverFvfEJEE13ExecuteIfSafeEf(+0x45)
 0x0000000004e2bc86 MyGameServer!_ZNK22TBaseMulticastDelegateIvJfEE9BroadcastEf(+0x85)
 0x000000000622f013 MyGameServer!_ZN6UWorld4TickE10ELevelTickf(+0xff2)
 0x00000000060df808 MyGameServer!_ZN11UGameEngine4TickEfb(+0x3f7)
 0x00000000046171a5 MyGameServer!_ZN14UArkGameEngine4TickEfb(+0x64)
 0x0000000003f28f0e MyGameServer!_ZN11FEngineLoop4TickEv(+0x38d)
 0x0000000003f29baa MyGameServer!_Z11GuardedMainPKw(+0x299)
 0x0000000006afb3d6 MyGameServer!_Z14CommonUnixMainiPPcPFiPKwE(+0xa65)
 0x00007f6710be0c05 libc.so.6!__libc_start_main(+0xf4)
 0x0000000003f24029 MyGameServer!UnknownFunction(0x3d24028)

if run server with gdb, gdb will load the debug file. and show clear crashstack with filename and linenum.

so, how can i make MyGameServer-Linux-Shipping-Binary load the MyGameServer-Linux-Shipping.debug

and there are

MyGameServer-Linux-Shipping.sym
MyGameServer-Linux-Shipping.debug
MyGameServer-Linux-Shipping

3 files in MyGame/Binaries/Linux directory

The callstack is being symbolicated via .sym, not .debug (.debug contains the debug information for gdb, while .sym contains a much more light-weight format for the runtime callstack symbolication). Make sure you have .sym right next to the server binary.

yes,i have the .sym file right next to the server binary, but crashstack still not contain filename and linenum.

finish use c++filt and addr2line cmdline tools to translation crashstack.
some bash cmd example:

[grissom@buildfarm ~]$c++filt "_ZN13UActorChannel14ReplicateActorEv" 
**UActorChannel::ReplicateActor()**
[grissom@buildfarm ~]$ addr2line -e /data1/grissom/TrunkServer/dist/MyGame/Binaries/Linux/MyGameServer-Linux-Shipping -f -C -p 0x000000000601a9b7
**UChannel::ReceivedNextBunch(FInBunch&, bool&) at E:\CI\UE\Engine\Source\Runtime\Engine\Private/DataChannel.cpp:474**

the result of addr2line may be mistake beacuse some C++ optiomize. check the c++filt result is much accurate

This is a workaround, not the proper course of action. Callstack symbolication works fairly reliable for us, so if it is not working for you we need to find the bug and fix it.

Interesting… Would you mind drilling a bit deeper into the problem?

  • Does the path to the server binary has spaces?
  • Does the .sym file have the appropriate permissions so the server can open it?
  • Are “MyGameServer-Linux-Shipping.sym” and “MyGameServer-Linux-Shipping” the actual names or just an example given? Particularly, if the file name for .sym contains more dots than just one (like "FooGameServer.MyGame.sym), it may not be found by the server.
  • Your callstack mentions MyGameServer module but the files you list are MyGameServer-Linux-Shipping. Would you mind checking that if you run MyGameServer binary, you also have MyGameServer.sym lying next to it?

= Does the path to the server binary has spaces?

- Yes,has many free space.

Does the .sym file have the appropriate permissions so the server can open it?

- Yes, 755 with MyGameServer-Linux-Shipping and 644 with MyGameServer-Linux-Shipping.debug and MyGameServer-Linux-Shipping.sym.

Are “MyGameServer-Linux-Shipping.sym” and “MyGameServer-Linux-Shipping” the actual names or just an example given? Particularly, if the file name for .sym contains more dots than just one (like "FooGameServer.MyGame.sym), it may not be found by the server.

Names are ok, all files are generated by UAT batch script,don’t modify it,just move it from windows (win10, cross-compile env) to linux (centos 7).

Your callstack mentions MyGameServer module but the files you list are MyGameServer-Linux-Shipping. Would you mind checking that if you run MyGameServer binary, you also have MyGameServer.sym lying next to it?

not understand it. i only have MyGameServer-Linux-Shipping binary file in shipping package. do you mean make a development package?

Does the path to the server binary has
spaces?
Yes,has many free space.

I meant space character in the path, like /path/with a space/character. This is sometimes not handled properly.

Your callstack mentions MyGameServer module but the files you list are MyGameServer-Linux->>Shipping. Would you mind checking that if you run MyGameServer binary, you also have >>MyGameServer.sym lying next to it?

not understand it. i only have MyGameServer-Linux-Shipping binary file in shipping package. do >you mean make a development package?

The callstack that you pasted in the beginning of the post seems to be from MyGameServer binary, not MyGameServer-Linux-Shipping. See for example this line:

0x000000000622f013 MyGameServer!_ZN6UWorld4TickE10ELevelTickf(+0xff2)

It has “MyGameServer” part, if it were from MyGameServer-Linux-Shipping it would be

0x000000000622f013 MyGameServer-Linux-Shipping!_ZN6UWorld4TickE10ELevelTickf(+0xff2)

As such, please make sure that you run the binary that you intend to run and that each binary also has .sym file located in the same directory.

1、no space in binary path.

2、maybe a problem.
in the callstack do has binary name ArkGameServer
and binary-file name is MyGameServer-Linux-Shipping

i do some test,found the server process start in our "DS Watcher " process written by golang, use process.start with params that Name hardcode as “MyGameServer” with Path is Shipping Binary Path. cause my problem.
just change it to MyGameServer-Linux-Shipping fix it.

thanks you RCL!