Weird behavior with std::stringstream on iPhone XS

Development Programming & Scripting
1

Hello, I found something very strange bug with std::stringstream on iPhone XS.
I’m still putting in work to reproduce on other iPhone devices but I found it impossible to reproduce on other devices and platforms. the followings are the detailed environment.

UE4 version : 4.21.1

target device : iPhone XS

iOS version : 12.1.1

Xcode version: 10.1

The test scenario is very simple, just put these 4 lines (or maybe 3 lines) into any project and run on iPhone XS.

std::stringstream stream;
// just put random long string into the stringstream value.
stream << "5sdocmoiatjiladmfc9euhxqincabiughjijdfjafdsjh9rpewiojdsicpklnjijowfvookjkfas";
stream << "5sdocmoiatjiladmfc9euhxq57456456456incabiughjijdfjafdsjh9rpewiojdsicpklnjijowfvookjkfas";
auto temp_string = stream.str(); // <<-- crash point.

The problem occurs in the line that I marked, the application keeps crashing on iPhone XS, right after calling the destructor of std::string that was created temporary while copying the value into the temp_string variable.

I found someone already struggled with the same issue before and fixed in the previous versions(Unreal crashes on two lines of extremely simple standard C++ code (only in Linux) - Platform & Builds - Epic Developer Community Forums). I’m guessing this bug is related to new/delete symbols in the linker script but I have no ideas.

2

c_str() does not return a string but a char const*. The buffer it points to is only valid during the lifetime of the std::string and even may be invalidated by subsequent operations on the string (s.a. docs). Knowing this I would consider it a rather expected behaviour to get access violations when trying to access this buffer after the string is deleted. That this is working on other platforms because the memory usually is not overwritten a few cycles later does not neither mean it is guaranteed to nor that it should.
To work around this just keep the string as long as you need it. This really shouldn’t mess up your memory footprint, if it does there probably is another issue elsewhere.

3

Yes. a temporal string that is created by str(), should be released after putting in the ‘temp_string’. but it still crashes even when I use str() instead of str().c_str(). The string should be deep-copied, not referred in this time. it only happens on iPhone XS. I’m going to dig deeper.

Currently, I think working around this is not an option because I already found there are several potential crash points like this in a lot of the third party libraries. I can’t just fix them all.

4

This sounds indeed weird. Can you also provide a complete code sample of the version using a deep copy and a stack trace of the crash(es)?
When in doubt of what types are used it might help to drop the auto keyword and fall back to static type usage.
I also would advice to perform some static code checking on your code. (I personally like to use Cppcheck to sanitize my code.)

Hard to believe it is such a common practice in 3rd party libs to use buffers of already deleted objects, because this is invalid behaviour on any platform (even if it might work in many cases) and there are established techniques around to deal with this sort of problems (e.g. smart/weak pointers).

5

Sorry for confusing. I mean, They use str(), not c_str(). The most cases are copying string that is created from std::stringstream, I didn’t see any c_str() code. As you know, Copying a string from std::stringstream is a common practice, and It doesn’t seem harmful to me.

The problem always happens when std::stringstream::str() is deallocated. even when I use std::copy() and std::move() with std::stringstream::str(). so I think any code will make the problem. I already removed auto keyword to make sure there’s no problem on the left side.

I heard iPhone XS is very sensitive at managing memory than other devices. Not sure whether it is true.

I’m going to update my post with call stack and some other things that might be useful, but You can reproduce it with the above code.

6

They use str(), not c_str(). […]
Copying a string from std::stringstream is a common practice, and It doesn’t seem harmful to me.
D’accord. Copying around strings should be safe to do in general and that is why I guess that the access violation is going on elsewhere even if it may be raised when the string is deallocated. Have you tried to run a sample program that does nothing else except for the lines from the sample above (not using c_str() ) to isolate the problem? I would be quite confident this alone should work as expected.
Please provide the code you are testing as complete and minimal as possible.

7

we have the same problem, and not fixed

8

We solved a similar problem by replacing Binned memory allocator (default choice on IOS build) with the ANSI one.
you can do this by modifying line 210 of ApplePlatformMemory.cpp (4.20 version).

9

Do we need to rebuild whole engine from source code after that ? Or we can set definition of FORCE_ANSI_ALLOCATOR variable by compilation parameters?

10

I confirmed that the problem can happen anytime when the std containers that are allocated by libc++ is deallocated in iOS. I doubt that this won’t cause any problem in Android OS because they use libstd++ instead of libc++.

I think the bug is somewhere between allocator and deallocator. at least one of these methods is using std::allocator/deallocator instead of FMemory::Malloc() and FMemory::Free().

My team solved this problem a month ago by wrapping the std containers so they can use FMemory::Malloc() and FMemory::Free() properly. You don’t need to rebuild the whole engine code.

Of course, you can see my team’s code since it is opened to anyone.
(Funapi 플러그인: funapi_allocator 클래스 추가 · iFunFactory/engine-plugin-ue4@bc6faab · GitHub)

My team tested it on iPhone XS and the other devices after the code is merged, No more crashes :slight_smile: I assume that you can use this workaround in the production environment until the Unreal Engine(or iOS) fix this.

  • Please note that this problem also can happen in the third-party libraries like JSON or Protobuf, If you’re struggling with that, you may have to make a script file that replaces these containers after your code is generated, or after you update the package.
11

Please see my answer. You don’t need to rebuild the whole source code.

12

Thanks a lot for your help, could you check my question please here https://answers.unrealengine.com/questions/885467/using-ue4-memory-allocation-in-third-party-libs.html ? Could you clarify how you build protobuf libs ? I tried your project from github and when try to compile under Win10 get a lot of errors about c++17 features support in UE4, couldn’t find how to resolve them :frowning:

13

@Clones1201 Could you confirm please that allocator change solved your issues with iPhone XS ?

14

yes, I can confirm this, that allocator change currently still in our code.

15

yes, we are using an engine built from source

16

Thanks a lot! And can you describe please how did you replace it ? Did you rebuild UE from source code with your changes ?

17

I have a similar problem。
I Build a plugin as static lib use in unreal 4.2.1. have used google proto buf
use xcode 9.4
and crash On the iphone A12 CPU series. like XR， XS Max
Got a lot of Log :
LogMemory: Warning: Attempting to free a pointer we didn’t allocate!
i check the code. protobuf generate .h files and include inline function use delete. it likes this delete will be use FMemory::Free()
so.i add c++ flag -fno-inline-functions to rebuild lib. it’s no log any more

next I Got a Crash. Dump Stack is std::basic_string::~basic_string()
It’s a struct member function that returns std::string
definition : std::string ToString() const;
implement :

	std::string tagCSHead::ToString() const {
		std::stringstream oss;
		oss << "CSHead"
			<< ", crypto_type: " << crypto_type
			<< ", compression_type: " << compression_type
			<< ", pkg_size: " << pkg_size
			<< ", version: " << (uint16_t)version
			<< ", msg_type: " << (uint16_t)msg_type
			<< ", flag: " << (uint16_t)flag
			<< ", cmd: " << cmd
			<< ", seq: " << seq
			<< ", session_id: " << session_id
			<< ", service_name_size: " << (uint16_t)service_name_size
			<< ", service_name: " << service_name
			<< ", target: " << target
			<< ", data_size: " << data_size;

		return oss.str();
	}

printf("send internal, cshead:%s", a_stCSHead.ToString().c_str());

it will print right log and then crash…
then i test sprintf .then it works…

 std::string tagCSHead::ToString() const {
        char szBuff[4096];
        sprintf(szBuff, "CSHead, crypto_type: %u, compression_type:%u, pkg_size:%u, version:%u, msg_type:%u, flag:%u, cmd: %u, seq:%u, session_id:%llu, service_name_size:%u, service_name:%s, target:%u, data_size:%u"
                , crypto_type, compression_type, pkg_size, (uint16_t)version, (uint16_t)msg_type, (uint16_t)flag, cmd, seq, session_id, (uint16_t)service_name_size, service_name, target, data_size);
        
        return szBuff;
 }

so …I Guess it Should be related to the implementation of std::stringstream
or Xcode stl library conflicts with A12？