x

Search in
Sort by:

Question Status:

Search help

  • Simple searches use one or more words. Separate the words with spaces (cat dog) to search cat,dog or both. Separate the words with plus signs (cat +dog) to search for items that may contain cat but must contain dog.
  • You can further refine your search on the search results page, where you can search by keywords, author, topic. These can be combined with each other. Examples
    • cat dog --matches anything with cat,dog or both
    • cat +dog --searches for cat +dog where dog is a mandatory term
    • cat -dog -- searches for cat excluding any result containing dog
    • [cats] —will restrict your search to results with topic named "cats"
    • [cats] [dogs] —will restrict your search to results with both topics, "cats", and "dogs"

Using UE4 memory allocation in third party libs

Hi guys,

I have a problem and need community help. I'm trying to use third party library (in particular google protobuf), but have an issue with iOS shipping build like described here https://answers.unrealengine.com/questions/869991/weird-behavior-with-stdstringstream-on-iphone-xs.html . According to workaround I decided to replace STL allocator and use UE allocation (FMemory::Malloc FMemory::Free etc.), but have an issue with compiling protobuf library - I have to link implementation of UE memory functions, how can I do this ? Does UE4 have compiled static libs or can I get it from source ?

Product Version: UE 4.21
Tags:
more ▼

asked Mar 27 '19 at 08:19 AM in C++ Programming

avatar image

kolobok1386
6 1 4 5

(comments are locked)
10|2000 characters needed characters left
Viewable by all users

1 answer: sort voted first

Hi, You don't need to rebuild the Protobuf library. it would be a tough work since you're linking Unreal's library with Protobuf so I'm not sure that it is a good way to fix this.

My Team made a script (.sh/.bat) to replace the std containers that are in generated code. (in pb.cc) You can also see that script files too in the repository.

If you're using Cmake or similar build system, You may have to add "add_custom_command()" and make it executes every time when you generate these pb.cc files.

more ▼

answered Mar 27 '19 at 08:48 AM

avatar image

gigone.lee
96 1 3 3

avatar image kolobok1386 Mar 27 '19 at 08:57 AM

Thanks a lot for your assistance! This issue is a real pain :( So as I understood you don't link protobuf library to your Plugin and just use generated protobuf code in your plugin ? I'm not expert of protobuf, but do this generated files contatin all necessary code ? I thought that they need some code from protobuf library

avatar image gigone.lee Mar 27 '19 at 09:30 AM

Yes. The only thing you have to do is, replacing containers in generated files(not Protobuf source code). Please make sure that you need a wrapping allocator before replacing(you can see the allocator class in my team's repository too).

avatar image kolobok1386 Mar 27 '19 at 01:07 PM

Hm... Just tried as you described - replace containers in generated files, but hten when I tried to build the plugin got linker errors that it can't find protobuf function implementations :( Maybe I need to geenerate protobuf files with some options ? Will be appreciate for your help.

avatar image gigone.lee Mar 27 '19 at 01:17 PM

Can I get your error message or piece of code that you replaced? It will be helpful to find out the solution.

avatar image kolobok1386 Mar 27 '19 at 01:37 PM

Something like this

Error LNK2019 unresolved external symbol "public: __cdecl google::protobuf::internal::LogMessage::LogMessage(enum google::protobuf::LogLevel,char const *,int)" (??0LogMessage@internal@protobuf@google@@QEAA@W4LogLevel@23@PEBDH@Z) referenced in function "public: __cdecl google::protobuf::internal::ArenaImpl::ArenaImpl (struct google::protobuf::ArenaOptions const &)" (??$?0UArenaOptions@protobuf@google@@@ArenaImpl@internal@protobuf@google@@QEAA@AEBUArenaOptions@23@@Z)

There are more then 100 such errors (145 to be precise), seems it depends on proto library, and looking into source code of generated files they have include to proto library and I believe they should depends on library code.

Just replace std::string from generated files to custom type with custom allocator, and replaced plugin dependency in Build.cs from protobuf.lib

avatar image kolobok1386 Mar 27 '19 at 01:53 PM

Hm... and one more question - I can see some protobuf source code in your plugin source directory, why did you put it there ? Seems it's for resolving necessary funtions ?

avatar image kolobok1386 Mar 28 '19 at 08:55 AM

@gigone.lee Could you assist please ?

avatar image kolobok1386 Mar 28 '19 at 01:20 PM

Like an example of errors

Error LNK2019 unresolved external symbol "public: __cdecl google::protobuf::internal::LogMessage::LogMessage(enum google::protobuf::LogLevel,char const ,int)" (??0LogMessage@internal@protobuf@google@@QEAA@W4LogLevel@23@PEBDH@Z) referenced in function "public: __cdecl google::protobuf::internal::ArenaImpl::ArenaImpl (struct google::protobuf::ArenaOptions const &)" (??$?0UArenaOptions@protobuf@google@@@ArenaImpl@internal@protobuf@google@@QEAA@AEBUArenaOptions@23@@Z) totalpoker D:\projects\totalpoker\Intermediate\ProjectFiles\poker-engine.lib(message_utils.obj) 1
Error LNK2019 unresolved external symbol "public: __cdecl google::protobuf::internal::LogMessage::~LogMessage(void)" (??1LogMessage@internal@protobuf@google@@QEAA@XZ) referenced in function "public: __cdecl google::protobuf::internal::ArenaImpl::ArenaImpl (struct google::protobuf::ArenaOptions const &)" (??$?0UArenaOptions@protobuf@google@@@ArenaImpl@internal@protobuf@google@@QEAA@AEBUArenaOptions@23@@Z) totalpoker D:\projects\totalpoker\Intermediate\ProjectFiles\poker-engine.lib(message_utils.obj) 1
Error LNK2019 unresolved external symbol "public: class google::protobuf::internal::LogMessage & __cdecl google::protobuf::internal::LogMessage::operator<<(char const
)" (??6LogMessage@internal@protobuf@google@@QEAAAEAV0123@PEBD@Z) referenced in function "public: __cdecl google::protobuf::internal::ArenaImpl::ArenaImpl (struct google::protobuf::ArenaOptions const &)" (??$?0UArenaOptions@protobuf@google@@@ArenaImpl@internal@protobuf@google@@QEAA@AEBUArenaOptions@23@@Z) totalpoker D:\projects\totalpoker\Intermediate\ProjectFiles\poker-engine.lib(message_utils.obj) 1

avatar image gigone.lee Mar 28 '19 at 02:02 PM

I was able to reproducible the same error in the home.

I've never experienced this in the office. It should be compiled without any errors...so weird. I'm going to tell this to my team so they can fix the issue. I'll let you know if I found the cause. thanks for the error anyway!

avatar image gigone.lee Mar 28 '19 at 09:12 AM

If 'Protobuf source code' means a bunch of pb.cc files, then yes. because iFun server needs the same protocol to receive the message from the client.

Since I'm not working as a client programmer, but As far as I know, We don't manage the Protobuf library source code in the repository.

avatar image kolobok1386 Mar 28 '19 at 09:35 AM

First of all thanks for your help! No I mean folder "google" with some protobuf sources in your plugin - "engine-plugin-ue4-experimental\Plugins\Funapi\Source\Funapi\Private\google", seems protobuf compiling with your plugin somehow...

avatar image gigone.lee Mar 28 '19 at 09:20 AM

Could you double check that you replace the std containers that are using std::allocator, to the custom allocator that is using Unreal's function?

You also need to add your custom allocator header in the Protobuf header so they can be compiled(Sorry I think I didn't mention this before)

We added the funapi_std_allocator.h file in protobuf/stubs/common.h like the bellow code.

...

#define GOOGLE_PROTOBUF_COMMON_H__

#include <funapi_std_allocator.h> ...

avatar image kolobok1386 Mar 28 '19 at 09:45 AM

Yes, I could see this line and made this change and in my version, but I beleieve it's not the reason of linker errors

avatar image gigone.lee Mar 28 '19 at 09:50 AM

ok, maybe I'm wrong :( which Prptobuf version do you have anyway? Please note that this plugin only supports version2 currently.

avatar image kolobok1386 Mar 28 '19 at 09:53 AM

We use 3.6.1, so you've added proto sources to your plugin, right? If yes then one more question - how did you compile it ? Because I tried to add proto source to Plugin, but got a lot of errors regarding c++17 features, how did you switch UE4 to compile c++17 code ?

avatar image gigone.lee Mar 28 '19 at 09:58 AM

Could you try with version2.6? I'm not sure version 3.6 will work because my team did not test it. As you can see those commits that are posted, we didnt touch any source code.

avatar image kolobok1386 Mar 28 '19 at 10:00 AM

I tried to add your proto sources and compile proto with protoc from 2.6, but again I got compilation errors regarding c++17 feature, couldn't find any useful information how to add c++ 17 feature compilation in google :(

avatar image gigone.lee Mar 28 '19 at 10:04 AM

okay. let me have some time to find out the issue. Any detilas(like protobuf version, build environment) will be helpful. :)

avatar image kolobok1386 Mar 28 '19 at 10:05 AM

Thanks a lot!!! Will wait for any news :)

avatar image kolobok1386 Mar 28 '19 at 02:19 PM

Sorry for disturbing, maybe you have any news on this issue ?

avatar image gigone.lee Mar 28 '19 at 02:30 PM

Sadly, No. I think I can spare more time tomorrow because It's almost 12:00 am In my country. I promise I'll bring good news tomorrow anyway.

avatar image kolobok1386 Mar 28 '19 at 02:31 PM

Thanks a lot for your assistance!!!

avatar image gigone.lee Mar 29 '19 at 05:35 AM

Hi, Could you retry again without unity build option? I heard that option is enabled by the default. There's a bug that Unreal Engine is not able to identify specific file extension(.cc) in 4.21 (fixed in 4.22).

You have two choices to avoid that. one is replacing the generated Protobuf file's extension to .cpp, Or turning off Unity build option.

avatar image kolobok1386 Mar 29 '19 at 07:39 AM

Yes, met this issue - replaced .cc to cpp.

avatar image gigone.lee Mar 28 '19 at 02:32 PM

What I found is that the problem is reproducible in any environment, which seems a high priority issue. I'll discuss with my coworker tomorrow to fix this issue.

avatar image kolobok1386 Mar 29 '19 at 07:40 AM

Hi, do you have any news on this issue ?

avatar image gigone.lee Mar 29 '19 at 07:52 AM

Did you try again without Unity build option?

avatar image kolobok1386 Mar 29 '19 at 07:56 AM

No, didn't try, I renamed .cc to .cpp and got issue with c++17 feature compilation.

avatar image gigone.lee Mar 29 '19 at 07:57 AM

It seems that the Unreal Engine is merging generated files when the Unity build option is enabled. so you have to turn off the option manually for now.

avatar image kolobok1386 Mar 29 '19 at 07:58 AM

Ok, thanks will try it, could you help please - how can I disable Unity build ?

avatar image gigone.lee Mar 29 '19 at 08:01 AM

Set the below flag in your Build.cs file.

BuildConfiguration.bUseUnityBuild = false;

If you're not familiar with this, you can find a similar setting on Github(search with the bUseUnityBuild keyword).

avatar image kolobok1386 Mar 29 '19 at 08:11 AM

Thanks a lot, yes I know what are you talking about :) Will try it! Will keep you updated.

avatar image kolobok1386 Mar 29 '19 at 10:06 AM

Unfortunately can't compile - getting a lot of errors. BTW can't find bUseUnityBuild = false; in your example project. Is your example from github compilable ?

avatar image gigone.lee Mar 29 '19 at 12:25 PM

You can add bUseUnityBuild = false; in {ProjectName}.Target.cs file too. Please try this if you didn't. Currently, I'm not available to use my computer but I was able to build the project in the office.

avatar image kolobok1386 Mar 29 '19 at 12:38 PM

Yes, tried this in project target.cs, but can't see this option in your .cs file. Which branch should I try from your poject - master or experimental ?

avatar image gigone.lee Mar 29 '19 at 12:45 PM

As far as I know, We didn't merge that fix in the master(stable) branch, so you should try with the experimental version.

You should put that option manually because we haven't touched it.

avatar image kolobok1386 Mar 29 '19 at 01:30 PM

Just tried to build experimental branch under Windows, and it doesn't compile :( Do you build under windows ?

avatar image kolobok1386 Mar 29 '19 at 08:14 PM

Error example:

   d:\engine-plugin-ue4-experimental\Plugins\Funapi\Source\Funapi\Private\google\protobuf\extension_set_heavy.cpp(151): error C2668: 'google::protobuf::internal::cpp_type': ambiguous call to overloaded function
   d:\engine-plugin-ue4-experimental\Plugins\Funapi\Source\Funapi\Private\google\protobuf\extension_set_heavy.cpp(127): note: could be 'google::protobuf::FieldDescriptor::CppType google::protobuf::internal::cpp_type(google::protobuf::internal::FieldType)'
 

avatar image kolobok1386 Mar 29 '19 at 08:14 PM

Just tried to build experimental branch of fun api project under win10. Firstly ran ChangeExtensions.bat to change .cc extensions and then simply tried to open project. Got a message that I have to compile modules : funapi_plugin_ue4 and Funapi. And answered yes to rebuild them. The build process started and I got a bunch of error messages like

  d:\engine-plugin-ue4-experimental\Plugins\Funapi\Source\Funapi\Private\google\protobuf\extension_set_heavy.cpp(151): error C2668: 'google::protobuf::internal::cpp_type': ambiguous call to overloaded function
   d:\engine-plugin-ue4-experimental\Plugins\Funapi\Source\Funapi\Private\google\protobuf\extension_set_heavy.cpp(127): note: could be 'google::protobuf::FieldDescriptor::CppType google::protobuf::internal::cpp_type(google::protobuf::internal::FieldType)'
   d:\engine-plugin-ue4-experimental\Plugins\Funapi\Source\Funapi\Private\google\protobuf\extension_set.cpp(60): note: or       'google::protobuf::internal::WireFormatLite::CppType google::protobuf::internal::`anonymous-namespace'::cpp_type(google::protobuf::internal::FieldType)'
   d:\engine-plugin-ue4-experimental\Plugins\Funapi\Source\Funapi\Private\google\protobuf\extension_set_heavy.cpp(151): note: while trying to match the argument list '(const google::protobuf::internal::FieldType)'
   d:\engine-plugin-ue4-experimental\Plugins\Funapi\Source\Funapi\Private\google\protobuf\extension_set_heavy.cpp(166): error C2668: 'google::protobuf::internal::cpp_type': ambiguous call to overloaded function
 
avatar image gigone.lee Mar 30 '19 at 09:55 AM

Did you add bUseUnityBuild = false; in both funapi_plugin_ue4.Target.cs and funapi_plugin_ue4Editor.Target.cs? Also, Make sure you execute ChangeExtensions file if you're working on 4.21 version.

avatar image kolobok1386 Apr 02 '19 at 08:52 AM

Hi, sorry was busy yesterday on another tasks, just checked with this flag and it works! Now compilation is fine - thanks a lot ! Will investigate our project and try to inject protobuf as you did. But one small question - As I understood you don't build/link protobuf library, you just put protobuf source to your plugin to have protobuf function, is this correct ?

avatar image kolobok1386 Apr 02 '19 at 02:56 PM

Could you check the question above please ?

avatar image kolobok1386 Apr 04 '19 at 06:34 AM

Hello, sorry for disturbing, could you clarify the question above please ?

(comments are locked)
10|2000 characters needed characters left
Viewable by all users
Your answer
toggle preview:

Up to 5 attachments (including images) can be used with a maximum of 5.2 MB each and 5.2 MB total.

Follow this question

Once you sign in you will be able to subscribe for any updates here

Answers to this question