x

Search in
Sort by:

Question Status:

Search help

  • Simple searches use one or more words. Separate the words with spaces (cat dog) to search cat,dog or both. Separate the words with plus signs (cat +dog) to search for items that may contain cat but must contain dog.
  • You can further refine your search on the search results page, where you can search by keywords, author, topic. These can be combined with each other. Examples
    • cat dog --matches anything with cat,dog or both
    • cat +dog --searches for cat +dog where dog is a mandatory term
    • cat -dog -- searches for cat excluding any result containing dog
    • [cats] —will restrict your search to results with topic named "cats"
    • [cats] [dogs] —will restrict your search to results with both topics, "cats", and "dogs"

Crash when Deserializing TArray into Json

So I am currently creating functions that will allow me to take in an array of bytes through a Network Socket, deserialize it back into Json, and then Parse said Json. However, when I try to use the deserialize Function, the entire Engine Crashes, and I am given a link to send a crash report.Here's the .rar file to my crash report.

The Code Chunk that is most relevant is below:

 TArray<uint8> ReceivedData;
     uint32 Size;
     uint8 receivedData[100];
     int32 Read = 0;
 
         while (Socket->HasPendingData(Size))
         {
             ReceivedData.SetNumUninitialized(FMath::Min(Size, 65507u));
 
             Read = 0;
             Socket->Recv(ReceivedData.GetData(), ReceivedData.Num(), Read);
 
         }
 
         if (ReceivedData.Num() <= 0)
         {
             VShow("Error, the Server isn't streaming data");
         }
         FString output = StringFromBinaryArray(ReceivedData);
         FString debugData = BytesToString(receivedData, Read);
         Print("Got Message" + debugData);
         TSharedPtr<FJsonObject> JsonObject;
         TSharedRef< TJsonReader<> > Reader = TJsonReaderFactory<>::Create(debugData);
         //If the line below this is commented out, the program doesn't crash
                 bool result = FJsonSerializer::Deserialize(Reader, JsonObject);
         if (!result)
             Print("Error, Json Deserialization Failed");

My Implementation in my Blueprint currently Connects to the other IP Address, delays for a second, then calls the above function. I know that the Socket works fine, as I am getting a connection, along with the data.

The Json being sent is being sent like this, according to the developers of the Software I am trying to access:

 {"animationValues":
 {
     "mouth_rightMouth_stretch":0.0000000000000000,
     "mouth_leftMouth_narrow":0.00000000000000000,
     "mouth_up":0.0000000000000000,
     "mouth_leftMouth_stretch":0.0000000000000000,
     "mouth_rightMouth_narrow":0.00000000000000000,
     "mouth_down":0.00000000000000000,
     "mouth_upperLip_left_up":0.0000000000000000,
     "mouth_upperLip_right_up":0.0000000000000000,
     "mouth_lowerLip_left_down":0.0000000000000000,
     "mouth_lowerLip_right_down":0.0000000000000000,
     "mouth_leftMouth_frown":0.0000000000000000,
     "mouth_rightMouth_frown":0.0000000000000000,
     "mouth_leftMouth_smile":0.00000000000000000,
     "mouth_rightMouth_smile":0.00000000000000000,
     
     "eyes_lookRight":0.0000000000000000,
     "eyes_lookLeft":0.00000000000000000,
     "eyes_lookDown":0.0000000000000000,
     "eyes_lookUp":0.00000000000000000,
     "eyes_leftEye_blink":0.00000000000000000,
     "eyes_rightEye_blink":0.00000000000000000,
     "eyes_leftEye_wide":0.0000000000000000,
     "eyes_rightEye_wide":0.0000000000000000,
     
     "brows_leftBrow_up":0.0000000000000000,
     "brows_leftBrow_down":0.00000000000000000,
     "brows_rightBrow_up":0.0000000000000000,
     "brows_rightBrow_down":0.00000000000000000,
     "brows_midBrows_up":0.0000000000000000,
     "brows_midBrows_down":0.00000000000000000,
     
     "jaw_open":0.0000000000000000,
     "jaw_left":0.0000000000000000,
     "jaw_right":0.00000000000000000,
     
     "mouth_phoneme_oo":0.0000000000000000,
     "mouth_right":0.0000000000000000,
     "mouth_left":0.00000000000000000,
     "mouth_phoneme_mbp":0.0000000000000000,
     "mouth_phoneme_ch":0.0000000000000000
     
 },
 
 "headRot":[0.0,0.0,0.0]}

I have no reason to assume there is anything wrong on their side, as they offer a plugin to do the same job. However, it costs $1500, so that's out of the question. I don't really see how I can be doing anything wrong in this situation, as the code compiles correctly without any errors.

Product Version: UE 4.10
Tags:
more ▼

asked Mar 22 '16 at 09:47 PM in Bug Reports

avatar image

IrishMTS
10 3 6 7

avatar image IrishMTS Mar 23 '16 at 01:02 PM

Can I bump this? Its really important that I get this resolved quickly

avatar image Adam Davis STAFF Mar 23 '16 at 01:49 PM

Hi IrishMTS,

Please do not bump threads that have not gone unanswered for at least 4 days. This will give us time to take a look at the question and determine what may be occurring. If after 4 days you have not heard from anyone, please feel free to bump the thread once.

I am assigning a member of our support team to this bug so we can assist you further.

avatar image IrishMTS Mar 23 '16 at 01:55 PM

Perfect, sorry.

avatar image gmpreussner STAFF Mar 23 '16 at 06:01 PM

What is the callstack of the crash?

avatar image IrishMTS Mar 24 '16 at 02:11 PM

Where would I find that information?

avatar image gmpreussner STAFF Mar 24 '16 at 03:29 PM

In Visual Studio's "Call Stack" window, in the CrashReporter dialog window (if shown), or in the game's log file (please attach to this thread!)

avatar image IrishMTS Mar 24 '16 at 07:43 PM
fypproject.log (23.3 kB)
fypproject.log (876.5 kB)
avatar image gmpreussner STAFF Mar 24 '16 at 07:47 PM

Can you reattach the second file? It seems like it didn't make it through.

avatar image IrishMTS Mar 24 '16 at 07:49 PM
fypproject.log (876.5 kB)
avatar image gmpreussner STAFF Mar 24 '16 at 07:58 PM

Yes, but these don't look like the right logs. A new log file is created each time you run the Engine (Editor or Game). Can you please reproduce your crash and then send the newest log file in /ProjectName/Saved/Logs ?

avatar image IrishMTS Mar 24 '16 at 08:12 PM

That should be it there, must have clicked the wrong one by accident

fypproject.log (883.6 kB)
avatar image IrishMTS Mar 28 '16 at 04:24 PM

Hi gmpreussner. I've been playing around with perhaps using a different library to deserialise the Json, and while I was playing around seeing how I could fit it in, I found that I got the same crash, and from the looks of things, the same fatal error:

LogPhysics:Warning: PHYSX: ....\PhysX\src\NpScene.cpp (2946) 8 : PxScene::unlockWrite() called without matching call to PxScene::lockWrite(), behaviour will be undefined.

I've attached the function as it is now, and as can be seen, the deserialising functions have been commented out. I'm not actually sure anymore whats causing the crash, but I know that if this function isn't called, the crash doesn't occur. Here's the new log file

 void ANetworkPlayerController::ReceiveData()
 {
     TArray<uint8> ReceivedData;
     uint32 Size;
     //uint8 receivedData[1000];
     int32 Read = 0;
 
         while (Socket->HasPendingData(Size))
         {
             ReceivedData.SetNumUninitialized(FMath::Min(Size, 65507u));
 
             Read = 0;
             Socket->Recv(ReceivedData.GetData(), ReceivedData.Num(), Read);
 
         }
 
         if (ReceivedData.Num() <= 0)
         {
             VShow("Error, the Server isn't streaming data");
             //return;
         }
         FString output = StringFromBinaryArray(ReceivedData);
         const char *char_pointer2 = reinterpret_cast<char*>(ReceivedData[0]); 
         VShow("Is it working?", char_pointer2);
         UE_LOG(LogTemp,Log,TEXT("%d"),ReceivedData[0]);
         //TSharedPtr<FJsonObject> JsonObject;
         //TSharedRef< TJsonReader<> > Reader = TJsonReaderFactory<>::Create(output);
         //bool result = FJsonSerializer::Deserialize(Reader, JsonObject);
         //if (!result)
         //    Print("Error, Json Deserialization Failed");
 }


fypproject.log (21.1 kB)
avatar image gmpreussner STAFF Mar 28 '16 at 04:31 PM

The crash is in PhysX and has nothing to do with your code. It seems that you have a memory corruption somewhere. How is StringFromBinaryArray() implemented?

avatar image IrishMTS Mar 28 '16 at 04:35 PM

Its a simple reinterpret_cast, just the one line essentially:

 FString ANetworkPlayerController::StringFromBinaryArray(const TArray<uint8>& BinaryArray)
 {
     //Create a string from a byte array!
     std::string cstr(reinterpret_cast<const char*>(BinaryArray.GetData()), BinaryArray.Num());
     return FString(cstr.c_str());
 }


What's the next stage? Is this something that can be fixed at my end?

avatar image gmpreussner STAFF Mar 28 '16 at 05:24 PM

You need to figure out which part of your code is corrupting memory. What does VShow() do?

Also, why are you copying the string data? TJsonReader can process the data directly. Create a FMemoryReader archive around the TArray and then create a Json reader from it. Pseudo-code:

 FMemoryReader MemoryReader(ReceivedData.GetData());
 TJsonReader<> JsonReader = TJsonReader<>::Create(&MemoryReader);
avatar image IrishMTS Mar 28 '16 at 05:59 PM

VShow is a function that just sets up ClientMessage in a particular way, nothing particularly unusual there.

Having set up the code using an FMemoryReader archive, the code doesn't crash, but I think that's more because the deserialize function now returns false. Would you have any idea why that might be? I unfortunately don't know whether thats due to my code, or due to something on the other end, but I have been assured that all works fine on the server program.

The New code:

 void ANetworkPlayerController::ReceiveData()
 {
     TArray<uint8> ReceivedData;
     uint32 Size;
     //uint8 receivedData[1000];
     int32 Read = 0;
 
         while (Socket->HasPendingData(Size))
         {
             ReceivedData.SetNumUninitialized(FMath::Min(Size, 65507u));
 
             Read = 0;
             Socket->Recv(ReceivedData.GetData(), ReceivedData.Num(), Read);
 
         }
 
         if (ReceivedData.Num() <= 0)
         {
             VShow("Error, the Server isn't streaming data");
             //return;
         }
         FMemoryReader MemoryReader(ReceivedData);
         TSharedRef< TJsonReader<> > Reader = TJsonReader<>::Create(&MemoryReader);
         bool result = FJsonSerializer::Deserialize(Reader, JsonObject);
         if (!result)
             Print("Error, Json Deserialization Failed");
 }
avatar image gmpreussner STAFF Mar 28 '16 at 07:56 PM

false usually indicates that the Json string is malformed. You can validate it here: http://jsonlint.com/

If the string is OK, you'll have to step through the code to see what's happening.

avatar image IrishMTS Mar 30 '16 at 04:31 PM

I'm wondering, whats the likelihood that I am taking too small a packet size in as the minimum before reading it? Is there anything in that function that might suggest that the Json is getting cut off in the middle somewhere?

avatar image gmpreussner STAFF Mar 30 '16 at 04:54 PM

What kind of socket is this? UDP? You may have to reassemble the string if it is longer than the buffer size.

Should be easy to test. Just log out the string.

avatar image IrishMTS Mar 30 '16 at 05:30 PM

This is the new Output I'm getting

I've also tried to ensure that it is Null-Terminated by altering adding a line to the streaming loop:

         while (Socket->HasPendingData(Size))
         {
             ReceivedData.SetNumUninitialized(Size);
 
             Read = 0;
             Socket->Recv(ReceivedData.GetData(), ReceivedData.Num(), Read);
 //I'm hoping that the below will null terminate the string, if it isn't already
             ReceivedData.Add((uint8)0);
             Print(StringFromBinaryArray(ReceivedData));
 
         }

capture.png (3.0 kB)
avatar image IrishMTS Mar 30 '16 at 05:37 PM

Its a TCP Socket, forgot to say that in the previous comment

avatar image gmpreussner STAFF Mar 30 '16 at 05:44 PM

Are you certain that the problem is caused by this particular code? Perhaps comment out everything and then add lines back in one by one. You should definitely log out the received Json string, and you should also step through the Json serializer to see where it fails.

avatar image IrishMTS Mar 30 '16 at 07:18 PM

I really don't understand what I'm doing anymore. I think I may have been trying to access the Json Object before it was filled in the deserialize function. Thats the explanation I have now, given that the code seems to run fine (even though the deserialise function returns -1).

However, while I was stepping through the the code, I got a weird number of strings being printed. It seems that I am getting the Json String coming in, but its in pieces, rather than full size that I need. Given the length of the Json Object that I am attempting to write in, what size should my buffer be? I'm not really familiar with low level things like that, its not really my expertise

avatar image gmpreussner STAFF Mar 30 '16 at 07:36 PM

TCP will reassemble the packets for you. The buffer size should be at least as large as the maximum expected Json string size (plus one, if the sender null-terminates it).

Is your server perhaps sending the Json string as multiple TCP packets? In that case you would have to reassemble the string yourself prior to parsing it.

avatar image IrishMTS Mar 30 '16 at 07:38 PM

I'm not sure how to determine what is a part of the string, and what is just noise. And how do I know when I've gotten the last packet? Again, networking really isn't my strong point, I'm quite lost in this

avatar image gmpreussner STAFF Mar 30 '16 at 07:54 PM

ReceivedData.Add((uint8)0);

This is not going to work. You are adding a zero to the end of the buffer, not the end of the received string (which may be less than the size of the buffer).

You probably want this:

 ReceivedData[Size] = 0;

Or you can null-terminate on the sender's side if you have control over the server's code.

avatar image gmpreussner STAFF Mar 30 '16 at 07:58 PM

Actually, do this instead:

 ReceivedData->RemoveAt(Read, ReceivedData->Num() - Read, false);

The Json reader may not check for null-termination.

avatar image IrishMTS Mar 30 '16 at 09:18 PM

Even with that line, the code is still spitting out that character set from above as if to say I don't know whether this is actually a string or not :(

avatar image gmpreussner STAFF Mar 30 '16 at 09:26 PM
 GLog->Logf(TEXT("Json string: %s"), (TCHAR*)ReceivedData->GetData);

avatar image IrishMTS Mar 30 '16 at 09:36 PM

Its still coming up with that undefined sign. alt text What does that refer to exactly? I don't understand what it means?

capture.png (22.9 kB)
avatar image gmpreussner STAFF Mar 30 '16 at 09:52 PM

Looks like a non-ASCII character.

Do this:

TCHAR* MyString = (TCHAR*)ReceivedData->GetData();

Then set a breakpoint on it and check the value in the Visual Studio debugger.

avatar image IrishMTS Mar 30 '16 at 11:13 PM

Yeah, there isn't anything ASCII about what I'm getting. Is there a chance that there's some sort of encoding to ensure there isn't any null characters inside the stream? Might that be changing the values?

avatar image gmpreussner STAFF Mar 30 '16 at 11:24 PM

Is the server reachable on the internet, or is it some program you're running locally? You could take a look at the TCP packets with Wireshark.

avatar image IrishMTS Mar 30 '16 at 04:55 PM

Okay, I spoke too soon. Opened it up today and its still crashing. Is Buffer Overflow something that I may have to be worried about? Does the string that I'm receiving cause memory corruption perhaps?

avatar image gmpreussner STAFF Mar 30 '16 at 05:00 PM

If the string is not null-terminated, the Json reader may read beyond the buffer length for sure.

(comments are locked)
10|2000 characters needed characters left

1 answer: sort voted first

To fix the errors that I had, I changed my code to the following:

 void ANetworkPlayerController::ReceiveData()
 {
     TArray<uint8> ReceivedData;
     uint32 Size;
     int32 Read = 0;
 
         while (Socket->HasPendingData(Size))
         {
             ReceivedData.SetNumUninitialized(FMath::Min(Size, 65507u));
 
             Read = 0;
             Socket->Recv(ReceivedData.GetData(), ReceivedData.Num(), Read);
             ReceivedData.RemoveAt(Read, ReceivedData.Num() - Read, false);
             TCHAR* MyString = (TCHAR*)ReceivedData.GetData();
             Print(MyString);
             GLog->Logf(TEXT("Json string: %s"), (TCHAR*)(ReceivedData.GetData()));
             Print(StringFromBinaryArray(ReceivedData));
 
         }
 
         if (ReceivedData.Num() <= 0)
         {
             VShow("Error, the Server isn't streaming data");
         }
         FMemoryReader MemoryReader(ReceivedData);
         TSharedRef< TJsonReader<> > Reader = TJsonReader<>::Create(&MemoryReader);
         bool result = FJsonSerializer::Deserialize(Reader, JsonObject);
         if (!result)
             Print("Error, Json Deserialization Failed");
 }

The main thing that was going wrong I believe was my setup of the JsonReader. Instead of converting to a string, and putting that through the reader, I set up an FMemoryReader object instead. That did stop the code from crashing.

However, my problem is still there, but its not my fault, aside from the fact that I was trying to access the Deserialised values, even when they didn't exist since the function had failed. That's to do with the software I was trying to interface with, nothing I can do now.

more ▼

answered Apr 02 '16 at 01:30 PM

avatar image

IrishMTS
10 3 6 7

(comments are locked)
10|2000 characters needed characters left
Your answer
toggle preview:

Up to 5 attachments (including images) can be used with a maximum of 5.2 MB each and 5.2 MB total.

Follow this question

Once you sign in you will be able to subscribe for any updates here

Answers to this question