Actor not relevant on server? Ownership/relevancy question

Hey guys. Its a larger question about networking I guess but I’ll present you my current situation. Still not entirely familiar with how networking works so bear with me.

Setup

I spawn a character for every controller (listen server scenario, so listen server and client get a char)
The character, in his bp, spawns a sword actor for himself behind an authority node (only on server, replicates to client)

Both sword and character are set to replicate and replicate movement.

IE. both players get a char and a sword, both created on server and replicated from server to client afaik.

My sword actor has a little collision box with physics.

So in this case, I attach a keypress that goes to a Run On Server event, which asks the “Get Physics Linear Velocity” of the sword (could be anything, I dont think this is where the problem lies) and prints it
The sword is moving with the idle animation because its tied to the skeleton of the character, so I should 100% get non-zero results.

Result

When I press the button on listen server window for the “listen servers chars sword” - I get correct results always.
When I press the button on the client, to ask the server what it thinks the velocity of the servers version of the client sword is, it returns 0 every now and then, and correct results other times. (1)
However, if I move my client character away from the “listenserver char”, or out of its vision (screen), it returns 0 always. (2) I’m assuming the latter has something to do with relevancy, but I dont get it.

1:

http://jordylakiere.com/upload/2016-04-24_11-33-16.png

2:

http://jordylakiere.com/upload/2016-04-24_11-34-14.png

So whats going on? Can someone ELI5 cause at this point I feel like I need that, heh. Shouldnt the server always be aware of everything? (ie no relevancy?)
Also can someone explains the steps that lead the 1st case scenario to return 0 every now and then?

Thank you so much, and if you need to see additional settings or bp or whatever, let me know.

Bump. (if that even works here) Anyone?