Hi Jeremy!
If you are someone who has recently jumped into blueprinting without having any previous experience with programming in general - or object-oriented programming (OOP) in particular - from using something like C++, Java, Python or (the list goes on) then I would imagine that a lot of this stuff must be very confusing and frustrating at times.
A full explanation would require a rather lengthy article, or indeed an entire book, and this isn’t really your typical AnswerHub material… but I’m having a slow morning here nursing a cup of tea, so I’ll try to clear some things up for you. Hopefully 
The main source of your problem is mostly down to the exact definition of what a Class is, versus what an Object (or an Instance) of a class is. It’s also about a core OOP concept known as Inheritance.
If we look at this from a Blueprint perspective: when you create a new Blueprint, let’s call it Robert, you are declaring a new Class. This class declaration, i.e. your blueprint, describes what a specific Robert could possibly know (data/variables) and what he could possibly do (methods/functions) if he is ever brought into existence.
That’s one key distinction: a blueprint class in itself will remain purely hypothetical and ethereal until an actual instance of it is created. As soon as you spawn a Robert, either by placing him in your level, or by dynamically spawning him at a later time in response to some in-game event, then you have an actual Robert that you can communicate with, but not until then.
Prior to spawning any Robert-instances, he will remain nebulous and impossible to talk to because he doesn’t exist yet. However, once you have spawned one or several Roberts, they are thought of as being objects or instances, or, in Blueprint lingo, references of the Robert type.
Next up we should look at inheritance. When you created the Robert blueprint, he was equipped with a lot of data and methods that you didn’t have to create yourself. If the blueprint was based on the built-in class called Actor, it means that any instance of the Robert class is not only that, but also an Actor. If it was based on Pawn, then any instance of your class is a Robert as well as a Pawn and also an Actor (again, because the Pawn class itself inherits from the Actor class). This means that any Robert-type object has all the capabilities that you have defined in your class, plus all the other capabilities inherited from Pawn and Actor. The most obvious way to think about this is to imagine a tree of different phyla, families, species and subspecies of animals: so… this dog I have next to me right now is a whippet. She’s also a canis domesticus, a mammal, a chordate, a tetrapod and an animal. A long list of classes upon classes, each one bringing in a specific set of abilities and properties. This means that I could theoretically cast my dog to a tetrapod or a chordate, but I would not be able to cast her to an arthropod for example, because she is not related to that class at all.
And that’s where casting comes in! Yes, this is going somewhere, mark my words…
Let’s say you have a reference to something in your level, that is a variable with some specific value identifying one particular actor that has been spawned somewhere in the world (that’s not a class per se, but an instance of a class). You know for sure that it is an Actor, on the most fundamental level, but you do not know what specific type it is. Let’s further assume that your blueprint logic needs to assume that it is in fact a Robert type actor, because it needs to call a certain function that only Roberts have.
The solution here is to use a Cast node. All that node does is simply to check the reference that you give it, look through its inheritance tree to see whether the Robert class is involved somewhere or not. If it is, then the cast is successful, and your Actor reference is now converted to a Robert reference, which in turn means you can call any function or look at any variable that has been defined in your Robert blueprint. On the other hand, if the original actor reference was spawned from some other class, say the DefaultPawn, then the cast will fail and you will not be able to call any Robert-specific functions on it, because, of course, it’s not actually a Robert at all.
You will have noticed by now that we always need a reference to a specific instance of a class in order to do any sort of casting. The class itself is not enough.
Anyway… before this goes totally out of hand - your specific questions:
I made a pawn blueprint class. Why can I drag a reference of that to my level blueprint and not any other blueprints?
The level blueprint is a bit special in that it can be made aware of all the objects that you have placed in your map at design time. So if you place an instance of your pawn in the level, the reference to it is already valid and known to the scene graph, which in turn is known by the level blueprint. All other blueprint types rely on a more dynamic approach. In a typical game, many actors are spawned as the game is played rather than being statically placed in the map from the beginning: enemies and projectiles are spawned and destroyed. That means that only the classes exist at the start, but there are no instances of them yet.
A typical Actor-based blueprint class, like your Pawn or Robert above, has no inherent knowledge of the world it will exist in, what level it might be spawned in or anything like that. An actor instance by default knows only its own functions and variables, and those that it has inherited from its base class. If you need it to know anything about some other actor, you must explicitly pass the information to it, typically by sending it a reference to some specific actor instance that you want it to be aware of or talk to.
Why do we have to reference something when we do casting? Why isn’t it enough to just go in blueprint2 and say “casting to blueprint1” and then get some variables from blueprint1?
This was hopefully answered in the general overview above, but the short summary is that without a reference, there is nothing to test for inheritance, so nothing can be cast.
On a tangent here though: in 4.9 (or thereabouts) there will apparently be a way to get access to the default values of some other blueprint class, presumably without explicitly spawning any instances, using a GetClassDefaults node. That could be helpful in some cases, and would not require a reference (if I understood things correctly, but I may very well be wrong) but it still wouldn’t be the same as having access to a specific, live instance of a particular class.
The only thing I can really cast to successfully is my player character
In UE, there is a bunch of classes that are referred to as Framework classes. These are fundamental classes of which at least one (sometimes exactly one) instance exists from the moment the game starts. These include GameMode, PlayerController and a couple of others, and anything related to them is typically always accessible from anywhere because they are created or generally available immediately after the game starts, before any other actors are spawned. The player character (technically the default pawn as specified by your GameMode) is therefore one of those things that you can easily get a reference to, and thereby the ability to cast to, just about anywhere. But that is an exception, and an ability that only the framework classes provide.
Generally speaking, you can successfully cast to anything anywhere, but only if 1) you have a valid reference to an instance of the class you want to cast to, and 2) it really is an instance of the class you are trying to cast to, or it is based on the class you are trying to cast to.
In one blueprint, I have a class array with 20 references in it. In that same blueprint I generate a thing with 7 randomly chosen instances from the class array. I want to make a widget that shows that randomly generated thing from my other blueprint. How can I find that array?
So, in this case you’ll probably see by now that you need to make at least one of these objects aware of the other, by a reference to the specific instance of the class you want to talk to. I don’t know exactly what you are trying to do here, so this will have to be a bit general:
You could go one way or the other here, either the blueprint with the 20 references (B20) would need a reference to the Widget, so it can call some function it has defined, say ShowTheseSevenThings(), or the Widget would need a reference to the B20 so it can call some function it has defined, say GiveMeSevenThings(). It’s either way.
In this case, someone must have created the Widget and placed it in the viewport. When that happens, it should have saved a reference to the widget. That reference could be passed on to the B20 instance, which in turn could use that to call the Widget and tell it to ShowTheseSevenThings(). All of it depends on an unbroken chain of references. Any actor that creates something should save a reference to what it has created so it can communicate with it or pass the reference to someone else for that same purpose.
As soon as some actor, any actor, has a reference to the B20 instance, it can easily cast to the B20 type and then just grab that array you were looking for. But if no such reference exists, you’re dead in the water.
A general approach (again, I don’t know what we’re talking about specifically, so this is a bit fuzzy) is to use some globally accessible class to handle the spawning of things like this. For example, in your scenario we are creating one object of the B20 type somewhere, and one Widget. Who should do this? One way is to let the PlayerController do it, maybe in response to some player input. If you do this, and make sure that the PlayerController saves the references to the things it spawned in a public, editable variable in your playercontroller blueprint, then any other actor can easily get the information because you can call GetPlayerController anywhere and cast that to your specific PlayerController class, at which point you can get the references it has stored for your Widget and your B20. Then they can all talk as much as they like, via those references once they have been cast to the proper type.
Oh well. I don’t know if this has helped you in any way, but my teacup is empty (had it refilled once though) and I have to get back to work.
Hopefully some things are clearer (or I’ve made it worse!)
At any rate, Blueprints are awesome and useful, but unfortunately they do not in any way negate the need to learn and understand the fundamental principles of programming in general. It’s just the way it is.