Why does deleting the double pointer in this code cause garbage collection crashes?

Here’s how my code works:
Every frame, I record a pointer to an actor, to a pointer in a vector.
When I choose, all the pointers in this vector, and their corresponding frame, now indirectly point to the same actor, because they were all pointing to the middleman pointer, which now points to an actor.
I then make a new middleman pointer that my upcoming pointer-pointers can point to, so that previous frames can keep their reference to one actor and upcoming frames have their new actor referenced.

Now, because I’m making new pointers on the heap, I should de-allocate them with delete, though I do not need to delete the actors they are pointing to. I just need the middleman pointers themselves de-allocated, and the pointer-pointers can just go out of scope because they reside in a local vector.

But the problem is that I seem to be deleting random memory in the engine; It will crash some time, or immediately, but never in the same place, though always because of access violations.

AllocateArrays() is called on begin play,
DeleteDataArrays() is called on end play,
SetHeldActor() is called whenever,
UnsetHeldActor() is called whenever,
and RecordData() is called every frame, with its parameter being the frame passed in.

Uncommenting the delete parts of the code will cause crashes, no matter which one.

.cpp:

#include "Traveler.h"
#include "TimeTravelManager.h"
#include "TimeTravelComponent_Pickuppable.h"

class TimeTravelManager;

void UTimeTravelComponent_Pickuppable::RecordData(int32 FrameToRecord)
{
	UE_LOG(GameInfo, Log, TEXT("Pickuppable: recording frame %i"), FrameToRecord);
	if (CurrentActorP == nullptr)
	{
		UnsetHeldActor();
	}
	ActorsHolding[FrameToRecord] = CurrentActorP;
}

void UTimeTravelComponent_Pickuppable::AllocateArrays(int32 size)
{
	ActorsHolding = vector<AActor**>(size);
	UnsetHeldActor();
}

void UTimeTravelComponent_Pickuppable::DeleteDataArrays()
{
	
	//all uses of this variable make sure that a pointer isnt deleted more than once.
	AActor* LastDeletedPointer = nullptr;

	for (int32 i = 0; i < ActorsHolding.size(); i++)
	{
		if (ActorsHolding[i] != nullptr)
		{
			if (*ActorsHolding[i] != LastDeletedPointer)
			{
				//delete LastDeletedPointer;
				LastDeletedPointer = *ActorsHolding[i];
				//delete ActorsHolding[i];
			}
			ActorsHolding[i] = nullptr;
		}
	}
	//delete LastDeletedPointer;
}

void UTimeTravelComponent_Pickuppable::SetHeldActor(AActor* Holder)
{
	//if (CurrentActorP == nullptr)
	{
		//unsetting will make a new pointer for the pp to point to
		UnsetHeldActor();
	}
	//the pointer is now equal to the pointer to the actor
	*CurrentActorP = Holder;
}

void UTimeTravelComponent_Pickuppable::UnsetHeldActor()
{
	//the pointer is now pointing to a new pointer, which will later point to the actor.
	CurrentActorP = new AActor*();
	*CurrentActorP = nullptr;
}

.h:

#pragma once

#include "TimeTravel/TimeTravelComponent.h"
#include "TimeTravelComponent_Pickuppable.generated.h"

/**
 * 
 */
UCLASS(ClassGroup = (Custom), meta = (BlueprintSpawnableComponent))
class TRAVELER_API UTimeTravelComponent_Pickuppable : public UTimeTravelComponent
{
	GENERATED_BODY()
	
	

public:

	virtual void AllocateArrays(int32 size) override;
	//virtual void PlaybackData(int32 FrameToPlayBack) override;
	virtual void RecordData(int32 FrameToRecord) override;
	virtual void DeleteDataArrays() override;
	UFUNCTION(BlueprintCallable, Category = "Pickups")  void SetHeldActor(AActor* Holder);
	UFUNCTION(BlueprintCallable, Category = "Pickups")	void UnsetHeldActor();
	virtual void Erase(int32 Frame) override;

	UFUNCTION(BlueprintCallable, Category = "Pickups")	bool CanBePickedUp();

	UFUNCTION(BlueprintCallable,BlueprintPure, Category = "Pickups")	AActor* GetTemporalActor();
	UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "Pickups")   bool IsTemporal;
	
protected:
	vector<AActor**> ActorsHolding;
	AActor** CurrentActorP;
	AActor** MostRecentOverWrittenP=nullptr;
	bool IsThereATemporalActor=false;
};

There are some other functions in this class which do stuff with the pointers, but they were not called during testing and isolation of the problem.

Issue was elsewhere in code.