Yes you should understand backprop

Andrej Karpathy on backpropagation: "The problem with Backpropagation is that it is a leaky abstraction. In other words, it is easy to fall into the trap of abstracting away the learning process — believing that you can simply stack arbitrary layers together and backprop will “magically make them work” on your data. So lets look at a few explicit examples where this is not the case in quite unintuitive ways."