> but when it comes to interpolate the image to register for the cost function value calculation, i could not use linear or bicubic it wouldn't make sense having, for example, .5 bit values...
Why not? The interpolation is only meant to serve the registration operation.
In any case, some non-iterative methods you might want to try, discussed in previous threads, are based on the Fourier and Radon transforms ( the latter requires the Image Proc. Toolbox)
If your two images are X and Y, then you can estimate the rotation by comparing
One of them will appear shifted with respect to the other in an amount corresponding to the rotation angle. By measuring the shift using your favorite method (e.g. cross-correlation), you get an estimate of rotation.
After undoing rotation, you can like at the phase plots to get the translation
or you can do a centroid measurement.
Another way to estimate rotation is to calculate an edge map of your two images and take their gradient vectors [gx,gy]using diff(). If you then histogram the angles of the gradient vectors atan2(gy,gx), you will see a shift in the histogram of one image with respect to the other. The shift again corresponds to the rotation angle and can again be extracted using your favorite method.