Consider the following 

x = 

-0.014724809271224 - 0.032408729988139i
 -0.000051794067055 + 0.000180228381251i
 -0.001150168703202 + 0.000733021908260i
  0.091778665639598 - 0.018733464111110i


A =  

 0.936888141790930 - 3.265019222175327i -0.994182138515699 - 3.064171560291264i
 -2.320204184527999 - 2.031168825411744i -0.130894036298698 - 3.296419190620100i
 -2.794653571146645 + 0.681212238406362i -2.876547877517820 - 0.778153738257779i
 -2.320204184527999 - 2.031168825411744i -2.930099068708969 - 0.115359046230159i

  Columns 3 through 4

 -2.803194782024307 - 1.075143068953146i -2.705065559467416 - 1.351928081115106i
 -0.602175391335523 - 3.200764679266519i -2.698088177121359 - 1.368820141257398i
 -1.196777838674177 - 2.970647939434055i -1.360680336252638 - 2.882107961949898i
 -2.907617880833896 - 0.584450977019415i -1.379176513445659 - 2.871338370331258i


b =

 -0.388588235529978 - 0.056528157461146i
 -0.301292991849427 + 0.033406652197330i
 -0.111802047441108 - 0.156423963924776i
 -0.208086913152924 - 0.134568648508758i


given A and x we can calculate b

b=A*x

now we should be able to go back given our newly calculated b and A from before to solve for x

x=A\b

the original x and the new x should agree.....................

However this only works for very small cases like the one shown above, when I start to increase the N size of the matrix where A is N by N then I have to add noise to make things work out properly. (Note the matrices will look similar to the 4 by 4 case above but just much larger size)

as such

A=A + (10^-6 + 1i*10^-6)* eye(size(A));

then 

b=A*x

then 

x=A\b

so that both the original x and newly calculated x agree, without adding the noise the two x's do not agree at all

Could someone help explain to me more clearly what is going on here?

I recently tried using factorize by Tim Davis http://www.mathworks.com/matlabcentral/fileexchange/24119-dont-let-that-inv-go-past-your-eyes-to-solve-that-system-factorize

instead so replaced x=A\b by x=factorize(A)\b but still no such help however this did remove the rcond stable or ill conditioned error