HI all,

For an optimization problem that is a weighted combination of cross entropy 
minimization and sum of least squares, are there good solvers specially 
designed for such problems? I've seen entropy minimization solvers around, 
and in MOSEK, etc., which made me thought there indeed exist solvers that 
specially tuned for the entropy minimization problems. Could anybody educate 
me and my friend what's the advantange for a solver that is specially for 
entropy minimization problems?

And I also knew that there are solvers specially for non-linear least 
squares. Again, could anybody tell us what's special about a non-linear 
least square solver?

But I didn't see solvers that handle a weighted combination of both. Are 
there? Please give us some pointers! Thanks a lot!

On Jul 19, 6:30 am, "Linus Utopia" <linus_uto...@gmail.com> wrote:
> HI all,
>
> For an optimization problem that is a weighted combination of cross entropy
> minimization and sum of least squares, are there good solvers specially
> designed for such problems? I've seen entropy minimization solvers around,
> and in MOSEK, etc., which made me thought there indeed exist solvers that
> specially tuned for the entropy minimization problems. Could anybody educate
> me and my friend what's the advantange for a solver that is specially for
> entropy minimization problems?

The engine in MOSEK is general optimizer. Well, it is specialized for
convex
problems. The entropy stuff in MOSEK is just a very nice interface for
entropy problems.

>
> And I also knew that there are solvers specially for non-linear least
> squares. Again, could anybody tell us what's special about a non-linear
> least square solver?
>
> But I didn't see solvers that handle a weighted combination of both. Are
> there? Please give us some pointers! Thanks a lot!

Nonlinear lest squares are not convex problems and hence that kind of
problems cannot be solved
by MOSEK.

"edadk" <e.d.andersen@mosek.com> wrote in message 
news:1184921975.697914.114780@r34g2000hsd.googlegroups.com...
> On Jul 19, 6:30 am, "Linus Utopia" <linus_uto...@gmail.com> wrote:
>> HI all,
>>
>> For an optimization problem that is a weighted combination of cross 
>> entropy
>> minimization and sum of least squares, are there good solvers specially
>> designed for such problems? I've seen entropy minimization solvers 
>> around,
>> and in MOSEK, etc., which made me thought there indeed exist solvers that
>> specially tuned for the entropy minimization problems. Could anybody 
>> educate
>> me and my friend what's the advantange for a solver that is specially for
>> entropy minimization problems?
>
> The engine in MOSEK is general optimizer. Well, it is specialized for
> convex
> problems. The entropy stuff in MOSEK is just a very nice interface for
> entropy problems.
>
>>
>> And I also knew that there are solvers specially for non-linear least
>> squares. Again, could anybody tell us what's special about a non-linear
>> least square solver?
>>
>> But I didn't see solvers that handle a weighted combination of both. Are
>> there? Please give us some pointers! Thanks a lot!
>
> Nonlinear lest squares are not convex problems and hence that kind of
> problems cannot be solved
> by MOSEK.
>


Why is minimization of nonlinear sum of squares not a convex problem?

So you are saying that our problem of minimization of "w1*cross_entropy + 
w2*sum_of_squares" is not a convex problem?

Thanks! 

Linus Utopia wrote:
> "edadk" <e.d.andersen@mosek.com> wrote in message 
> news:1184921975.697914.114780@r34g2000hsd.googlegroups.com...
>> On Jul 19, 6:30 am, "Linus Utopia" <linus_uto...@gmail.com> wrote:
>>> HI all,
>>>
>>> For an optimization problem that is a weighted combination of cross 
>>> entropy
>>> minimization and sum of least squares, are there good solvers specially
>>> designed for such problems? I've seen entropy minimization solvers 
>>> around,
>>> and in MOSEK, etc., which made me thought there indeed exist solvers that
>>> specially tuned for the entropy minimization problems. Could anybody 
>>> educate
>>> me and my friend what's the advantange for a solver that is specially for
>>> entropy minimization problems?
>> The engine in MOSEK is general optimizer. Well, it is specialized for
>> convex
>> problems. The entropy stuff in MOSEK is just a very nice interface for
>> entropy problems.
>>
>>> And I also knew that there are solvers specially for non-linear least
>>> squares. Again, could anybody tell us what's special about a non-linear
>>> least square solver?
>>>
>>> But I didn't see solvers that handle a weighted combination of both. Are
>>> there? Please give us some pointers! Thanks a lot!
>> Nonlinear lest squares are not convex problems and hence that kind of
>> problems cannot be solved
>> by MOSEK.
>>
> 
> 
> Why is minimization of nonlinear sum of squares not a convex problem?
> 
> So you are saying that our problem of minimization of "w1*cross_entropy + 
> w2*sum_of_squares" is not a convex problem?

This is convex if w1 is negative and w2 is positive, and the terms 
inside the square are linear (i.e. the the least-squares term actually 
is something like (A*x-b)'*(A*x-b)

What Erling means is that the general nonlinear least-squares problem 
with terms of the type (f(x) - b)'*(f(x)-b) typically is non-convex

Using a dedicated solver basically means that the solver can take 
short-cuts when computing gradients and Hessians etc, since these are 
known. More-over, a dedicated solver often work in the primal-dual 
space, which typically yields better performance. Of course, you can get 
pretty good performance using a general nonlinear solver also, if you 
supply code for gradients and such.