See comments/questions below.<div><br></div><div>Ross<br><br><div class="gmail_quote">On Sat, Jun 29, 2013 at 4:13 PM, Brian G. Peterson <span dir="ltr"><<a href="mailto:brian@braverock.com" target="_blank">brian@braverock.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
You're absolutely on the right track, except for this one part:<br>
* if a constraint is violated, omit that set of weights<br>
<br>
Depending on your constraints, your chance of constructing a valid portfolio may be very very slim.<br>
<br>
In random portfolios, we could try this, just keep drawing until you get ones that match, but this could take forever.<br>
<br>
In stochastic solvers, you typically only have support for box constraints, so again your chance of having truly feasible vectors is pretty small in many cases.<br>
<br>
This leads to the need for a transformation function, to bring a random/stochastic weights vector into compliance, or close, with all/most of your constraints.<br>
<br>
In DEoptim, we added the possibility of fnMap to be called directly on each generation's population, to 'fix' the random population and give it a full population.<br>
<br>
In PortfolioAnalytics, we can support an 'fnMap'-like transform directly in DEoptim and in random portfolios.<br></blockquote><div>[Ross] Would the while loops in randomize_portfolio have to be modified for additional constraints or is it simple enough to apply the fnMap function at the end of the function to modify the portfolio that is returned?</div>
<div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
We can support it *indirectly* for any arbitrary solver inside constrained_objective (as long as we can store the transformed weight vector to know what the objective was actually calculated on, which we can get to later).<br>
<br>
So i think we still need to write a fnMap transform function.<br>
<br>
some other notes:<br>
You're correct that volatility is usually dealt with as an objective. In fact, since it requires returns, I only think of this as an objective.<br></blockquote><div>[Ross] I added target volatility as a constraint based on the email from Peter Carl, but planned to implement it like an objective in constrained_objective. Is this ok or should I remove volatility as a constraint type?</div>
<div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Diversification I need to look at and see what you're calling 'diversification' to see if that requires the returns, or if it is just a measure of *weights* concentration, that word can mean either.<br></blockquote>
<div>[Ross] I am calculating diversification as just a measure of weights concentration.</div><div>diversification <- function(weights){</div><div> return(1 - sum(weights^2))</div><div>} </div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Turnover can be handled either as an objective or as a constraint.<br></blockquote><div>[Ross] Great, I added support for both.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Penalization is likely necessary in constrained_objective when we have to relax constraints somewhat to get to anything feasible, hopefully Doug can provide more guidance here, as the literature on relaxing constraints is pretty thin, to the best of my knowledge.<br>
<br>
Regards,<br>
<br>
Brian<div><br>
<br>
On 06/29/2013 03:32 PM, Ross Bennett wrote:<br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>
Brian,<br>
<br>
Thanks for sharing the paper and your ideas for the mapping function.<br>
<br>
One of the main point I got from the methodology described in the paper<br>
is that the sets of weights are omitted instead of transformed if they<br>
do not meet the constraints.<br>
<br></div>
* generate random portfolios<br>
* test each set of portfolio weights<br>
* if a constraint is violated, omit that set of weights<br>
* compute the objective function on each remaining set of weights<br>
* select the set of optimal weights<div><br>
<br>
Is this what you were getting at here?<br>
"A slightly more rigorous treatment of the problem is given here:<br>
</div><a href="http://papers.ssrn.com/sol3/__papers.cfm?abstract_id=1680224" target="_blank">http://papers.ssrn.com/sol3/__<u></u>papers.cfm?abstract_id=1680224</a><div><br>
<<a href="http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1680224" target="_blank">http://papers.ssrn.com/sol3/<u></u>papers.cfm?abstract_id=1680224</a><u></u>><br>
It is possible that can use this method directly for random portfolios."<br>
<br>
I could add a block of code at the end of random_portfolios that tests<br>
each set of weights against the constraints and only keeps the weights<br>
that satisfy the constraints. Thoughts?<br>
<br>
If we have to transform the weights, here are my thoughts looking at the<br>
specific cases for constraint types<br>
Constraint Type<br>
<br></div>
* Leverage (min_sum and max_sum)<br>
o This is done in randomize_portfolio by randomly permuting and<div><br>
increasing or decreasing an individual element (asset weight)<br>
until min_sum and max_sum constraints are satisfied while taking<br>
into account box constraints. These constraints are satisfied<br>
based on the way random portfolios are constructed.<br></div>
o This is done in constrained_objective by transforming the entire<br>
vector (<br>
+ if(sum(w)>max_sum) { w<-(max_sum/sum(w))*w } # normalize to<br>
max_sum<br>
+ if(sum(w)<min_sum) { w<-(min_sum/sum(w))*w } # normalize to<br>
min_sum<br>
o Implement by moving this into the mapping function... correct?<br>
* Box (min and max)<br>
o This is done in randomize_portfolio by construction<br>
o This is done in constrained_objective by penalizing weights<br>
outside box constraints.<br>
o Implement by using logic from randomize_portfolio to transform<div><br>
weights instead of penalizing them. Goal is to transform the<br>
weights vector instead of penalize... correct?<br></div>
* Group (cLO and cUP)<br>
o One approach is to normalize the weights in each given group<div><br>
that violate cLO or cUP so that the group weights sum to cLO or<br>
cUP. This changes the sum of weights, so when the weights vector<br>
is normalized the group constraints will likely be violated, but<br>
it gets us close. See sandbox/testing_constrained_<u></u>group.R<br></div>
o Another approach is to add this to randomize_portfolio so the<div><br>
group constraints as well as box and leverage are satisfied by<br>
construction. Need to spend more time understanding code in<br>
randomize_portfolio to see how feasible this is.<br></div>
* turnover<br>
o Could we include this in constrained_objective as a penalty?<br>
* diversification<br>
o Could we include this in constrained_objective as a penalty?<br>
* volatility<br>
o Could we include this in constrained_objective as a penalty?<br>
* position_limit<br>
o This may be able to be implemented in randomize_portfolio by<div><br>
generating portfolios with the number of non-zero weights equal<br>
to max.pos, then fill in weights of zero so the length of the<br>
weights vector is equal to the number of assets, then scramble<br>
the weights vector. The number of non-zero weights could also be<br>
random so that the number of non-zero weights is not always<br>
equal to max.pos. This could be implemented in the DEoptim<br>
solver with the mapping function. This might be do-able in Rglpk<br>
for max return and min ETL. Rglpk supports mixed integer types,<br>
but solve.QP does not. May be able to use branch-and-bound<br>
technique using solve.QP, but needs more research.<br>
<br>
<br>
Regards,<br>
Ross<br>
<br>
<br>
On Sat, Jun 29, 2013 at 8:45 AM, Brian G. Peterson <<a href="mailto:brian@braverock.com" target="_blank">brian@braverock.com</a><br></div><div><div>
<mailto:<a href="mailto:brian@braverock.com" target="_blank">brian@braverock.com</a>>> wrote:<br>
<br>
Based on side conversations with Ross and Peter, I thought I should<br>
talk a little bit about next steps related to the mapping function.<br>
<br>
Apologies for the long email, I want to be complete, and I hope that<br>
some of this can make its way to the documentation.<br>
<br>
The purpose of the mapping function is to transform a weights vector<br>
that does not meet all the constraints into a weights vector that<br>
does meet the constraints, if one exists, hopefully with a minimum<br>
of transformation.<br>
<br>
In the random portfolios code, we've used a couple of techniques<br>
pioneered by Pat Burns. The philosophical idea is that your optimum<br>
portfolio is most likely to exist at the edges of the feasible space.<br>
<br>
At the first R/Finance conference, Pat used the analogy of a<br>
mountain lake, where the lake represents the feasible space. With a<br>
combination of lots of different constraints, the shore of the lake<br>
will not be smooth or regular. The lake (the feasible space) may<br>
not take up a large percentage of the terrain.<br>
<br>
If we randomly place a rock anywhere in the terrain, some of them<br>
will land in the lake, inside the feasible space, but most will land<br>
outside, on the slopes of the mountains that surround the lake. The<br>
goal should be to nudge these towards the shores of the lake (our<br>
feasible space).<br>
<br>
Having exhausted the analogy, let's talk details.<br>
<br>
A slightly more rigorous treatment of the problem is given here:<br></div></div>
<a href="http://papers.ssrn.com/sol3/__papers.cfm?abstract_id=1680224" target="_blank">http://papers.ssrn.com/sol3/__<u></u>papers.cfm?abstract_id=1680224</a><div><div><br>
<<a href="http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1680224" target="_blank">http://papers.ssrn.com/sol3/<u></u>papers.cfm?abstract_id=1680224</a><u></u>><br>
It is possible that can use this method directly for random<br>
portfolios (and that we could add the ectra constraint types to<br>
DEoptim). If so, much of the rest of what I'll write here is<br>
irrelevant. I strongly suspect that there will be some constraint<br>
types that will still need to be 'adjusted' via a mapping method<br>
like the one laid out below, since a stochastic solver will hand us<br>
a vector that needs to be transformed at least in part to move into<br>
the feasible space. It's alsom not entirely clear to me that the<br>
methods presented in the paper can satisfy all our constraint types.<br>
<br>
<br>
I think our first step should be to test each constraint type, in<br>
some sort of hierarchy, starting with box constraints (almost all<br>
solvers support box constraints, of course), since some of the other<br>
transformations will violate the box constraints, and we'll need to<br>
transform back again.<br>
<br>
Each constraint can be evaluated as a logical expression against the<br>
weights vector. You can see code for doing something similar with<br>
time series data in the sigFormula function in quantstrat. It takes<br>
advantage of some base R functionality that can treat an R object<br>
(in this case the weights vector) as an environment or 'frame'. This<br>
allows the columns of the data to be addressed without any major<br>
manipulation, simply by column name (asset name in the weights<br>
vector, possibly after adding names back in).<br>
<br>
The code looks something like this:<br>
eval(parse(text=formula), data)<br>
<br>
So, 'data' is our weights vector, and 'formula' is an expression<br>
that can be evaluated as a formula by R. Evaluating this formula<br>
will give us TRUE or FALSE to denote whether the weights vector is<br>
in compliance or in violation of that constraint. Then, we'll need<br>
to transform the weight vector, if possible, to comply with that<br>
constraint.<br>
<br>
Specific Cases:<br>
I've implemented this transformation for box constraints in the<br>
random portfolios code. We don't need the evaluation I'll describe<br>
next for box constraints, because each single weight is handled<br>
separately.<br>
<br>
min_sum and max_sum leverage constraints can be evaluated without<br>
using the formula, since the formula is simple, and can be expressed<br>
in simple R code. The transformation can be accomplished by<br>
transforming the entire vector. There's code to do this in both the<br>
random portfolios code and in constrained_objective. It is probably<br>
preferable to do the transformation one weight at a time, as I do in<br>
the random portfolios code, to end closer to the edges of the<br>
feasible space, while continuing to take the box constraints into<br>
account.<br>
<br>
linear (in)equality constraints and group constraints can be<br>
evaluated generically via the formula method I've described above.<br>
Then individual weights can be transformed taking the value of the<br>
constraint (<,>,=) into account (along with the box constraints and<br>
leverage constraints).<br>
<br>
and so on...<br>
<br>
Challenges:<br>
- recovering the transformed vector from a optimization solver that<br>
doesn't directly support a mapping function. I've got some tricks<br>
for this using environments that we can revisit after we get the<br>
basic methodology working.<br>
<br>
-allowing for progressively relaxing constraints when the<br>
constraints are simply too restrictive. Perhaps Doug has some<br>
documentation on this as he's done it in the past, or perhaps we can<br>
simply deal with it in the penalty part of constrained_objective()<br>
<br>
Hopefully this was helpful.<br>
<br>
Regards,<br>
<br>
Brian<br>
<br>
--<br>
Brian G. Peterson<br>
<a href="http://braverock.com/brian/" target="_blank">http://braverock.com/brian/</a><br></div></div>
Ph: <a href="tel:773-459-4973" value="+17734594973" target="_blank">773-459-4973</a> <tel:<a href="tel:773-459-4973" value="+17734594973" target="_blank">773-459-4973</a>><br>
IM: bgpbraverock<br>
______________________________<u></u>___________________<br>
GSoC-PortA mailing list<br>
<a href="mailto:GSoC-PortA@lists.r-forge.r-__project.org" target="_blank">GSoC-PortA@lists.r-forge.r-__<u></u>project.org</a><br>
<mailto:<a href="mailto:GSoC-PortA@lists.r-forge.r-project.org" target="_blank">GSoC-PortA@lists.r-<u></u>forge.r-project.org</a>><br>
<a href="http://lists.r-forge.r-__project.org/cgi-bin/mailman/__listinfo/gsoc-porta" target="_blank">http://lists.r-forge.r-__<u></u>project.org/cgi-bin/mailman/__<u></u>listinfo/gsoc-porta</a><br>
<<a href="http://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/gsoc-porta" target="_blank">http://lists.r-forge.r-<u></u>project.org/cgi-bin/mailman/<u></u>listinfo/gsoc-porta</a>><div><br>
<br>
<br>
<br>
<br>
______________________________<u></u>_________________<br>
GSoC-PortA mailing list<br>
<a href="mailto:GSoC-PortA@lists.r-forge.r-project.org" target="_blank">GSoC-PortA@lists.r-forge.r-<u></u>project.org</a><br>
<a href="http://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/gsoc-porta" target="_blank">http://lists.r-forge.r-<u></u>project.org/cgi-bin/mailman/<u></u>listinfo/gsoc-porta</a><br>
<br>
</div></blockquote><div><div>
<br>
<br>
-- <br>
Brian G. Peterson<br>
<a href="http://braverock.com/brian/" target="_blank">http://braverock.com/brian/</a><br>
Ph: <a href="tel:773-459-4973" value="+17734594973" target="_blank">773-459-4973</a><br>
IM: bgpbraverock<br>
______________________________<u></u>_________________<br>
GSoC-PortA mailing list<br>
<a href="mailto:GSoC-PortA@lists.r-forge.r-project.org" target="_blank">GSoC-PortA@lists.r-forge.r-<u></u>project.org</a><br>
<a href="http://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/gsoc-porta" target="_blank">http://lists.r-forge.r-<u></u>project.org/cgi-bin/mailman/<u></u>listinfo/gsoc-porta</a><br>
</div></div></blockquote></div><br>
</div>