## Friday, December 30, 2016

### rJava: The Gift That Keeps On Giving

I've written not once but twice before (in 2011 and 2015) about the hassles of getting the rJava package to work with R. Every time I think I have a fix for it, someone changes something somewhere and the previous fix no longer works. I had to reinstall rJava today (from the Canonical repositories) after a system upgrade on PC. So, naturally it declined to work.

The symptoms, in both R terminal sessions and when running RStudio, were the same as in 2015. Executing library(rJava) in an R session earned me an error message containing the same nugget
libjvm.so: cannot open shared object file: No such file or directory
as before. Executing Sys.getenv(c("JAVA_HOME", "LD_LIBRARY_PATH")) showed me that the proximate cause was the same as in 2015: the library path was missing the "/jre" piece near its middle. So, same problem implies same cure, right? Wrong! The paths in /usr/lib/R/etc/ldpaths were correct; they just were not working as expected.

The key lines in  /usr/lib/R/etc/ldpaths were as follows.
: ${JAVA_HOME=/usr/lib/jvm/java-8-oracle/jre} :${R_JAVA_LD_LIBRARY_PATH=${JAVA_HOME}/lib/amd64/server}  They look correct, but apparently they are only executed if the corresponding environment variables (JAVA_HOME and R_JAVA_LD_LIBRARY_PATH) are not already defined. I suspect this is due to the colon at the start of each line, but I'm no expert on BASH syntax. You can test this in a terminal by running something like the following: execute JAVA_HOME="silly" R at the terminal prompt (which temporarily resets the JAVA_HOME environment variable to something, well, silly), and in the R session execute Sys.getenv("LD_LIBRARY_PATH") which prints a path whose Java portion starts with "silly". The source of the predefined value of JAVA_HOME, at least on my system, is /etc/profile.d/jdk.sh, which contains the following lines. export J2SDKDIR=/usr/lib/jvm/java-8-oracle export J2REDIR=/usr/lib/jvm/java-8-oracle/jre export PATH=$PATH:/usr/lib/jvm/java-8-oracle/bin:/usr/lib/jvm/java-8-oracle/db/bin:/usr/lib/jvm/java-8-oracle/jre/bin
export JAVA_HOME=/usr/lib/jvm/java-8-oracle
export DERBY_HOME=/usr/lib/jvm/java-8-oracle/db


One solution would be to append "/jre" to JAVA_HOME in that file (making it identical to J2REDIR), but that has two problems. The first is that whatever installer created /etc/profile.d/jdk.sh is likely to overwrite the change the next time the installer is run. The second, potentially more serious, problem is that JAVA_HOME is presumably being set to its current value for a reason. Changing it might have unforeseen consequences with another program.

So my "solution" (hack) is to modify one line in /usr/lib/R/etc/ldpaths as seen below.
## : ${R_JAVA_LD_LIBRARY_PATH=${JAVA_HOME}/lib/amd64/server}
: ${R_JAVA_LD_LIBRARY_PATH=${JAVA_HOME}/jre/lib/amd64/server}

That's still subject to being overwritten by an installer in the future, but at least it only affects R (and it works).

## Tuesday, December 27, 2016

### Mint Upgrade Woes

As I mentioned a couple of months ago, I upgraded a laptop from Linux Mint 17.3 ("Rebecca") to 18.0 ("Sarah") with minimal difficulties. My laptop serves as a guinea pig for these things. Once I'm sure things work fine on the laptop, I'll consider upgrading my desktop as well.

A few days ago I finally undertook the desktop upgrade. By this time the current Mint version was 18.1 ("Serena"). I was expecting a smooth ride, because I forgot one key difference between laptop and desktop: the desktop has an NVIDIA display card. I can't believe I forgot to take that into account, because I've written before about problems with NVIDIA hardware drivers.

Shortening (somewhat) a long and painful story, the initial upgrade left the system with an unusable display, forcing me to do a fresh install of 18.1 from CD (which, among other things, removed a whole lot of packages that will need to be reinstalled). Even after that, the display was an adventure.

The initial installation gave me a choice between the open-source Nouveau driver and version 304.132 of the NVIDIA "legacy" driver. This is the same version that previously caused major headaches. Last time it cause a black screen. This time, presumably due to updates in some system modules, I actually got the background for my desktop (with the Mint logo) and ... something. At first it showed only the upper left corner of my desktop, but after assorted uninstall/reinstall/desperately-try-something machinations, I got to the point where it showed the entire desktop ... except the start button and bottom panel. Oh, and windows did not have the maximize/minimize/close controls, some windows could be dragged while others could not, I don't think I could resize windows, and for some reason the menu button in Firefox didn't work. Other than that, things were peachy. I tried using the sgfxi script to change driver versions, but there was no joy to be had. Of the subset of listed versions that I tried (there were too many to mess with every one), the older ones were allegedly incompatible with one of the libraries (Xorg, I think) and the newer ones had the same problems (or were incompatible).

So I decided to stick with the Nouveau drivers, which install with Mint and are the default choice. Their behavior was also a bit odd. Depending on where I was coming from (fresh install, switching from NVIDIA to Nouveau, something else), they might recognize my 1920x1080 Samsung monitor correctly or they might think that my only monitor was a laptop display (limited to 640x480 resolution). The latter resulted in a butt-ugly and more or less unusable result. Eventually I got Nouveau working at 1920x1080 and thought I was good to go. In fact, I almost was.

The one deal-breaker with Nouveau came when I clicked links in email messages (using Thunderbird) to open them in the Firefox browser. Seemingly randomly, about 3/4 or so would work, but occasionally one would lock the system solid (no response to keyboard or mouse) with a corrupted display ("white blizzard"). The only recourse was power off/power on. Google searches suggested changing a couple of settings in Firefox regarding hardware acceleration. One setting no longer exists; I toggled the other, but it did no good.

I put up with that for a day or so before deciding that, absent any cure, I would have to find a way to use the NVIDIA driver. It occurred to me that the things it was screwing up (window controls, desktop panel, menu button ...) were things under the purview of the window manager (which was Cinnamon). So I installed MATE alongside Cinnamon and switched to it. Sure enough, I got back the main menu and bottom panel, window controls and so forth. It's been stable for a few days now (with no freezes from clicking links in email messages), so I think I've confirmed my thesis that the legacy NVIDIA driver does not get along with Cinnamon, at least on my machine.

Bottom line: anyone who's getting screen freezes using an NVIDIA driver with Mint and Cinnamon might want to consider trying an alternate desktop environment.

## Monday, December 26, 2016

### A Panel Launcher Menu for MATE

A recent upgrade to my Linux Mint PC forced me to switch the desktop environment from Cinnamon (which I've been using for years) to MATE. For the most part, that was painless, but a few things from my old desktop did not translate well.

I had the icons on my Cinnamon desktop organized in a way that made sense to me (but would baffle anyone else); MATE decided they needed to be alphabetized and snapped to a grid. That will be easy (but tedious) to fix. The trickier part was recreating the mix of indicators and applets I wanted in the panel at the bottom of the screen.

One feature I've gotten used to is having an applet that pops open a customize menu of launchers. There's a really excellent Gnome shell extension named MyLauncher that is easy to add in Cinnamon but apparently not available for MATE. Clicking the MyLauncher panel icon opens a pop-up menu that is fully customizable. Menu entries are text, not icons, and you select what the text says and what it does. As one example, I have an entry reading "Search files" that launches Searchmonkey (which I heartily recommend).

I'm a bit finicky (anal?) about how such a menu should work.
1. I want it to list just selected applications, not every application on the system.
2. I want text prompts, not icons. (If I'm thinking "search for a file", my brain is not translating that to "look for a picture of a monkey".)
3. I want to choose the prompts myself. (Before the morning coffee sinks in, I may not remember that "searchmonkey" is the command I'm looking for.)
4. I want a single list; I don't want to have to wade through categories (as one does in the main Mint menu).
5. Adding/editing the menu needs to be reasonably easy.
MyLauncher met all those criteria. After the switch to MATE (and after discovering I could not use MyLauncher any more), I went looking for something equivalent but struggled to find an adequate substitute. It is easy to add an instance of "Main Menu", "Menu Bar" or "mintMenu" to the panel, but all of those default to listing every application (violating item 1 above) and use the same categories as the main menu (violating item 4). I think I could edit my way past that, but deleting a zillion applications added by default is more work than manually adding the handful I want. Also, when a new application is installed, it is added to the main menu automatically, and I'm not sure if it would wind up added to my custom menu as well (forcing me to edit out each newly installed application).

Anyway, after a bit of searching I found a solution with which I'm comfortable. It combines two programs. Rofi is something of a Swiss army knife, with many functions, one of which is providing menus (with search-as-you-type functionality). Rofi is available from the Canonical repositories, so I was able to install it using Synaptic. The program that actually produces the menus is Menugen, which is actually a set of BASH scripts. There's not much to installation: you just download it, unzip it and store it someplace.

With Rofi and Menugen installed, I just had to load my old MyLauncher menu into a text editor, massage it into the format Menugen wants, and park it someplace. I made the new menu an executable script that calls menugen to interpret it. I then added to the panel a custom launcher invoking my menu script, and that was that.

In case it helps anyone, here's an abbreviated version of my script:

#!/home/paul/Software/menugen-master/menugen
#begin main
name="Quick Launch"
numbered=false
add_exec "Freeguide"  'freeguide'
add_exec "Text editor"  "xed"
add_nop "-----------------"
add_exec "Synaptic"  "gksudo synaptic"
add_nop "-----------------"
add_exec "Terminal"  "terminator"
#end main


The add_nop commands add "no-op" lines (separators), while the add_exec commands add menu entries. There is also an add_item command for adding menu entries and an add_link command for linking to a submenu. The primary difference between add_item and add_exec (at least as far as I'm concerned) is that add_item returns you to the menu after you make a choice and add_exec closes the menu after you make a choice (which is the behavior I want).

## Sunday, December 18, 2016

### Using an MCE Remote with Mythbuntu

As I chronicled in a previous post, I've been using MythMote on my (Android) cell phone to serve as a remote control for my MythTV installation (running on Mythbuntu) when watching recordings  or live TV. For the most part this works fine, but there are little inconveniences associated with it, so I decided it was time buckle down and get one of my (many) remote controls to do the job. As it happens, I have Windows Media Center infrared receiver attached to the machine via USB drive, so the associated remote control was the logical choice.

I took the better part of a day to get it working, so I'll chronicle here the things I learned. Everything here is specific to MythTV running on some version of Linux, and much of it may require adjustment if MythTV is not installed on Mythbuntu, Ubuntu or maybe Linux Mint or Debian.

### Ignore LIRC

As with any of my MythTV adventures, I started by consulting Google. It turns out a lot of the information out there is obsolete if you have a remotely recent operating system. Many (most?) of the posts I found discussed how to work with Linux Infrared Remote Control (LIRC), an add-on package for Linux. The catch is that infrared remote support is baked into recent kernels, so you don't need LIRC unless you want to use certain extended capabilities that it apparently provides. I don't know what they are, and I don't need them.

### One Ring to Control Them All

Get to know the ir-keytable command, which will be your friend through all this.  Running it (with no options) in a terminal told me that my system understood that I had an MCE receiver attached (on /dev/input/event3, as it turns out ... yours may differ). Running

ir-keytable --device /dev/input/event3 --test

then let me test which key events were detected when each button on the remote was pressed.

### BIOS Nonsense

This led to the first "minor" problem: according to ir-keytable, none of the buttons on my remote were registering. The LEDs on the remote and receiver both lit with each button press, so the receiver was "hearing" the remote; it just wasn't being believed by the system. Further consultation with Google led me to this post, which suggested that the problem lies in how some BIOSes support USB 3. The IR receiver was plugged into a USB 2 hub, and I don't have anything attached to the machine via USB 3 (I'm not even sure it has a USB 3 jack), but whatever. I rebooted, got into the BIOS editor, found the setting for USB 3 support (every damn BIOS seems to hide it in a different place, so I can't tell you exactly where to look), and turned it off, then booted back into Mythbuntu. This achieved a partial success: ir-keytable in test mode could see button presses! That's the good news. The bad news is that only a few were interpreted correctly.

### Remapping the Buttons

The next step is to remap the buttons on the remote. The drill is to map buttons to the keys you would use if you were controlling MythTV with a keyboard. An important note here is that I'm only concerned with "playback" controls, not with controlling operating system menus or other applications. Here are some helpful links for that.
• The MythTV wiki Keybindings page lists keyboard keys associated with various operations (e.g., "P" for pause/resume playback). Keep this handy as a reference.
• "HOWTO: Get MCE USB remote working in Ubuntu without using LIRC for MythTV or Kodi (xbmc)" is a blog post with detailed, step-by-step instructions. Use this as your main guide. I will say that I deviated a bit here and there. In particular, the author has you generate the default button mapping and save it directly its final destination. This is the line

sudo ir-keytable --read --device=/dev/input/event10 > /etc/rc_keymaps/myrc6_mce

in his post. I found it much easier to write the file to the desktop, edit it there, experiment until I got it right, and only then move it to /etc/rc_keymaps. (The move requires superuser rights; editing on the desktop does not.)
• "MythTV: Use All Buttons of Your Remote Control - Without LIRC" is another handy post on the same subject. In particular, it has a table at the end of all possible key codes, which you may find useful as a reference (to tell, for instance, that the escape key is KEY_ESC rather than KEY_ESCAPE).

### My Remapping

There was some trial-and-error getting to a mapping I liked, but ultimately I was successful. The "HOWTO" blog post identifies an issue with key codes greater than 255 (those with three digit hexadecimal codes). Apparently X11 does not like them. I found, however, that some of the two digit key codes did not work as specified, either, and had to be remapped to other two digit codes. My remote is an RC6 type (the first of the two black ones shown here -- although I'm in the US, not the UK or AU). Reproduced below are just the key mappings I changed in the "myrc6_mce" file.

scancode 0x800f0400 = KEY_0 # was KEY_NUMERIC_0 (0x200)
scancode 0x800f0401 = KEY_1 # was KEY_NUMERIC_1 (0x201)
scancode 0x800f0402 = KEY_2 # was KEY_NUMERIC_2 (0x202)
scancode 0x800f0403 = KEY_3 # was KEY_NUMERIC_3 (0x203)
scancode 0x800f0404 = KEY_4 # was KEY_NUMERIC_4 (0x204)
scancode 0x800f0405 = KEY_5 # was KEY_NUMERIC_5 (0x205)
scancode 0x800f0406 = KEY_6 # was KEY_NUMERIC_6 (0x206)
scancode 0x800f0407 = KEY_7 # was KEY_NUMERIC_7 (0x207)
scancode 0x800f0408 = KEY_8 # was KEY_NUMERIC_8 (0x208)
scancode 0x800f0409 = KEY_9 # was KEY_NUMERIC_9 (0x209)
scancode 0x800f040a = KEY_BACKSPACE # was KEY_DELETE (0x6f)
scancode 0x800f040d = KEY_M # was KEY_MEDIA (0xe2)
scancode 0x800f040e = KEY_F9 # was KEY_MUTE (0x71)
scancode 0x800f040f = KEY_I # was KEY_INFO (0x166)
scancode 0x800f0410 = KEY_F11 # was KEY_VOLUMEUP (0x73)
scancode 0x800f0411 = KEY_F10 # was KEY_VOLUMEDOWN (0x72)
scancode 0x800f0412 = KEY_UP # was KEY_CHANNELUP (0x192)
scancode 0x800f0413 = KEY_DOWN # was KEY_CHANNELDOWN (0x193)
scancode 0x800f0414 = KEY_RIGHT # was KEY_FASTFORWARD (0xd0)
scancode 0x800f0415 = KEY_LEFT # was KEY_REWIND (0xa8)
scancode 0x800f0416 = KEY_P # was KEY_PLAY (0xcf)
scancode 0x800f0418 = KEY_P # was KEY_PAUSE (0x77)
scancode 0x800f0419 = KEY_SPACE # was KEY_STOP (0x80)
scancode 0x800f041a = KEY_END # was KEY_NEXT (0x197)
scancode 0x800f041b = KEY_HOME # was KEY_PREVIOUS (0x19c)
scancode 0x800f0422 = KEY_ENTER # was KEY_OK (0x160)
scancode 0x800f0423 = KEY_ESC # was KEY_EXIT (0xae)
scancode 0x800f0426 = KEY_S # was KEY_EPG (0x16d)
scancode 0x800f046e = KEY_P # was KEY_PLAYPAUSE (0xa4)
scancode 0x800f0481 = KEY_P # was KEY_PLAYPAUSE (0xa4)


### Stuttering

Update 12/28/16: The one issue I discovered with the remote was a tendency for button presses to be counted twice (occasionally, not always). This can be moderately annoying when running through menus or trying to pause/resume and terribly annoying when trying to fast forward or reverse.

Fortunately, the solution is simple, and hinges on the fact that the button presses are being treated as keyboard input. In Mythbuntu's main menu, navigate to Settings > Accessibility > Keyboard > Slow Keys. Select the check box for "Use slow keys" and set the "Acceptance delay" slider to something around 100 ms. The menu location may vary with other distributions, and you may need to tinker with the delay. Caveat: This will also affect typing on your keyboard. If you are a "hunt and peck" typist, the impact may not be too bad. If you are a touch typist, it can get really annoying, possibly more annoying than the remote button stutter.

## Friday, December 2, 2016

### Support for Benders Decomposition in CPLEX

As of version 12.7, CPLEX now has built-in support for Benders decomposition. For details on that (and other changes to CPLEX), I suggest you look at this post on J-F Puget's blog and Xavier Nodet's related slide show. [Update 12/7/16: There is additional information about the Benders support in a presentation by IBM's Andrea Tramontani at the 2016 INFORMS national meeting, "

I've previously posted Java code for a simple example of Benders decomposition (a fixed-cost warehousing/distribution problem). To get a feel for the new features related to Benders, I rewrote that example. The code for the revised example is in this Git repository, and is free for use under a Creative Commons 3.0 license. There is also an issue tracker, if you bump into any bugs.

Going through the code here would be pretty boring, but there are a few bits and pieces I think are worth explaining, and I'll show some far from definitive timing results.

### Modeling Approaches

I have five modeling approaches in the code.
• STANDARD: This is just a single MIP model, using no decomposition. It's present partly for benchmark purposes and partly because a few of the newer approaches build on it.
• MANUAL: This is essentially a reprise of my earlier code. I write two separate models, a master problem (MIP) and a subproblem (LP), just as one would do prior to CPLEX 12.7.
• ANNOTATED: This is the first of three methods exploiting the new features of CPLEX 12.7 I create a single model (by duplicating the STANDARD code) and then annotate it (using the annotatiion method added to CPLEX 12.7) to tell CPLEX how to split it into a master problem and a single subproblem. The annotation method would also let me create multiple disjoint subproblems, but the example we are using only needs one subproblem.
• WORKERS: This is the ANNOTATED method again, but with a parameter setting giving CPLEX the option to split the single subproblem into two or more subproblems if it sees a structure in the subproblem that suggests partitioning is possible.
• AUTOMATIC: Here I create a single model (identical to the STANDARD method) and, via a parameter setting, let CPLEX decide how to split it into a master problem and one or more subproblems.

### Benders Strategy Parameter

The key to all this is a new parameter, whose Java name is IloCplex.Benders.Strategy. It is integer valued, but for some reason is not part of the IloCplex.IntParam class hierarchy (which threw me off at first when I tried to use it in my code). The values defined for it are:
• -1 = OFF: Use standard branch and cut, doing no Benders decomposition (even if annotations are present). Note that this will not stop manually coded Benders from working (as in my MANUAL method); it just stops CPLEX for creating a decomposition. This is the default value.
• 0 = AUTO: If no annotations are present, do standard branch and cut (akin to the OFF value). If the user supplies annotations, set up a Benders decomposition using those annotations, but partition the subproblems further if possible (akin to the WORKERS behavior below). If the user supplies incorrect annotations, throw an exception. This is the default value.
• 1 = USER: Decompose the problem, adhering strictly to the user's annotations (with no additional decomposition of subproblems). If the user fails to annotate the model, or annotates it incorrectly, throw an exception.
• 2 = WORKERS: This is similar to USER, but gives CPLEX permission to partition subproblems into smaller subproblems if it identifies a way to do so.
• 3 = FULL: CPLEX ignores any user annotations and attempts to decompose the model. If either all the variables are integer (no LP subproblem) or none of them are (no MIP master problem), CPLEX throws an exception.
There are also two other Benders-related parameters, in Java IloCplex.Param.Benders.Tolerances.feasibilitycut and IloCplex.Param.Benders.Tolerances.optimalitycut, that set tolerances for feasibility and optimality cuts. (In my code I leave those at default values. If it ain't broke, don't fix it.)

#### Syntax

Coding the strategy parameter looks like the following.
IloCplex cplex;      // declare a model object
// ... build the model ...
int strategy = ...;  // pick the strategy value to use
cplex.setParam(IloCplex.Param.Benders.Strategy, strategy);


### Annotations

Annotations can be specified in a separate file (if you are reading in a model) or added in code (which is what I do in my demo program). IBM apparently intends annotations to be used more generally than just for Benders decomposition. To use them for Benders, what you do is annotate each model variable with index number of the problem into which it should be placed (where problem 0 is the master problem and problems 1, 2, ... are subproblems). Only variables are annotated, not constraints or objective functions. If you fail to annotate a variable, it is given a default annotation (discussed further below). If you assign a negative integer as an annotation, the universe will implode spectacularly, and CPLEX with throw an exception just before it does.

You start by creating a single MIP model, as if you were not going to use Benders decomposition. Assuming again that your model exists in a variable named cplex, you begin the decomposition process with a line like the following:
IloCplex.LongAnnotation benders =
cplex.newLongAnnotation("cpxBendersPartition");
The name "benders" for the variable in which the annotation is stored is arbitrary, but the name cpxBendersPartition for the annotation must be used verbatim. This version of the annotation constructor sets the default value to 0, so that variables lacking annotations are assumed to belong to the master problem. An alternate version of newLongAnnotation uses a second argument to set the default value.

Next, you annotate each variable with the index of the problem into which it should be placed. Suppose that we have two variables defined as follows.
IloIntVar x = cplex.boolVar("Use1");
// open warehouse 1?
IloNumVar y = cplex.numVar("Ship12")
// amount shipped from warehouse 1 to customer 2

Assuming that we want x to belong to the master problem (index 0) and y to belong to the first and only subproblem (index 1), we would add the following code:
cplex.setAnnotation(benders, x, 0);
cplex.setAnnotation(benders, y, 1);
where benders is the variable holding our annotation.

There are versions of setAnnotation that take vector arguments, so that you can annotate a vector of variables in a single call. If the model in question has a fairly small number of integer variables and a rather large number of continuous variables (which is pretty common), you might want to use the two-argument version of newLongAnnotation to set the default value at 1, and then annotate only the integer variables, letting the continuous variables take the default annotation (i.e., be assigned to subproblem 1).

That's all there is to creating a Benders decomposition from a standard MIP model. Note, in particular, that you do not need to create callbacks. CPLEX handles all that internally. You can still add things like lazy constraint callbacks if you have a reason to do so; but for a "typical" Benders decomposition (dare I use that phrase?), you just need to create a single MIP model, annotate it, and set the Benders strategy parameter. What could be easier than that? Well, actually, one thing. If you set the Benders strategy to FULL, you don't even have to annotate the model! You can let CPLEX figure out the decomposition on its own.

### Performance

Test runs of a single model (especially one as simple as the fixed-charge warehouse problem) on a single computer (especially one with a measly four cores) written by a single programmer (who happens not to be terribly good at it) don't prove anything. I'll leave it to someone else to do serious benchmarking. That said, I was curious to see if there were any gross differences in performance, and I also wanted to see if all methods led to correct answers. (I'm not what you would call a trusting soul.) So I ran 25 replications of each method, using a different problem instance (and a different random seed for CPLEX) each time. In an attempt to level the playing field, I forced Java to collect garbage after each model was solved (and timing stopped), to minimize the impact of garbage collection on run times. All problem instances were the same size: 50 warehouses serving 4,000 customers. The basic model has these dimensions: 4,050 rows; 200,050 columns (of which 50 are binary variables, the rest continuous); and 400,050 nonzero constraint coefficients.

The first plot below shows the wall-clock time spent setting up (and, where relevant, decomposing) each model.
None of the methods strikes me as being overly slow, ignoring a couple of outliers. Using annotations (the "Annotated" and "Workers" methods) does seem to impose a nontrivial burden on model construction.

The second plot shows solution times. For each problem instance, all five methods achieved identical objective values (and proved optimality), and all used the same number of warehouses. (I did not check whether the values of the flow variables were identical.) So any concerns I might have had about validity were assuaged.
I'm not sure how much to read into this, but "manual" decomposition (the old fashioned approach, using explicit callbacks) seems to have the highest variance in solution time. The three approaches using new features of CPLEX 12.7 ("Annotated", "Automatic" and "Workers") had very similar run times. I'm fairly certain that the "Workers" method, wherein CPLEX tries to further partition the single subproblem I specified, wound up sticking with a single subproblem (due to the structure of the model).

### Which Is Better?

Assume that you have a problem that is amenable to "standard" Benders decomposition (as opposed to some of the funky variants, such as combinatorial Benders, Benders with MIP subproblems, Benders with subproblems that are not necessarily optimization problems, ...). The easiest approach from the user perspective is clearly the automatic option (Benders strategy = FULL), in which CPLEX literally does all the work. Runner up is the annotation approach, which is both much easier and much less prone to coding errors than the "manual" approach (defining separate problems and writing a lazy constraint callback).

On the performance side, things are a bit less clear. If you dig through Xavier Nodet's slides, you'll see that CPLEX use somewhat sophisticated techniques, based on recent research, to generate Benders cuts. I suspect that, on the test problem, their cuts would be a bit stronger than the ones I generate in the "manual" approach, which may account for some of the difference (in their favor) in median run times seen in the second plot. Also, since they can access the subproblems and add cuts to the master directly, rather than having to go through callbacks, using the annotation feature may result in a bit less "drag".

With other cases, I suspect that if you have a particular insight into the model structure that lets you generate problem-specific Benders cuts that are tighter than the generic cuts, you may do better sticking to the manual approach. Fortunately, since you can turn on automatic Benders decomposition just by tweaking a single parameter, it should be easy to tell whether your cuts really do improve performance.