September 20, 2017 Problem Writing Opportunities and How-Tos

Many Topcoder members enjoy solving SRM and Marathon tasks. These 2 kinds of competitions feature tasks prepared mostly by the members themselves. This means that a member may also give it a try and become a problem writer. We decided to ask t-mac and Nickolas about the opportunities to become a problem writer.
gorbunov: Is it right that everyone can become a problem writer, or there are some prerequisites?
Nickolas: https://apps.topcoder.com/wiki/display/tc/Write+Problems+for+TopCoder lists two requirements for Algorithm writers: 18+ years old and 1500+ Algo rating. Makes sense – the hardest problems to come up with are Div 1 Medium and Hard, and they are probably out of reach for lower-rated members. I’m firmly in 1500+ area myself, and I managed to write only a couple of Div 1 Hards over the years.
For Marathons, I didn’t have 1500+ rating when I started writing problems, but I already had writer’s application approved, so I guess I had some credibility 🙂
t-mac: On marathons, we have a little bit more flexibility with requirements for problem writers. Unlike algo, success as a writer does not depend as much upon the specific difficulty of coming up with the solution, since there are usually many ways competitors will approach the problem, and ultimately none of them will be “perfect”. Instead, some creativity with the task is a lot more important here.
gorbunov: What are the steps to follow in order to create a set of tasks that can be used for an Algorithm competition?
Nickolas:

• Fill problem writer’s application in MPSQAS to get access to problem creation.
• Create problem stubs (problem statement only) in MPSQAS.
• Work with Algorithm coordinator to pick a set of problems that will work together well in an SRM.
• Once the problems are approved, develop them. Each problem needs a reference solution, a checkData function which checks whether input data satisfies the constraints described in the problem, and a set of tests (including examples).

t-mac:
Adding to what Nickolas already said, spending some time confirming that the problems don’t closely mirror something we have already used, and that the solution isn’t easily Googleable. Though we usually catch it early in the process, it is frustrating to find out that a problem has a solution well described on OEIS, for instance.
For some types of SRM problems, in addition to the reference solution, there is a solution checker that is written also. This is only the case for a somewhat-more-recent problem type where one is allowed to return any solution that works, e.g. “return the set of players that is expected to receive the highest score, if there is more than one such set you may return any of them”.
gorbunov: What about Marathon competitions?
Nickolas:
The process is pretty much the same, except there’s only one problem to be prepared.
The set of deliverables is different for Marathon problems, though. Initial problem submission is just an idea description, with no details. I start by writing a visualizer, which includes test case generation, simulation/scoring routines and the actual visualization; this allows to test and tweak the problem easily. Once I’m happy with the implementation, I write up the full statement, the tester code in MPSQAS and the supplementary files like sample solutions and visualizer manual.
gorbunov: I have heard that there is the MPSQAS software that should be used for problem writing. Is there any tutorial for it?
Nickolas: I don’t think there’s any tutorial on the web or in MPSQAS itself. But when my writer’s application was approved, I got several emails from OlexiyO (Algorithm coordinator back in 2007) with a bunch of pointers about Algo writing, description of the process etc.
t-mac: Admittedly MPSQAS is something of an arcane system that is showing its age, however, at the heart of it, it does really well what it was designed to do: allow multiple individuals to contribute to problem creation by running/testing it against our test framework.
gorbunov: How long does it usually take to prepare a problem set?
Nickolas: With or without procrastination? 🙂 Anything from two evenings for a Marathon with a short clean implementation (MegaParty or KnightsAttacks) to several weeks for an SRM or a Marathon heavy with implementation details and/or geometry.
To put this into perspective, in the first 8 months of this year I’ve written 10 contests (2 of them outside of Topcoder), and will have written 4 more in the next two months.
t-mac: Something that happens over time, especially on the algorithm side, is what I call “writer fatigue”. After a while, coming up with new and interesting tasks becomes increasingly challenging as it is hard not to repeat the same ideas. A decade ago, I was one of the most common algo problem writers, now I write maybe one contest a year–and even then, often it’s for private events or special clients. For marathons, there was a time when I was one of two individuals that wrote almost every contest (going way back here, the other was lbackstrom). Though new ideas are a bit harder to come by, it’s not as bad as with algo problems, and you can still find me writing contests semi-regularly.
gorbunov: Who determines if and when a task will be used?
Nickolas: Algorithm coordinator for Algorithms track ([[cgy4ever]] these days).
A group of concerned citizens for Marathon track (t-mac, [Jaco] and me).
t-mac: On the algo side, [[cgy4ever]] does a fantastic job at this… probably the biggest challenge is making sure the level of difficulty is right. To be frank, for a while the SRM difficulty in Div 1 was spiralling out of control. We are aware of things like this, and trying to correct them, though we have some ways to go still. On the other hand, there are some cases–namely late-round TCO contests– where very high difficulty is exactly what we want. Other contests–like the regional events– pose a unique challenge where we know there may be some very talented competitors, but we still want to have a contest that is enjoyable for newcomers as well.
For marathons, Nickolas is probably selling herself short, as she really puts a lot of effort into suggesting good tasks, and really serves as a good motivator for the group.
gorbunov: What is the percentage of rejected tasks?
Nickolas: For Algorithm, I have 71 problems currently entered in MPSQAS: 45 used, 3 scheduled to be used in the near future, 4-5 will probably be used at some point but not scheduled yet. The rest will probably never be used at Topcoder – several I’ve used at other websites, one became a Marathon, and others are just not interesting enough. That’s about 25% rejection rate over 10 years.
But that can vary depending on the writer, the number of writers you’re competing against, the personal taste of Algorithm coordinator and the events for which the problems are selected. For example, with the previous coordinator [[rng_58]] I had a 100% rejection rate for TCO’12, and I have yet to come with a problem to be used at TCO finals in Algorithm track 🙂
For Marathons, the process is slightly different – typically I propose a bunch of ideas, we pick several best ones and use them for the nearest matches, and then revisit the list of ideas (which might gain a couple new ones or have old ones tweaked in the meantime) to pick the best ones again. MazeFixing and RollingBalls spent five years in this limbo before being chosen, CutTheRoots – almost seven! Only a few problems were hard-rejected through the years, and there are typically the ones which are impossible to implement. The list currently has 14 problems, 6 or 7 of them usable, others requiring tweaks.
t-mac: Some of this also once again depends on your individual experience as a problem writer. In many cases, rather than an outright rejection, there may need to be some “massaging” of the problem statement, size limits, or implementation details. Perhaps to make the problem more interesting, less silly, easier to understand, or to better match the target difficulty level.
gorbunov: Thanks for the answers. Topcoder is always looking for new problem writers. We are always in need of fresh and original tasks.