How to do Research in Computer Science

Here are a few guidelines for new Computer Science researchers (written by David Ingram). These are the kind of things I wish I'd known when starting my research! Much of this is likely to be relevant to research in any subject.

Contents

1. Read
2. Stop reading
3. Decide what interests you
4. Balance programming and research
5. Talk to people
6. Write
7. Learn how to simulate
8. Read again
9. Time management

1. Read

You obviously need to start by doing a great deal of reading, so that you are aware of previous work in your area. This is the literature search phase.

This task is a lot easier now than it used to be, because almost any paper you want to look up will have full text available for free on the web (the author's or research group's home page is the most likely location). There are still some journals that prohibit this as well as repositories which charge a subscription fee (that your institution may or may not have registered for), but they are growing fewer. This is a good thing, since the old model of ordering paper copies wasn't really suitable for modern day computing research and slowed things down unnecessarily.

It's now reasonable to assume that if you can't get hold of a paper online, it can safely be ignored, which saves an awful lot of time with libraries and so on. Finding things is easy too - you don't need a special catalogue since if you have a decent citation (title and the surnames of some authors) Google will invariably find what you want very quickly indeed.

Be prepared to do plenty of "chasing references" via bibliographies - you will often find a mediocre paper which is useful by providing citations to several of the important papers in the area. The only drawback to this is that references obviously only go backward in time - so you may need to scan the contents of the proceedings for relevant conferences in the last few years for the latest work. If you've read the earlier stuff you'll know from the titles and people/groups responsible what are likely to be interesting developments and significant improvements to the established methods.

When you are reading a research paper, write on the margin in red pen to indicate the most relevant parts, and when you have finished scribble a quick summary on the front page including whether it was useful or not. This will save you having to read it again later! Keep track of the references to any papers you may need to cite by typing them into your bibliography file.

2. Stop reading

OK, you've read hundreds of research papers, and every idea you've come up with is either impossible to implement, not useful, or has been done before. It's time to stop reading and start doing. By designing systems (and starting to code them, if appropriate) you will gain valuable insights beyond what you've read about. You also need the practical experience of building things so it's a good idea to get started on that early. The code you write doesn't have to be perfect or solve any really important problems initially, so be prepared to throw it away or redesign it later. It's still useful because it makes you focus on how things really work and this will inform your later reading and thinking. Sometimes you find you've only just started programming when you realise there are some tricky details you haven't worked out yet!

3. Decide what interests you

It's unusual for your first idea to be the one you stick with. Be prepared to change direction completely in the first six months. When picking your project, the most important thing is that it is something you are passionately interested in. This is more important than originality, difficulty and relevance (although all of those matter too). If you don't care that much about your area, you won't keep up the momentum during months or years of research. If you do care, you also have a great metric for deciding how to solve the problem. Whenever you are unsure what to do, you can ask yourself if that is the way you would like it to work. Satisfying your own needs (or those of a well identified target market) is a powerful motivator and will leave you in no doubt as to whether your project has succeeded or not.

4. Balance programming and research

You should be aware that doing programming isn't the same thing as doing research, but it is a skill that you should practice and will need. The only sensible conclusion is that you should seek a good balance between theoretical work and hacking!

5. Talk to people

One thing that differentiates the research world from industry is the degree to which ideas are shared, and this is fundamental torwards making progress. On the one hand, you must read what others have done to make sure your work is original and doesn't duplicate something else. But equally there's no way you could create much if you had to invent the whole subject from scratch yourself. It's accepted that progress is made by standing on the shoulders of giants, using all the results known so far as well as your own ideas. This is why you should read past work and also why you need to talk to others. If you have access to lots of other clever brains they can help you avoid getting stuck on some issues for too long. An important part of research is therefore chatting to other people!

There are lots of different ways of collaborating, from incidental remarks over lunch to group meetings, e-mail exchanges, brainstorming sessions, chatting in the corridor in front of a whiteboard and asking people questions. They are all useful. Who you talk to depends on your environment. An ideal environment is one where you have local experts on different subjects (e.g. certain people who know a lot about networks, security, a particular programming language, hardware etc), you know who they are, and you can approach any of them at will. This is very productive indeed. Often people are only too happy to advise you.

Unfortunately many environments don't have such a good culture of interaction and you may feel reluctant to approach people because they seem too busy/senior/reluctant to help. If that's the case then (apart from trying to change the culture) you should form a small group of close collaborators (in the same office or group perhaps) who you talk to on a daily basis. You need people you can quickly bounce ideas off or ask silly questions without prior introduction. Simply explaining your idea to someone else is a great way of clarifying it yourself, even if they don't have any useful suggestions. They may also be able to point you to a local expert or relevant textbook/paper/course.

Usually of course you will have one or more supervisors who are explicitly responsible for advising you. You will arrange with them when to meet; too frequently and you will both be overburdened with meetings and preparing reports, too infrequently and you may waste time before getting help. A good arrangement is to meet regularly every two weeks and in addition to have a supervisor who is prepared to see you at short notice (say within a day) if unexpected questions arise.

Another important thing to do is to present formal talks on your work. A good research group will give you lots of opportunities to do so and you should take advantage of this. Giving a talk to your colleagues isn't too intimidating and is excellent practice for presenting papers at conferences, as well as a generally useful skill and a source of feedback. Preparing for a talk is fun, lets you straighten your ideas out and gives you a useful milestone to aim for.

6. Write

It's all too easy to do lots of good work without imparting that knowledge to others or getting any recognition for it. For this reason it's vital to publish as much as you can. This can seem very tedious because editing and typesetting papers happens after you have finished with an idea and appears to take time away from "real work", but the project isn't finished until it is published!

The solution to this is to write continuously. Type up all your notes as you go, and then you can gradually edit them into the forms needed for talks and publications. Of course this is an equally good idea when applied to documentation for source code as any software engineer knows (otherwise it doesn't get written). It's better to write up things which aren't perfected (and sometimes even to publish things which don't entirely work!) than to put off writing until there's no time left. You can always revise it later, and if something turns out to be a bad idea it may still be instructive to include it as a rejected solution, so that others can understand the reasoning behind your improved design.

7. Learn how to simulate

When you write up your research, you are going to need lots of nice numbers and pretty graphs to back up what you are saying. Sometimes this is an unfortunate aspect of our subject since it biases research away from important areas like HCI which can't be "measured" so easily. It's easier to assess work which is quantitative in nature. In any case, the graphs do at least provide some evidence that you have got the thing to work in practice and that there are no unnoticed "gotchas" in your wonderful theory.

To generate your results you have a choice between implementing your design or simulating it. The difference is more subtle than you think - simulating isn't just about removing extraneous detail to make it quicker to code or go faster. Measuring an implementation (the alternative way of generating performance figures) may be less accurate if you cannot reproduce exactly the right environment, which is common if your design is meant to scale to large sets of principals or data, involve real users, or be geographically distributed. A simulation uses a model of exactly how long it would take in practice for things such as network messages, so that you can speed up time but still know exactly when key events would take place. It's worth taking a course or reading something about "Discrete Event Simulation" if you need to do this. On the other hand, a real implementation highlights lots of practical details which a simulation can't; it may be better for verifying your design is serviceable than for generating performance figures.

8. Read again

If your project is of a significant length (say 18 months and up) it's immensely useful to go and read some more of other people's work part way through. It's easy to overlook this, because you may think that you've now committed to your design so further reading is of no use and will just slow you down. It's certainly true that whilst you are busy programming there's little point being distracted by research papers - you will only want reference manuals on your desk! The time to go back to reading is when you've reached a natural conclusion and are unsure what to do next.

There are two reasons why this is a good idea. The first and more obvious one is that the state-of-the-art will have progressed somewhat since your first literature search. Even if there's no time to incorporate newer ideas you can at least make sure your bibliography is up to date so you don't get caught out by being unaware of similar things other people are working on. The second, more subtle reason is that over the course of the project you are sure to have changed direction from what was originally envisioned. Therefore, your literature search won't have covered the problems you are now working on! Looking things up now can be a great time saver because it's likely there are some well-established theories or methods which can help significantly with your eventual design.

9. Time management

There's plenty of advice on good time management available, most of which is common sense. If you need to keep anything in mind I'll offer two opposing pieces of advice: seize the moment, and keep things ticking over. If you've got a great idea, it's your prerogative to drop everything and write it down or code it there and then. On the other hand, if there's nothing which particularly demands your time try to make progress on several fronts and do a bit of each every day.

Also note that you can be surprisingly productive on the train, in the shower, stuck at home with a cold, on trips out of town etc. At times like this you have fewer everyday distractions (e-mail, people to chat to) as well as a change of scenery and more time to think. Even better, if you're just killing time it's free progress and doesn't feel like real work either! Take your notes and a laptop/PDA/sketch pad with you and you can work on your ideas whilst mobile.