First, you need to have a clear goal, vision, or problem that needs solving,
along with a criterium of success.|
Second, you need to know some basic principles of language design,
so that your thoughts are clear and you don't repeat previous mistakes.
Designing a language is largely about designing abstractions, and that
is an acquired skill (like painting, playing music, or writing novels) that
is helped by looking at how others have done it.
I suggest reading CTM, although SICP, Noonan & Tucker, and some
other books are also good possibilities.
Third, iterate the following approach for your specific domain:
It's important to find a balance between designing and reading papers.
It's important to start designing before searching the literature.
This will make your literature search much more focused: you will have
first-hand experience what the real issues are.
Otherwise you can spend all your time reading
papers without getting anywhere.
In the other direction, don't forget to look diligently for papers,
otherwise you will waste a lot of time reinventing things.
You don't need to find many papers; a few classic ones are all
that you will need to get the gist of a good idea.
- Design the simplest possible language to solve your problem and
write programs in it. Start with a simple set of concepts and see how
far you can push them before making things more complicated.
When you start needing technicalities that are unrelated to the problem,
it's a sign that you are stretching the concepts too far.
- Search the literature to see how others have tackled the problem
and merge their good ideas with yours. Lots of clever people have
worked for many years in CS and the field is littered with good ideas
(many of which have never been exploited properly!).