Designing a new programming language (cost: 45 RP)

zombe on 7/8/2014 at 16:22

Did some more thinking today, now that it is not so darn hot here. *Curse you sun*

A few things i have mentioned that i have decided to change:
* "member"/"static": this distinction does absolutely nothing, the compiler is forced/bound to discover it anyway - why bother the programmer then? "function" is the new replacement.
* "out": this is an unnecessary subset of stuff that already exists (artifact from the time when "()" was used instead of "{}"). Multiple outputs used. Bonus: one can not declare new variables in function parameter, but sure can in deconstructors.

+ several more that have not come up yet, so not much point to say. Especially given that most of them are not final answers, more like considerations/open-questions/possible-solutions-to-be-scrutinized.

Qooper on 7/8/2014 at 17:49

Heh, it's been dingity dangity darn hot here too, I'd say pretty close to 30C. I've been going through the stuff you've written and you've given me a whole lot to think about. Two main things:

1. The functor issue
2. The stuff with the getters and setters

I was going to reconsider the functor thingy anyways, but now I'm unsure whether or not I'm going to have it in the language at all. And if I am, how am I going to include it (was thinking about something like a bind-keyword). But it's really difficult to sit in front of a computer and think about heavy stuff like this in this heat :D One thing is certain, I need to go back to the basics and rethink what this language is going to be designed for and what the philosophy behind it will be.

Before I head out for a walk, a couple of questions about your language.

You seem to have a pretty flexible foreach. Since you can say foreach i in [0..10] \ {...}, can you also use that [0..10] in other contexts? You know, like list comprehensions in Python for instance.

I was wondering about this: foreach i in [blah.min < blah.max]
What does that line do? Is [blah.min < blah.max] considered a list or is that part of the foreach syntax? I'm referring to the brackets [].

You asked about the class Tree -example. Yeah the close -keyword looks a lot like class :D I need to consider using something like end instead. close Tree basically means that Tree cannot be inherited from, and it also closes all of its descendants. This is basically the same as when in discrete mathematics you say:
1. a node is a tree with a left child and a right child, both of which are trees
2. a leaf is a tree with a value
3. nothing else is a tree

That's what I'm trying to model with that. Also, your pseudo-C++ code was spot on, that's what the Tree example would become, more or less. The reason why those class definitions looked that way is because in my language there are two types of classes, kinda. The ones you saw there are called case classes, pretty much the same as the ones in Scala. They are otherwise normal classes but they have some special stuff in them that allow them to be unpacked by functions, match-expressions and such. This is a nod in the direction of functional programming because I've found these things to be of particular use in certain situations, for example making a scene graph implementation in a game engine :)

But there's already so much to read, so much to write and so very much to think about. I'm overwhelmed and a bit depressed because I don't know what to do anymore :( :D :( :confused: :bored:

I wish there was a logical route, guided by laws of logic, to "the right language spec". Then I'd only need to think until a proof is found. But designing a language for fun should be fun, which is what I started to do in the first place. Now I'm trying to reach perfection again, and everything falls apart. That's the wrong approach in design and programming. The correct approach is to identify the objective, keep the objective in sight at all times and then devise a solution that is good.

I have a lot more questions for you zombe, about your lang and about LL1 and such, if you have the patience to guide me :D But now for a walk.

Thank you, sorry, and I'll be back shortly.

zombe on 7/8/2014 at 19:41

Quote Posted by Qooper

I was going to reconsider the functor thingy anyways, but now I'm unsure whether or not I'm going to have it in the language at all.

My stance is that C++ lambda style functors are worth it only when:
* they support closure
* you have nothing better to do and would like the convenience of having anonymous functions

Creating a functor from another function by injecting one of the parameters can be for both (the provided parameter could be given as an external variable and not by value -> closure). But, in my case, i think it would be more trouble than worth - given that you can do the exact same thing and more with lambda:

Code:

function doStuff(Int a, b) \ return a + b

// returns a functor that takes one integer parameter and adds whatever "param" contained
function test(Int param)
return $(Int b)\ return doStuff(param, b)

PS. this lambda stuff is one of the open questions. For example: should i use by-value or by-reference closure - or leave it for programmer to decide (a'la: C++xx, can not remember which one added that stuff. 11?). As my goal is to support hot-swap then by-reference and its implied lifetime constraints ("test" in this case would not be able to return the functor as "param" is only valid in its scope -> compile time error) would be very nice ... then again. Hmmm.

Quote Posted by Qooper

You seem to have a pretty flexible foreach. Since you can say foreach i in [0..10] \ {...}, can you also use that [0..10] in other contexts? You know, like list comprehensions in Python for instance.

Nope, no list comprehension. That would be more useful in functional programming languages (IMHO), if i want to do something special with list then i use a procedure.

[0..10] is actually two things: [] and 0..10. The later one describes extent (mathcad, IIRC, uses it). Extent is also used in "switch" (but syntactically slightly more limited):

Code:

switch something
@1, 5, 7..10: dostuff // @ means label (named labels can be used to break out from specific while loops etc, or just as goto targets). Values of 1, 5, 7,8,9,10 will land here
@namedLabel 42..: dootherstuff // a label can be named also
@foo + bar .. foo + meh: domore // the extent is from (foo+bar) til (foo+meh) - both must be compile time constants
@ .. : doless // extent encompassing everything else (the "default" of C++)

// trivia: all cases "break" by default, if you want to continue elsewhere then you must explicitly use ... erm ... "continue 7" / "continue namedLabel"

Extent can also contain metadata:
* are the values incrementing or decrementing (switch: does not care)
* is the end of the extent also part of the extent. Ie, is [1..10] from one to nine or ten? (switch: always part of the extent)

Which brings us to ...

Quote Posted by Qooper

I was wondering about this: foreach i in [blah.min < blah.max]
What does that line do? Is [blah.min < blah.max] considered a list or is that part of the foreach syntax? I'm referring to the brackets [].

"<" gives the metadata:
* blah.min is smaller than blah.max => ++i should be used (instead of --i)
* blah.max is NOT part of the extent (a'la: for(i=blah.min; i < blah.max; i++) ...)

Note that similarly to switch, one can leave extent borders undefined: "foreach i in [ <= 10] \ ...", this would start with whatever value "i" had, and end with 10 (inclusive).

All-in-all, there is only one foreach:
" foreach 'extractedValue' in 'iteratorSource' "

Any object that implements "iterator" property (function/variable) can be used:
* Host provided dynamic arrays / maps
* Anonymous structures: {1, 2, 3} - iterator is over properties (NB! only if all properties agree on type), arrays can be initialized with things-with-iterators too: "var foo[] = {1, 2, 3}" or "var foo[7] = myDynamicArray"
* Extents have it too. "[]" is needed only to prevent confusion for reader and compiler (would break LL1 too, i'm fairly sure)

What the iterator does is left open for the programmer: generate consecutive primary numbers? random number generator?

In a sense "foreach" does this (half-pseudo code given in my language):

Code:

// 'is' = given iterator source, 'el' = element extracted
scope
var it = 'is'.iterator // get an iterator
while 'it'.hasNext // while there are more elements to give (extent defines one itself based on its metadata)
defer? it.next // (again, extent defines one itself based on its metadata)
'el' = it.getElement
doForeachBodyCode

Iterator, of course, is just some class/struct that implements what is needed, here: "hasNext", "next" and "getElement".

That is all the magic there is.

Quote Posted by Qooper

But there's already so much to read, so much to write and so very much to think about. I'm overwhelmed and a bit depressed because I don't know what to do anymore :( :D :( :confused: :bored:

I hope i did not scare you off from this undertaking. It can be quite a challenge if getting stuck in feature-creep/overengineering/overextending.

Quote Posted by Qooper

I have a lot more questions for you zombe, about your lang and about LL1 and such, if you have the patience to guide me :D

Given that i am in the process of making a lang then your questions sure can not hurt :D ... feel free to ask.

Explaining my shit helps me see more clearly over the heap of that shit. Oh, so much to do ... but it is interesting at least.

edit: now that you made me think about it ... i do not see any reason not to expose the Extent object to be used where-ever one wants to. "[]" is even non-ambiguous (LL1) as i originally used it for lambda and have not reused since changing to "$" for lambda.

Hm... my extent border thing has changed - it is a bit finicky in regards to initial value now. Have to rethink that ... tomorrow.

edit2: erm, nope, that does not actually change much - just that "foreach" must provide the border itself and it can do that.

Vernon on 8/8/2014 at 12:26

I'll do it for 40

Qooper on 8/8/2014 at 13:34

Quote Posted by Vernon

I'll do it for 40

I'm afraid the project has experienced an incline in complexity. It now requires 75 research points.

Qooper on 8/8/2014 at 13:36

Quote Posted by zombe

Tokenizer - cuts out the possibly meaningful parts
Lexer - adds extra information to tokens (a'la - this is a number / namespace [as only thous can start with a capital letter]). After lexer there is only a generic token for integer numbers where the number is in its extra information slot.

Tokenizer itself is rather unusable - and i and probably you too abuse its name to mean a lexer :D.

You're right, when I say tokenizer I actually mean lexer :D I really don't have any academic background when it comes to compiler stuff. But I stand corrected. Basically I do lexing, then the syntax tree thingy and then finally the target code.

Quote:

My planned route:
* tokenizer (well, lexer with no prior separate tokenizer step)
* LL1 descent parser to generate abstract syntax tree
> LL1 using leftmost element to recursively descend on ( (http://en.wikipedia.org/wiki/Recursive_descent_parser) ). 1 = no lookahead needed, you know exactly what language construct is being described at every point. (http://en.wikipedia.org/wiki/LL_parser)
> It is very simple and i suspect that even if you are not familiar with this stuff then you probably are intuitively using it.

This caught my interest when I read it. So LL1 needs no lookahead? Intriguing, I need to read about it and see if I can refit my language with minimal modifications. It would make parsing a whole lot easier.

Quote:

> PS. do you have a list of production rules for your language? ( (http://en.wikipedia.org/wiki/Backus-Naur_form) ? )

Not yet, but now that I read about that BNF you linked I realize I really really should study the subject. Currently, I have a very messy and non-formal description of the rules in a doodle.txt :D Some people doodle squares and mutants and whatnot, I doodle languages. Must be due to brain damage I guess.

Quote:

* analyze the tree and convert it into an internal tree (as stuff that exists in language does not necessarily exist in compiler. For example - "aliases"/"contexts" are meaningless for my compiler)

Hmm interesting! This is one place where our strategies seem to diverge, at least I think so. I have only one tree structure before I emit target code. But you make a good point here. So if I wanted to add namespaces like in C++ (I'm not saying I would), would they require/benefit from this internal tree structure?

Quote:

Random thoughts on code example:
Wow, this was confusing (probably less so than for you to decipher my info dump). Only noticed that the last line has "close" on my third read x_x - what can i say, it's 30C+ here.
... and i am still not completely sure i get what it exactly represents. How does my ~pseudo-C++ sound?

struct Tree {};
struct Node : Tree { Tree *left, *right; };
struct Leaf : Tree { Anything data; };
+ disallow further extending Tree (i assume one is still free to extend Node and Leaf).

Right?

Just wanted to recap. You're absolutely correct here, except that close Tree closes Node and Leaf too. And I'm going to replace close with end, so it would become end Tree.

Quote:

* "val", "var". A bit concerning that it is a bit hard to visually distinguish the two. Why not use "const" (by your description, that seems to match what i use it for)?

These I just ripped straight from Scala. They seem to work pretty well after you get used to them. The visual difference between the letters l and r is big enough for a trained eye to make a distinction between them even after a quick glance. But your point is still worth considering, so I'll think about an alternative. I don't want to use const because it is slightly longer and I want the lang to be as quick to write as possible. Since vals and vars are used so frequently, I don't want the keywords to be longer than 3 characters.

Quote:

* "var xp: int" - i assume you can omit "int" if it can be inferred? Then why not "int xp" or "var xp" (which is the way i went)? That said, type names and identifiers are easy to distinguish, in my case, which helps ("Int foo" instead of requiring "var Int foo" to stay LL1).
* does the xp initialization to zero have to be in constructor. In my case i can initialize (even using the member functions of the class being initialized) in class declaration and override it in constructor if needed (the compiler reorders all the tidbits (relatively simple/intuitive and deterministic rule) to make sense or cry error if it can not be done).

You're right about this one too, the type could've been omitted. In fact I could have even written var xp = 0, which would have resulted in automatic code generation in the constructor. I don't want to put the type name before the variable name because of the way that I've already begun to write the compiler. Besides, it's pretty straight forward to parse a var or a def. When you hit those keywords, you always know what to expect next. I'm trying to fit every definition into a single format, which is something like [element] = [definition], where element can either be a variable or a function, and definition can be an expression or a list of statements.

Quote:

* "this" vs "self", my native language is not English, so i am intrigued on your choice. Is one better than the other in your case? Why?

There's really no well-justified reason. I chose self because I've been using Python for a long time and in my opinion self sounds more descriptive than this. The same goes for void and none. I think void describes nothingness better than none, but these are rather subjective statements. All I can say is: my lang, my choice :D And besides, it's a toy project and won't ever be used by others.

Quote:

* "getName", i take that default access restriction is ~= C++ private? ... interesting, i highly prefer the opposite because prototyping/trying-things becomes before setting-things-in-stone. Private is an unnecessary burden till then. Scripting language eccentricity i guess.

Accessibility is something I haven't yet decided on. And what makes it more complicated is that there are two axes: self/others, read/write. Luckily this is simpler than it seems, because if something is read-only to self, it wouldn't make sense for it to be read/write to others. But for instance in your lang something can be write-only, which made me think further about these things. At the moment I haven't set anything in stone yet, but I'm thinking how I should approach this.

I want to make immutability the default way of doing things, so that the lang encourages mathematical solutions to problems. But at the same time I want the lang to be as efficient as hackish C, so I'm going to provide ways to override default behavior in order to get as close to the metal as possible and make necessary shortcuts. I really need to write the philosophy and set of objectives somewhere so that this doesn't get out of control :D

Quote:

* "match t" ... Speaking of keywords, what do you have? Mine (current):

Code:

Keywords usable as identifiers:
alias, struct, class, context, namespace,
override, abstract, member, static,
final, as, bye, const, interface
Keywords that cannot be used as identifiers:
foreach, in, do, while,
switch, if, else,
defer, scope,
return, break, continue, goto,
var, null, new, read, out, is, instanceof, assume

Good that you ask, because I haven't even written a list like this. All I have is the code of the compiler, where things are rather messy. But extracting keywords from the code, I get:

Code:

Keywords used for definitions:
val, var, def, class, object
Keywords used for expressions:
to, if, for, while, match, null, inf
Miscellaneous:
return, close, import

Quote:

* "def... : void = for _ <- 0 to count {...}" - ah, "_" usage :). My version: "foreach _ in [0 <= count] \ code" ... assuming yours is not exclusive to "count". A bit ambiguous.

In my lang, [int] to [int] means range and it can be used in for-loops or to generate lists. In for-loops they become a standard C for-loop, but if you say val l = 0 to 10 you get an actual list with elements 0 thru 9. Or maybe I'll change this so that the endpoint is included in the list. I don't know yet. I've used list comprehensions in Python and found them to be extremely powerful and useful, so I'm aiming to make a similar implementation. Also I've been thinking about something like val l = yield 0 to 10, which would mean the same thing except that the list would be generated lazily.

Quote:

PS. What is "void" :P ... and why do you seem to declare that the function returns it?

Because otherwise type inference would infer that it returns the type of: for _ <- 0 to count {...}. void means that the type is nothing. In my lang ifs, fors and whiles all have a type. They are expressions. You can say val a = for i <- 0 to 5 {if i == 2 {1000}, {i}}, which would assign a to the value of [0, 1, 1000, 3, 4].

Quote:

PS. Be mindful when using "<-". Seems fine here, but can easily become ambiguous (and break LL1).

Can you give me an example where it might become ambiguous?

Quote:

* "val f = printStuff("Hello world!") #this creates a functor with the first argument bound" interesting, but i don't think i would be ok with it.
I assume the functor generation is triggered by missing parameter (or alternatively by missing invocation if it had no parameters to begin with)? What happens when someone down the line adds a printStuff with one parameter? Stuff breaks in a bad manner (possibly non-compiler detectable semantic change because of unrelated changes elsewhere - add inheritance to that and shit is bound to hit the fan). That is a no-no :/.
I strongly recommend adding explicitness (i like the idea itself and will at some point consider adding something like that myself).

I'm thinking of adding a bind-keyword, which would work so, that you select a function or a method like this: bind(int, int) funcName(params_go_here), where the parameters to bind are the types of the parameters to the function, and the parameters to the function itself are the parameters you want to bind. This way function overloading can be done. The parameters to the bind can also be omitted, but if there is ambiguity, the compiler will give an error: bind funcName(7). And then I could add something like an addr-keyword that simply takes the address of a function without creating a functor: addr(int, int) funcName.

Quote:

Slightly related: i can use function invocations as variables - which your mentioned stuff would disallow (a'la javascript getters/setters with non-bonkers syntax):

Code:

context Example // ie. i do not care what Example is (class? struct? namespace?), i just want to conveniently add my junk to it, so set it as current context.

Int #hidden_int = 42 // a "jailed"/restricted integer (PS. Int is alias for Int32, there is no such thing as just Int)

static i_am_int_var \ return hidden_int // getter (with no additional logic for brevity)
static i_am_int_var(Int val) \ hidden_int = val; return val // setter (no additional logic also)

static i_am_another_int_var \ return &hidden_int // setter and getter in one (as the value is returned as reference).

static i_am_writeonly_int_var(Int val) \ hidden_int = val // setter only, and does not even return what the value was actually set at.

static i_am_readonly_int_var \ return hidden_int // getter only

// thous are all functions, just that you do not have to use them as such.

static test
Int i_am_a_real_int_var = 10
i_am_a_real_int_var -= i_am_int_var // OK
i_am_int_var += 10 // OK

i_am_another_int_var += i_am_another_int_var + i_am_readonly_int_var // OK

i_am_writeonly_int_var = 0 // OK
i_am_writeonly_int_var += 666 // ERROR!

i_am_readonly_int_var = i_am_a_real_int_var // ERROR!

This intrigues me. I might later on add a special keyword that allows me to use methods as getters and setters. I need to think about it.

Quote:

Since you mentioned memory/object-lifetime management i mention mine :). Since my language is meant for scripting and not to be a standalone language then it will use whatever the host language provides. So far i already have (not made with the language in mind, just made prior for whatever):
* Incremental mark-and-sweep garbage collector implemented as optional trait that can be added to any C++ class.
* Reference counting, similarly as a optional trait.
* Single-owner/Unique
On memory management part there is also:
* Non-freeable allocator as trait.
* Pool as trait.
* Single-owner random chunks of memory.

I intend to expose it on scripting side by generic means (ie. rule and code lists). "class" uses GC by default, "struct" does not and in fact cannot as it has an immutable trait of "trivial-copy".

What if you want a class to use several different memory models depending on the situation? For instance in case of Vec3, you might want them to perform as efficiently as possible, so pass them on the stack. But then there might be the rare case that you want to pass a reference to a Vec3 to some object for instance, so it would be very handy to have it be reference counted in that single case. Is this possible in your lang?

In my lang this whole thing works slightly differently, but this scenario is possible. For each class the compiler makes a "raw struct", and also a "header tag" that is used for boxing objects. The raw struct is exactly what you get when you write a simple C++-class. The header tag is something that can be attached or detached as needed. When there is need for reference counting, the tag is attached. When there is need for type identification at runtime, the tag is attached. Basically whenever there is need for boxing an object, the tag is attached and the object gets all the benefits of a managed object. This can be done for base types such as int, char, float and double, but also for structs. However, the tag is not needed and is not used in cases where the compiler knows about the type and scope of the object.

Quote:

That is still translation :P (One could say that no human language can be directly translated to another - they are different with their own quirks and features, adding stuff that was not in the original is unavoidable when translating).
PS. do not pay much attention to me on that subject - i am just expressing my frustration on the master-hair-splitter teacher i had nearly 20y ago The reality is that any translation in programming language context is compiling as that is how the word is defined (ex: translation of C laguage into machine code language). "Compiler": a tad bit superfluous word nowadays - it used to be badly defined word and now it is just kinda pointless.

Some teachers just tend to be that way I guess :D

Quote:

Yes. While C++14 and its prior versions have made the language less atrocious (... sorta) - it is still shit. ... Iteresting. Inheritance, and well, all of the OOP related stuff, is one of the most fucked up parts of C++ (fucked up way beyond any hope of redemption IMNSHO) :/

Seconded and applauded. Sometimes shit gets coated with all kinds of things, but the essence of it never ceases to smell. Instead it gets used as manure for healthy things to grow.

Quote:

I would say it, as and anti example, is a must-read for everyone going out to write a new language. In addition to specialization questions like thous: (http://www.redmountainsw.com/wordpress/2007/06/11/considerations-when-designing-your-own-programmingscripting-language/) (list seems a bit too short, but could not find anything better :( ). Not the mention the number one advice: DON'T WRITE ANOTHER BLOODY LANGUAGE! ... unless you just want to - then it is fine.

This is a great read. Thank you for sharing this!

zombe on 8/8/2014 at 20:23

Quote Posted by Qooper

You're right, when I say tokenizer I actually mean lexer :D I really don't have any academic background when it comes to compiler stuff.

Last i had any professional supervision was almost exactly 10 years ago - i do not really remember much. Be wary of trusting me ;)

Quote Posted by Qooper

So LL1 needs no lookahead?

Yep, all you need is the next token to know which way it is going - that obviously also means that you do no backtracking. Whenever you encounter a token you did not expect => parse error.

Quote Posted by Qooper

Hmm interesting! This is one place where our strategies seem to diverge, at least I think so. I have only one tree structure before I emit target code. But you make a good point here.

It is somewhat required in my specific case - YMMV. If you do not see much use for it then you would probably better off by not complicating the matters with an additional layer.

Quote Posted by Qooper

So if I wanted to add namespaces like in C++ (I'm not saying I would), would they require/benefit from this internal tree structure?

Namespaces are mandatory in my language. Also, only other namespaces can exist in global scope => there is no such thing as global variables / functions.
... when i said the compiler knows nothing about aliases and type-context - i mean that the parser (in its analysis / conversion phase) removes all aliases and propagates the context to be explicit. Why in my case such an odd thing is needed would be a bit time intensive to explain - so, i won't. It is extremely unlikely any of it applies in your case anyway.

Quote Posted by Qooper

Accessibility is something I haven't yet decided on.

It is not a done subject for me either. Current problem: virtual protected functions cannot be done ... no idea what and if i need to do about it.

Quote Posted by Qooper

Can you give me an example where it might become ambiguous?

"... -10 < -1 ..." vs "... -10 <- 1 ..."

Quote Posted by Qooper

Never had any problems on C++ side - it is just never needed.

Also, my language is supposed to track the usage over function boundaries - so, usually lifetime management code would be omitted. My default parameter passing rule is "by-value", but that does not mean that any actual copies will be made (in fact, no "class" has a default "clone" implementation to begin with - only if the compiler detects that an actual copy is needed will it attempt to do so. Ie. it is legal to pass classes by value even if it is actually impossible to copy the class).

-------------------------------------
Noticed a few fuckups my changes have introduced, so re-updated (well, just rewrote from 0) my production rules to track them down and fix ... still work todo, but enough for today. So, here it is in its current form (some corner-cutting in there too as i intend to use recursive descent parser and a full-bown BNF is just not practical):

Code:

Qooper on 10/8/2014 at 21:15

Quote Posted by zombe

Yep, all you need is the next token to know which way it is going - that obviously also means that you do no backtracking. Whenever you encounter a token you did not expect => parse error.

Quote:

"... -10 < -1 ..." vs "... -10 <- 1 ..."

The way I parse the input code into tokens would actually distinguish between those two cases. < - (with the space in between) will be parsed into two different tokens of type operator, whereas <- (without the space) will be parsed into just one token of type operator. I require spaces between operators, which makes things slightly easier for me.

I think I should, at least at some point, do something similar to that BNF-esque thing you listed. Like I stated earlier, the rules of my language are currently in a rather informal shape. I'm starting to realize this has quite a bit potential for a lot of things. I'm not designing simply a language, but also the "platform" for it. Platform in this case means the different C constructs that my syntax tree gets translated into. The thing is, I could make several different languages that could all communicate with each other without any complicated bridging code, as long as I keep the last step (syntax tree -> C) the same.

R Soul on 10/8/2014 at 21:53

I have a syntax suggestion. Allow this:
if(4 < n < 7) ... as well as the usual if(n > 4 && n < 7) ...

zombe on 10/8/2014 at 22:47

Quote Posted by Qooper

Yeah, though that it likely is not a problem for you - just to be wary.

Btw. i too use greedy tokenizer (Without requiring spaces between tokens tho. With one exception: all concatenations of "+", "-" ... so, "+-" and "---" will fail at tokenization time. To name an atrocious unambiguous/well-defined example: "--x -= --x - - - - x--"). When parsing, due to LL1 nature, whenever i encounter a token that was not expected i "degrade" the token if allowed and would provide the token the parser asks for. Ex: ">>=" -> ">" ">" "=", ">>" -> ">" ">".

------------------------------
On my front: did not have time to do anything yesterday, but today was quite fruitful.

Decided that i need to write down the pseudocode for the lexer (the part pertinent to the "Off-side rule" to ensure sanity in my production rules):

Code:

tokenizer pseudocode
indent = [0]
while !EOF
indent_at <- find_beginning_of_next_token // skip whitespace/comments
tok <- read_and_create_token
if new_line
if tok in continuation_tokens ("," "+" "-" "/" "*" "&&" "||" "^^" "?" ":" "->")
if indent_at > indent.last
WARNING: bad indentation for continuation // can be safely ignored
else
if indent_at == indent.last
if not out.empty // eat leading "\n"
out.push "\n"
else if indent_at < indent.last // start of block
indent.push indent_at
out.push "\OPEN"
else // end of one or more blocks
do
out.push "\CLOSE", "\n"
indent.pop
while indent_at < indent.last
if indent_at != indent.last
ERROR: indentation mismatch // ignore via: indent.push indent_at; out.push "\OPEN" -> will cause a parse error
out.push tok // or if last != 0: indent.last = indent_at -> might work
while indent.last != 0 // close open blocks
out.push "\CLOSE", "\n"
indent.pop
if out.last == "\n" // eat trailing "\n"
out.pop
out.push "\EOF"

Also, the production rules have seen quite a lot of changes and fixes - almost all done. And the followup (Writing down what and how happens in the end. Also writing down what and why i discarded.) is progressing nicely too.

Code:

'??' - nonterminal symbol
<??> - terminal symbol representing a group (a'la: numbers, names, etc)
"??" - terminal symbol
() - mandatory choice list
[] - optional choice list
| - option separator
.. - optional repetition of preceding element without separators
?? - optional repetition of preceding element with ?? as separator
note : degradable tokens: ">>", ">>=" - nothing else is degradeable and the longest match for a token is taken.
note : nonsensical strings of "+" and "-" (a'la "---", "+-7") generate Lexer errors and do not generate tokens.
================================================================================ various bits an pieces
'typeVar' : ["read] ( "var" | 'type' ) // note: "var" is valid even in 'defContext'! (especially good for: "var {a, b] = ...")
'type' : 'typeBase' ['typeStore']

'typeStore' : ( "&" | "*" | "[" ['expr'] ]" ) .. // note: degrades "&&"

'typeBase' : [<Type> . ".] <Type> ["<" 'typeSub' , ">] // note: degrades ">>", ">>="
'typeSub' : ('typeBase' | "_") // can omit trailing ",", ex: "Foo<T1 = A, T2, T3 = B>" -> "Foo<_, C>"
: 'typeAlias' ["=" 'typeBase'] // only valid in 'defContext'. note: storage is not part of type. degrades ">>", ">>="
'typeAlias' : "<" <Type> ">"

--------------------------------------------------------------------------------
'varName' : ["#] <name> [<tag>] // "#" and <tag> are only valid in 'defStatic' and 'defContext'
'varBlock' : ( "\" 'varList' ; | "\OPEN" ('varList' ; ) \n "\CLOSE" )
'varList' : 'typeVar' 'varElem' ,
'varElem' : 'varName' ['varConstruct' | "=" 'expr']
: 'deconstruct' ["=" 'expr']
'varConstruct' : 'construct'
: ["." <name>] "(" 'expr' , ")"

--------------------------------------------------------------------------------
'fnParams' : ( ['typeVar' | 'typeAlias'] 'varElem' ) ,
'fnResult' : "->" 'typeVar' ["=" 'expr']

--------------------------------------------------------------------------------
'constructObj' : "new" 'typeBase' 'construct'
: "new" 'typeBase' ["." <name>] [ "(" 'expr' , ")]
: 'typeBase' 'construct'
'deconstruct' : "{" (['typeVar'] 'varName') , "}" // ex: "var {a, b} = {1, 3.14}" "Int a, {Float f} = {3.14}"
'construct' : "{" ([ ['typeVar'] "=" ] 'expr') , "}" // ex: "return Foo{Int& = a, b}"
'extent' : ['expr'] (".." | "<=" | ">=" | "<" | ">") ['expr']

--------------------------------------------------------------------------------
'alias' : ("!" | "as" <Type>) // ex: "alias Foo.Bar.Snafu!" == "alias Foo.Bar.Snafu as Snafu"

================================================================================ SOURCE
'source' : ['def'] \n "\EOF"

--------------------------------------------------------------------------------
'def' : 'defFunction'
: 'defContext'
: 'defAlias'
: 'defStatic'
: 'defInterface'
: 'defOperator'
: 'defUnwrap'

--------------------------------------------------------------------------------
'defContext' : ("context" | "namespace") 'typeBase' ['alias']
: ("class" | "struct") ['typeBase' "->] 'typeBase' [<unit>] ['alias'] [<tag>] ['varBlock']

'defAlias' : "alias" ('typeBase' [<unit>] 'alias') ,

'defStatic' : ["const] 'varList'

'defFunction' : ["final] ("function" | "override" | "abstract") ['typeBase' ".] <name> ["(" 'fnParams' ")] [<tag>] ['fnResult'] 'codeBlockTrail'

'defInterface' : "interface" ['typeBase' ".] <name> "->" (["#] <name> ["(" 'typeBase' , ")]) ,

'defOperator' : ["final] "operator" <name> ['typeBase' ".] ("+" | "-" | "*" | "/" | "%" | "&" | "|" | "^" | "<<" | ">>" | "==" | "!=" | ">=" | "<=" | ">" | "<" | "&&" | "||" | "^^")
['type' | 'typeAlias'] <name> ['fnResult'] 'codeBlockTrail'
: ["final] "operator" ['typeBase' ".] ("++" | "--" | "!" | "~" | "-" | "+") <name> ['fnResult'] 'codeBlockTrail'
: ["final] "operator" "[" (['type'] <name>) , ]" ['fnResult'] 'codeBlockTrail'
: ["final] "operator" <name> ['typeBase' ".] ("++" | "--") 'codeBlockTrail' // clones source object before calling this (note: no return value)
: "operator" ['typeBase' ".] "new" ["." <name>] ["(" 'fnParams' ")] ["clone] ['fnResult'] 'codeBlockTrail' // "clone" -> define clone operator using this constructor (one parameter, no conversions - good direct/unwrap match)
: "operator" ['typeBase' ".] "bye" ["." <name>] ["(" 'fnParams' ")] 'codeBlockTrail'
: "operator" ['typeBase' ".] "clone" [ <name> 'codeBlockTrail' ] // this = new_nonbuilt_obj, name = source_obj. missing-code: "operator clone obj \ this(obj)" -> use copy-constructor

'defUnwrap' : "unwrap" ['typeBase'] "->" ['fnResult'] 'codeBlockTrail'
: "unwrap" "return" 'expr'
: "unwrap" <name>

================================================================================ code
'statements' : ('statementC' | 'statementV') ;
'statementsV' : 'statementV' ; // variable declarations and expressions only
'statementBlock': "\OPEN" 'statements' \n "\CLOSE"

'codeBlock' : ('statementBlock' | 'statements')
'codeBlockTrail': ['statementBlock' | "\" 'statements']
'codeBlockCase' : ['statements'] ['statementBlock'] // combine - 'statements' begins where 'statementBlock' indent is

--------------------------------------------------------------------------------
'statementC' : 'jumpContinue' // note: all begin with a terminal
: 'jumpBreak'
: 'jumpReturn'
: 'jumpGoto'
: 'flowIf'
: 'flowWhile'
: 'flowDo'
: 'flowForeach'
: 'flowSwitch'
: 'flowScope' // not allowing 'statementBlock' directly
: 'flowDefer'
: 'label'
: 'assume'

'statementV' : 'varList' // both might begin with 'baseType', but will diverge immediately afterwards
: 'expr' // ...

--------------------------------------------------------------------------------
'jumpContnue' : "continue" ['exprTern']
'jumpBreak' : "break" [<name> | <value>]
'jumpReturn' : "return" ['expr']
'jumpGoto' : "goto" <name>

'flowIf' : "if" 'statementsV' 'codeBlockTrail' ["else" 'codeBlock']
'flowWhile' : "while" ['statementsV'] 'codeBlockTrail'
'flowDo' : "do" 'codeBlock' "while" 'statementsV'
'flowForeach' : "foreach" 'statementsV' "in" 'expr' // ie. "foreach var a = 10; a.foo() - 42; _ in ..."
'flowSwitch' : "switch" 'statementsV' "\n" 'swCase' \n
'flowScope' : "scope" 'codeBlock'
'flowDefer' : "defer" ["?] 'codeBlock'

'label' : "@" <name> ['statement']

'assume' : "assume" 'exprBool'

--------------------------------------------------------------------------------
'swCase' : "case" ["@" <name>] 'swOption' , ":" 'codeBlockCase'
'swOption' : ('extent' | 'expr') // Note: 'extent' begins with 'expr' (ie. first build 'expr' and then 'extent' if possible)

================================================================================ expression: precedence rules
'expr' : 'deconstruct' "=" 'exprAssign'
: 'exprAssign'
'exprAssign' : 'exprTern' [("+=" | "-=" | "*=" | "/=" | "%=" | "&=" | "|=" | "^=" | "<<=" | ">>=" | "=" | ":=") 'exprAssign'] // note: ":=" makes anything besides just <name> before it invalid

'exprTern' : 'exprBool' ["?" 'expr' ":" 'expr']

'exprBool' : 'exprIs' [("&&" | "||" | "^^") 'exprBool']
'exprIs' : 'exprCmp' [("is" | "instanceof") 'typeBase'] // "is" -> is-a (public interfaces), "instanceof" -> exact type match
'exprCmp' : 'exprAdd' [("==" | "!=" | ">=" | "<=" | ">" | "<") 'exprCmp']
'exprAdd' : 'exprMul' [("-" | "+") 'exprAdd']
'exprMul' : 'exprBin' [("*" | "/" | "%") 'exprMul']
'exprBin' : 'exprMov' [("&" | "|" | "^") 'exprBin']
'exprMov' : 'exprUn' [("<<" | ">>") 'exprMov']
'exprUn' : ("!" | "~" | "-" | "+") 'value'
: "&" 'value' // note: there can be only one (cannot degrade "&&" and "& &foo" is also wrong)! "&" means that value should maintain its indirection ("new" does that too), ex: "Int&& foo, bar = &foo".
: ("++" | "--") 'value'
: 'value' ("++" | "--")
: 'value'

================================================================================ value
'value' : <const>
: "(" 'expr' ")" // note: value is expression - if expression would also be value then infinite recursion on syntax error ... that would be bad
: "null" ["(" 'type' ")]
: 'lambda'
: "_" // anonymous/internal variable
: 'construct' // anonymous structure: "{1, 2}[0] == 1" -> because properties have same type and hence the structure can be treated as an array
: "[" 'extent' ]"
: 'constructObj' // both might begin with 'baseType', but will diverge immediately afterwards
: 'lookup' // ...

--------------------------------------------------------------------------------
'lambda' : "$" ["&] [<name> ,] ["(" 'fnParams' ")] ['fnResult'] 'codeBlockTrail' // type: Handle (exact type is tracked by compiler)

-------------------------------------------------------------------------------- lots of cutting corners here
'lookup' : ['typeBase' | "this] ["." "super] . 'lookupBody' // 'typeBase' only valid on initial lookup, ex: "Foo.handlerVar.this.super.getter(7)" calls handlerVar(7) with this = this.super.getter
'lookupBody' : 'varConstruct' // call dependent constructors (will be done automatically first if not done explicitly)
: "." ('fnOperator' | <name>) "(" ['typeBase' ,] ")" // ex: "&Foo.bar.dooh(Int, Int)" - not allowed to omit the signature!
: "." 'fnOperator' "(" 'expr' , ")" ['fnInvokers'] ["." 'lookup'] ??? extra bloody dots everywhere
: [".] <name> ['fnInvokers'] ['lookupBody'] // "." can only be omitted if there is nothing prior it, 'varConstruct' is not applicable anymore.

... eh, redo this crap

'fnInvokers' : ( "(" 'expr' , ")" | "[" 'expr' , ]" ) ..

'fnOperator' : ("+" | "-" | "*" | "/" | "%" | "&" | "|" | "^" | "<<" | ">>"
| "==" | "!=" | ">=" | "<=" | ">" | "<" | "&&" | "||"
| "^^" | "++" | "--" | "!" | "~" | "-" | "+" | "[" ]"
| "\" ("++" | "--")) // post- increment/decrement

... still some crap to straighten.

----------------------------------

Quote Posted by R Soul

I have a syntax suggestion. Allow this:
if(4 < n < 7) ... as well as the usual if(n > 4 && n < 7) ...

That would be nice ... but i can not think of a way how to do it :/

Not anything clean at least - and ugly hacks might lead us back to C++ nightmares.