Thinking of implementing socket support into Plotscript

[Chronoboy]

This message is mainly for James and TMC, but I wouldn't mind hearing from the rest of the community as well on this.

I am sort of thinking of diving into the plotscript engine and adding network TCP socket support. It will only support client-side sockets for obvious reasons, but this can enable games within plotscript to connect to a remote server via raw TCP to send/receive data. This could be used for say keeping a central high score board for platform type games, perhaps implementing some sort of multiplayer, where you can transmit your players location to the server, and receive the location of other players from the server, and perhaps to copy protect your awesome games using a license server. There are actually way too many uses to count.

Now, I know in the wiki that multiplayer was never ever going to be a considered feature, however, this wouldn't be a "multiplayer" feature, but rather a raw TCP socket, which game designers would be free to use however they please. This may open up potential privacy issues, so if I were to implement this feature into the engine, there will be a notice upon launching a game which uses TCP commands to inform the player that this game may connect to an Internet server to transmit/receive data.

So, my actual question to James and TMC, is how difficult would it be to add new plotscript commands into the engine? I haven't worked much with either the compiler or the interpreter yet. Also, does the FreeBASIC port to Android and OUYA support TCP connections?

Here is the related FreeBASIC documentation on sockets: http://ext.freebasic.net/dev-docs/files ... cp-bi.html

I also cannot stress this enough, this isn't a feature request, I plan on implementing this entirely on my own for the benefit of the community as a whole, regardless if it is used by anyone else besides myself. I have been lots of network programming in Python recently, and would love to take my skills to the next level by implementing a full socket client into OHR. As this would open up many new possibilities and potential for the engine.

I would also like to note, that regardless how much backlash or negativity the community might have against adding such plotscript commands to the engine, I absolutely want to add such an ability. You don't have to use the feature if you don't want to, but at least it will be there for designers who want to exploit networking in their games in some manner.

Oh, another interesting socket idea would be a Puckamon monster trading component. So you can trade your collected Puckamon with other players online. As one of the goal of Pokemon was to trade with players with the opposite game as your own. Blue never had some Pokemon Red had, for example. It could also be used for battling other players with your collected Puckamon. This would be a very primitive form of multiplayer a socket client would enable. It wouldn't be much work either on the server-side to implement.

If this feature ends up becoming science fact, rather than science fiction. I'd be more than happy to help game designers out with developing the required TCP server to handle what they want to enable within their game. After I complete said feature, I plan on providing various tech demos, in the form of client RPG files, and fully working servers which you can play around with. I will create a high score board server with web interface, among other interesting examples of how you could exploit network sockets within your game.

Anyways, that's a wrap of what I want to start working on. If the plotscript engine is easy to manipulate, you should be able to look forward to something within the next month. I hope to have the actual feature ready for the next official release of the engine, so that it can be officially included in a stable version.

Without further ado, I am going to dive into the OHR source code again to see how the plotscript engine works.

UPDATE: Okay, so I just dived into the yetmore.bas source file, where most of the newer commands are, namely the slice commands, and this is pretty straight forward... I don't even think this will take me a month to implement, but I still stand on that month just in case my personal life gets in the way. But I think I will see about focusing on this for the next couple evenings after work and on the upcoming weekend. I hope to actually test it by maybe as early as next week. Now, I need to look at the HamsterSpeak compiler, as that might be a whole other thing... Oh, and this implementation will only allow a single TCP connection at a time, so you won't be-able to connect to multiple servers at once. I don't think this will be a problem tho, as I don't see anybody needing to connect to more than a single server. You can of course close an existing connection and open a new one to a different server. It will use the string space available within the plotscript engine to tell it which address to connect to. The port will be a standard integer variable. Data can then be sent by a null-terminated string, or by single bytes using integer variables. I may enable an option to switch the string terminator to say a linefeed instead of a null.

UPDATE 2: Wow, from what I can see the HamsterSpeak compiler really does nothing in terms of knowing the commands, so it appears that new commands are just added into plotscr.hsd with a new ID and required parameters for the compiler. Then, after adding the command here, all I need to do is add what the command does into yetmore.bas, so I don't really need to recompile hspeak.exe, I just need to compile GAME.EXE and test my new HamsterSpeak commands. This is actually much easier than I originally thought it would be. I may document the entire process of adding completely new HamsterSpeak commands that run custom FreeBASIC code after I am able to implement this feature so that anybody else who might have ideas on what they would like to see implemented into HamsterSpeak can easily refer to a document, and add code directly into game. I may consider adding other interesting commands to HamsterSpeak as a result if this process goes well. This may improve the overall development cycle of speed of adding new requested features into HamsterSpeak if another develop(such as myself) is able to work with the code. Such the much anticipated Battle Script code... Also, since I do have programming experience when it comes with writing Virtual Machines, I might be-able to help out in terms of cleaning up the interpreter to allow adding much needed features such as Battle Scripting easier to implement. However, it might just be best to re-implement the entire battle system directly in HamsterSpeak, now that slices are a reality. Anyways, I am going to start work with adding networking support into HamsterSpeak.

[Bob the Hamster]

I always wanted to save network support until after plotscripting gained support for real strings and real arrays and structs.

Maybe that isn't a reasonable prerequisite, I don't know.

I used to feel pretty strongly opposed to network support, but now I am a bit more open to it, but still cautious.

I am glad you were able to figure out how the command implementations in yet more.bad work :)

[sheamkennedy]

On a kind of related note, I've always wanted to know how new features get coded in to the engine. I'd love it if you were to perhaps make videos documenting how to do this sort of thing. The simpler the feature you add in this sort of tutorial video the better. This has always been something that interested me but I really have no idea where to start nor much time usually to figure it out and ask around. I'm sure others may be interested in knowing these inner workings too, so it would be great if there were short reference videos showing how it's done.

[TMC]

I definitely want to see network support too. This is one feature though that especially deserves careful planning of interface and implementation. However I think that rapid prototyping is a great/the best way to plan something, as we can then play around with it (not publically exposed) and make changes.

I want to have a network API that's as high level and easy to use as possible. Let's think about how can we can make common tasks like highscore tables simple. However many or most of the things we would want won't be possible until we have a new script interpreter. We can add some network commands before then if it's done in a cleanly compatible way, though I imagine most of them would end up being obsolete anyway. I'm also in favour of supporting UDP packets (or rather, something like TCP with weaker guarantees; there are many libraries for that) and peer-to-peer connections for those people who want to make fast-paced multiplayer games, but that's very much optional and can be added later.

Making it as simple as possible to use means the server side (if it exists, as opposed to direct peer-to-peer, if we add that) also needs to be simple. Ideally almost no one would need to set up a web server, because we could set up central servers and give out accounts. It would also be ideal if you could avoid having to write any code in a language other than HamsterSpeak, at least for common patterns like the server acting as a match-making service following by just forwarding packets between clients (if peer-to-peer isn't implemented). It would also be possible to run the HS interpreter on the server (sounds like fun!), though of course anyone who knows any other language wouldn't want to use HS for that.

I've done very little network programming. But I feel like a request and response interface like HTTP will cover many use cases. I started jotting done some ideas on the wiki (without deleting the bad ones :), and would like to see other's ideas too.

http://rpg.hamsterrepublic.com/ohrrpgce ... rk_support

After playing around a bit, I feel like there isn't really any difference between request-response and a message stream unless you have the ability to send multiple parallel requests.

It would even make sense to be able to send objects like NPCs and slices over the network, which will have a lot of fun applications.

Regarding library, I don't know what to use. The TCP functions in FB.ext are badly documented (eg it doesn't say how to open a socket) and I don't like the look of it. (As far as FB sockets libraries go, chiSocket was a lot better, but that's disappeared.) SDL_net seems far better. It's just 1500 lines of C, but it sure hides a lot of messy detail. However it provides very few features on top of TCP and UDP sockets, so I wonder whether there's something more featureful available. However, for prototyping it just doesn't matter.

----
I've written an article to document the process: How to add a new script command, as well as putting more information and lots of cross links everywhere.

I'd love it if you were to perhaps make videos documenting how to do this sort of thing.

James put together a video on fixing a bug:
https://www.youtube.com/watch?v=KSO2Hc4ua2Y
Even that is quite long (it can take a while to figure out a bug). I'd like to put up a video showing a simple feature implementation, though that could easily be less watchable: when you're debugging at least you run the game a lot.
Alternatively, James or I could put up something more like a tutorial rather than a raw recording, cutting out all the time wasted figuring out how to implement something and starting over halfway through. You can use your imagination for those bits ;)

[Bob the Hamster]

If I was going to use networking in a game, here is the first project that would come to my mind.

I would want to write a turn-based 2-player strategy game, similar to Hero Academy. I imagine not many of you have played it, so to make the analogy easier, lets pretend instead that I was implementing a 2-player OHR Chess game.

The game would connect you to a server with a username and a password. You would request a game, and the server would match you with an opponent, either randomly, or specifically by requested username.

I would want a choice of a generic in-game account creation tool, or something more customizable. It might also be nice to have an option for a nickname/password reservation system more like how names work in IRC (Minetest uses this so people can have persistent usernames on servers without the need for a central name server)

I would write a script that lets the player make their chess move, and then I would sent that data off to the server. Ideally I would be able to create a struct-like object (like a reload tree) with my script, have a reference to it stored in a variable, and then have a command that sends that structure to the server

The server would then be responsible for verifying if it is a legal chess move.

When my opponent plays next, the script would ask the server to send any pending moves, and it would reply with either false, or a reload-esque data structure which would be saved into a variable, and then the script would animate my move. Then that player could make their move, and it would be sent back to me the same way.

That is a very specific, very high-level way of doing asynchronous multiplayer, and I am sure there is a whole world of other ways people would want to do it, but that is the one that I would be most interested, and that is why I want to be able to have complex data structures in plotscripts before I would attempt a networked game.

If I have to send and receive single integers and patch them together somehow in my script, I am just not interested, personally.

[TMC]

(In your example does the server actually need to verify anything? It sounds like the opponent can do that.)

Exposing RELOAD to scripting would be neat for defining and editing data in Custom, using the reload editor and editedit, or a tabular interface. But plain dicts, arrays, and objects (class instances) would probably be easier to deal with, so I wonder whether there's any place for RELOAD in scripts at run time. I don't see why it would be preferred for network transfer.

In practice, if you're scripting a game with structured data today, then you're probably storing it in slices. Which is why being able to transfer slices over the network would be extra useful.

[Bob the Hamster]

Yeah, I didn't really mean I needed reload support directly in scripts, I just meant something like it. Arrays, dicts, and objects would be more appropriate.

In a game like that, I would actually keep track of a lot of state in the server to prevent cheating. The ohrrpgce client would just do move-input interface and move animation replay. I would probably actually have the server keep track of the whole gameboard. It would mean more work on the server (which I could do in python, or some other more suitable language) and it would have the benefit that you couldn't just hack your client so that your pawn can teleport across the board next to the king.

Now I am focused on asynchronous multiplayer, but it would also be interesting to try and think out what would be needed by somebody trying to do real-time multiplayer. For example, if Gunpigs was to become network multiplayer, What would be needed to keep the players in sync? How would you avoid allowing lag to give one player an advantage over the other?

Those are puzzles that smart people have already solved (MANY times over) but I know nothing about the answers, having zero real-time network multiplayer experience (Spitwars IPX multiplayer doesn't count, Thorbrian did all the heavy lifting on that one)

[sheamkennedy]

I would love to see some edited tutorials showing the implementation of a simple feature. I realize this is a bit of work on whoever throws the tutorials together so only do it if you think it would be helpful to others too. Also I feel like a lot of background information may be needed to explain things since I would consider myself an absolute beginner.

[TMC]

Hmm, in that case sending slices through network streams could be less than ideal. But having a server that does anything nontrivial probably always complicates things.

I've never done any netcode programming; I've only read a few articles on it. Here's two great starting points:

https://news.ycombinator.com/item?id=7508350

https://news.ycombinator.com/item?id=8399767 - in particular, see this short thread which explains what the Source engine does. It's very interesting that there are two "opposite" solutions to the problem. To quote:

In a laggy environment, you can either have dodging work 100% correctly client side, if you dodged it, it didn't hit you, no questions asked . . . or you can have aiming work that way, if you hit it, you hit it, no questions asked.

You can have one or the other. You cannot have both. (And in a server-based setup, you generally get neither).

Synchronising between multiple clients is a hard problem, so it would be ideal if we could provide it somehow through built-in features --- few users would be able to get it right themselves. It's an interesting question. But I suspect it's not possible to do without applying a sledgehammer like record-and-replay logging of the intrepreter.

[Chronoboy]

I apologize for being absent for a bit here, I was moving into my new place and finally got my home Internet installed... Anyways, I read through all your replies, and here's some thoughts on your thoughts...

While I do like the idea of high-level network programming in HamsterSpeak, where you can literally send objects over the wire. My main worry about this, is having to write overly complex server-side code to decode and reencode said objects... This can add processing lag on both ends. Low-level networking on the binary level is much quicker for a CPU to process and parse. So, the initial networking support will focus on low-level networking, where you can send integers and strings over the wire. The strings will be in Pascal format to keep server-side and client-side parsing simple. Pascal strings are stored exactly how OHR stores most of it's own strings. Where the first byte is the length, followed by the actual string. This is much easier to handle than C-style null-terminated strings when working with streaming data.

At a later date, once we figure out the low-level networking portion and things are going smoothly, then if the community believes that being able to send OHR objects, such as slices, and RELOAD data over the wire, we add a couple new script commands to handle that. So, here's the initial command-set I am thinking of:

Code: Select all

connect
close
send
recv
connected

Along with some new plotscript triggers, for when data is received, and when a socket disconnects. The connect command should be blocking initially, or at least during the development and testing. The return value will state if the socket was opened/connected successfully. Connecting should never take longer than 5 seconds on most broadband connections, as not really much data is sent in the initial packets.

Initially we should support TCP, as it's easier to code on both the server and client side, and it's more reliable than UDP. I've only personally coded TCP sockets, so I will need more time to get myself familiar with UDP before I'd add support for that.

In summary, initially we should only support low-level sockets, like the sending of integers and strings. Later on, we can add in support for high-level networking, but low-level will always be supported for games/apps that would want to use it.

Also, to keep things simple... We should really only allow a single socket connection at a time. I honestly don't see why a game would need to connect to 2 different servers and ports. So, the game engine itself will manage the underlying socket variable, and if connect is called a second time, the previous socket is closed, or we can return an error. The latter is most likely better. This way, we don't need to hand over a variable to each function, all send will need is either an integer or a string, and recv will only take one parameter, which says what we are trying to recv, either an integer or a string.

[TMC]

It seems difficult to avoid ending up with an old networking interface and a new one later, however I would still hope the two can be as similar as possible, including sharing commands that don't need to change, to avoid confusion and code duplication. Likewise the protocol should be extended later rather than replaced.

I'm not sure what the objection to encoding/decoding objects is. I'm not proposing anything more complicated than adding a byte with the type of every piece of data, which probably isn't actually different to what you're suggesting. In future more types aside from strings and ints can be added, including structured types like arrays and objects. Encode/decode times are going to be on the order of a million times too small to be significant. A typed BNF description:

Code: Select all

packet_data := header array_datum
array_datum := int8:3 int32:length datum*   # The toplevel data in a packet is stored in an array
datum := int_datum | string_datum    # Arrays not allowed
int_datum := int8:1 int32:value
string_datum := int8:2 int32:length (int8:char)*

Should use "receive" instead of "recv", as we have very few existing abbreviations in command names. They should also have "socket", "network", or something else in the name. But anyway, what arguments did you have in mind for 'send' and 'receive'?

[Bob the Hamster]

How is a low-level network interface with only send/receive going to deal with the fact that we don't have script multitasking yet?

The running script causes everything to wait, so does that mean that the whole game pauses while we wait for the server to reply?

or will you "send" some data, and then have a command that checks to see if a reply is waiting, and you have to set up some system of looping or repeated timers to know when it is okay to call "receive"?

The thing about a "low level" network interface, is that it is the exact opposite of "simple". It may be the simplest to implement in the engine, but it is also the hardest possible interface to use in a script.

[Bob the Hamster]

I wrote up a mock-up of what I think it might be like to work with a low-level interface like this

Code: Select all

script, check if connection was opened okay, socket, begin
  if(not(socket is connected(socket))) then(
    show text box(5) # can't connect to server
    wait for text box
    exit returning(false)
  )
  exit returning(true)
end

script, wait for request to be acknowledged, socket, begin
  if(not(wait for reply(socket))) then(
    show text box(6) # reply timed out
    wait for text box
    exit returning(false)
  )
  exit returning(true)
end

script, check for server acknowledgement, socket, valid reply=1, begin
  if&#40;socket receive&#40;socket&#41; <> valid reply&#41; then&#40;
    show text box&#40;7&#41; # uexpected reply, server didn't understand
    wait for text box
    exit returning&#40;false&#41;
  &#41;
  exit returning&#40;true&#41;
end

script, send high score to server, score, name string id, begin
  # Return value is the slot that the high score was saved in, 1 for first place, 2 for second place, etc...
  # or -1 to indicate that saving the score failed
  return&#40;-1&#41;
  variable&#40;socket&#41;
  socket &#58;= connect to server&#40;server&#58;foo&#41; # Servers are configured in a lump and referred to by ID number
  if&#40;check if connection was opened okay&#40;socket&#41; == false&#41; then&#40;exit script&#41;
  while&#40;true&#41; do&#40; #single pass loop so we can break out
    socket send&#40;socket, 37&#41; # command 37 tells the server that the next data sent will be a high score
    if&#40;wait for request to be acknowledged&#40;socket&#41; == false&#41; then&#40;break&#41;
    if&#40;check for server acknowledgement&#40;socket&#41; == false&#41; then&#40;break&#41;
    # Server acknowledged, which means it understood, and is ready.
    # Now we send the score.
    socket send&#40;socket, score&#41;
    if&#40;wait for request to be acknowledged&#40;socket&#41; == false&#41; then&#40;break&#41;
    if&#40;check for server acknowledgement&#40;socket&#41; == false&#41; then&#40;break&#41;
    # Server acknowledged to indicate it received and understood the score
    # Now the server is waiting for the name
    socket send string&#40;socket, name string id&#41;
    if&#40;wait for request to be acknowledged&#40;socket&#41; == false&#41; then&#40;break&#41;
    if&#40;check for server acknowledgement&#40;socket&#41; == false&#41; then&#40;break&#41;
    # Server acknowledged to indicate that the name was received.
    # Now the server is waiting for us to confirm that we are ready to receive the score slot
    socket send&#40;socket, 1&#41;
    if&#40;wait for request to be acknowledged&#40;socket&#41; == false&#41; then&#40;break&#41;
    variable&#40;slot&#41;
    slot &#58;= socket receive&#40;socket&#41;
    return&#40;slot&#41;
    break
  &#41;
  disconnect from server&#40;socket&#41;
end

It would be painful to write, but it could certainly work.

This example just assumes that the script will block the game until the server communication is finished. A version that runs in the background would require some timer voodoo.

[TMC]

Sending and receiving are both asynchronous operations, and importantly sending a packet to the server doesn't imply that you expect to get a reply, so the 'send' command has nothing to wait for anyway (aside from the TCP acknowledgement). Server messages could also arrive out of the blue. Let's call this packet-oriented.

On the wiki page (which I haven't looked at since) I mentioned the request-and-response paradigm. It's easy to implement on top of the raw packet-orientated interface, and it's what people will in practice normally be using, except for realtime things. We should add builtin commands for this: to send a request and wiat the response. The main difficulty is when multiple scripts at once try to communicate with the server. Then it's necessary to look at the packets coming from the server and figure out where they should go back to. Also, packets that are responses to requests should get removed from the stream so that they're not visible to scripts. (That's why these should be built-ins rather than scripts in plotscr.hsd.) That will let you mix and match packet- and request-orientated interfaces completely freely.

Also, we talked about adding script triggers for when data is received, but I've thought some more about that and feel more comfortable with specifying a callback script instead. The only difference is that you pass a script id to the 'connect' command instead of setting it in Custom. We could even allow callbacks for the replies to individual requests. Callbacks and triggers and scripts waiting on a socket would have to be delayed until the start of the next tick. (By then the data might have already been read by another script)

Code: Select all

request := send request(...)
wait for request(request)
... := read reply(request, ...)

Well, on to James' objections. The only problem here is that we don't have script multitasking yet. Well, I can't see any way around that but I also don't see any special reason to complain, as it's not an inherent flaw in the interface, and there are many other things (any other wait statement!) that cause the same problem and necessitate a workaround where you don't call any wait statement. Incidentally script multitasking is one of those things I've been delaying until Callipygous is released...

[Bob the Hamster]

Oh!

I totally missed the wiki article. I will go read that.

And the callback idea sounds fantastic :)