We are working on a Top-Down-RPG-like Multiplayer game for learning purposes (and fun!) with some friends. We already have some Entities in the Game and Inputs are working, but the network implementation gives us headache :D
The Issues
When trying to convert with dict some values will still contain the pygame.Surface, which I dont want to transfer and it causes errors when trying to jsonfy them. Other objects I would like to transfer in a simplyfied way like Rectangle cannot be converted automatically.
Already functional
Client-Server connection
Transfering JSON objects in both directions
Async networking and synchronized putting into a Queue
Situation
A new player connects to the server and wants to get the current game state with all objects.
Data-Structure
We use a "Entity-Component" based architecture, so we separated the game logic very strictly into "systems", while the data is stored in the "components" of each Entity. The Entity is a very simple container and has nothing more than a ID and a list of components. Example Entity (shorten for better readability):
Entity
|-- Component (Moveable)
|-- Component (Graphic)
| |- complex datatypes like pygame.SURFACE
| `- (...)
`- Component (Inventory)
We tried different approaches, but all seems not to fit very well or feel "hacky".
pickle
Very Python near, so not easy to implement other clients in future. And I´ve read about some security risks when creating items from network in this dynamic way how pickle it offers. It does not even solve the Surface/Rectangle issue.
__dict__
Still contains the reference to the old objects, so a "cleanup" or "filter" for unwanted datatypes happens also in the origin. A deepcopy throws Exception.
...\Python\Python36\lib\copy.py", line 169, in deepcopy
rv = reductor(4)
TypeError: can't pickle pygame.Surface objects
Show some code
The method of the "EnitityManager" Class which should generate the Snapshot of all Entities, including their components. This Snapshot should be converted to JSON without any errors - and if possible without much configuration in this core-class.
class EnitityManager:
def generate_world_snapshot(self):
""" Returns a dictionary with all Entities and their components to send
this to the client. This function will probably generate a lot of data,
but, its to send the whole current game state when a new player
connects or when a complete refresh is required """
# It should be possible to add more objects to the snapshot, so we
# create our own Snapshot-Datastructure
result = {'entities': {}}
entities = self.get_all_entities()
for e in entities:
result['entities'][e.id] = deepcopy(e.__dict__)
# Components are Objects, but dictionary is required for transfer
cmp_obj_list = result['entities'][e.id]['components']
# Empty the current list of components, its going to be filled with
# dictionaries of each cmp which are cleaned for the dump, because
# of the errors directly coverting the whole datastructure to JSON
result['entities'][e.id]['components'] = {}
for cmp in cmp_obj_list:
cmp_copy = deepcopy(cmp)
cmp_dict = cmp_copy.__dict__
# Only list, dict, int, str, float and None will stay, while
# other Types are being simply deleted including their key
# Lists and directories will be cleaned ob recursive as well
cmp_dict = self.clean_complex_recursive(cmp_dict)
result['entities'][e.id]['components'][type(cmp_copy).__name__] \
= cmp_dict
logging.debug("EntityMgr: Entity#3: %s" % result['entities'][3])
return result
Expectation and actual results
We can find a way to manually override elements which we dont want. But as the list of components will increase we have to put all the filter logic into this core class, which should not contain any components specializations.
Do we really have to put all the logic into the EntityManager for filtering the right objects? This does not feel good, as I would like to have all convertion to JSON done without any hardcoded configuration.
How to convert all this complex data in a most generic approach?
Thanks for reading so far and thank you very much for your help in advance!
Interesting articles which we were already working threw and maybe helpful for others with similar issues
https://gafferongames.com/post/what_every_programmer_needs_to_know_about_game_networking/
http://code.activestate.com/recipes/408859/
https://docs.python.org/3/library/pickle.html
UPDATE: Solution - thx 2 sloth
We used a combination of the following architecture, which works really great so far and is also good to maintain!
Entity Manager now calls the get_state() function of the entity.
class EntitiyManager:
def generate_world_snapshot(self):
""" Returns a dictionary with all Entities and their components to send
this to the client. This function will probably generate a lot of data,
but, its to send the whole current game state when a new player
connects or when a complete refresh is required """
# It should be possible to add more objects to the snapshot, so we
# create our own Snapshot-Datastructure
result = {'entities': {}}
entities = self.get_all_entities()
for e in entities:
result['entities'][e.id] = e.get_state()
return result
The Entity has only some basic attributes to add to the state and forwards the get_state() call to all the Components:
class Entity:
def get_state(self):
state = {'name': self.name, 'id': self.id, 'components': {}}
for cmp in self.components:
state['components'][type(cmp).__name__] = cmp.get_state()
return state
The components itself now inherit their get_state() method from their new superclass components, which simply cares about all simple datatypes:
class Component:
def __init__(self):
logging.debug('generic component created')
def get_state(self):
state = {}
for attr, value in self.__dict__.items():
if value is None or isinstance(value, (str, int, float, bool)):
state[attr] = value
elif isinstance(value, (list, dict)):
# logging.warn("Generating state: not supporting lists yet")
pass
return state
class GraphicComponent(Component):
# (...)
Now every developer has the opportunity to overlay this function to create a more detailed get_state() function for complex types directly in the Component Classes (like Graphic, Movement, Inventory, etc.) if it is required to safe the state in a more accurate way - which is a huge thing for maintaining the code in future, to have these code pieces in one Class.
Next step is to implement the static method for creating the items from the state in the same Class. This makes this working really smooth.
Thank you so much sloth for your help.
Do we really have to put all the logic into the EntityManager for filtering the right objects?
No, you should use polymorphism.
You need a way to represent your game state in a form that can be shared between different systems; so maybe give your components a method that will return all of their state, and a factory method that allows you create the component instances out of that very state.
(Python already has the __repr__ magic method, but you don't have to use it)
So instead of doing all the filtering in the entity manager, just let him call this new method on all components and let each component decide that the result will look like.
Something like this:
...
result = {'entities': {}}
entities = self.get_all_entities()
for e in entities:
result['entities'][e.id] = {'components': {}}
for cmp in e.components:
result['entities'][e.id]['components'][type(cmp).__name__] = cmp.get_state()
...
And a component could implement it like this:
class GraphicComponent:
def __init__(self, pos=...):
self.image = ...
self.rect = ...
self.whatever = ...
def get_state(self):
return { 'pos_x': self.rect.x, 'pos_y': self.rect.y, 'image': 'name_of_image.jpg' }
#staticmethod
def from_state(state):
return GraphicComponent(pos=(state.pos_x, state.pos_y), ...)
And a client's EntityManager that recieves the state from the server would iterate for the component list of each entity and call from_state to create the instances.
Can someone provide a simple explanation of methods vs. functions in OOP context?
A function is a piece of code that is called by name. It can be passed data to operate on (i.e. the parameters) and can optionally return data (the return value). All data that is passed to a function is explicitly passed.
A method is a piece of code that is called by a name that is associated with an object. In most respects it is identical to a function except for two key differences:
A method is implicitly passed the object on which it was called.
A method is able to operate on data that is contained within the class (remembering that an object is an instance of a class - the class is the definition, the object is an instance of that data).
(this is a simplified explanation, ignoring issues of scope etc.)
A method is on an object or is static in class.
A function is independent of any object (and outside of any class).
For Java and C#, there are only methods.
For C, there are only functions.
For C++ and Python it would depend on whether or not you're in a class.
But in basic English:
Function: Standalone feature or functionality.
Method: One way of doing something, which has different approaches or methods, but related to the same aspect (aka class).
'method' is the object-oriented word for 'function'. That's pretty much all there is to it (ie., no real difference).
Unfortunately, I think a lot of the answers here are perpetuating or advancing the idea that there's some complex, meaningful difference.
Really - there isn't all that much to it, just different words for the same thing.
[late addition]
In fact, as Brian Neal pointed out in a comment to this question, the C++ standard never uses the term 'method' when refering to member functions. Some people may take that as an indication that C++ isn't really an object-oriented language; however, I prefer to take it as an indication that a pretty smart group of people didn't think there was a particularly strong reason to use a different term.
In general: methods are functions that belong to a class, functions can be on any other scope of the code so you could state that all methods are functions, but not all functions are methods:
Take the following python example:
class Door:
def open(self):
print 'hello stranger'
def knock_door():
a_door = Door()
Door.open(a_door)
knock_door()
The example given shows you a class called "Door" which has a method or action called "open", it is called a method because it was declared inside a class. There is another portion of code with "def" just below which defines a function, it is a function because it is not declared inside a class, this function calls the method we defined inside our class as you can see and finally the function is being called by itself.
As you can see you can call a function anywhere but if you want to call a method either you have to pass a new object of the same type as the class the method is declared (Class.method(object)) or you have to invoke the method inside the object (object.Method()), at least in python.
Think of methods as things only one entity can do, so if you have a Dog class it would make sense to have a bark function only inside that class and that would be a method, if you have also a Person class it could make sense to write a function "feed" for that doesn't belong to any class since both humans and dogs can be fed and you could call that a function since it does not belong to any class in particular.
Simple way to remember:
Function → Free (Free means it can be anywhere, no need to be in an object or class)
Method → Member (A member of an object or class)
A very general definition of the main difference between a Function and a Method:
Functions are defined outside of classes, while Methods are defined inside of and part of classes.
The idea behind Object Oriented paradigm is to "treat" the software is composed of .. well "objects". Objects in real world have properties, for instance if you have an Employee, the employee has a name, an employee id, a position, he belongs to a department etc. etc.
The object also know how to deal with its attributes and perform some operations on them. Let say if we want to know what an employee is doing right now we would ask him.
employe whatAreYouDoing.
That "whatAreYouDoing" is a "message" sent to the object. The object knows how to answer to that questions, it is said it has a "method" to resolve the question.
So, the way objects have to expose its behavior are called methods. Methods thus are the artifact object have to "do" something.
Other possible methods are
employee whatIsYourName
employee whatIsYourDepartmentsName
etc.
Functions in the other hand are ways a programming language has to compute some data, for instance you might have the function addValues( 8 , 8 ) that returns 16
// pseudo-code
function addValues( int x, int y ) return x + y
// call it
result = addValues( 8,8 )
print result // output is 16...
Since first popular programming languages ( such as fortran, c, pascal ) didn't cover the OO paradigm, they only call to these artifacts "functions".
for instance the previous function in C would be:
int addValues( int x, int y )
{
return x + y;
}
It is not "natural" to say an object has a "function" to perform some action, because functions are more related to mathematical stuff while an Employee has little mathematic on it, but you can have methods that do exactly the same as functions, for instance in Java this would be the equivalent addValues function.
public static int addValues( int x, int y ) {
return x + y;
}
Looks familiar? That´s because Java have its roots on C++ and C++ on C.
At the end is just a concept, in implementation they might look the same, but in the OO documentation these are called method.
Here´s an example of the previously Employee object in Java.
public class Employee {
Department department;
String name;
public String whatsYourName(){
return this.name;
}
public String whatsYourDeparmentsName(){
return this.department.name();
}
public String whatAreYouDoing(){
return "nothing";
}
// Ignore the following, only set here for completness
public Employee( String name ) {
this.name = name;
}
}
// Usage sample.
Employee employee = new Employee( "John" ); // Creates an employee called John
// If I want to display what is this employee doing I could use its methods.
// to know it.
String name = employee.whatIsYourName():
String doingWhat = employee.whatAreYouDoint();
// Print the info to the console.
System.out.printf("Employee %s is doing: %s", name, doingWhat );
Output:
Employee John is doing nothing.
The difference then, is on the "domain" where it is applied.
AppleScript have the idea of "natural language" matphor , that at some point OO had. For instance Smalltalk. I hope it may be reasonable easier for you to understand methods in objects after reading this.
NOTE: The code is not to be compiled, just to serve as an example. Feel free to modify the post and add Python example.
In OO world, the two are commonly used to mean the same thing.
From a pure Math and CS perspective, a function will always return the same result when called with the same arguments ( f(x,y) = (x + y) ). A method on the other hand, is typically associated with an instance of a class. Again though, most modern OO languages no longer use the term "function" for the most part. Many static methods can be quite like functions, as they typically have no state (not always true).
Let's say a function is a block of code (usually with its own scope, and sometimes with its own closure) that may receive some arguments and may also return a result.
A method is a function that is owned by an object (in some object oriented systems, it is more correct to say it is owned by a class). Being "owned" by a object/class means that you refer to the method through the object/class; for example, in Java if you want to invoke a method "open()" owned by an object "door" you need to write "door.open()".
Usually methods also gain some extra attributes describing their behaviour within the object/class, for example: visibility (related to the object oriented concept of encapsulation) which defines from which objects (or classes) the method can be invoked.
In many object oriented languages, all "functions" belong to some object (or class) and so in these languages there are no functions that are not methods.
Methods are functions of classes. In normal jargon, people interchange method and function all over. Basically you can think of them as the same thing (not sure if global functions are called methods).
http://en.wikipedia.org/wiki/Method_(computer_science)
A function is a mathematical concept. For example:
f(x,y) = sin(x) + cos(y)
says that function f() will return the sin of the first parameter added to the cosine of the second parameter. It's just math. As it happens sin() and cos() are also functions. A function has another property: all calls to a function with the same parameters, should return the same result.
A method, on the other hand, is a function that is related to an object in an object-oriented language. It has one implicit parameter: the object being acted upon (and it's state).
So, if you have an object Z with a method g(x), you might see the following:
Z.g(x) = sin(x) + cos(Z.y)
In this case, the parameter x is passed in, the same as in the function example earlier. However, the parameter to cos() is a value that lives inside the object Z. Z and the data that lives inside it (Z.y) are implicit parameters to Z's g() method.
Historically, there may have been a subtle difference with a "method" being something which does not return a value, and a "function" one which does.Each language has its own lexicon of terms with special meaning.
In "C", the word "function" means a program routine.
In Java, the term "function" does not have any special meaning. Whereas "method" means one of the routines that forms the implementation of a class.
In C# that would translate as:
public void DoSomething() {} // method
public int DoSomethingAndReturnMeANumber(){} // function
But really, I re-iterate that there is really no difference in the 2 concepts.
If you use the term "function" in informal discussions about Java, people will assume you meant "method" and carry on. Don't use it in proper documents or presentations about Java, or you will look silly.
Function or a method is a named callable piece of code which performs some operations and optionally returns a value.
In C language the term function is used. Java & C# people would say it a method (and a function in this case is defined within a class/object).
A C++ programmer might call it a function or sometimes method (depending on if they are writing procedural style c++ code or are doing object oriented way of C++, also a C/C++ only programmer would likely call it a function because term 'method' is less often used in C/C++ literature).
You use a function by just calling it's name like,
result = mySum(num1, num2);
You would call a method by referencing its object first like,
result = MyCalc.mySum(num1,num2);
Function is a set of logic that can be used to manipulate data.
While, Method is function that is used to manipulate the data of the object where it belongs.
So technically, if you have a function that is not completely related to your class but was declared in the class, its not a method; It's called a bad design.
In OO languages such as Object Pascal or C++, a "method" is a function associated with an object. So, for example, a "Dog" object might have a "bark" function and this would be considered a "Method". In contrast, the "StrLen" function stands alone (it provides the length of a string provided as an argument). It is thus just a "function." Javascript is technically Object Oriented as well but faces many limitations compared to a full-blown language like C++, C# or Pascal. Nonetheless, the distinction should still hold.
A couple of additional facts: C# is fully object oriented so you cannot create standalone "functions." In C# every function is bound to an object and is thus, technically, a "method." The kicker is that few people in C# refer to them as "methods" - they just use the term "functions" because there isn't any real distinction to be made.
Finally - just so any Pascal gurus don't jump on me here - Pascal also differentiates between "functions" (which return a value) and "procedures" which do not. C# does not make this distinction explicitly although you can, of course, choose to return a value or not.
Methods on a class act on the instance of the class, called the object.
class Example
{
public int data = 0; // Each instance of Example holds its internal data. This is a "field", or "member variable".
public void UpdateData() // .. and manipulates it (This is a method by the way)
{
data = data + 1;
}
public void PrintData() // This is also a method
{
Console.WriteLine(data);
}
}
class Program
{
public static void Main()
{
Example exampleObject1 = new Example();
Example exampleObject2 = new Example();
exampleObject1.UpdateData();
exampleObject1.UpdateData();
exampleObject2.UpdateData();
exampleObject1.PrintData(); // Prints "2"
exampleObject2.PrintData(); // Prints "1"
}
}
Since you mentioned Python, the following might be a useful illustration of the relationship between methods and objects in most modern object-oriented languages. In a nutshell what they call a "method" is just a function that gets passed an extra argument (as other answers have pointed out), but Python makes that more explicit than most languages.
# perfectly normal function
def hello(greetee):
print "Hello", greetee
# generalise a bit (still a function though)
def greet(greeting, greetee):
print greeting, greetee
# hide the greeting behind a layer of abstraction (still a function!)
def greet_with_greeter(greeter, greetee):
print greeter.greeting, greetee
# very simple class we can pass to greet_with_greeter
class Greeter(object):
def __init__(self, greeting):
self.greeting = greeting
# while we're at it, here's a method that uses self.greeting...
def greet(self, greetee):
print self.greeting, greetee
# save an object of class Greeter for later
hello_greeter = Greeter("Hello")
# now all of the following print the same message
hello("World")
greet("Hello", "World")
greet_with_greeter(hello_greeter, "World")
hello_greeter.greet("World")
Now compare the function greet_with_greeter and the method greet: the only difference is the name of the first parameter (in the function I called it "greeter", in the method I called it "self"). So I can use the greet method in exactly the same way as I use the greet_with_greeter function (using the "dot" syntax to get at it, since I defined it inside a class):
Greeter.greet(hello_greeter, "World")
So I've effectively turned a method into a function. Can I turn a function into a method? Well, as Python lets you mess with classes after they're defined, let's try:
Greeter.greet2 = greet_with_greeter
hello_greeter.greet2("World")
Yes, the function greet_with_greeter is now also known as the method greet2. This shows the only real difference between a method and a function: when you call a method "on" an object by calling object.method(args), the language magically turns it into method(object, args).
(OO purists might argue a method is something different from a function, and if you get into advanced Python or Ruby - or Smalltalk! - you will start to see their point. Also some languages give methods special access to bits of an object. But the main conceptual difference is still the hidden extra parameter.)
for me:
the function of a method and a function is the same if I agree that:
a function may return a value
may expect parameters
Just like any piece of code you may have objects you put in and you may have an object that comes as a result. During doing that they might change the state of an object but that would not change their basic functioning for me.
There might be a definition differencing in calling functions of objects or other codes. But isn't that something for a verbal differenciations and that's why people interchange them? The mentions example of computation I would be careful with. because I hire employes to do my calculations:
new Employer().calculateSum( 8, 8 );
By doing it that way I can rely on an employer being responsible for calculations. If he wants more money I free him and let the carbage collector's function of disposing unused employees do the rest and get a new employee.
Even arguing that a method is an objects function and a function is unconnected computation will not help me. The function descriptor itself and ideally the function's documentation will tell me what it needs and what it may return. The rest, like manipulating some object's state is not really transparent to me. I do expect both functions and methods to deliver and manipulate what they claim to without needing to know in detail how they do it.
Even a pure computational function might change the console's state or append to a logfile.
From my understanding a method is any operation which can be performed on a class. It is a general term used in programming.
In many languages methods are represented by functions and subroutines. The main distinction that most languages use for these is that functions may return a value back to the caller and a subroutine may not. However many modern languages only have functions, but these can optionally not return any value.
For example, lets say you want to describe a cat and you would like that to be able to yawn. You would create a Cat class, with a Yawn method, which would most likely be a function without any return value.
To a first order approximation, a method (in C++ style OO) is another word for a member function, that is a function that is part of a class.
In languages like C/C++ you can have functions which are not members of a class; you don't call a function not associated with a class a method.
IMHO people just wanted to invent new word for easier communication between programmers when they wanted to refer to functions inside objects.
If you are saying methods you mean functions inside the class.
If you are saying functions you mean simply functions outside the class.
The truth is that both words are used to describe functions. Even if you used it wrongly nothing wrong happens. Both words describe well what you want to achieve in your code.
Function is a code that has to play a role (a function) of doing something.
Method is a method to resolve the problem.
It does the same thing. It is the same thing. If you want to be super precise and go along with the convention you can call methods as the functions inside objects.
Let's not over complicate what should be a very simple answer. Methods and functions are the same thing. You call a function a function when it is outside of a class, and you call a function a method when it is written inside a class.
Function is the concept mainly belonging to Procedure oriented programming where a function is an an entity which can process data and returns you value
Method is the concept of Object Oriented programming where a method is a member of a class which mostly does processing on the class members.
I am not an expert, but this is what I know:
Function is C language term, it refers to a piece of code and the function name will be the identifier to use this function.
Method is the OO term, typically it has a this pointer in the function parameter. You can not invoke this piece of code like C, you need to use object to invoke it.
The invoke methods are also different. Here invoke meaning to find the address of this piece of code. C/C++, the linking time will use the function symbol to locate.
Objecive-C is different. Invoke meaning a C function to use data structure to find the address. It means everything is known at run time.
TL;DR
A Function is a piece of code to run.
A Method is a Function inside an Object.
Example of a function:
function sum(){
console.log("sum")l
}
Example of a Method:
const obj = {
a:1,
b:2,
sum(){
}
}
So thats why we say that a "this" keyword inside a Function is not very useful unless we use it with call, apply or bind .. because call, apply, bind will call that function as a method inside object ==> basically it converts function to method
I know many others have already answered, but I found following is a simple, yet effective single line answer. Though it doesn't look a lot better than others answers here, but if you read it carefully, it has everything you need to know about the method vs function.
A method is a function that has a defined receiver, in OOP terms, a method is a function on an instance of an object.
A class is the collection of some data and function optionally with a constructor.
While you creating an instance (copy,replication) of that particular class the constructor initialize the class and return an object.
Now the class become object (without constructor)
&
Functions are known as method in the object context.
So basically
Class <==new==>Object
Function <==new==>Method
In java the it is generally told as that the constructor name same as class name but in real that constructor is like instance block and static block but with having a user define return type(i.e. Class type)
While the class can have an static block,instance block,constructor, function
The object generally have only data & method.
Function - A function in an independent piece of code which includes some logic and must be called independently and are defined outside of class.
Method - A method is an independent piece of code which is called in reference to some object and are be defined inside the class.
General answer is:
method has object context (this, or class instance reference),
function has none context (null, or global, or static).
But answer to question is dependent on terminology of language you use.
In JavaScript (ES 6) you are free to customising function context (this) for any you desire, which is normally must be link to the (this) object instance context.
In Java world you always hear that "only OOP classes/objects, no functions", but if you watch in detailes to static methods in Java, they are really in global/null context (or context of classes, whithout instancing), so just functions whithout object. Java teachers could told you, that functions were rudiment of C in C++ and dropped in Java, but they told you it for simplification of history and avoiding unnecessary questions of newbies. If you see at Java after 7 version, you can find many elements of pure function programming (even not from C, but from older 1988 Lisp) for simplifying parallel computing, and it is not OOP classes style.
In C++ and D world things are stronger, and you have separated functions and objects with methods and fields. But in practice, you again see functions without this and methods whith this (with object context).
In FreePascal/Lazarus and Borland Pascal/Delphi things about separation terms of functions and objects (variables and fields) are usually similar to C++.
Objective-C comes from C world, so you must separate C functions and Objective-C objects with methods addon.
C# is very similar to Java, but has many C++ advantages.
In C++, sometimes, method is used to reflect the notion of member function of a class. However, recently I found a statement in the book «The C++ Programming Language 4th Edition», on page 586 "Derived Classes"
A virtual function is sometimes called a method.
This is a little bit confusing, but he said sometimes, so it roughly makes sense, C++ creator tends to see methods as functions can be invoked on objects and can behave polymorphic.
Can I map Scala functions to JSON; or perhaps via a different way than JSON?
I know I can map data types, which is fine. But I'd like to create a function, map it to JSON send it via a REST method to another server, then add that function to a list of functions in another application and apply it.
For instance:
def apply(f: Int => String, v: Int) = f(v)
I want to make a list of functions that can be applied within an application, over different physical locations. Now I want to add and remove functions to the list. By means of REST calls.
Let's assume I understand security problems...
ps.. If you downvote, you might as well have the decency to explain why
If I understand correctly, you want to be able to send Scala code to be executed on different physical machines. I can think of a few different ways of achieving that
Using tools for distributed computing e.g. Spark. You can set up Spark clusters on different machines and then select to which cluster you want to submit Spark jobs. There are a lot of other tools for distributed computing that might also be worth looking into.
Pass scala code as a string and compile it either within your server side code (here's an example) or by invoking scalac as an external process.
Send the functions as byte code and execute the byte code on the remote machine.
If it fits with what you want to do, I'd recommend option #1.
Make sure that you can really trust the code that you want to execute, to not expose yourself to malicious code.
The answer is you can't do this, and even if you could you shouldn't!
You should never, never, never write a REST API that allows the client to execute arbitrary code in your application.
What you can do is create a number of named operations that can be executed. The client can then pass the name of the operation which the server can look up in a Map[String, <function>] and execute the result.
As mentioned in my comment, here is an example of how to turn a case class into JSON. Things to note: don't question the implicit val format line (it's magic); each case class requires a companion object in order to work; if you have Optional fields in your case class and define them as None when turning it into JSON, those fields will be ignored (if you define them as Some(whatever), they will look like any other field). If you don't know much about Scala Play, ignore the extra stuff for now - this is just inside the default Controller you're given when you make a new Project in IntelliJ.
package controllers
import javax.inject._
import play.api.libs.json.{Json, OFormat}
import play.api.mvc._
import scala.concurrent.Future
#Singleton
class HomeController #Inject()(cc: ControllerComponents) extends AbstractController(cc) {
case class Attributes(heightInCM: Int, weightInKG: Int, eyeColour: String)
object Attributes {
implicit val format: OFormat[Attributes] = Json.format[Attributes]
}
case class Person(name: String, age: Int, attributes: Attributes)
object Person {
implicit val format: OFormat[Person] = Json.format[Person]
}
def index: Action[AnyContent] = Action.async {
val newPerson = Person("James", 24, Attributes(192, 83, "green"))
Future.successful(Ok(Json.toJson(newPerson)))
}
}
When you run this app with sbt run and hit localhost:9000 through a browser, the output you see on-screen is below (formatted for better reading). This is also an example of how you might send JSON as a response to a GET request. It isn't the cleanest example but it works.
{
"name":"James",
"age":24,
"attributes":
{
"heightInCM":187,
"weightInKG":83,
"eyeColour":"green"
}
}
Once more though, I would never recommend passing actual functions between services. If you insist though, maybe store them as a String in a Case Class and turn it into JSON like this. Even if you are okay with passing functions around, it might even be a good exercise to practice security by validating the functions you receive to make sure they're not malicious.
I also have no idea how you'll convert them back into a function either - maybe write the String you receive to a *.scala file and try to run them with a Bash script? Idk. I'll let you figure that one out.
I've been playing around with the reflect package, and I notice how limited the functionality of functions are.
package main
import (
"fmt"
"reflect"
"strings"
)
func main() {
v := reflect.ValueOf(strings.ToUpper)
fmt.Printf("Address: %v\n", v) // 0xd54a0
fmt.Printf("Can set? %d\n", v.CanSet()) // False
fmt.Printf("Can address? %d\n", v.CanAddr()) // False
fmt.Printf("Element? %d\n", v.Elem()) // Panics
}
Playground link here.
I've been taught that functions are addresses to memory with a set of instructions (hence v prints out 0xd54a0), but it looks like I can't get an address to this memory address, set it, or dereference it.
So, how are Go functions implemented under the hood? Eventually, I'd ideally want to manipulate the strings.ToUpper function by making the function point to my own code.
Disclaimers:
I've only recently started to delve deeper into the golang compiler, more specifically: the go assembler and mapping thereof. Because I'm by no means an expert, I'm not going to attempt explaining all the details here (as my knowledge is most likely still lacking). I will provide a couple of links at the bottom that might be worth checking out for more details.
What you're trying to do makes very, very little sense to me. If, at runtime, you're trying to modify a function, you're probably doing something wrong earlier on. And that's just in case you want to mess with any function. The fact that you're trying to do something with a function from the strings package makes this all the more worrying. The reflect package allows you to write very generic functions (eg a service with request handlers, but you want to pass arbitrary arguments to those handlers requires you to have a single handler, process the raw request, then call the corresponding handler. You cannot possibly know what that handler looks like, so you use reflection to work out the arguments required...).
Now, how are functions implemented?
The go compiler is a tricky codebase to wrap ones head around, but thankfully the language design, and the implementation thereof has been discussed openly. From what I gather, golang functions are essentially compiled in pretty much the same way as a function in, for example, C. However, calling a function is a bit different. Go functions are first-class objects, that's why you can pass them as arguments, declare a function type, and why the reflect package has to allow you to use reflection on a function argument.
Essentially, functions are not addressed directly. Functions are passed and invoked through a function "pointer". Functions are effectively a type like similar to a map or a slice. They hold a pointer to the actual code, and the call data. In simple terms, think of a function as a type (in pseudo-code):
type SomeFunc struct {
actualFunc *func(...) // pointer to actual function body
data struct {
args []interface{} // arguments
rVal []interface{} // returns
// any other info
}
}
This means that the reflect package can be used to, for example, count the number of arguments and return values the function expects. It also tells you what the return value(s) will be. The overall function "type" will be able to tell you where the function resides, and what arguments it expects and returns, but that's about it. IMO, that's all you really need though.
Because of this implementation, you can create fields or variables with a function type like this:
var callback func(string) string
This would create an underlying value that, based on the pseudo code above, looks something like this:
callback := struct{
actualFunc: nil, // no actual code to point to, function is nil
data: struct{
args: []interface{}{string}, // where string is a value representing the actual string type
rVal: []interface{}{string},
},
}
Then, by assigning any function that matches the args and rVal constraints, you can determine what executable code the callback variable points to:
callback = strings.ToUpper
callback = func(a string) string {
return fmt.Sprintf("a = %s", a)
}
callback = myOwnToUpper
I hope this cleared 1 or 2 things up a bit, but if not, here's a bunch of links that might shed some more light on the matter.
Go functions implementation and design
Introduction to go's ASM
Rob Pike on the go compiler written in go, and the plan 9 derived asm mapping
Writing a JIT in go asm
a "case study" attempting to use golang ASM for optimisation
Go and assembly introduction
Plan 9 assembly docs
Update
Seeing as you're attempting to swap out a function you're using for testing purposes, I would suggest you not use reflection, but instead inject mock functions, which is a more common practice WRT testing to begin with. Not to mention it being so much easier:
type someT struct {
toUpper func(string) string
}
func New(toUpper func(string) string) *someT {
if toUpper == nil {
toUpper = strings.ToUpper
}
return &someT{
toUpper: toUpper,
}
}
func (s *someT) FuncToTest(t string) string {
return s.toUpper(t)
}
This is a basic example of how you could inject a specific function. From within your foo_test.go file, you'd just call New, passing a different function.
In more complex scenario's, using interfaces is the easiest way to go. Simply implement the interface in the test file, and pass the alternative implementation to the New function:
type StringProcessor interface {
ToUpper(string) string
Join([]string, string) string
// all of the functions you need
}
func New(sp StringProcessor) return *someT {
return &someT{
processor: sp,
}
}
From that point on, simply create a mock implementation of that interface, and you can test everything without having to muck about with reflection. This makes your tests easier to maintain and, because reflection is complex, it makes it far less likely for your tests to be faulty.
If your test is faulty, it could cause your actual tests to pass, even though the code you're trying to test isn't working. I'm always suspicious if the test code is more complex than the code you're covering to begin with...
Underneath the covers, a Go function is probably just as you describe it- an address to a set of instructions in memory, and parameters / return values are filled in according to your system's linkage conventions as the function executes.
However, Go's function abstraction is much more limited, on purpose (it's a language design decision). You can't just replace functions, or even override methods from other imported packages, like you might do in a normal object-oriented language. You certainly can't do dynamic replacement of functions under normal circumstances (I suppose you could write into arbitrary memory locations using the unsafe package, but that's willful circumvention of the language rules, and all bets are off at that point).
Are you trying to do some sort of dependency injection for unit testing? If so, the idiomatic way to do this in Go is to define interface that you pass around to your functions/methods, and replace with a test version in your tests. In your case, an interface may wrap the call to strings.ToUpper in the normal implementation, but a test implementation might call something else.
For example:
type Upper interface {
ToUpper(string) string
}
type defaultUpper struct {}
func (d *defaultUpper) ToUpper(s string) string {
return strings.ToUpper(s)
}
...
// normal implementation: pass in &defaultUpper{}
// test implementation: pass in a test version that
// does something else
func SomethingUseful(s string, d Upper) string {
return d.ToUpper(s)
}
Finally, you can also pass function values around. For example:
var fn func(string) string
fn = strings.ToUpper
...
fn("hello")
... but this won't let you replace the system's strings.ToUpper implementation, of course.
Either way, you can only sort of approximate what you want to do in Go via interfaces or function values. It's not like Python, where everything is dynamic and replaceable.
We have a package that we are looking to convert to kotlin from python in order to then be able to migrate systems using that package.
Within the package there are a set of classes that are all variants, or 'flavours' of a common base class.
Most of the code is in the base class which has a significant number of optional parameters. So consider:
open class BaseTree(val height:Int=10,val roots:Boolean=true, //...... lots more!!
class FruitTree(val fruitSize, height:Int=10, roots:Boolean=true,
// now need all possible parameters for any possible instance
):BaseTree(height=height, roots=roots //... yet another variation of same list
The code is not actually trees, I just thought this was a simple way to convey the idea. There are about 20 parameters to the base class, and around 10 subclasses, and each subclass effectively needs to repeat the same two variations of the parameter list from the base class. A real nightmare if the parameter list ever changes!
Those from a Java background may comment "20 parameters is too many", may miss that this is optional parameters, the language features which impacts this aspect of design. 20 required parameters would be crazy, but 10 or even 20 optional parameters is not so uncommon, check sqlalchemy Table for example.
In python, you to call a base class constructor you can have:
def __init__(self, special, *args, **kwargs):
super().__init(*args, **kwargs) # pass all parameters except special to base constructor
Does anyone know a technique, using a different method (perhaps using interfaces or something?) to avoid repeating this parameter list over and over for each subclass?
There is no design pattern to simplify this use case.
Best solution: Refactor the code to use a more Java like approach: using properties in place of optional parameters.
Use case explained: A widely used class or method having numerous optional parameters is simply not practical in Java, and kotlin is most evolved as way of making java code better. A python class with 5 optional parameters, translated to Java with no optional parameters, could have 5! ( and 5 factorial is 60) different Java signatures...in other words a mess.
Obviously no object should routinely be instanced with a huge parameter list, so normall python classes only evolve for classes when the majority of calls do not need to specify these optional parameters, and the optional parameters are for the exception cases. The actual use case here is the implementation of a large number of optional parameters, where it should be very rare for any individual object to be instanced using more than 3 of the optional parameter. So a class with 10 optional parameters that is used 500 times in an application, would still expect 3 of the optional parameters to be the maximum ever used in one instance. But this is simply a design approach not workable in Java, no matter how often the class is reused.
In Java, functions do hot have optional parameters, which means this case where an object is instanced in this way in a Java library simply could never happen.
Consider an object with one mandatory instance parameter, and five possible options. In Java these options would each be properties able to be set by setters, and objects would then be instanced, and the setter(s) called for setting any relevant option, but infrequently required change to the default value for that option.
The downside is that these options cannot be set from the constructor and remain immutable, but the resultant code reduces the optional parameters.
Another approach is to have a group of less 'swiss army knife' objects, with a set of specialised tools replacing the one do-it-all tool, even when the code could be seen as just slightly different nuances of the same theme.
Despite the support for Optional parameters in kotlin, The inheritance structure in kotlin is not yet optimised for heavier use of this feature.
You can skip the name like BaseTree(height, roots) by put the variable in order but you cannot do things like Python because Python is dynamic language.
It is normal that Java have to pass the variables to super class too.
FruitTree(int fruitSize, int height, boolean root) {
super(height, root);
}
There are about 20 parameters to the base class, and around 10 subclasses
This is most likely a problem of your classes design.
Reading your question I started to experiment myself and this is what I came up with:
interface TreeProperties {
val height: Int
val roots: Boolean
}
interface FruitTreeProperties: TreeProperties {
val fruitSize: Int
}
fun treeProps(height: Int = 10, roots: Boolean = true) = object : TreeProperties {
override val height = height
override val roots = roots
}
fun TreeProperties.toFruitProperty(fruitSize: Int): FruitTreeProperties = object: FruitTreeProperties, TreeProperties by this {
override val fruitSize = fruitSize
}
open class BaseTree(val props: TreeProperties)
open class FruitTree(props: FruitTreeProperties): BaseTree(props)
fun main(args: Array<String>){
val largTree = FruitTree(treeProps(height = 15).toFruitProperty(fruitSize = 5))
val rootlessTree = BaseTree(treeProps(roots = false))
}
Basically I define the parameters in an interface and extend the interface for sub-classes using the delegate pattern. For convenience I added functions to generate instances of those interface which also use default parameters.
I think this achieves the goal of repeating parameter lists quite nicely but also has its own overhead. Not sure if it is worth it.
If your subclass really has that many parameters in the constructur -> No way around that. You need to pass them all.
But (mostly) it's no good sign, that a constructor/function has that many parameters...
You are not alone on this. That is already discussed on the gradle-slack channel. Maybe in the future, we will get compiler-help on this, but for now, you need to pass the arguments yourself.