ZMQ python programming - deep-learning

I have got an object detection which reads an input image and runs the inference then it outputs the classIDs[] (class name) and confidence levels of the detected object confidences[].

In case one has never worked with ZeroMQ,one may here enjoy to first look at "ZeroMQ Principles in less than Five Seconds"before diving into further details
Q : Could you please tell me how can I communicate the output ... from deep learning system?
May use a socket.send( pickle.dumps( [ classIDs[i], confidences[i], ] ) )
Both the first O/P-topic creeping comment, posted 15 minutes after this answer did answer the O/P-problem definition (and was deleted later) and also the second O/P-topic creeping comment, posted about an hour after a due answer was in-place, did not change the game :
whatever you try to pass over the ZeroMQ channel has to be SER/DES-handled. If willing to make things complex, ok, it still goes the same way :
socket.send( pickle.dumps( <whateverBLOBneeded> ) )
If starting to have new problems, due to SER/DES-collisions ( as object-instances and Class()-es have so often in attempts to have 'em pickle'd ), feel free to try to salvage the so often Exceptions "vomiting" pickle module with :import dill as pickle a smarter SER/DES dill module from Mike McKerns
and,
again the rest goes the same way :
socket.send( pickle.dumps( <whateverBLOBneeded> ) )
A bonus part
May rather want to prototype with PUSH/PULL it does not block in a mutual deadlock as all REQ/REP do.

Related

How to convert string to json and vice versa in flutter?

Hi below is the json that I need to extract data from.
[
"http://activepeersai.computing.dcu.ie/feedback_participant/114",
"http://activepeersai.computing.dcu.ie/peerLearningPrompter/4",
"{\"0\":{\"feedback_id\":114,\"timer_used\":1,\"timer\":10.0,\"question1\":\"How to break ice with strangers (in social gathering & in formal events) ?\",\"question2\":\"How to build networking (with those from different age, culture, education background, language\\u2026)(in university & in workplace) ?\",\"question3\":\"\",\"question4\":\"\",\"question5\":\"\",\"question6\":\"\",\"question7\":\"\",\"question8\":\"\",\"question9\":\"\",\"question10\":\"\"},\"1\":{\"feedback_id\":115,\"timer_used\":1,\"timer\":10.0,\"question1\":\"How to deal with difficult teammates (dominating \\/ debater character \\/ negative \\/ lack of confidence \\/ free rider\\u2026) ?\",\"question2\":\"How to build mutual trust with teammates ?\",\"question3\":\"\",\"question4\":\"\",\"question5\":\"\",\"question6\":\"\",\"question7\":\"\",\"question8\":\"\",\"question9\":\"\",\"question10\":\"\"}}",
"{\"id\":{\"0\":1,\"1\":2,\"2\":3,\"3\":4,\"4\":5,\"5\":6,\"6\":7,\"7\":8,\"8\":9,\"9\":10,\"10\":11,\"11\":12,\"12\":13,\"13\":14,\"14\":15,\"15\":16,\"16\":17,\"17\":18,\"18\":19,\"19\":20,\"20\":21,\"21\":22,\"22\":23,\"23\":24,\"24\":25,\"25\":26,\"26\":27,\"27\":28,\"28\":29,\"29\":30,\"30\":31,\"31\":32,\"32\":33,\"33\":34,\"34\":35,\"35\":40,\"36\":41,\"37\":42,\"38\":43,\"39\":44,\"40\":45,\"41\":46,\"42\":47,\"43\":48,\"44\":49,\"45\":50,\"46\":51,\"47\":52,\"48\":53,\"49\":54,\"50\":55,\"51\":56,\"52\":57,\"53\":58,\"54\":59,\"55\":60,\"56\":61,\"57\":62,\"58\":63,\"59\":36,\"60\":37,\"61\":38,\"62\":39},\"name\":{\"0\":\"Why did the sharer choose this skill as the most confident skill and why did the learner wants to improve this skill?\",\"1\":\"Can you share another situation whereby using this skill made a very big difference to the outcome? What did you do? What was the result? What might have happened if this skill wasn't brought to the situation?\",\"2\":\"What advice do you have to give to others about becoming better at using this skill in their work and in their lives generally?\",\"3\":\"What can hold people back from being better at this skill? How can you encourage them to overcome these challenges?\",\"4\":\"What would be the outcome if this session\\/series was a huge success? How would we know if this happened?\",\"5\":\"What is working for you now? What is stopping you from moving forward?\",\"6\":\"What do your role models do that you would love to learn\\/incorporate as habits\\/adopt as a mindset?\",\"7\":\"\\\"What got you here won't get you there.\\\" What is your reaction to that statement? What actions have got you to this point that may not serve you if you move forward? What new behaviours do you need to adopt?\",\"8\":\"What would be the outcome if this session\\/series was a huge success? How would we know if this happened?\",\"9\":\"If you had all the time, people, money, resources that you might possibly need, what would you do differently? Does that energise you, frighten you or a bit of both?\",\"10\":\"What do you think you need to do get a better result (or closer to your goal)?\",\"11\":\"What is the worst that could happen and how could you handle it? What is the best that could happen and how could you handle it?\",\"12\":\"What have been the most impactful decisions that you've made in your career? What led to making those decisions? Were you aware of the impact those decisions would have? What would you say to others in a similar position?\",\"13\":\"What habits have stood you in good stead? Are there things that you do regularly, daily or very often that have made a big different over time?\",\"14\":\"Can you share an experience that was transformational for you, taught you a long-lasting lesson or was particularly memorable for its benefits and challenges?\",\"15\":\"How do you think experience affects somebody's perspective, way of making decisions or their feelings about taking risks?\",\"16\":\"Can you describe your target market for your business or who might your ideal employer be?\",\"17\":\"Can you share details of a project that you worked on recently? Why were you or your business chosen to work with this client\\/employer? What were the success and challenges along the way?\",\"18\":\"What makes you\\/your product\\/your service different to others? Why would a client or employer choose what you do over others?\",\"19\":\"What might I be able to do to help you?\",\"20\":\"What is the key problem that you are facing currently?\",\"21\":\"If that problem was solved, what impact would it have?\",\"22\":\"In order to get to that outcome, what do we need to do, what resources need to be invested and any other changes to be made?\",\"23\":\"Is there a willingness to take that action to get towards that outcome?\",\"24\":\"What assumptions are being made? How likely are each one to happen? What would be the impact of those assumptions happening or not happening?\",\"25\":\"What application of the SCAMPER technique could be useful to help get another perspective?\\n*SCAMPER is an acronym formed from the abbreviation of: Substitute, Combine, Adapt, Modify (Also magnify and minify), Put to another use, Eliminate and Reverse.\",\"26\":\"If we had double the budget, half the time and were living in another country trying to make this happen, how might we think differently?\",\"27\":\"If any idea generated already in this discussion was to become a reality, what would be the impact on e year later in terms of benefits, ongoing coasts, sustained behavioural change and a launchpad for further growth?\",\"28\":\"Leadership\",\"29\":\"Communication\",\"30\":\"Adaptability\",\"31\":\"Team Work\",\"32\":\"Problem Solving\",\"33\":\"Conflict Management\",\"34\":\"Productivity\",\"35\":\"How would you describe good\\/effective communication? Please share an example of a time that you've seen it in action and an example of when you saw that good communication skills were clearly lacking.\",\"36\":\"How have you handled working under someone you felt was not good at communicating?\",\"37\":\"If you're trying to get your point across or convince somebody that your idea is the right one, what do you do?\",\"38\":\"Who do you think is a good communicator and why? What can we learn from them?\",\"39\":\"Talk about a time that you needed to adapt to a new situation. What did you find difficult and how did you work through that? How can somebody prepare to be more adaptable in future situations?\",\"40\":\"When you're in a situation where it feels like you have no control over it (i.e. a new manager, starting in a new job, government-led changes etc), what do you do to focus on what you can do?\",\"41\":\"How do you handle having multiple priorities at the same time?\",\"42\":\"How do you adjust to different work settings? For example, working with different teams, switching between logic and creativity, learning new processes, tools or technologies?\",\"43\":\"How do you feel about working in a team? What do you think are the key things that need to happen to make good teamwork?\",\"44\":\"What has been your experience of working in teams where there were problems? Did these arise due to strong personalities, somebody not sharing the workload, miscommunication, the wrong support or technology systems etc?\",\"45\":\"How do you keep a team motivated? Share your story about a rewarding team experience.\",\"46\":\"When you're in a team situation, what role do you usually play?\",\"47\":\"Describe a situation where you had to solve a problem. What did you do? what was the result? What might you have done differently?\",\"48\":\"What steps do you take before making a decision on how to solve a problem, and why?\",\"49\":\"Give an example of a situation in which you saw an opportunity in a potential problem. What did you do? What was the outcome?\",\"50\":\"Can you tell me about a situation where you overcame a problem using a creative solution?\",\"51\":\"Have you ever had a team member who kept raising objections on projects? How did you (or would you) manage them?\",\"52\":\"You have noticed that a team member is aggressive or arrogant toward the rest of the team. How would you approach this person?\",\"53\":\"What would you do if your manager gave you negative feedback on the way you approached a problem? How do give negative\\/constructive feedback to others?\",\"54\":\"How could you use a situation with conflict to have a better relationship with all involved?\",\"55\":\"How would you describe a typical working day in your current role? How you manage importance versus urgency? How do you maintain a work-life balance also?\",\"56\":\"If you have your day planned out to achieve a goal, how do you manage distractions or other things that can happen along the way?\",\"57\":\"What do you think very productive people do differently than others?\",\"58\":\"What holds people back from being more productive? How can people turn those around so that by doing the opposite, they can become more productive?\",\"59\":\"Have you ever had a team member who kept raising objections on projects? How did you (or would you) manage them?\",\"60\":\"How do you describe your leadership style?\",\"61\":\"What was a difficult decision you had to make as a leader, and how did you come to that decision?\",\"62\":\"What are the most important attributes of successful leaders today? Who do you think is a good leader and why?\"},\"session_type_id\":{\"0\":1.0,\"1\":1.0,\"2\":1.0,\"3\":1.0,\"4\":2.0,\"5\":2.0,\"6\":2.0,\"7\":2.0,\"8\":3.0,\"9\":3.0,\"10\":3.0,\"11\":3.0,\"12\":4.0,\"13\":4.0,\"14\":4.0,\"15\":4.0,\"16\":5.0,\"17\":5.0,\"18\":5.0,\"19\":5.0,\"20\":6.0,\"21\":6.0,\"22\":6.0,\"23\":6.0,\"24\":7.0,\"25\":7.0,\"26\":7.0,\"27\":7.0,\"28\":8.0,\"29\":8.0,\"30\":8.0,\"31\":8.0,\"32\":8.0,\"33\":8.0,\"34\":8.0,\"35\":null,\"36\":null,\"37\":null,\"38\":null,\"39\":null,\"40\":null,\"41\":null,\"42\":null,\"43\":null,\"44\":null,\"45\":null,\"46\":null,\"47\":null,\"48\":null,\"49\":null,\"50\":null,\"51\":null,\"52\":null,\"53\":null,\"54\":null,\"55\":null,\"56\":null,\"57\":null,\"58\":null,\"59\":null,\"60\":null,\"61\":null,\"62\":null},\"session_type_name\":{\"0\":\"Learning and Development\",\"1\":\"Learning and Development\",\"2\":\"Learning and Development\",\"3\":\"Learning and Development\",\"4\":\"Mentoring\",\"5\":\"Mentoring\",\"6\":\"Mentoring\",\"7\":\"Mentoring\",\"8\":\"Coaching\",\"9\":\"Coaching\",\"10\":\"Coaching\",\"11\":\"Coaching\",\"12\":\"Experience Sharing\",\"13\":\"Experience Sharing\",\"14\":\"Experience Sharing\",\"15\":\"Experience Sharing\",\"16\":\"Networking\",\"17\":\"Networking\",\"18\":\"Networking\",\"19\":\"Networking\",\"20\":\"Establishing Buy-In\",\"21\":\"Establishing Buy-In\",\"22\":\"Establishing Buy-In\",\"23\":\"Establishing Buy-In\",\"24\":\"Idea Exploration\",\"25\":\"Idea Exploration\",\"26\":\"Idea Exploration\",\"27\":\"Idea Exploration\",\"28\":\"Seven Soft Skills\",\"29\":\"Seven Soft Skills\",\"30\":\"Seven Soft Skills\",\"31\":\"Seven Soft Skills\",\"32\":\"Seven Soft Skills\",\"33\":\"Seven Soft Skills\",\"34\":\"Seven Soft Skills\",\"35\":null,\"36\":null,\"37\":null,\"38\":null,\"39\":null,\"40\":null,\"41\":null,\"42\":null,\"43\":null,\"44\":null,\"45\":null,\"46\":null,\"47\":null,\"48\":null,\"49\":null,\"50\":null,\"51\":null,\"52\":null,\"53\":null,\"54\":null,\"55\":null,\"56\":null,\"57\":null,\"58\":null,\"59\":null,\"60\":null,\"61\":null,\"62\":null},\"parent_id\":{\"0\":null,\"1\":null,\"2\":null,\"3\":null,\"4\":null,\"5\":null,\"6\":null,\"7\":null,\"8\":null,\"9\":null,\"10\":null,\"11\":null,\"12\":null,\"13\":null,\"14\":null,\"15\":null,\"16\":null,\"17\":null,\"18\":null,\"19\":null,\"20\":null,\"21\":null,\"22\":null,\"23\":null,\"24\":null,\"25\":null,\"26\":null,\"27\":null,\"28\":null,\"29\":null,\"30\":null,\"31\":null,\"32\":null,\"33\":null,\"34\":null,\"35\":30.0,\"36\":30.0,\"37\":30.0,\"38\":30.0,\"39\":31.0,\"40\":31.0,\"41\":31.0,\"42\":31.0,\"43\":32.0,\"44\":32.0,\"45\":32.0,\"46\":32.0,\"47\":33.0,\"48\":33.0,\"49\":33.0,\"50\":33.0,\"51\":34.0,\"52\":34.0,\"53\":34.0,\"54\":34.0,\"55\":35.0,\"56\":35.0,\"57\":35.0,\"58\":35.0,\"59\":29.0,\"60\":29.0,\"61\":29.0,\"62\":29.0}}"
]
I need to extract data from the 3rd line.
The below code tries to extract the first set of questions from the 3rd line.
import 'dart:core';
import 'dart:convert';
import 'package:http/http.dart' as http;
class Question {
late final String question;
Question({ required this.question});
}
void getData() async{
List<Question> sample_data = [];
String link = 'https://activepeersai.computing.dcu.ie/API/631';
var url = Uri.parse(link);
var result = await http
.get(url,headers: {"Accept": "application/json","Access-Control_Allow_Origin": "*"});
if(result.statusCode==200){
var data = jsonDecode(result.body);
var res = jsonDecode(data[2]);
String round1 = json.encode(res);
var data1 = jsonDecode(round1);
var questions = jsonDecode(data1[0]);
for(var k in questions.keys){
if(((k=='question1')|| (k=='question2') || (k=='question3') || (k=='question4') || (k=='question5') || (k=='question6') || (k=='question7') || (k=='question8') || (k=='question9') || (k=='question10'))){
Question question = Question(question: res[k]);
sample_data.add(question);
}
}
print('$sample_data');
}
}
void main() {
getData();
}
Now when I print round1 I get the below output.
{"0":{"feedback_id":114,"timer_used":1,"timer":10,"question1":"How to break ice with strangers (in social gathering & in formal events) ?","question2":"How to build networking (with those from different age, culture, education background, language…)(in university & in workplace) ?","question3":"","question4":"","question5":"","question6":"","question7":"","question8":"","question9":"","question10":""},"1":{"feedback_id":115,"timer_used":1,"timer":10,"question1":"How to deal with difficult teammates (dominating / debater character / negative / lack of confidence / free rider…) ?","question2":"How to build mutual trust with teammates ?","question3":"","question4":"","question5":"","question6":"","question7":"","question8":"","question9":"","question10":""}}
Which when I checked on Json Checker came to be valid.
But after this I get an error.
Error: Expected a value of type 'String', but got one of type 'Null'
at Object.throw_ [as throw] (http://localhost:56003/dart_sdk.js:5080:11)
at Object.castError (http://localhost:56003/dart_sdk.js:5039:15)
at Object.cast [as as] (http://localhost:56003/dart_sdk.js:5356:17)
at String.as (http://localhost:56003/dart_sdk.js:46240:19)
at getData (http://localhost:56003/packages/testing_for_data/main.dart.lib.js:74:56)
at getData.next (<anonymous>)
at http://localhost:56003/dart_sdk.js:40641:33
at _RootZone.runUnary (http://localhost:56003/dart_sdk.js:40511:59)
at _FutureListener.thenAwait.handleValue (http://localhost:56003/dart_sdk.js:35438:29)
at handleValueCallback (http://localhost:56003/dart_sdk.js:35999:49)
at _Future._propagateToListeners (http://localhost:56003/dart_sdk.js:36037:17)
at [_completeWithValue] (http://localhost:56003/dart_sdk.js:35872:23)
at async._AsyncCallbackEntry.new.callback (http://localhost:56003/dart_sdk.js:35906:35)
at Object._microtaskLoop (http://localhost:56003/dart_sdk.js:40778:13)
at _startMicrotaskLoop (http://localhost:56003/dart_sdk.js:40784:13)
at http://localhost:56003/dart_sdk.js:36261:9
Can someone tell me what I am doing wrong?
Its coming due to the line
var data1 = jsonDecode(round1);
I believe this line is wrong
var questions = jsonDecode(data1[0]);
Firstly, because you probably need data1['0'] instead of data1[0]. They key is a string, not a number.
Secondly. You don't need to decode it again. It's already decoded. So try
var questions = data1['0'];
The next thing is that
Question question = Question(question: res[k]);
needs to be changed to
Question question = Question(question: questions[k]);
Because you actually want it from the questions.
If you want it to print the question instead of Instance of 'Question' you can add a toString method to it, for example:
class Question {
late final String question;
Question({ required this.question});
#override
String toString() {
return question;
}
}

Status of in-place `rfft` and `irfft` in Julia

So I'm doing some hobby-related stuff which involves taking Fourier transforms of large real arrays which barely fit in memory, and was curious to see if there was an in-place version of rfft and irfft that saved RAM, since RAM consumption is important to me. These transforms are possible despite the input-vs-output-type mismatch, and require an extra row of padding.
In Implement in-place rfft! and irfft!, Tim Holy said he was working on an in-place rfft! and irfft! that made use of a buffer-containing RCpair object, but then Steven Johnson said that he was implementing something equivalent using A_mul_B!(y, plan, x), which he elaborated on here.
Things get a little weird from then on. In the documentation for both 0.3.0 and 0.4.0 there is no mention of A_mul_B!, although A_mul_B is listed. But when I try entering them into Julia, I get
A_mul_B!
A_mul_B! (generic function with 28 methods)
A_mul_B
ERROR: A_mul_B not defined
which suggests that the situation is actually the opposite of what the documentation currently describes.
So since A_mul_B! seems to exist, but isn't documented anywhere, I tried to guess how to test it in-place as follows:
A = rand(Float32, 10, 10);
p = plan_rfft(A);
A_mul_B!(A,p,A)
which resulted in
ERROR: `A_mul_B!` has no method matching A_mul_B!(::Array{Float32,2}, ::Function, ::Array{Float32,2})
So...
Are in-place real FFTs still a work in progress? Or am I using A_mul_B! wrong?
Is there a mismatch between the 0.3.0 documentation and 0.3.0's function library?
That pull request from Steven Johnson is listed as open, not merged; that means the work hasn't been finished yet. The one from me is closed, but if you want the code you can grab it by clicking on the commits.
The docs indeed omit mention of A_mul_B!. A_mul_B is equivalent to A*B, and so isn't exported independently now. A_mul_B! would be used like this: instead of C = A*B, you could say A_mul_B!(C, A, B).
Can you please edit the docs to fix these issues? (You can edit files here in your webbrowser.)

What is pass-reference-by-value?

I came across this yesterday and was wondering what the relationship of pass-by-value, pass-by-reference and pass-reference-by-value is. How do they all behave? Which programming language uses which?
Additionally:
I have also occasionally heard other terms like pass-by-pointer, pass-by-copy, pass-by-object...how do they all fit in?
It's a made-up term that refers to something that is actually pass-by-value, but where people always get confused because the only values in the language are references (pointers to things). Many modern languages work this way, including Java (for non-primitives), JavaScript, Python, Ruby, Scheme, Smalltalk, etc.
Beginners always ask questions on StackOverflow about why it's not pass-by-reference, because they don't understand what pass-by-reference really is. So to satisfy them some people make up a new term that they also don't understand, but makes it seem like they learned something, and which sounds vaguely like both pass-by-value and pass-by-reference.
Use of terminology varies widely between different language communities. For example, the Java community almost always describes this semantics as pass-by-value, while the Python and Ruby communities rarely describe it as pass-by-value, even though they are semantically identical.
Let's start with the basics. You probably already know this or at least think you do. Surprisingly many people are actually misusing these terms quite a bit (which certainly doesn't help with the confusion).
What is pass-by-value and pass-by-reference?
I am not going to tell you the answer just yet and explain it using examples instead. You will see why later.
Imagine you are trying to learn a programming language and a friend tells you that he knows a great book on the topic. You have two options:
borrow the book from him or
buy your own copy.
Case 1: If you borrow his book and you make notes in the margins (and add some bookmarks, references, drawings, ...) your friend can later benefit from those notes and bookmarks as well. Sharing is caring. The world is great!
Case 2: But if, on the other hand, your friend is a bit of a neat freak or holds his books very dear he might not appreciate your notes and even less the occasional coffee stain or how you fold the corners for bookmarking. You should really get your own book and everybody can happily treat his book the way he likes. The world is great!
These two cases are - you guessed it - just like pass-by-reference and pass-by-value.
The difference is whether the caller of the function can see changes that the callee makes.
Let's look at a code example.
Say you have the following code snippet:
function largePrint(text):
text = makeUpperCase(text)
print(text)
myText = "Hello"
largePrint(myText)
print(myText)
and you run it in a pass-by-value language the output will be
HELLO
Hello
while in a pass-by-reference language the output will be
HELLO
HELLO
If you pass-by-reference the variable "myText" gets changed by the function. If you just pass the value of the variable the function cannot change it.
If you need another example, this one using websites got quite a few upvotes.
Okay, now that we have the basics down let's move on to...
What is pass-reference-by-value?
Let's use our book example again. Assume that you have decided to borrow the book from your friend although you know how much he loves his books. Let us also assume that you then - being the total klutz that you are - spilled a gallon of coffee over it. Oh, my...
Not to worry, your brain starts up and realizes that you can just buy your friend a new book. It was in such good condition he won't even notice!
What does that story have to do with pass-reference-by-value?
Well, imagine on the other hand that your friend loved his books so muchs he doesn't want to lend it to you. He does however agree to bring the book and show it to you. The problem is, you can still spill a gallon of coffee over it but now you can't buy him a new book anymore. He would notice!
While this may sound dreadful (and probably has you sweating as if you had drunk that gallon of coffee) but it is actually quite common and a perfectly good way to share books call functions. It is sometimes called pass-reference-by-value.
How does the code example fare?
In a pass-reference-by-value function the output of our snippet is
HELLO
Hello
just like in the pass-by-value case. But if we change the code a little bit:
function largePrint(text):
text.toUpperCase()
print(text)
myText = "Hello"
largePrint(myText)
print(myText)
The output becomes:
HELLO
HELLO
just like in the pass-by-reference case. The reason is the same as in the book example: We can change the variable (or book) by calling text.toUpperCase() (or by spilling coffee on it). BUT we cannot change the object that the variable references anymore (text = makeUppercase(text) has no effect); in other words we cannot use a different book.
So summarizing we get:
pass-by-value:
You get your own book, your own copy of the variable and whatever you do with it the original owner will never know.
pass-by-reference:
You use the same book as your friend or the same variable as the caller. If you change the variable it changes outside your function.
pass-reference-by-value
You use the same book as your friend but he doesn't let it out of his sight. You can change the book but not EXchange it. In variable terms that means that you cannot change what the variable references but you can change that which the variable references.
So now when I show you the following Python code:
def appendSeven(list):
list.append(7)
def replaceWithNewList(list):
list = ["a", "b", "c"]
firstList = [1, 2, 3, 4, 5, 6]
print(firstList)
appendSeven(firstList)
print(firstList)
secondList = ["x", "y", "z"]
print(secondList)
replaceWithNewList(secondList)
print(secondList)
and tell you that the output is this
[1, 2, 3, 4, 5, 6]
[1, 2, 3, 4, 5, 6, 7]
['x', 'y', 'z']
['x', 'y', 'z']
what do you say?
Indeed, you are correct! Python uses pass-reference-by-value! (And so is Java and Ruby and Scheme and...)
Conclusion:
Was this extremely difficult? No.
Then why are so many people confused by it? (I'm not going to link to all the other questions about "How do I pass a variable by reference in Python" or "Is Java pass-by-reference?" and so on...)
Well, I see several reasons:
Saying pass-value-by-reference is really long (and it behaves almost the same as pass-by-reference anyway so get of my back and stop splitting hairs!)
There are nasty special cases that complicate the issue
like primitive types (which are often pass-by-value in pass-value-by-reference languages)
immutable types (which cannot be changed so passing them by-reference doesn't get you anywhere)
Some people just like to use different names:
pass-by-copy for pass-by-value because if you paid attention, you need your own book, so a copy has to be made. (But that is an implementation detail, other tricks like copy-on-write might be used to only sometimes make a copy.)
pass-by-reference for pass-reference-by-value because, after all, you did get a reference and that means you can change the object that the caller had (even if you cannot EXchange it)
pass-by-reference for pass-reference-by-value because almost no language actually has pass-by-reference (notable Exceptions (that I know): C++, C# but you have to use special syntax)
pass-by-sharing for pass-reference-by-value (remember our book example? that name is actually way better and everybody should use it!)
pass-by-object for pass-reference-by-value because allegedly Barbara Liskov first named it and used that, so that's its birthname)
pass-by-pointer for pass-by-reference because C doesn't have pass-by-reference and that is how it allows you to accomplish the same thing.
If there is anything you think I should add, please let me know. If I made a mistake, PLEASE LET ME KNOW!
For those of you who still haven't got enough: http://en.wikipedia.org/wiki/Evaluation_strategy

Spartan Programming

I really enjoyed Jeff's post on Spartan Programming. I agree that code like that is a joy to read. Unfortunately, I'm not so sure it would necessarily be a joy to work with.
For years I have read about and adhered to the "one-expression-per-line" practice. I have fought the good fight and held my ground when many programming books countered this advice with example code like:
while (bytes = read(...))
{
...
}
while (GetMessage(...))
{
...
}
Recently, I've advocated one expression per line for more practical reasons - debugging and production support. Getting a log file from production that claims a NullPointer exception at "line 65" which reads:
ObjectA a = getTheUser(session.getState().getAccount().getAccountNumber());
is frustrating and entirely avoidable. Short of grabbing an expert with the code that can choose the "most likely" object that was null ... this is a real practical pain.
One expression per line also helps out quite a bit while stepping through code. I practice this with the assumption that most modern compilers can optimize away all the superfluous temp objects I've just created ...
I try to be neat - but cluttering my code with explicit objects sure feels laborious at times. It does not generally make the code easier to browse - but it really has come in handy when tracing things down in production or stepping through my or someone else's code.
What style do you advocate and can you rationalize it in a practical sense?
In The Pragmatic Programmer Hunt and Thomas talk about a study they term the Law of Demeter and it focuses on the coupling of functions to modules other than there own. By allowing a function to never reach a 3rd level in it's coupling you significantly reduce the number of errors and increase the maintainability of the code.
So:
ObjectA a = getTheUser(session.getState().getAccount().getAccountNumber());
Is close to a felony because we are 4 objects down the rat hole. That means to change something in one of those objects I have to know that you called this whole stack right here in this very method. What a pain.
Better:
Account.getUser();
Note this runs counter to the expressive forms of programming that are now really popular with mocking software. The trade off there is that you have a tightly coupled interface anyway, and the expressive syntax just makes it easier to use.
I think the ideal solution is to find a balance between the extremes. There is no way to write a rule that will fit in all situations; it comes with experience. Declaring each intermediate variable on its own line will make reading the code more difficult, which will also contribute to the difficulty in maintenance. By the same token, debugging is much more difficult if you inline the intermediate values.
The 'sweet spot' is somewhere in the middle.
One expression per line.
There is no reason to obfuscate your code. The extra time you take typing the few extra terms, you save in debug time.
I tend to err on the side of readability, not necessarily debuggability. The examples you gave should definitely be avoided, but I feel that judicious use of multiple expressions can make the code more concise and comprehensible.
I'm usually in the "shorter is better" camp. Your example is good:
ObjectA a = getTheUser(session.getState().getAccount().getAccountNumber());
I would cringe if I saw that over four lines instead of one--I don't think it'd make it easier to read or understand. The way you presented it here, it's clear that you're digging for a single object. This isn't better:
obja State = session.getState();
objb Account = State.getAccount();
objc AccountNumber = Account.getAccountNumber();
ObjectA a = getTheUser(AccountNumber);
This is a compromise:
objb Account = session.getState().getAccount();
ObjectA a = getTheUser(Account.getAccountNumber());
but I still prefer the single line expression. Here's an anecdotal reason: it's difficult for me to reread and error-check the 4-liner right now for dumb typos; the single line doesn't have this problem because there are simply fewer characters.
ObjectA a = getTheUser(session.getState().getAccount().getAccountNumber());
This is a bad example, probably because you just wrote something from the top of your head.
You are assigning, to variable named a of type ObjectA, the return value of a function named getTheUser.
So let's assume you wrote this instead:
User u = getTheUser(session.getState().getAccount().getAccountNumber());
I would break this expression like so:
Account acc = session.getState().getAccount();
User user = getTheUser( acc.getAccountNumber() );
My reasoning is: how would I think about what I am doing with this code?
I would probably think: "first I need to get the account from the session and then I get the user using that account's number".
The code should read the way you think. Variables should refer to the main entities involved; not so much to their properties (so I wouldn't store the account number in a variable).
A second factor to have in mind is: will I ever need to refer to this entity again in this context?
If, say, I'm pulling more stuff out of the session state, I would introduce SessionState state = session.getState().
This all seems obvious, but I'm afraid I have some difficulty putting in words why it makes sense, not being a native English speaker and all.
Maintainability, and with it, readability, is king. Luckily, shorter very often means more readable.
Here are a few tips I enjoy using to slice and dice code:
Variable names: how would you describe this variable to someone else on your team? You would not say "the numberOfLinesSoFar integer". You would say "numLines" or something similar - comprehensible and short. Don't pretend like the maintainer doesn't know the code at all, but make sure you yourself could figure out what the variable is, even if you forgot your own act of writing it. Yes, this is kind of obvious, but it's worth more effort than I see many coders put into it, so I list it first.
Control flow: Avoid lots of closing clauses at once (a series of }'s in C++). Usually when you see this, there's a way to avoid it. A common case is something like
:
if (things_are_ok) {
// Do a lot of stuff.
return true;
} else {
ExpressDismay(error_str);
return false;
}
can be replaced by
if (!things_are_ok) return ExpressDismay(error_str);
// Do a lot of stuff.
return true;
if we can get ExpressDismay (or a wrapper thereof) to return false.
Another case is:
Loop iterations: the more standard, the better. For shorter loops, it's good to use one-character iterators when the variable is never used except as an index into a single object.
The particular case I would argue here is against the "right" way to use an STL container:
for (vector<string>::iterator a_str = my_vec.begin(); a_str != my_vec.end(); ++a_str)
is a lot wordier, and requires overloaded pointer operators *a_str or a_str->size() in the loop. For containers that have fast random access, the following is a lot easier to read:
for (int i = 0; i < my_vec.size(); ++i)
with references to my_vec[i] in the loop body, which won't confuse anyone.
Finally, I often see coders take pride in their line number counts. But it's not the line numbers that count! I'm not sure of the best way to implement this, but if you have any influence over your coding culture, I'd try to shift the reward toward those with compact classes :)
Good explanation. I think this is version of the general Divide and Conquer mentality.

How can I program a simple chat bot AI?

I want to build a bot that asks someone a few simple questions and branches based on the answer. I realize parsing meaning from the human responses will be challenging, but how do you setup the program to deal with the "state" of the conversation?
It will be a one-to-one conversation between a human and the bot.
You probably want to look into Markov Chains as the basics for the bot AI. I wrote something a long time ago (the code to which I'm not proud of at all, and needs some mods to run on Python > 1.5) that may be a useful starting place for you: http://sourceforge.net/projects/benzo/
EDIT: Here's a minimal example in Python of a Markov Chain that accepts input from stdin and outputs text based on the probabilities of words succeeding one another in the input. It's optimized for IRC-style chat logs, but running any decent-sized text through it should demonstrate the concepts:
import random, sys
NONWORD = "\n"
STARTKEY = NONWORD, NONWORD
MAXGEN=1000
class MarkovChainer(object):
def __init__(self):
self.state = dict()
def input(self, input):
word1, word2 = STARTKEY
for word3 in input.split():
self.state.setdefault((word1, word2), list()).append(word3)
word1, word2 = word2, word3
self.state.setdefault((word1, word2), list()).append(NONWORD)
def output(self):
output = list()
word1, word2 = STARTKEY
for i in range(MAXGEN):
word3 = random.choice(self.state[(word1,word2)])
if word3 == NONWORD: break
output.append(word3)
word1, word2 = word2, word3
return " ".join(output)
if __name__ == "__main__":
c = MarkovChainer()
c.input(sys.stdin.read())
print c.output()
It's pretty easy from here to plug in persistence and an IRC library and have the basis of the type of bot you're talking about.
Folks have mentioned already that statefulness isn't a big component of typical chatbots:
a pure Markov implementations may express a very loose sort of state if it is growing its lexicon and table in real time—earlier utterances by the human interlocutor may get regurgitated by chance later in the conversation—but the Markov model doesn't have any inherent mechanism for selecting or producing such responses.
a parsing-based bot (e.g. ELIZA) generally attempts to respond to (some of the) semantic content of the most recent input from the user without significant regard for prior exchanges.
That said, you certainly can add some amount of state to a chatbot, regardless of the input-parsing and statement-synthesis model you're using. How to do that depends a lot on what you want to accomplish with your statefulness, and that's not really clear from your question. A couple general ideas, however:
Create a keyword stack. As your human offers input, parse out keywords from their statements/questions and throw those keywords onto a stack of some sort. When your chatbot fails to come up with something compelling to respond to in the most recent input—or, perhaps, just at random, to mix things up—go back to your stack, grab a previous keyword, and use that to seed your next synthesis. For bonus points, have the bot explicitly acknowledge that it's going back to a previous subject, e.g. "Wait, HUMAN, earlier you mentioned foo. [Sentence seeded by foo]".
Build RPG-like dialogue logic into the bot. As your parsing human input, toggle flags for specific conversational prompts or content from the user and conditionally alter what the chatbot can talk about, or how it communicates. For example, a chatbot bristling (or scolding, or laughing) at foul language is fairly common; a chatbot that will get het up, and conditionally remain so until apologized to, would be an interesting stateful variation on this. Switch output to ALL CAPS, throw in confrontational rhetoric or demands or sobbing, etc.
Can you clarify a little what you want the state to help you accomplish?
Imagine a neural network with parsing capabilities in each node or neuron. Depending on rules and parsing results, neurons fire. If certain neurons fire, you get a good idea about topic and semantic of the question and therefore can give a good answer.
Memory is done by keeping topics talked about in a session, adding to the firing for the next question, and therefore guiding the selection process of possible answers at the end.
Keep your rules and patterns in a knowledge base, but compile them into memory at start time, with a neuron per rule. You can engineer synapses using something like listeners or event functions.
I think you can look at the code for Kooky, and IIRC it also uses Markov Chains.
Also check out the kooky quotes, they were featured on Coding Horror not long ago and some are hilarious.
I think to start this project, it would be good to have a database with questions (organized as a tree. In every node one or more questions).
These questions sould be answered with "yes " or "no".
If the bot starts to question, it can start with any question from yuor database of questions marked as a start-question. The answer is the way to the next node in the tree.
Edit: Here is a somple one written in ruby you can start with: rubyBOT
naive chatbot program. No parsing, no cleverness, just a training file and output.
It first trains itself on a text and then later uses the data from that training to generate responses to the interlocutor’s input. The training process creates a dictionary where each key is a word and the value is a list of all the words that follow that word sequentially anywhere in the training text. If a word features more than once in this list then that reflects and it is more likely to be chosen by the bot, no need for probabilistic stuff just do it with a list.
The bot chooses a random word from your input and generates a response by choosing another random word that has been seen to be a successor to its held word. It then repeats the process by finding a successor to that word in turn and carrying on iteratively until it thinks it’s said enough. It reaches that conclusion by stopping at a word that was prior to a punctuation mark in the training text. It then returns to input mode again to let you respond, and so on.
It isn’t very realistic but I hereby challenge anyone to do better in 71 lines of code !! This is a great challenge for any budding Pythonists, and I just wish I could open the challenge to a wider audience than the small number of visitors I get to this blog. To code a bot that is always guaranteed to be grammatical must surely be closer to several hundred lines, I simplified hugely by just trying to think of the simplest rule to give the computer a mere stab at having something to say.
Its responses are rather impressionistic to say the least ! Also you have to put what you say in single quotes.
I used War and Peace for my “corpus” which took a couple of hours for the training run, use a shorter file if you are impatient…
here is the trainer
#lukebot-trainer.py
import pickle
b=open('war&peace.txt')
text=[]
for line in b:
for word in line.split():
text.append (word)
b.close()
textset=list(set(text))
follow={}
for l in range(len(textset)):
working=[]
check=textset[l]
for w in range(len(text)-1):
if check==text[w] and text[w][-1] not in '(),.?!':
working.append(str(text[w+1]))
follow[check]=working
a=open('lexicon-luke','wb')
pickle.dump(follow,a,2)
a.close()
here is the bot
#lukebot.py
import pickle,random
a=open('lexicon-luke','rb')
successorlist=pickle.load(a)
a.close()
def nextword(a):
if a in successorlist:
return random.choice(successorlist[a])
else:
return 'the'
speech=''
while speech!='quit':
speech=raw_input('>')
s=random.choice(speech.split())
response=''
while True:
neword=nextword(s)
response+=' '+neword
s=neword
if neword[-1] in ',?!.':
break
print response
You tend to get an uncanny feeling when it says something that seems partially to make sense.
I would suggest looking at Bayesian probabilities. Then just monitor the chat room for a period of time to create your probability tree.
I'm not sure this is what you're looking for, but there's an old program called ELIZA which could hold a conversation by taking what you said and spitting it back at you after performing some simple textual transformations.
If I remember correctly, many people were convinced that they were "talking" to a real person and had long elaborate conversations with it.
If you're just dabbling, I believe Pidgin allows you to script chat style behavior. Part of the framework probably tacks the state of who sent the message when, and you'd want to keep a log of your bot's internal state for each of the last N messages. Future state decisions could be hardcoded based on inspection of previous states and the content of the most recent few messages. Or you could do something like the Markov chains discussed and use it both for parsing and generating.
If you do not require a learning bot, using AIML (http://www.aiml.net/) will most likely produce the result you want, at least with respect to the bot parsing input and answering based on it.
You would reuse or create "brains" made of XML (in the AIML-format) and parse/run them in a program (parser). There are parsers made in several different languages to choose from, and as far as I can tell the code seems to be open source in most cases.
You can use "ChatterBot", and host it locally using - 'flask-chatterbot-master"
Links:
[ChatterBot Installation]
https://chatterbot.readthedocs.io/en/stable/setup.html
[Host Locally using - flask-chatterbot-master]: https://github.com/chamkank/flask-chatterbot
Cheers,
Ratnakar