We're deserializing large JSON payloads from an external API, each response being roughly 30MB.
On my developer machine it takes 0.3s to call JsonConvert.DeserializObject(jsonString) with those 30MB in a single thread. But if I deserialize in multiple threads the time spent per deserialization drastically increases as the number of threads increases, up to 5-10s sometimes. My test code looks like this:
var json = System.IO.File.ReadAllText("c:/largejson.json");
var count = 100;
var ops = 0;
var results = new ConcurrentBag<TimeSpan>();
while (ops < count)
{
var tasks = new List<Task>();
for (var i = 0; i < Environment.ProcessorCount; i++)
{
tasks.Add(Task.Run(() =>
{
var stopwatch = new Stopwatch();
stopwatch.Start();
JsonConvert.DeserializeObject(json);
ops++;
stopwatch.Stop();
results.Add(stopwatch.Elapsed);
}
));
}
await Task.WhenAll(tasks);
}
var totalSeconds = results.Sum(r => r.TotalSeconds);
var secondsPerOp = totalSeconds / ops;
And here's the file I'm using: https://github.com/andersekdahl/files/blob/master/largejson.json
I've tested both with and without passing in a new JsonSerializerSettings() as a second argument but it doesn't make any difference.
My first instinct was that there was some locking inside Json.NET but when I profiled it using dotTrace nothing really sticks out. A lot of time is spent in JTokenWriter.WritePropertyName but that doesn't look like it's shared between threads/deserializations.
Anyone has any clues what's causing the slowdown?
Related
I'm currently working with a folder of JSON files which are collected through a tracking experiment with a drone. The data contains position, rotation and timestamp of the drone while it's moving and levitating inside the tracking system.
What I'm currently doing is trying to simulate the movement of the drone inside Unity using those data. So far, I've managed to parse the position and rotation from the data to an object inside Unity and extract the timestamp to System.DateTime in Unity.
However, I don't how to work with the timestamp. I want to use the timestamp to match the position and rotation of the object (i.e: at this timestamp, the drone should be at this position(x,y,z) and has the rotation(x,y,z,w)). Can someone help me with this problem, really appreciate your help :D Here is my current code:
void Update()
{
if (loaded)
{
for(int i = 0; i <= pos_data.Count; i+= 10)
{
Cube.transform.position = pos_data[i];
Cube.transform.rotation = rot_data[i];
}
}
else
{
LoadJson();
//startTime = datetime[0];
loaded = true;
}
}
public void LoadJson()
{
string HeadPath = #Application.dataPath + "/Data/" + "drone_data_1.json";
string HeadJsonhold = File.ReadAllText(HeadPath);
var data_ = JSON.Parse(HeadJsonhold);
for (int rows = 0; rows <= data_.Count; rows += 10)
{
pos_data.Add(new Vector3(data_[rows]["location"]["x"].AsFloat, data_[rows]["location"]["y"].AsFloat, data_[rows]["location"]["z"].AsFloat));
rot_data.Add(new Quaternion(data_[rows]["rotation"]["x"].AsFloat, data_[rows]["rotation"]["y"].AsFloat, data_[rows]["rotation"]["z"].AsFloat, data_[rows]["rotation"]["w"].AsFloat));
Time = System.DateTime.ParseExact(data_[rows]["Timestamp"], "yyyyMMddHHmmss",null);
//Debug.Log(Time);
}
}
If I understand you correctly what you are getting are samples of a real-world drone that at some rate stores keyframes of its movement.
Now you already successfully load that json data but wonder how to animate the Unity object accordingly.
The timestamp itself you can't use at all! ^^
It most probably lies somewhere in the past ;) And you can't just assign something to Time.
What you can do, however, is take the timestamp of the first sample (I will just assume that your samples are all already ordered by the time) and calculate the difference to the next sample and so on.
Then you can use that difference in order to always interpolate between the current and next sample transforms using the given time delta.
Currently you are just doing all samples in one single frame so there won't be any animation at all.
Also just as sidenote:
for(int i = 0; i <= pos_data.Count; i+= 10)
is wrong twice:
a) you already skipped 10 samples when loading the data -> are you sure you now want to again skip 10 of these => In total every time skipping 100 samples?
b) since indices are 0 based the last accessible index would be pos_data.Count - 1 so in general when iterating Lists/arrays it should be i < pos_data.Count ;)
First of all I would suggest you use a better data structure and use one single list holding the information that belongs together instead of multiple parallel lists and rather load your json like e.g.
[Serializable]
public class Sample
{
public readonly Vector3 Position;
public readonly Quaternion Rotation;
public readonly float TimeDelta;
public Sample(Vector3 position, Quaternion rotation, float timeDelta)
{
Position = position;
Rotation = rotation;
TimeDelta = timeDelta;
}
}
And then
// Just making this serialized so you can immediately see in the Inspector
// if your data loaded correctly
[SerializeField] private readonly List<Sample> _samples = new List<Sample>();
public void LoadJson()
{
// start fresh
_samples.Clear();
// See https://learn.microsoft.com/dotnet/api/system.io.path.combine
var path = Path.Combine(Application.dataPath, "Data", "drone_data_1.json");
var json = File.ReadAllText(path);
var data = JSON.Parse(json);
DateTime lastTime = default;
for (var i = 0; i <= data.Count; i += 10)
{
// First I would pre-cache these values
var sample = data[i];
var sampleLocation = sample["location"];
var sampleRotation = sample["rotation"];
var sampleTime = sample["Timestamp"];
// Get your values as you did already
var position = new Vector3(sampleLocation["x"].AsFloat, sampleLocation["y"].AsFloat, sampleLocation["z"].AsFloat));
var rotation = new Quaternion(sampleRotation["x"].AsFloat, sampleRotation["y"].AsFloat, sampleRotation["z"].AsFloat, sampleRotation["w"].AsFloat));
var time = System.DateTime.ParseExact(sampleTime, "yyyyMMddHHmmss", null);
// Now for the first sample there is no deltaTime
// for all others calculate the difference in seconds between the
// last and current sample
// See https://learn.microsoft.com/dotnet/csharp/language-reference/operators/conditional-operator
var deltaTime = i == 0 ? 0f : GetDeltaSeconds(lastTime, time);
// and of course store it for the next iteration
lastTime = time;
// Now you can finally add the sample to the list of samples
// instead of having multiple parallel lists
_samples.Add(new Sample(position, rotation, deltaTime));
}
}
private float GetDeltaSeconds(DateTime first, DateTime second)
{
// See https://learn.microsoft.com/dotnet/api/system.datetime.op_subtraction#System_DateTime_op_Subtraction_System_DateTime_System_DateTime_
var deltaSpan = second - first;
// See https://learn.microsoft.com/dotnet/api/system.timespan.totalseconds#System_TimeSpan_TotalSeconds
return (float)deltaSpan.TotalSeconds;
}
So now what to do with this information?
You now have samples (still assuming ordered by time) holding all required information to be able to interpolate between them.
I would use Coroutines instead of Update, in my eyes they are easier to understand and maintain
// Do your loading **once** in Start
private void Start()
{
LoadJson();
// Then start the animation routine
// I just make it a method so you could also start it later e.g. via button etc
StartAnimation();
}
// A flag just in case to avoid concurrent animations
private bool alreadyAnimating;
// As said just making this a method so you could also remove it from Start
// and call it in any other moment you like
public void StartAnimation()
{
// Only start an animation if there isn't already one running
// See https://docs.unity3d.com/ScriptReference/MonoBehaviour.StartCoroutine.html
if(!alreadyAnimating) StartCoroutine(AnimationRoutine());
}
private IEnumerator AnimationRoutine()
{
// Just in case abort if there is already another animation running
if(alreadyAnimating) yield break;
// Block concurrent routine
alreadyAnimating = true;
// Initially set your object to the first sample
var lastSample = _samples[0];
Cube.transform.position = lastSample.Position;
Cube.transform.rotation = lastSample.Rotation;
// This tells Unity to "pause" the routine here, render this frame
// and continue from here in the next frame
yield return null;
// then iterate through the rest of samples
for(var i = 1; i < _samples.Count; i++)
{
var lastPosition = lastSample.Position;
var lastRottaion = lastSample.Rottaion;
var currentSample = _samples[i];
var targetPosition = sample.Position;
var targetRotation = sample.Rotation;
// How long this interpolation/animation will take
var duration = currentSample.TimeDelta;
// You never know ;)
// See https://docs.unity3d.com/ScriptReference/Mathf.Approximately.html
if(Mathf.Approximately(duration, 0f))
{
Cube.transform.position = targetPosition;
Cube.transform.rotation = targetRotation;
lastSample = currentSample;
continue;
}
// And this is where the animation magic happens
var timePassed = 0f;
while(timePassed < duration)
{
// this factor will be growing linear between 0 and 1
var factor = timePassed / duration;
// Interpolate between the "current" transforms (the ones it had at beginning of this iteration)
// towards the next sample target transforms using the factor between 0 and 1
// See https://docs.unity3d.com/ScriptReference/Vector3.Lerp.html
Cube.transform.position = Vector3.Lerp(lastPosition, targetPosition, factor);
// See https://docs.unity3d.com/ScriptReference/Quaternion.Slerp.html
Cube.transform.rotation = Quaternion.Slerp(lastRotation, targetRotation, factor);
// This tells Unity to "pause" the routine here, render this frame
// and continue from here in the next frame
yield return null;
// increase by the time passed since the last frame was rendered
timePassed += Time.deltaTime;
}
// just to be sure to end with clean values
Cube.transform.position = targetPosition;
Cube.transform.rotation = targetRotation;
lastSample = currentSample;
}
// Allow the next animation to start (or restart this one)
alreadyAnimating = false;
// Additional stuff to do once the animation is done
}
I'm a bit new with AS3 (but not really with coding) so please forgive my ignorance. I'm creating a small function that will be called by a Main Function to update the position of 52 Pointers that have the x and y position of multiple point objects (empty movie clips). It will also then update two global arrays with those values (one array for the x and one for the y).
The problem is, as there are 52 of them, and they will probably grow in quantity, I'd like to be able to use a FOR function to do it, but I can't seem to be able to figure it out.
I get this error: Access of undefined property _point.
Here is a piece of the code that dream about:
function happyFunc():void
{
var avpointers:int = 52;
var vpointx:Array = new Array();
var vpointy:Array = new Array();
for (aa=0; aa<vpointers; aa++)
{
vpointx[aa] = _point[aa].x;
vpointy[aa] = _point[aa].y;
}
}
And this is the code that I'm stuck with...
function reallySadFunc():void
{
_point1 = localToGlobal(new Point(point1.x,point1.y));
//...
_point52 = localToGlobal(new Point(point52.x,point1.y));
vpointx[0] = _point1.x;
vpointx[1] = _point2.x;
//...
//oh god there are 104 lines of this why do I have to suffer
}
Thank you!
If I read your question correctly, I think this is what you need:
public static const CLIP_COUNT:int = 52;
// ...
private function happyFunc(parentClip:DisplayObjectContainer):void
{
var vpointx:Vector.<Number> = new Vector.<Number>(CLIP_COUNT, true);
var vpointy:Vector.<Number> = new Vector.<Number>(CLIP_COUNT, true);
var clipPoint:Point = new Point ();
var globalPoint:Point;
for (var i:int = 0; i < CLIP_COUNT; i++)
{
var childClip:DisplayObject = parentClip.getChildByName("point" +
(i + 1).toString());
clipPoint.x = childClip.x;
clipPoint.y = childClip.y;
globalPoint = parentClip.localToGlobal(clipPoint);
vpointx[i] = globalPoint.x;
vpointy[i] = globalPoint.y;
}
// do something with vpointx and vpointy - they're local variables
// and will go out of scope unless you declare them as class members
// or somehow return them from this function.
}
This function works by taking the parent display object (the one that contains the 52 movie clips - this could be the Stage) and iterates through them by getting each movie clip by name. The code assumes that your movie clips are called point1, point2, ..., point52.
To speed up the local-to-global coordinate conversion, two Point objects are created and then reused during the for loop to avoid creating many temporary Point instances.
Instead of using Array, use Vector.<Number> - this class has better performance than Array does.
Suppose we have some custom object type:
class SomeObjectType {
public var intProperty1:int;
public var intProperty2:int;
public var intProperty3:int;
public var stringProperty1:String;
public var stringProperty2:String;
public var stringProperty3:String;
public var stringPropertyThatActuallyIsInt1:String;
public var stringPropertyThatActuallyIsInt2:String;
public var stringPropertyThatActuallyIsInt3:String;
...
%ABOUT_20_ANOTHER_PROPERTIES_THAT_I_WON'T_USE%
}
We have a collection of more than 20k instances of these objects. And we have just 1 text input that is actually search filter. User can type in this filter field anything he want and if his filter matches with ANY of first 9 fields I described before we should leave this object in collection. Just simple items filtering.
And let me describe how it works in our project now. This algorithm casts all these properties to Strings, concatenate them, and search using indexOf() != -1 method. This is really slow. It takes about 500-900ms on my dev machine and about 3-4s on iPad on every filter change. Horrible statistics, isn't it?
Small note: we use this algorithm in different 3 places in app and objects differs from object I described above but idea is same. I believe that it is a good idea to compare int to int, string to string (implementing some of fast algorithms(there are lots if them)), and convert string that is actually to int and compare them as int to int, but object differs a lot so I need some common algorithm.
If by collection you mean ArrayCollection, I would recommend to use Vector instead.
Vectors are around 50 times faster then ArrayCollections.
If you need databinding, you could have a look at VectorCollection, but I can't imagine the performance to be anywhere close to Vector.
Also if you are not extending class SomeObjectType anywhere, you could gain some performance (especially on iOS) by making it final class SomeObjectType.
You can use dictionary to search, I think it will much faster, you just need to init for one time.
var dict:Dictionary = new Dictionary();
//get properties,in you obj, they are intProperty1,intProperty2 etc,exclude prototype
var properties:Array = ObjectUtil.getClassInfo(SomeObjectType, ["prototype"]).properties as Array;
for each (var obj:SomeObjectType in yourArrayCollection) {
for (var i:int = 0; i < properties.length; i++) {
var qname:Object = properties[i];
var k:String = qname.localName;
var v:String = obj[k];
if (dict[v] == null) {
dict[v] = [];
}
//add only one time
if ((dict[v] as Array).indexOf(obj) == -1) {
(dict[v] as Array).push(obj);
}
}
}
So if you want return all objs contains "5", just return dict[5]
I have an object that is nested several levels deep which I'm referencing multiple times. Is it faster to create and set a reference to that object or does it matter?
Context
I heard that it is faster to create a local reference. This is years ago and for a Visual Basic project. But this is Flash. And the output is a SWF / bytecode. And the compiler could look at that reference and do what I'm doing so that the object does not have to be looked up each time.
For example:
public function doStuff():void {
Model.instance.view1.button1 = button1;
Model.instance.view1.button2 = button2;
Model.instance.view1.button3 = button3;
Model.instance.view1.button4 = button4;
Model.instance.view1.button5 = button5;
Model.instance.view1.button6 = button6;
// more fake code referencing more something.something.something like things
for (var i:int;i<something.something.something.length;i++) {
var result:Object = Controller.staticMethod(button1);
var result2:Object = Controller.staticMethod(button1);
}
}
Vs:
public function doStuff():void {
var view1:View = Model.instance.view1;
view1.button1 = button1;
view1.button2 = button2;
view1.button3 = button3;
view1.button4 = button4;
view1.button5 = button5;
view1.button6 = button6;
}
.
Taking Baris suggestion I tested it for myself. Below are the results (though I don't know how to interpret them into the actual difference ie 'method 1 is .000001 faster than method 2').
Results
Test 1
var instance:Model = Model.instance;
var startTime:int = getTimer();
for(var i:int = 0; i<10000000; i++){
instance.url = "";
}
trace(getTimer()-startTime); // 826, 929, 823
var startTime:int = getTimer();
for(var i:int = 0; i<10000000; i++){
Model.instance.url = "";
}
trace(getTimer()-startTime); // 3483, 3976, 3539
Test 2
var instance:Model = Model.instance;
var localLogo:BitmapImage = Model.instance.logo;
var startTime:int = getTimer();
for(var i:int = 0; i<10000000; i++){
localLogo = logo;
}
trace(getTimer()-startTime); // 2070, 2083, 2110
var startTime:int = getTimer();
for(var i:int = 0; i<10000000; i++){
instance.logo = logo;
}
trace(getTimer()-startTime); // 2028, 2509, 2154
Generally, the as3 compiler is not that smart as we think. I think it does that to reduce compilation times.
This page http://upshots.org/actionscript/20-tips-to-optimize-your-actionscript gives us a rule:
5: The one-dot rule When accessing nested variables, anytime a
variable requires even one level of scope shift to discover, and is
referenced more than once, save it to local variable. In drawing
classes, you’ll often see Math.PI / 2 referenced within loops that
might iterated hundreds of times each frame – that value should be
stored in a local variable. Note that this is not true when
referencing member methods. Saving Math.PI to a variable is
appropriate; saving Math.round to a variable is not.
It does make difference, but to a software building perspective, the more readable code would be the propper way to do it, unless it's critical code, like a physics section of your app.
Other than making time benchmarks, you can inspect the bytecode too, using this tool http://www.swfwire.com/
I prefer :
public function doStuff():void {
var view1:View = Model.instance.view1;
view1.button1 = button1;
view1.button2 = button2;
view1.button3 = button3;
view1.button4 = button4;
view1.button5 = button5;
view1.button6 = button6;
}
I think it looks cleaner, you can profile and see which one is faster but it won't matter performance wise if you are not doing this tons of times in some inner loop.
public function doStuff():void {
var startTime:int = getTimer();
for(var i:int = 0; i<1000000; i++){
Model.instance.view1.button1 = button1;
Model.instance.view1.button2 = button2;
Model.instance.view1.button3 = button3;
Model.instance.view1.button4 = button4;
Model.instance.view1.button5 = button5;
Model.instance.view1.button6 = button6;
}
trace(getTimer()-startTime);
}
public function doStuff():void {
var startTime:int = getTimer();
for(var i:int = 0; i<1000000; i++){
var view1:View = Model.instance.view1;
view1.button1 = button1;
view1.button2 = button2;
view1.button3 = button3;
view1.button4 = button4;
view1.button5 = button5;
view1.button6 = button6;
}
trace(getTimer()-startTime);
}
The time to access is probably irrelevant, because you seem to be doing some UI work, and fractions of milliseconds it will take are of no concern. There are other concerns though:
You are writing repetitious code - this is bad, always. If it must be repeated, you should write a program that repeats it from a single source everywhere, if it may not be repeated - opt not to repeat. The concern is simple. Once you discover that the buttons are in ModelB.instance.view1 you will need to copy and paste the same code multiple times - of course while doing repetitive work you will make a mistake.
AS3 code not only allows, it also welcomes side effects, which, in this case may create undesirable effect. For example, Model.instance.view1 may be a getter that returns a new copy of view every time it is accessed, or, when accessed, it may alter something else in the internal state of the instance, whatever that is, and performing this change multiple times is undesirable.
There's Demeter's rule. When simplified beyond possible, it says that you shouldn't use more than a single dot operator in succession. I.e. an object may only be aware of the property of its own property (or local variable), but never access the property of the property of the property and never further down the line. This imposes unnecessary restriction on the structure of your program. However, sometimes it looks like it might be difficult to avoid, you should strive for a better, self-contained code. More on Demeter's rule here: http://en.wikipedia.org/wiki/Law_of_Demeter
It is not a function of compiler to "optimize" such code, especially because of the side effects. Compiler will faithfully follow the bad code written by a programmer and generate a lot of repetitious assembly because it trusts the programmer that this code is actually needed. Note, I'm not talking about Adobe compiler, which is not an optimizing compiler, and wouldn't remove this code even if the static analysis suggested that. I'm talking about some abstract "ideal" compiler for the language, that would optimally compile the code in question.
Suppose there is a live WAV stream that can be reached at a certain URL, and we need to stream it with as little latency as possible. Using HTML5 <audio> for this task is a no-go, because browsers attempt to pre-buffer several seconds of the stream, and the latency goes up accordingly. That's the reason behind using Flash for this task. However, due to my inexperience with this technology, I only managed to get occasional clicks and white noise. What's wrong in the code below? Thanks.
var soundBuffer: ByteArray = new ByteArray();
var soundStream: URLStream = new URLStream();
soundStream.addEventListener(ProgressEvent.PROGRESS, readSound);
soundStream.load(new URLRequest(WAV_FILE_URL));
var sound = new Sound();
sound.addEventListener(SampleDataEvent.SAMPLE_DATA,playSound);
sound.play();
function readSound(event:ProgressEvent):void {
soundStream.readBytes(soundBuffer, 0, soundStream.bytesAvailable);
}
function playSound(event:SampleDataEvent):void {
/* The docs say that if we send too few samples,
Sound will consider it an EOF */
var samples:int = (soundBuffer.length - soundBuffer.position) / 4
var toadd:int = 4096 - samples;
try {
for (var c: int=0; c < samples; c++) {
var n:Number = soundBuffer.readFloat();
event.data.writeFloat(n);
event.data.writeFloat(n);
}
} catch(e:Error) {
ExternalInterface.call("errorReport", e.message);
}
for (var d: int = 0; d < toadd; d++) {
event.data.writeFloat(0);
event.data.writeFloat(0);
}
}
Like The_asMan pointed out, playing a wav file is not that easy. See as3wavsound for an example.
If your goal is low latency, the best option would be to convert to MP3, so you can use just use a SoundLoaderContext.