Netlogo: most efficient way to code with gis:apply-coverage? - gis

I'm copying variables from a SHP file to patches with the gis:apply-coverage command. The issue is that they are 147 different variables, so I want to write code as efficient as possible. The variables are:
patches-own [
SUBSECTOR HARE HAUN HAMU TUSAPAB TUPLPBSC TUECOTEA TUAVSCVB TURUEATF
INMIAA INMIEM INIBRBR INIBRBA INIRMT INIRP INIRPM INIRYD INIRP1 INIARIR
INAAGCU INAGAEG INAGACE INAGAES INAGAFO INAGAAA INAGAAP INAAFSI INAAFVH
INAPCPE INAPCCC ESCSVMS ESCSVTE ESCSBTE ESCSBAP ESCSBAL ESCSBTI ESCSBLV
ESCSBEL ESCSBCA ESCSBRM ESCSBAI ESCSDTE ESCSDCC ESCSDTA ESCSDMP ESCSDSF
ESCSDDA ESCSDHM ESCSDTP ESCSDCA ESCSDAC ESCSDCV ESCSDSS ESCSDCS ESCSCRE
ESCSCMA ESECVJN ESECVPR ESECVSG ESECBEA ESECBAI ESECBBG ESECBTE ESECDET
ESECDGA ESECCLG ESECCPO ESECCPH ESECCAJ ESECCTC ESECVUU ESSSBCS ESSSBHO
ESSSBHE ESSSDCC ESSSDOR ESSSDCI ESSSCCA ESSSRHR ESDRVPC ESDRVJV ESDRVJI
ESDRVCD ESDRBPV ESDRBSF ESDRBUC ESDRBCC ESDRBSD ESDRBED ESDRDCG ESDRDAF
ESDRDED ESDRDBP ESDRDCI ESDRDCN ESDRDCE ESDRCZO ESDRCCC ESDRCCI ESAPBMP
ESAPBOG ESAPBJC ESAPDDM ESAPCPM ESAPCRP ESAPCAD ESAPCEM ESSUVCP ESSUBCS
ESSUB1 ESSUBCE ESSUCIE ESSUCEP ESCTVAS ESCTVOT ESCTBAC ESCTBAT ESCTBCT
ESCTBET ESCTB1 ESCTBES ESCTBAP ESCTDOT ESCTDTT ESCTDHH ESCTD1 ESCTCTT
ESCTC1 ESCTRTV ESCTRTA ESCTRTF ESCTRTI ESCTRPU ESCTRTE ESCTRMA ESCTRIP
UEGCC UECAH UEPES UETMA UEMON UEAER UEPTAN UEPD UECB UERSPT UEPRE
]
and the code I wrote is:
to copy-values-to-patch
gis:apply-coverage Subsectores_090121-dataset "SUBSECTOR" Subsector
gis:apply-coverage Subsectores_090121-dataset "HARE" HARE
gis:apply-coverage Subsectores_090121-dataset "HAUN" HAUN
gis:apply-coverage Subsectores_090121-dataset "HAMU" HAMU
....
gis:apply-coverage Subsectores_090121-dataset "UEPRE" UEPRE
end
In this format, there are 149 lines of code, but I think there must be a way to write it in a more practical way (for example, not needing to write the gis:apply-coverage command 147 times). As anyone can see, I'm still new at coding, so any help would be greatly appreciated. Regards,
Javier

Related

Octave buffering console output

I'm new to Octave.
I'm working with a huge matrix with thousands of rows and just 2 columns.
When I print it the terminal seems to be cutting part of it, but I want the whole thing to be shown.
Any way to do that or maybe to redirect the output in a file or something?
I had the same in problem in Eclipse but it was much easier to solve the same issue back then.

to redirect a Matrix to a file use save
A=rand(5,5)
A =
0.552789 0.570720 0.896715 0.982821 0.622758
0.704791 0.514302 0.380680 0.344768 0.459588
0.230478 0.744468 0.668264 0.659316 0.414710
0.819015 0.572315 0.404128 0.074092 0.813156
0.099461 0.031186 0.499781 0.758091 0.243281
save("-ascii","file-ascii.txt","A")
save("file-text.txt","A")
pure text
$ cat file-ascii.txt
5.52789005e-01 5.70719716e-01 8.96714571e-01 9.82821412e-01 6.22757836e-01
7.04791031e-01 5.14301972e-01 3.80679512e-01 3.44768074e-01 4.59587827e-01
2.30478499e-01 7.44467506e-01 6.68264066e-01 6.59315817e-01 4.14709954e-01
8.19014881e-01 5.72314594e-01 4.04128230e-01 7.40916214e-02 8.13155903e-01
9.94607828e-02 3.11857103e-02 4.99780576e-01 7.58091470e-01 2.43280663e-01
octave default style
$ cat file-text.txt
# Created by Octave 5.2.0, Wed May 27 21:43:22 2020 CEST <Marco#LAPTOP-82F08ILC>
# name: A
# type: matrix
# rows: 5
# columns: 5
0.55278900467056769 0.57071971558810353 0.89671457115004172 0.98282141203898321 0.622757835845834
0.70479103140202992 0.51430197227101204 0.3806795123567624 0.34476807355856237 0.45958782700965189
0.2304784985452572 0.74446750644064241 0.66826406647042169 0.65931581744685042 0.41470995365627888
0.81901488130574696 0.57231459443213173 0.40412823039438278 0.074091621362711677 0.81315590327055964
0.099460782832668723 0.031185710267227097 0.49978057628118927 0.75809147040922908 0.24328066285306477

Hyperparameter tuning using tensorboard.plugins.hparams api with custom loss function

I am building a neural network with my own custom loss function (pretty long and complicated). My network is unsupervised so my input and expected output are identical and also at the moment I am using one single input (just trying to optimize the loss for a single input).
I am trying to use tensorboard.plugins.hparams api for hyperparameter tuning and don't know how to incorporate my custom loss function there. I'm trying to follow the code suggested on the Tensorflow 2.0 website.
This is what the website suggests:
HP_NUM_UNITS = hp.HParam('num_units', hp.Discrete([16, 32]))
HP_DROPOUT = hp.HParam('dropout', hp.RealInterval(0.1, 0.2))
HP_OPTIMIZER = hp.HParam('optimizer', hp.Discrete(['adam', 'sgd']))
METRIC_ACCURACY = 'accuracy'
with tf.summary.create_file_writer('logs/hparam_tuning').as_default():
hp.hparams_config(
hparams=[HP_NUM_UNITS, HP_DROPOUT, HP_OPTIMIZER],
metrics=[hp.Metric(METRIC_ACCURACY, display_name='Accuracy')],
)
I need to change that as I don't want to use the dropout layer, so I can just delete that. In terms of the METRIC_ACCURACY, I don't want to use accuracy as that has no use in my model but rather use my custom loss function. If I were to do the regular fit model it would look like this:
model.compile(optimizer=adam,loss=dl_tf_loss, metrics=[dl_tf_loss])
So I tried to change the suggested code into the following code but I get an error and am wondering how I should change it so that it suits my needs. Here is what I tried:
HP_NUM_UNITS = hp.HParam('num_units', hp.Discrete([16, 32]))
HP_OPTIMIZER = hp.HParam('optimizer', hp.Discrete(['adam', 'sgd']))
#METRIC_LOSS = dl_tf_loss
with tf.summary.create_file_writer('logs/hparam_tuning').as_default():
hp.hparams_config(hparams=[HP_NUM_UNITS, HP_OPTIMIZER],metrics=
[hp.Metric(dl_tf_loss, display_name='Loss')])
It gives me the following error:
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-26-27d079c6be49> in <module>()
5
6 with tf.summary.create_file_writer('logs/hparam_tuning').as_default():
----> 7 hp.hparams_config(hparams=[HP_NUM_UNITS, HP_OPTIMIZER],metrics=[hp.Metric(dl_tf_loss, display_name='Loss')])
8
3 frames
/usr/local/lib/python3.6/dist-packages/tensorboard/plugins/hparams/summary_v2.py in hparams_config(hparams, metrics, time_created_secs)
127 hparams=hparams,
128 metrics=metrics,
--> 129 time_created_secs=time_created_secs,
130 )
131 return _write_summary("hparams_config", pb)
/usr/local/lib/python3.6/dist-packages/tensorboard/plugins/hparams/summary_v2.py in hparams_config_pb(hparams, metrics, time_created_secs)
161 domain.update_hparam_info(info)
162 hparam_infos.append(info)
--> 163 metric_infos = [metric.as_proto() for metric in metrics]
164 experiment = api_pb2.Experiment(
165 hparam_infos=hparam_infos,
/usr/local/lib/python3.6/dist-packages/tensorboard/plugins/hparams/summary_v2.py in <listcomp>(.0)
161 domain.update_hparam_info(info)
162 hparam_infos.append(info)
--> 163 metric_infos = [metric.as_proto() for metric in metrics]
164 experiment = api_pb2.Experiment(
165 hparam_infos=hparam_infos,
/usr/local/lib/python3.6/dist-packages/tensorboard/plugins/hparams/summary_v2.py in as_proto(self)
532 name=api_pb2.MetricName(
533 group=self._group,
--> 534 tag=self._tag,
535 ),
536 display_name=self._display_name,
TypeError: <tensorflow.python.eager.def_function.Function object at 0x7f9f3a78e5c0> has type Function, but expected one of: bytes, unicode
I also tried running the following code:
with tf.summary.create_file_writer('logs/hparam_tuning').as_default():
hp.hparams_config(hparams=[HP_NUM_UNITS, HP_OPTIMIZER],metrics=
[dl_tf_loss])
but got the following error:
AttributeError Traceback (most recent call last)
<ipython-input-28-6778bdf7f1b1> in <module>()
8
9 with tf.summary.create_file_writer('logs/hparam_tuning').as_default():
---> 10 hp.hparams_config(hparams=[HP_NUM_UNITS, HP_OPTIMIZER],metrics=[dl_tf_loss])
2 frames
/usr/local/lib/python3.6/dist-packages/tensorboard/plugins/hparams/summary_v2.py in <listcomp>(.0)
161 domain.update_hparam_info(info)
162 hparam_infos.append(info)
--> 163 metric_infos = [metric.as_proto() for metric in metrics]
164 experiment = api_pb2.Experiment(
165 hparam_infos=hparam_infos,
AttributeError: 'Function' object has no attribute 'as_proto'
Would greatly appreciate any help.
Thanks in advance!

I figured it out.
The original METRIC_ACCURACY that I changed to METRIC_LOSS is apparently just the name, I needed to write 'tf_dl_loss' as a string and not as the function.
In the proceeding parts of the tuning, I needed to anyway write my fit command, there I inserted the actual loss function as I showed in my example of the regular fit function.
Highly recommend this as a way of tuning the hyperparameters.

You might be interested by this demo. Compiling the model with dl_tf_loss in the metric will waste time. It is possible to let hp.Metric know about other recorded summaries in different directories using the group argument.

JQ 1.5: Timestamp / date transformation produce huge file

I using jq 1.5 under a Windows 10 powershell enviroment to transform json files and import them to a MS SQL database. The original json file is around 1,1mb. I stored the file here: Json origin file. I use following jq command to transform the data:
[.legs[] | {Legid: .legId, Farecode: .fareBasisCode, Travelduration: .travelDuration, Traveldistance: .totalTravelDistance, Distanceunit: .totalTravelDistanceUnits, Refundable: .isRefundable , Nonstop: .isNonStop, Departure_Airport: .segments[].departureAirportName, Departure_Code: .segments[].departureAirportCode, Arrival_Airport: .segments[].arrivalAirportName, Arrival_Code: .segments[].arrivalAirportCode, Departure_Time: .segments[].departureTimeEpochSeconds, Arrival_Time: .segments[].arrivalTimeEpochSeconds, Airline: .segments[].airlineName, Airline_Code: .segments[].airlineCode, Flight_Number: .segments[].flightNumber, Equipment: .segments[].equipmentDescription}]
That command produce following file transformed file. Now i had to transform the UNIX Timestamps to Dates. So i modified the command:
[.legs[] | {Legid: .legId, Farecode: .fareBasisCode, Travelduration: .travelDuration, Traveldistance: .totalTravelDistance, Distanceunit: .totalTravelDistanceUnits, Refundable: .isRefundable , Nonstop: .isNonStop, Departure_Airport: .segments[].departureAirportName, Departure_Code: .segments[].departureAirportCode, Arrival_Airport: .segments[].arrivalAirportName, Arrival_Code: .segments[].arrivalAirportCode, Departure_Time: .segments[].departureTimeEpochSeconds, Arrival_Time: .segments[].arrivalTimeEpochSeconds, Airline: .segments[].airlineName, Airline_Code: .segments[].airlineCode, Flight_Number: .segments[].flightNumber, Equipment: .segments[].equipmentDescription}] | .[].Departure_Time |= todate | .[].Arrival_Time |= todate
The transformed file without date Transformation have around 3 mb. After the date Transformation the file have around 40 mb. I think i have a logical error in my command, but cant find it. Tips?
Regards
Timo

Your use of iteration (.segments[]) causes multiplicative behavior: in your case, since there are four cases in which .segments|length is 2, you get a 2^10 expansion locally, four times.
In situations such as this, it would make sense to use a small but well-chosen subset of the data (or maybe more easily, an artificial dataset) to check the code.
Perhaps what you intended is something more like:
[ .legs[] | range(0; .segments|length) as $i | .... ]

OpenAI gym: How to get complete list of ATARI environments

I have installed OpenAI gym and the ATARI environments. I know that I can find all the ATARI games in the documentation but is there a way to do this in Python, without printing any other environments (e.g. NOT the classic control environments)

You can achieve this by calling list_games() on atari_py:
>>> import atari_py as ap
>>> game_list = ap.list_games()
>>> print(sorted(game_list))
<<< ['adventure',
'air_raid',
'alien',
'amidar',
'assault',
'asterix',
'asteroids',
'atlantis',
'bank_heist',
'battle_zone',
'beam_rider',
'berzerk',
'bowling',
'boxing',
'breakout',
'carnival',
'centipede',
'chopper_command',
# ...
]

To save others the bother:
'adventure', 'air_raid', 'alien', 'amidar', 'assault', 'asterix', 'asteroids', 'atlantis', 'bank_heist', 'battle_zone', 'beam_rider', 'berzerk', 'bowling', 'boxing', 'breakout', 'carnival', 'centipede', 'chopper_command', 'crazy_climber', 'defender', 'demon_attack', 'double_dunk', 'elevator_action', 'enduro', 'fishing_derby', 'freeway', 'frostbite', 'gopher', 'gravitar', 'hero', 'ice_hockey', 'jamesbond', 'journey_escape', 'kaboom', 'kangaroo', 'krull', 'kung_fu_master', 'montezuma_revenge', 'ms_pacman', 'name_this_game', 'phoenix', 'pitfall', 'pong', 'pooyan', 'private_eye', 'qbert', 'riverraid', 'road_runner', 'robotank', 'seaquest', 'skiing', 'solaris', 'space_invaders', 'star_gunner', 'tennis', 'time_pilot', 'tutankham', 'up_n_down', 'venture', 'video_pinball', 'wizard_of_wor', 'yars_revenge', 'zaxxon'
Updated in 10.12.17

For anyone looking for ALE specific, use the following code.
import gym
tree = gym.envs.registry.all()._mapping.tree["ALE"]
env_names = [f"ALE/{name}-v5" for name, value in tree.items() if "ram" not in name]
This will output:
['ALE/Tetris-v5', 'ALE/Adventure-v5', 'ALE/AirRaid-v5', 'ALE/Alien-v5', 'ALE/Amidar-v5', 'ALE/Assault-v5', 'ALE/Asterix-v5', 'ALE/Asteroids-v5', 'ALE/Atlantis-v5', 'ALE/Atlantis2-v5', 'ALE/Backgammon-v5', 'ALE/BankHeist-v5', 'ALE/BasicMath-v5', 'ALE/BattleZone-v5', 'ALE/BeamRider-v5', 'ALE/Berzerk-v5', 'ALE/Blackjack-v5', 'ALE/Bowling-v5', 'ALE/Boxing-v5', 'ALE/Breakout-v5', 'ALE/Carnival-v5', 'ALE/Casino-v5', 'ALE/Centipede-v5', 'ALE/ChopperCommand-v5', 'ALE/CrazyClimber-v5', 'ALE/Crossbow-v5', 'ALE/Darkchambers-v5', 'ALE/Defender-v5', 'ALE/DemonAttack-v5', 'ALE/DonkeyKong-v5', 'ALE/DoubleDunk-v5', 'ALE/Earthworld-v5', 'ALE/ElevatorAction-v5', 'ALE/Enduro-v5', 'ALE/Entombed-v5', 'ALE/Et-v5', 'ALE/FishingDerby-v5', 'ALE/FlagCapture-v5', 'ALE/Freeway-v5', 'ALE/Frogger-v5', 'ALE/Frostbite-v5', 'ALE/Galaxian-v5', 'ALE/Gopher-v5', 'ALE/Gravitar-v5', 'ALE/Hangman-v5', 'ALE/HauntedHouse-v5', 'ALE/Hero-v5', 'ALE/HumanCannonball-v5', 'ALE/IceHockey-v5', 'ALE/Jamesbond-v5', 'ALE/JourneyEscape-v5', 'ALE/Kaboom-v5', 'ALE/Kangaroo-v5', 'ALE/KeystoneKapers-v5', 'ALE/KingKong-v5', 'ALE/Klax-v5', 'ALE/Koolaid-v5', 'ALE/Krull-v5', 'ALE/KungFuMaster-v5', 'ALE/LaserGates-v5', 'ALE/LostLuggage-v5', 'ALE/MarioBros-v5', 'ALE/MiniatureGolf-v5', 'ALE/MontezumaRevenge-v5', 'ALE/MrDo-v5', 'ALE/MsPacman-v5', 'ALE/NameThisGame-v5', 'ALE/Othello-v5', 'ALE/Pacman-v5', 'ALE/Phoenix-v5', 'ALE/Pitfall-v5', 'ALE/Pitfall2-v5', 'ALE/Pong-v5', 'ALE/Pooyan-v5', 'ALE/PrivateEye-v5', 'ALE/Qbert-v5', 'ALE/Riverraid-v5', 'ALE/RoadRunner-v5', 'ALE/Robotank-v5', 'ALE/Seaquest-v5', 'ALE/SirLancelot-v5', 'ALE/Skiing-v5', 'ALE/Solaris-v5', 'ALE/SpaceInvaders-v5', 'ALE/SpaceWar-v5', 'ALE/StarGunner-v5', 'ALE/Superman-v5', 'ALE/Surround-v5', 'ALE/Tennis-v5', 'ALE/TicTacToe3D-v5', 'ALE/TimePilot-v5', 'ALE/Trondead-v5', 'ALE/Turmoil-v5', 'ALE/Tutankham-v5', 'ALE/UpNDown-v5', 'ALE/Venture-v5', 'ALE/VideoCheckers-v5', 'ALE/Videochess-v5', 'ALE/Videocube-v5', 'ALE/VideoPinball-v5', 'ALE/WizardOfWor-v5', 'ALE/WordZapper-v5', 'ALE/YarsRevenge-v5', 'ALE/Zaxxon-v5']
I think you can get the version 5 in "-v5" by using value, but I was too lazy to add that.
And I can't explain what is what because I hacked by way through so I don't understand my code either. If anyone do understand, please add more!

scikit learn - converting features stored as strings into numbers

I'm currently dipping my toes into machine learning using the scikit-learn python library and am trying to use some .CSV data in the format
Date Name Average_Price_SA
1995-01-01 Barking And Dagenham 70885.331285935
1995-01-01 Barnet 99567.4268042005
1995-01-01 Barnsley 49608.33494746
....
....
....
2005-01-01 Barking And Dagenham 13294.12321312
I have read them in using panda using the line
data = pd.read_csv('data.csv')
From what I have learned so far, I think I'm supposed to convert those 'Name' category strings into floats so that they can be accepted into a model.
I'm not sure how to go about this. Any help would be greatly appreciated.
Thanks

You can use scikit's LabelBinarizer to convert the strings to one hot vectors. These have N zeros (where N is the number of unique strings) with a one at a single component.
from __future__ import print_function
from sklearn import preprocessing
names = ["Barking And Dagenham", "Barnet", "Barnsley"]
lb = preprocessing.LabelBinarizer()
vectors = lb.fit_transform(names)
for name, vector in zip(names, vectors):
print("%s => %s" % (name, str(vector)))
Output:
Barking And Dagenham => [1 0 0]
Barnet => [0 1 0]
Barnsley => [0 0 1]