I am trying to pull data from TSP.gov for the Thrift Savings Plan closing prices. The proper code should be:
import pandas_datareader.tsp as tsp # https://pandas-datareader.readthedocs.io/en/v0.6.0/remote_data.html
tspreader = tsp.TSPReader(start='2015-10-1', end='2015-12-31')
tspreader.read()
It appears that TSP did an update to their site, and this no longer works. I was wondering if anyone found a workaround?
https://github.com/pydata/pandas-datareader/issues/824
Related
so I have a lot of GPXs of users driving data from a game project where object which are placed on the road and then the user collects it. I want to somehow analyze these data to find out how users tend to drive given different objects, which ones draw them the most, which ones draw least. I have not done any data analysis before, so how can I analyze these data to get this sort of information? This might sound very novice, but yeah any help is appreciated.
You would probably like to do this in Python if you are novice, and then you can use a library like this one (gpxpy) to explore your data.
That is a GPX parser, I believe it will provide you with the data you like to see.
In their documentation you can see that you can use it like that :
import gpxpy
import gpxpy.gpx
# Open a file
gpx_file = open('yourfile.gpx', 'r')
# Parse the file
gpx = gpxpy.parse(gpx_file)
# Iterate over the tracks
for track in gpx.tracks:
for segment in track.segments:
for pt in segment.points:
print(f'Point at ({pt.latitude},{pt.longitude}) -> {pt.elevation}')
for waypoint in gpx.waypoints:
print(f'waypoint {waypoint.name} -> ({waypoint.latitude},{waypoint.longitude})')
for route in gpx.routes:
print('Route:')
for pt in route.points:
print(f'Point at ({pt.latitude},{pt.longitude}) -> {pt.elevation}')
Once you have those points you can calculate the distances, speeds, etc. from the coordinates.
I have a hazard model on a Weibull distribution, using the Stata command streg with the options nohr and time appended to that line of code. At least, that's the code from the do file I downloaded from a replication file.
If I have a new sliver of data, how do I compute the value of the model for that specific sliver of data? I would solve by hand in Excel (my wheelhouse is R or Python) but the closed form of the regression eludes me. I'm not sure from the documentation on the command exactly how they're adding in the other regressors and the Weibull regression has a lot of parameters that I'd rather not manually chug at. I'm hoping someone can help with what I believe is a simple out-of-sample forecast in a language I simply do not use.
infile warnum frstyear lastyear ccode1 ccode2 length logleng censor oadm oada oadp omdm omda omdp opdm opda opdp durscale rterrain rterrstr summperb sumpopbg popratbg bofadjbg qualratb salscale reprsumb demosumb surpdiff nactors adis3010 using 1perwarf.raw
stset length, id(warnum) fail(censor)
streg oadm oada oadp opda rterrain rterrstr bofadjbg summperb sumpopbg popratbg qualratb surpdiff salscale reprsumb demosumb adis3010 nactors, dist(weibull) nohr time
I've recently been introduced to R and trying the heatwaveR package. I get an error when loading erddap data ... Here's the code I have used so far:
library(rerddap)
library(ncdf4)
info(datasetid = "ncdc_oisst_v2_avhrr_by_time_zlev_lat_lon", url = "https://www.ncei.noaa.gov/erddap/")
And I get the following error:
Error in curl::curl_fetch_memory(x$url$url, handle = x$url$handle) :
schannel: next InitializeSecurityContext failed: SEC_E_INVALID_TOKEN (0x80090308) - The token supplied to the function is invalid
Would like some help in this. I'm new to this website too so I apologize if the above question is not as per standards (codes to be typed in a grey box, etc.)
Someone directed this post to my attention from the heatwaveR issues page on GitHub. Here is the answer I provided for them:
I do not manage the rerddap package so can't say exactly why it may be giving you this error. But I can say that I have noticed lately that the OISST data are often not available on the ERDDAP server in question. I (attempt to) download fresh data every day and am often denied with an error similar to the one you posted. It's gotten to the point where I had to insert some logic gates into my download script so it tells me that the data aren't currently being hosted before it tries to download them. I should also point out that one may download the "final" data from this server, which have roughly a two week delay from present day, as well as the "preliminary (prelim)" data, which are near-real-time but haven't gone through all of the QC steps yet. These two products are accounted for in the following code:
# First download the list of data products on the server
server_data <- rerddap::ed_datasets(which = "griddap", "https://www.ncei.noaa.gov/erddap/")$Dataset.ID
# Check if the "final" data are currently hosted
if(!"ncdc_oisst_v2_avhrr_by_time_zlev_lat_lon" %in% server_data)
stop("Final data are not currently up on the ERDDAP server")
# Check if the "prelim" data are currently hosted
if(!"ncdc_oisst_v2_avhrr_prelim_by_time_zlev_lat_lon" %in% server_data)
stop("Prelim data are not currently up on the ERDDAP server")
If the data are available I then check the times/dates available with these two lines:
# Download final OISST meta-data
final_info <- rerddap::info(datasetid = "ncdc_oisst_v2_avhrr_by_time_zlev_lat_lon", url = "https://www.ncei.noaa.gov/erddap/")
# Download prelim OISST meta-data
prelim_info <- rerddap::info(datasetid = "ncdc_oisst_v2_avhrr_prelim_by_time_zlev_lat_lon", url = "https://www.ncei.noaa.gov/erddap/")
I ran this now and it looks like the data are currently available. Is your error from today, or from a day or two ago? The availability seems to cycle over the week but I haven't quite made sense of any pattern yet. It is also important to note that about a day before the data go dark they are filled with all sorts of massive errors. So I've also had to add error trapping into my code that stops the data aggregation process once it detects temperatures in excess of some massive number. In this case it is something like1^90, but the number isn't consistent meaning it is not a missing value placeholder.
To manually see for yourself if the data are being hosted you can go to this link and scroll to the bottom:
https://www.ncei.noaa.gov/erddap/griddap/index.html
All the best,
-Robert
first post on here!
I have been playing around with postman automation and managed to copy over ID strings and FAB Id's.
I have seen similar questions asked but didn't quite highlight how I was hoping to do it...
Availability Response;
<SessionInfo FABSessionId="FAB123456789"/>
<VehicleSearchResponse Success="true" ModifiedRequest="false">
<ViewInfo Offset="0" Length="30"/>
<ResultSetInfo SortCode="cost" SortAscending="true" MaxItems="900" NumItems="30"/>
<Vehicles>
<Vehicle StartDate="20181002" EndDate="20181006" PickUpPoint="XXX" DropOffPoint="XXX">
<Image URL="https://picture.net/example.jpg"/>
<MoreInfoTexts>
<MoreInfo Header="..."/>
<MoreInfo Header="..."/>
<MoreInfo Header="..."/>
</MoreInfoTexts>
<PickUpDepotDetails Telephone="012345 67890">
<Address Address1="..." CityOrTown="..." Country="..."/>
</PickUpDepotDetails>
<DropOffDepotDetails Telephone="012345 67890">
<Address Address1="..." CityOrTown="..." Country="..."/>
</DropOffDepotDetails>
</Vehicle>
........
</Vehicles>
</VehicleSearchResponse>
See above... I want to be able to copy the whole "Vehicle" segment from the availability response to the costing request... It is an array so there is 20 other results below .......
Costing Request:
<Itinerary EndDate="20181006" ItineraryId="V!XXX!12345!ABC123!XXX!ABC321!XXX" StartDate="20181002">
<Vehicles>
{{vehicleSetter}}
</Vehicles>
<OptionalExtras>
So the whole Vehicle - /Vehicle segment needs to be copied to {{vehicleSetter}}.
Is there a way to do this using similar (Costing) Pre-RQ scripts to;
pm.globals.get("fabSessionSetter");
pm.globals.get("vehicleSetter");
and maybe something in (Avail.) Tests script like
pm.globals.set("vehicleSetter", jsonObject.FAB_VehicleAvailRS.VehicleSearchResponse.Vehicles.Vehicle[2]);
Let me know if you need any more information... Thanks!
First of all, I would ask devs to use json type data for passing to instead of XML. it's much more convenient, I went this way on the begging of the development of a new project, it brought me many benefits. But in case it's possible in your case.
it's not necessary to use global variables.Environment variables are enough.
But try this, I never did it, but perhaps it will help you.
https://github.com/cheeriojs/cheerio
https://www.w3schools.com/xml/xpath_syntax.asp
I think I had a similar problem to you, having to convert json into xml, either way I had a fantastic response from Sivcan Singh
Convert XML to JSON for extracting data then convert JSON back to XML in Postman
good moring.
first of all. This is the most impressive community i ever saw!
Well several days i mused about the three-folded job of
a. getting
b. parsing
c. storing a number of pages.
Two days ago i thought that getting the pages would be the major-task. No this isnt the case - i guess that the parser-job would be a heroic task. Each of the pages that are intended to be parsed is a png-image.
So the question is - after getting all them. How to parse them!? This seems to be the issue. Guess that there are some perl-modules out there - that can help in doing this...
Well - i think that this job only can be done with some OCR embedded! Question: is there a perl-module that can be use here to support this task:
BTW: see the result-pages.
BTW;: and as i thought i can find all 790 resultpages within a certain range between
Id= 0 and Id= 100000 i thought, that i can go the way with a loop:
http://www.foundationfinder.ch/ShowDetails.php?Id=11233&InterfaceLanguage=&Type=Html
http://www.foundationfinder.ch/ShowDetails.php?Id=927&InterfaceLanguage=1&Type=Html
http://www.foundationfinder.ch/ShowDetails.php?Id=949&InterfaceLanguage=1&Type=Html
http://www.foundationfinder.ch/ShowDetails.php?Id=20011&InterfaceLanguage=1&Type=Html
http://www.foundationfinder.ch/ShowDetails.php?Id=10579&InterfaceLanguage=1&Type=Html
i thought i can go the Perl-Way but i am not very very sure:
I was trying to use LWP::UserAgent on the same URLs [see below]
with different query arguments, and i am wondering if LWP::UserAgent provides a
way for us to loop through the query arguments? I am not sure that LWP::UserAgent has a method for us to do that. Well - i sometimes heard that it is easier to use Mechanize. But is it really easier!?
But - to be frank; The first task " GETTING all the pages is not very difficult - if we compare this task with the parsing... How can this be done!?
Any ideas - suggestions -
look forward to hear from you...
zero
You do not need a Perl module, you only need the system function.
system qw[ tesseract.exe foo.png foo.txt ];
my $text = read_file('foo.txt');
You may need to preprocess the images to help Tesseract, say using ImageMagick like:
system qw[ convert.exe -resize 200% image.jpg foo.png ];